Posts
Comments
Yes, that sounds much more normal to me.
Though in the particular case here, something else seems off: when you write you would normally italicize both the "f" and the "x", as you can see in the rendering in this very paragraph. I can't think of any situation in actual mathematical writing where you would italicize one and not the other in order to make some distinction between function-names and variable names.
For that matter, I'm not wild about making a distinction between "variables" and "functions". If you write and also then it would be normal for "f" and "x" to be italicized and not "sin". I was going to say that the reason is that f and x are in fact both variables, and it just happens that one of them takes values that are functions, whereas sin is a fixed function and you'll never see anything like "let sin = 3" or "let sin = cos" -- but actually that isn't quite right either, because named mathematical constants like e are usually italicized. I think the actual distinction is that single-letter names-of-things get italicized and multiple-letter ones usually don't.
(Brief self-review for LW 2023 review.)
Obviously there's nothing original in my writeup as opposed to the paper it's about. The paper still seems like an important one, though I haven't particularly followed the literature and wouldn't know if it's been refuted or built upon by other later work. In particular, in popular AI discourse one constantly hears things along the lines of "LLMs are just pushing symbols around and don't have any sort of model of the actual world in them", and this paper seems to me to be good evidence that transformer networks, even quite small ones, can build internal models that aren't just symbol-pushing.
There's a substantial error in the post as it stands (corrected in comments, but I never edited the post): I claim that the different legal-move-prediction abilities of the "network trained on a smallish number of good games" and "network trained on a much larger number of random games" cases is because the network isn't big enough to capture both "legal" and "good strategy" well, when in fact it seems more likely that the difference is mostly because the random-game training set is so much larger. I'm not sure what the etiquette is around making edits now in such cases.
I'm confused by what you say about italics. Mathematical variables are almost always italicized, so how would italicizing something help to clarify that it isn't a variable?
It seems like a thing that literally[1] everyone does sometimes. "Let's all go out for dinner." "OK, where shall we go?" As soon as you ask that question you're "optimizing for group fun" in some sense. Presumably the question is intending to ask about some more-than-averagely explicit, or more-than-averagely sophisticated, or more-than-averagely effortful, "optimizing for group fun", but to me at least it wasn't very clear what sort of thing it was intending to point at.
[1] Almost literally.
Yeah, I do see the value of keeping things the same across multiple years, which is why I said "might be worth" rather than "would be a good idea" or anything of the sort.
To me, "anti-agathics" specifically suggests drugs or something of the kind. Not so strongly that it's obvious to me that the question isn't interested in other kinds of anti-aging measures, but strongly enough to make it not obvious whether it is or not.
There is arguably a discrepancy between the title of the question "P(Anti-Agathics)" and the actual text of the question; there might be ways of "reaching an age of 1000 years" that I at least wouldn't want to call "anti-agathics". Uploading into a purely virtual existence. Uploading into a robot whose parts can be repaired and replaced ad infinitum. Repeated transfer of consciousness into some sort of biological clones, so that you get a new body when the old one starts to wear out.
My sense is that the first of those is definitely not intended to be covered by the question, and the second probably isn't; I'm not sure about the third. "Magical" options like survival of your immortal soul in a post-mortem heaven or hell, magical resurrection of your body by divine intervention, and reincarnation, are presumably also not intended.
In future years, it might be worth tweaking the wording by e.g. inserting the word "biological" or some wording like "in something that could credibly be claimed to be the body they are now living in". Or some other thing that better matches the actual intent of the question.
I have just noticed something that I think has been kinda unsatisfactory about the probability questions since for ever.
There's a question about the probability of "supernatural events (including God, ghosts, magic, etc.)" having occurred since the beginning of the universe. There's another question about the probability of there being a god.
I notice an inclination to make sure that the first probability is >= the second, for the obvious reason. But, depending on how the first question is interpreted, that may be wrong.
If the existence of a god is considered a "supernatural event since the beginning of the universe" then obviously that's the case. But note that one thing a fair number of people have believed is that a god created the universe and then, so to speak, stepped away and let it run itself. (It would be hard to distinguish such a universe from a purely natural one, but perhaps you could identify some features of the universe that such a being would be more or less likely to build in.) In that case, you could have a universe that (1) created by a god in which (2) no supernatural events have ever occurred or will ever occur.
The unsatisfactory thing, to be clear, is the ambiguity.
For future years, it might be worth considering either (1) replacing "God" with something like "acts of God" or "divine interventions" in the first question, or (2) adding an explicit clarification that the mere existence of a god should be considered a "supernatural event" in the relevant sense even if what that god did was to make a universe that runs naturally, or (3) tweaking the second question to explicitly exclude "deistic" gods.
"Trauma" meaning psychological as opposed to physical damage goes back to the late 19th century.
I agree that there's a widespread tendency to exaggerate the unpleasantness/harm done by mere words. (But I suggest there's an opposite temptation too, to say that obviously no one can be substantially harmed by mere words, that physical harm is different in kind from mere psychological upset, etc., and that this is also wrong.)
I agree that much of the trans community seems to have embraced what looks to me like a severely hyperbolic view of how much threat trans people are under. (But, usual caveats: it's very common for the situation of a minority group to look and feel much worse from the inside than from the outside, and generally this isn't only a matter of people on the inside being oversensitive, it's also a matter of people on the outside not appreciating how much unpleasantness those on the inside face. So my guess is that that view is less hyperbolic than it looks to me.)
I agree that the term "deadname" is probably popular partly because "using my deadname" has more of an obviously-hostile-move sound than "using my old name" or similar. But if we avoid every term with any spin attached, we'll have to stop calling people liberals (as if no one else cared about freedom) or conservatives (as if their opponents were against preserving valuable things) or Catholics (the word means "universal") or pro-life or pro-choice or, or, or, or. For my part, I avoid some spinny terms but not others, on the basis of gut feeling about how much actual wrongness is baked into them and how easy it is to find other language, which (I don't know how coherently) cashes out as being broadly OK with "liberal" and "conservative", preferring to avoid "pro-life" and "pro-choice" or at least making some snarky remarks about the terms before using them, avoiding the broadest uses of the term "transphobia", etc. And for me "deadname" seems obviously basically OK even though, yes, the term was probably chosen partly for connotations one might take issue with. Your mileage may vary.
I don't think "deadname" is a ridiculous term just because no one died. The idea is that the name is dead: it's not being used any more. Latin is a "dead language" because (roughly speaking) no one speaks or writes in Latin. "James" is a "dead name" because (roughly speaking) no one calls that person "James" any more.
This all seems pretty obvious to me, and evidently it seems the opposite way to you, and both of us are very smart [citation needed], so probably at least one of us is being mindkilled a bit by feeling strongly about some aspect of the issue. I don't claim to know which of us it is :-).
I think you're using "memetic" to mean "of high memetic fitness", and I wish you wouldn't. No one uses "genetic" in that way.
An idea that gets itself copied a lot (either because of "actually good" qualities like internal consistency, doing well at explaining observations, etc., or because of "bad" (or at least irrelevant) ones like memorability, grabbing the emotions, etc.) has high memetic fitness. Similarly, a genetically transmissible trait that tends to lead to its bearers having more surviving offspring with the same trait has high genetic fitness. On the other hand, calling a trait genetic means that it propagates through the genes rather than being taught, formed by the environment, etc., and one could similarly call an idea or practice memetic if it comes about by people learning it from one another rather than (e.g.) being instinctive or a thing that everyone in a particular environment invents out of necessity.
When you say, e.g., "lots of work in that field will be highly memetic despite trash statistics, blatant p-hacking, etc." I am pretty certain you mean "of high memetic fitness" rather than "people aware of it are aware of it because they learned of it from others rather than because it came to them instinctively or they reinvented it spontaneously because it was obvious from what was around them".
(It would be possible, though I'd dislike it, to use "memetic" to mean something like "of high memetic fitness for 'bad' reasons" -- i.e., liable to be popular for the sort of reason that we might not appreciate without the notion of memes. But I don't think that can be your meaning in the words I quoted, which seem to presuppose that the "default" way for a piece of work to be "memetic" is for it to be of high quality.)
Unless I misread, it said "mRNA" before.
Correction: the 2024 Nobel Prize in Medicine was for the discovery of microRNA, not mRNA which is also important but a different thing.
I think it's more "Hinton's concerns are evidence that worrying about AI x-risk isn't silly" than "Hinton's concerns are evidence that worrying about AI x-risk is correct". The most common negative response to AI x-risk concerns is (I think) dismissal, and it seems relevant to that to be able to point to someone who (1) clearly has some deep technical knowledge, (2) doesn't seem to be otherwise insane, (3) has no obvious personal stake in making people worry about x-risk, and (4) is very smart, and who thinks AI x-risk is a serious problem.
It's hard to square "ha ha ha, look at those stupid nerds who think AI is magic and expect it to turn into a god" or "ha ha ha, look at those slimy techbros talking up their field to inflate the value of their investments" or "ha ha ha, look at those idiots who don't know that so-called AI systems are just stochastic parrots that obviously will never be able to think" with the fact that one of the people you're laughing at is Geoffrey Hinton.
(I suppose he probably has a pile of Google shares so maybe you could squeeze him into the "techbro talking up his investments" box, but that seems unconvincing to me.)
Pedantic correction: you have some sizes where you've written e.g. 20' x 20' and I'm pretty sure you mean 20" x 20".
(Also, the final note saying pixel art is good for crisp upscaling and you should start with the lowest-resolution version seems very weird to me, though the way it's worded makes it unlikely that this is a mistake; another sentence or so elaborating on why this is a good idea would be interesting to me.)
So maybe e.g. the (not very auto-) autoformalization part produced a theorem-statement template with some sort of placeholder where the relevant constant value goes, and AlphaProof knew it needed to find a suitable value to put in the gap.
I'm pretty sure what's going on is:
- The system automatically generates candidate theorems it might try to prove, expressing possible answers, and attempts to prove them.
- In this case, the version of the theorem it ended up being able to prove was the one with 2 in that position. (Which is just as well, since -- I assume, not having actually tried to solve the problem for myself -- that is in fact the unique number for which such a theorem is true.)
- So the thing you end up getting a proof of includes the answer, but not because the system was told the answer in advance.
It would be nice to have this more explicitly from the AlphaProof people, though.
[EDITED to add:] Actually, as per the tweet from W T Gowers quoted by "O O" elsewhere in this thread, we do have it explicitly, not from the AlphaProof people but from one of the mathematicians the AlphaProof people engaged to evaluate their solutions.
The AlphaZero algorithm doesn't obviously not involve an LLM. It has a "policy network" to propose moves, and I don't know what that looks like in the case of AlphaProof. If I had to guess blindly I would guess it's an LLM, but maybe they've got something else instead.
I don't think this [sc. that AlphaProof uses an LLM to generate candidate next steps] is true, actually.
Hmm, maybe you're right. I thought I'd seen something that said it did that, but perhaps I hallucinated it. (What they've written isn't specific enough to make it clear that it doesn't do that either, at least to me. They say "AlphaProof generates solution candidates", but nothing about how it generates them. I get the impression that it's something at least kinda LLM-like, but could be wrong.)
Looks like this was posted 4 minutes before my https://www.lesswrong.com/posts/TyCdgpCfX7sfiobsH/ai-achieves-silver-medal-standard-solving-international but I'm not deleting mine because I think some of the links, comments, etc. in my version are useful.
Nothing you have said seems to make any sort of conspiracy theory around this more plausible than the alternative, namely that it's just chance. There are 336 half-hours per week; when two notable things happen in a week, about half a percent of the time one of them happens within half an hour of the other. The sort of conspiracies you're talking about seem to me more unlikely than that.
(Why a week? Arbitrary choice of timeframe. The point isn't a detailed probability calculation, it's that minor coincidences happen all the time.)
Given how much harm such an incident could do CrowdStrike, and given how much harm it could do an individual at Crowdstrike who turned out to have caused it on purpose, your second explanation seems wildly improbable.
The third one seems pretty improbable too. I'm trying to imagine a concrete sequence of events that matches your description, and I really don't think I can. Especially as Trump's formal acceptance of the GOP nomination can hardly have been any sort of news to anyone.
(Maybe I've misunderstood your tone and your comment is simply a joke, in which case fair enough. But if you're making any sort of serious suggestion that the incident was anything to do with Mr Trump, I think you're even crazier than he is.)
Right: as I said upthread, the discussion is largely about whether terms like "spending" are misleading or helpful when we're talking about time rather than money. And, as you point out (or at least it seems clearly implied by what you say), whether a given term is helpful to a given person will depend on what other things are associated with that term in that person's mind, so it's not like there's even a definite answer to "is it helpful or misleading?".
(But, not that it matters all that much, I think you might possibly not have noticed that Ruby and Raemon are different people?)
In such a world we'd presumably already have vocabulary adapted to that situation :-). But yes, I would feel fine using the term "spending" (but then I also feel fine talking about "spending time") but wouldn't want to assume that all my intuitions from the present world still apply.
(E.g., in the actual world, for anyone who is neither very rich or very poor spending always has saving as an alternative[1], and how much you save can have a big impact on your future well-being. In the hypothetical spend-it-or-lose-it world, that isn't the case, and that feels like a pretty important difference.)
[1] For the very poor, much of their spending is basically compulsory and saving instead isn't an option. For the very rich, the choice is still there but for normal-size purchases matters less because they're going to have ample resources whether they spend or save.
I am not convinced by the analogy. If you have $30 in your bank account you spend it on a book, you are $30 poorer; you had the option of just not doing that, in which case you would still have the $30. If you have 60 minutes ahead of you in the day and you spend it with a friend, then indeed you're 60 minutes older = poorer at the end of that time; but you didn't have the option of not spending those 60 minutes; they were going to pass by one way or another whatever you did.
You might still have given up something valuable! If you'd have preferred to devote those 60 minutes to earning money, or sleeping, or discovering timeless mathematical truths, then talking to your friend instead has had an opportunity cost. But the valuable thing you've foregone is that other thing you'd prefer to have done, not the time itself. That was always going to pass you by; it always does.
There aren't exactly definite right and wrong answers here; everyone agrees that "spending" time is not exactly the same thing as spending money, ditto for "giving" time, and the question is merely whether it's similar enough that the choice of language isn't misleading us. And it seems to me that, because the time is going to go anyway, the least-misleading way to think of it is that the default, no-action, case to compare with -- analogous to simply not spending money and having it sit in the bank -- is whatever you'd have been doing with your time otherwise. If you help a friend out by doing some tedious task that makes their life better, then you are "giving" or "spending" time. If you sit and chat with a friend because you both enjoy talking with one another, then you're not "giving" or "spending" in a sense that much resembles "giving" or "spending" money: you aren't worse off afterwards than if you hadn't done that; if you hadn't, then you'd likely have occupied the same time doing something no better.
I'm in two minds as to whether I believe what I just wrote. The counter-argument goes like this: comparing the situation after spending an hour with your friend with the situation after spending an hour doing something else begs the question; it's like comparing the situation after spending the $30 on a book with that after spending the $30 on something else. But I don't think the counter-argument really works, precisely because you have the option of just leaving the $30 in the bank and never spending it on something else, and there is no corresponding option for money.
You would be in the same situation if you'd done something else during that hour. You're only "paying", in the sense of giving up something valuable to you, in so far as you would have preferred to do something else.
That's sometimes true of time spent with friends -- maybe your friend is moving house and you help them unload a lot of boxes or something -- but by and large we human beings tend to enjoy spending time with our friends. (Even when unloading boxes, actually.)
Agreed. I was going for "explain how he came to say the false thing", not "explain why it's actually true rather than false".
I took it as a kinda-joking backformation from "patisserie".
On the other hand, it seems like Adam is looking at breadmaking that uses a sourdough starter, and that does have both yeasts and bacteria in it. (And breadmaking that uses it is correspondingly more error-prone and in need of adjustment on the fly than most baking that just uses commercial yeast, though some of what makes it more error-prone isn't directly a consequence of the more complicated leavener.)
I tried them all. My notes are full of spoilers.
Industrial Revolution (_2):
I didn't catch the deliberately-introduced error; instead I thought the bit about a recession in the 1830s was likely false. I took "the Industrial Revolution" to include the "Second Industrial Revolution" (as, indeed, the article seems to) and it seems to me that electricity was important for the progress of that, so I'm not entirely convinced that what was introduced to the article was exactly a falsehood.
Price gouging:
It seemed utterly unbelievable to me that a gathering of economists would disapprove wholeheartedly of "price gouging" and, lo, I was correct. (I feel like this one was pretty easy.)
World economy:
A few things seemed dubious and nothing seemed obviously wrong. There's a statement that economists usually use PPP to translate currencies and not official exchange rates, but then a table that (if I understand it right) uses both of those. (But this is kinda reconcilable: usually PPP is used for comparisons, and it's much better, but no harm in including nominal figures as well.) There's a claim that it's usual to consider only "human economic activity" and I don't know what that means -- what's the alternative? E.g., if it doesn't include making things with automatic machines then this is obviously false; if it's contrasting with aliens or something then it's so obviously true as not to be worth saying; probably it means something else but I don't know what, and looking it up feels a bit cheaty. This feels odd but my epistemic state is more "I don't understand something here" than "I know what this means and it's wrong". (Oh, I guess maybe it means "human economic activity as opposed to other human activities not classified as economic, like parents taking care of their own children", in which case it's not content-free and is probably correct.) There's a claim that before the Industrial Revolution "global output was dominated by China and India" and that India was the largest economy circa 1CE which seems iffy -- bigger than the Roman Empire? -- but, I dunno, those both had large populations and China at least was pretty advanced in some ways. There's a claim that the African Union has an economy bigger than $2T/year; I'm not sure just who's in the African Union but most of Africa is pretty poor and I'd have thought that if e.g. Nigeria and/or South Africa is at ~$1T then there ought to be a bigger list of $2T individual countries. The first of these feels like a nitpick; for the second, it feels like maybe I just don't understand a technical term; the third and fourth feel like memorization more than understanding. The thing that seems most likely wrong is the claim about China and India, though I wouldn't be astonished to find it's completely true. ... The actual fabrication turns out to be none of those things. I did wonder about errors introduced in the table that was falsified, but didn't consider the thing that was actually done. It's a fair cop.
Cell (biology):
Surely 1-5nm is waaaay too small for even a small prokaryotic cell. The 100nm given as biggest typical size for a eukaryotic cell feels way too small too. Are these really microns rather than nanometres, perhaps? Of course some eukaryotic cells get much larger than 100um -- consider a hen's egg, for instance. But 100um feels like a plausible "seldom much bigger than this" figure. (Yup, this is spot on, and indeed these were originally microns.)
Fundamental theorems of welfare economics:
This is not a field I'm aware of knowing anything about, but the two theorems sound familiar (though I hadn't associated them with the specific title here). The only thing that seems obviously fishy is that the article says the theorems don't ensure existence of the equilibria, but also says that the second theorem guarantees that "any Pareto optimum can be supported as a competitive equilibrium for some initial set of endowments", which sure sounds like ensuring the existence of equilibria to me. There are a couple of other things that look strange but I think they're more likely just badly-written-article strange: "The production economy is more general and entails additional assumptions" (surely to be more general is to require fewer assumptions, but maybe what's meant is that in the broader setting you need extra assumptions that automatically hold in the narrower setting or something), and "The assumptions are all based on the standard graduate microeconomics textbook" which is just a really weird sentence. There are lots of technicalities wherein errors could be hiding, but I'm no economist and wouldn't notice. So, my guess is that it's the "don't ensure existence" thing. -- Nope, all those are apparently correct, and actually there are some further technical conditions that I didn't know about. This seems very much in the "memorization" rather than "understanding" category to me; e.g., for all I know "perfect information" is implied by "rationality" as those terms are used in economics. Maybe given infinite intelligence I could have figured out that there needed to be some sort of condition about externalities, but: bah.
List of causes of death by rate:
This says that neurological disorders are the biggest single cause of death. Ahead of both cardiovascular and respiratory problems? Really? Can't possibly be right. And, indeed, it isn't. On the one hand, this is "memorization"; on the other, surely this is pretty well known.
Natural selection:
Surely that description of "microevolution" and "macroevolution" can't be right; the distinction is about the scale of the changes, not the nature of the environmental pressures that cause them. ... Huh, apparently that isn't the issue; the actual error is one I should have spotted if I'd been reading more carefully and hadn't seen something else I was sure was wrong. I have to say I still think those sentences about microevolution and macroevolution are wrong. Aren't they? (I tend to avoid those terms, which I mostly associate with creationists who argue that one is real and the other not, so maybe I'm misunderstanding them, but e.g. the Wikipedia pages for them don't seem to match what's claimed in this article.)
Working time:
Surely the decreases in working hours shown in that table are too dramatic. Unless, I guess, they're the result of e.g. increasing numbers of part-time workers, rather than changes to typical full-time hours or something? (If so, then right next to the table there should be a load of caveats about this.) But this claims that, for instance, in France in 1950 the average worker worked nearly 3000 hours per year, versus about half that now. And can it really be true that Greece has longer-working workers than any other EU country? It's not that any of the individual figures is completely unbelievable, but in the aggregate something seems really off. I'm sure there has been a decrease in working hours, but it surely can't be this big. (I may have been encouraged toward this conclusion by noticing that the directory containing this example had what looked like data-processing code, suggesting that something quantitative has been changed in a nontrivial way.) Anyway, cheaty or not, I was spot on with this.
So, I got exactly half of them right (and alternated between right and wrong!). For two of the wrong ones I'm prepared to argue that the thing I thought I'd spotted is wrong even though it wasn't deliberately introduced as a test; for another, I'm maybe prepared to argue that the deliberate error introduced isn't exactly an error.
There appear to be two edited versions of the Industrial Revolution article. Which one is recommended? (My guess: the _2 one because it's more recent.)
Let's suppose it's true, as Olli seems to find, that most not-inconsequential things in Wikipedia are more "brute facts" than things one could reasonably deduce from other things. Does this tell us anything interesting about the world?
For instance: maybe it suggests that reasoning is less important than we might think, that in practice most things we care about we have to remember rather than working out. It certainly seems plausible that that's true, though "reasonining is less important than we might think" feels like a slightly tendentious way of putting it. (I suggest: Reasoning is very important on those occasions when you actually need to do it, but those occasions are rarer than those of us who are good at reasoning might like to think.)
Unfortunately, not being a NYT subscriber I think I can't see the specific video you mention (the only one with Biden allegedly being led anywhere that I can see before the nag message has his wife doing the alleged leading, and the point of it seems to have been not that someone was leading him somewhere but that he walked off instead of greeting veterans at a D-Day event, and there isn't anything in the text I can see that calls anything a cheap fake).
(Obviously my lack of an NYT subscription isn't your problem, but unless there's another source for what they're claiming I can't actually tell whether I agree with your take on it or not.)
Again, I wasn't claiming that the people spinning conspiracy theories about the NYT wanting Biden out of the way to lower billionaires' taxes are right. (I think they're almost certainly wrong.) But when I see, within a week of one another, one claim that you can tell the NYT is politically biased because of how they went out of their way to defend Biden from claims about his age/frailty/... and another claim that you can tell the NYT is politically biased because of how they went out of their way to propagate and dramatize claims about Biden's age/frailty/..., my instinctive reaction is to be skeptical about both those claims.
... Ah, I've found what I think is the text of the NYT article. What it actually says about "cheap fakes" is this:
Some of the videos of Mr. Biden circulating during this year's campaign are clearly manipulated to make him look old and confused. Others cut out vital context to portray him in a negative light, a process sometimes known as a "cheap fake" because it requires little expense or technological skill to create.
(Which is not at all what I assumed "cheap fake" meant on reading your comment, for what it's worth.) But that text doesn't say anything about Obama and doesn't use the word "fundraiser" (and doesn't include any of the videos) so I still can't tell what video it is you're saying isn't misleading and therefore have no opinion on whether or not it actually is.
I had a look at a YouTube video from "Sky News AU" which was about the D-Day thing and it looked to me like a mixture of places where Biden's behaviour was genuinely troubling and short clips of the sort that I'm pretty sure you could find for anyone of his age whether or not there was anything much wrong with them, if you did a bit of cherry-picking. The latter seems like the sort of thing the NYT article called "cheap fakes" and whatever Biden's current state it seems pretty clear to me that there was some of that going on in the (pretty mainstream, I take it) video I looked at.
Again, I don't know exactly what video you're talking about or what the NYT said about it, since in what looks like a copy of the article's text there's nothing about Obama leading him anywhere at any fundraiser. But from what I've looked at so far, I'm not seeing how the NYT article is misinformation.
(As for the actual question of Biden's current state of physical and/or cognitive health, which is somewhat relevant to this, I'm not sure what to think. The very worst things look pretty alarming; on the other hand, I took a sort of random sampling of short segments from That Debate and I thought that in them Trump was significantly more incoherent than Biden. And I know that Biden is a lifelong stutterer which will make any sort of slowness look more alarming, and which is a fact that never seems to be mentioned in any of the articles about how decrepit he allegedly is. It's not relevant to all the things that people are making noise about -- e.g., if he calls someone by the wrong name, that's probably nothing to do with his stutter. On the other hand, calling people by the wrong name happens all the time and e.g. Trump does it a lot too.)
What in that article is misinformation?
Elsewhere on the internet, people are complaining vociferously that the NYT's more recent articles about Biden's age and alleged cognitive issues show that the NYT is secretly doing the bidding of billionaires who think a different candidate might tax them less. I mention this not because I think those people are right but as an illustration of the way that "such-and-such a media outlet is biased!" is a claim that often says more about the position of the person making the complaint than about the media outlet in question.
I gave it a few paragraphs from something I posted on Mastodon yesterday, and it identified me. I'm at least a couple of notches less internet-famous than Zvi or gwern, though again there's a fair bit of my writing on the internet and my style is fairly distinctive. I'm quite impressed.
(I then tried an obvious thing and fed it a couple of Bitcoin-white-paper paragraphs, but of course it knew that they were "Satoshi Nakamoto" and wasn't able to get past that. Someone sufficiently determined to identify Satoshi and with absurd resources could do worse than to train a big LLM on "everything except writings explicitly attributed to Satoshi Nakamoto" and then see what it thinks.)
If it's true that models are "starting to become barely capable of noticing that they are falling for this pattern" then I agree it's a good sign (assuming that we want the models to become capable of "general intelligence", of course, which we might not). I hadn't noticed any such change, but if you tell me you've seen it I'll believe you and accordingly reduce my level of belief that there's a really fundamental hole here.
I'm suggesting that the fact that things the model can't do produce this sort of whack-a-mole behaviour and that the shape of that behaviour hasn't really changed as the models have grown better at individual tasks may indicate something fundamental that's missing from all models in this class, and that might not go away until some new fundamental insight comes along: more "steps of scaling" might not do the trick.
Of course it might not matter, if the models become able to do more and more difficult things until they can do everything humans can do, in which case we might not be able to tell whether the whack-a-mole failure mode is still there. My highly unreliable intuition says that the whack-a-mole failure mode is related to the planning and "general reasoning" lacunae you mention, and that those might turn out also to be things that models of this kind don't get good at just by being scaled further.
But I'm aware that people saying "these models will never be able to do X" tend to find themselves a little embarrassed when two weeks later someone finds a way to get the models to do X. :-) And, for the avoidance of doubt, I am not saying anything even slightly like "mere computers will never be truly able to think"; only that it seems like there may be a hole in what the class of models that have so far proved most capable can be taught to do, and that we may need new ideas rather than just more "steps of scaling" to fill those holes.
That seems reasonable.
My impression (which isn't based on extensive knowledge, so I'm happy to be corrected) is that the models have got better at lots of individual tasks but the shape of their behaviour when faced with a task that's a bit too hard for them hasn't changed much: they offer an answer some part of which is nonsense; you query this bit; they say "I'm sorry, I was wrong" and offer a new answer some different part of which is nonsense; you query this bit; they say "I'm sorry, I was wrong" and offer a new answer some different part of which is nonsense; rinse and repeat.
So far, that pattern doesn't seem to have changed much as the models have got better. You need to ask harder questions to make it happen, because they've got better at the various tasks, but once the questions get hard enough that they don't really understand, back comes the "I'm sorry, I was wrong" cycle pretty much the same as it ever was.
It's pretty good. I tried it on a few mathematical questions.
First of all, a version of the standard AIW problem from the recent "Alice in Wonderland" paper. It got this right (not very surprisingly as other leading models also do, at least much of the time). Then a version of the "AIW+" problem which is much more confusing. Its answer was wrong, but its method (which it explained) was pretty much OK and I am not sure it was any wronger than I would be on average trying to answer that question in real time.
Then some more conceptual mathematical puzzles. I took them from recent videos on Michael Penn's YouTube channel. (His videos are commonly about undergraduate or easyish-olympiad-style pure mathematics. They seem unlikely to be in Claude's training data, though of course other things containing the same problems might be.)
One pretty straightforward one: how many distinct factorials can you find that all end in the same number of zeros? It wrote down the correct formula for the number of zeros, then started enumerating particular numbers and got some things wrong, tried to do pattern-spotting, and gave a hilariously wrong answer; when gently nudged, it corrected itself kinda-adequately and gave an almost-correct answer (which it corrected properly when nudged again) but I didn't get much feeling of real understanding.
Another (an exercise from Knuth's TAOCP; he rates its difficulty HM22, meaning it needs higher mathematics and should take you 25 minutes or so; it's about the relationship between two functions whose Taylor series coefficients differ by a factor H(n), the n'th harmonic number) it solved straight off and quite neatly.
Another (find all functions with (f(x)-f(y))/(x-y) = f'((x+y)/2) for all distinct x,y) it initially "solved" with a solution with a completely invalid step. When I said I couldn't follow that step, it gave a fairly neat solution that works if you assume f is real-analytic (has a Taylor series expansion everywhere). This is also the first thing that occurred to me when I thought about the problem. When asked for a solution that doesn't make that assumption, it unfortunately gave another invalid solution, and when prodded about that it gave another invalid one. Further prompting, even giving it a pretty big hint in the direction of a nice neat solution (better than Penn's :-)), didn't manage to produce a genuinely correct solution.
I rate it "not terribly good undergraduate at a good university", I think, but -- as with all these models to date -- with tragically little "self-awareness", in the sense that it'll give a wrong answer, and you'll poke it, and it'll apologize effusively and give another wrong answer, and you can repeat this several times without making it change its approach or say "sorry, it seems I'm just not smart enough to solve this one" or anything.
On the one hand, the fact that we have AI systems that can do mathematics about as well as a not-very-good undergraduate (and quite a bit faster) is fantastically impressive. On the other hand, it really does feel as if something fairly fundamental is missing. If I were teaching an actual undergraduate whose answers were like Claude's, I'd worry that there was something wrong with their brain that somehow had left them kinda able to do mathematics. I wouldn't bet heavily that just continuing down the current path won't get us to "genuinely smart people really thinking hard with actual world models" levels of intelligence in the nearish future, but I think that's still the way I'd bet.
(Of course a system that's at the "not very good undergraduate" level in everything, which I'm guessing is roughly what this is, is substantially superhuman in some important respects. And I don't intend to imply that it doesn't matter whether Anthropic are lax about what they release just because the latest thing happens not to be smart enough to be particularly dangerous.)
Even though it would have broken the consistent pattern of the titling of these pieces, I find myself slightly regretting that this one isn't called "Grace Notes".
A nitpick: you say
fun story, I passed the C2 exam and then I realized I didn’t remember the word faucet when I went to the UK to visit a friend
but here in the UK I don't think I have ever once heard a native speaker use the word "faucet" in preference to "tap". I guess the story is actually funnier if immediately after passing your C2 exam you (1) thought "faucet" was the usual UK term and (2) couldn't remember it anyway...
(I liked the post a lot and although I am no polyglot all the advice seems sound to me.)
Please don't write comments all in boldface. It feels like you're trying to get people to pay more attention to your comment than to others, and it actually makes your comment a little harder to read as well as making the whole thread uglier.
It looks to me as if, of the four "root causes of social relationships becoming more of a lemon market" listed in the OP, only one is actually anything to do with lemon-market-ness as such.
The dynamic in a lemon market is that you have some initial fraction of lemons but it hardly matters what that is because the fraction of lemons quickly increases until there's nothing else, because buyers can't tell what they're getting. It's that last feature that makes the lemon market, not the initial fraction of lemons. And I think three of the four proposed "root causes" are about the initial fraction of lemons, not the difficulty of telling lemons from peaches.
- urbanization: this one does seem to fit: it means that the people you're interacting with are much less likely to be ones you already know about, so you can't tell lemons from peaches.
- drugs: this one is all about there being more lemons, because some people are addicts who just want to steal your stuff.
- MLM schemes: again, this is "more lemons" rather than "less-discernible lemons".
- screens: this is about raising the threshold below which any given potential interaction/relationship becomes a lemon (i.e., worse than the available alternative), so again it's "more lemons" not "less-discernible lemons".
Note that I'm not saying that "drugs", "MLM", and "screens" aren't causes of increased social isolation, only that if they are the way they're doing it isn't quite by making social interactions more of a lemon market. (I think "screens" plausibly is a cause of increased social isolation. I'm not sure I buy that "drugs" and "MLM" are large enough effects to make much difference, but I could be convinced.)
I like the "possible solutions" part of the article better than the section that tries to fit everything into the "lemon market" category, because it engages in more detail with the actual processes involved by actual considering possible scenarios in which acquaintances or friendships begin. When I think about such scenarios in the less-isolated past and compare with the more-isolated present, it doesn't feel to me like "drugs" and "MLM" are much of the difference, which is why I don't find those very plausible explanations.
I think this is oversimplified:
High decouplers will notice that, holding preferences constant, offering people an additional choice cannot make them worse off. People will only take the choice if its better than any of their current options.
This is obviously true if somehow giving a person an additional choice is literally the only change being made, but you don't have to be a low-decoupler to notice that that's very very often not true. For a specific and very common example: often other people have some idea what choices you have (and, in particular, if we're talking about whether it should be legal to do something or not, it is generally fairly widely known what's legal).
Pretty much everyone's standard example of how having an extra choice that others know about can hurt you: threats and blackmail and the like. I might prefer not to have the ability to pay $1M to avoid being shot dead, or to prove I voted for a particular candidate to avoid losing my job.
This is pretty much parallel to a common argument for laws against euthanasia, assisted suicide, etc.: the easier it is for someone with terrible medical conditions to arrange to die, the more opportunities there are for others to put pressure on them to do so, or (this isn't quite parallel, but it seems clearly related) to make it appear that they've done so when actually they were just murdered.
Then it seems unfortunate that you illustrated it with a single example, in which A was a single (uniformly distributed) number between 0 and 1.
I think this claim is both key to OP's argument and importantly wrong:
But a wavefunction is just a way to embed any quantum system into a deterministic system
(the idea being that a wavefunction is just like a probability distribution, and treating the wavefunction as real is like treating the probability distribution of some perhaps-truly-stochastic thing as real).
The wavefunction in quantum mechanics is not like the probability distribution of (say) where a dart lands when you throw it at a dartboard. (In some but not all imaginable Truly Stochastic worlds, perhaps it's like the probability distribution of the whole state of the universe, but OP's intuition-pumping example seems to be imagining a case where A is some small bit of the universe.)
The reason why it's not like that is that the laws describing the evolution of the system explicitly refer to what's in the wavefunction. We don't have any way to understand and describe what a quantum universe does other than in terms of the evolution of the wavefunction or something basically equivalent thereto.
Which, to my mind, makes it pretty weird to say that postulating that the wavefunction is what's real is "going further away from quantum mechanics". Maybe one day we'll discover some better way to think about quantum mechanics that makes that so, but for now I don't think we have a better notion of what being Truly Quantum means than to say "it's that thing that wavefunctions do".
I have the impression -- which may well be very unfair -- that at some early stage OP imbibed the idea that what "quantum" fundamentally means is something very like "random", so that a system that's deterministic is ipso facto less "quantum" than a system that's stochastic. But that seems wrong to me. We don't presently have any way to distinguish random from deterministic versions of quantum physics; randomness or something very like it shows up in our experience of quantum phenomena, but the fact that a many-worlds interpretation is workable at all means that that doesn't tell us much about whether randomness is essential to quantum-ness.
So I don't buy the claim that treating the wavefunction as real is a sort of deterministicating hack that moves us further away from a Truly Quantum understanding of the universe.
(And, incidentally, if we had a model of Truly Stochastic physics in which the evolution of the system is driven by what's inside those probability distributions -- why, then, I would rather like the idea of claiming that the probability distributions are what's real, rather than just their outcomes.)
I don't know exactly what the LW norms are around plagiarism and plagiarism-ish things, but I think that introducing that basically-copied material with
I learned this by observing how beginners and more experienced people approach improv comedy.
is outright dishonest. OP is claiming to have observed this phenomenon and gleaned insight from it, when in fact he read about it in someone else's book and copied it into his post.
I have strong-downvoted the post for this reason alone (though, full disclosure, I also find the one-sentence-per-paragraph style really annoying and that may have influenced my decision[1]) and will not find it easy to trust anything else I see from this author.
[1] It feels to me as if the dishonest appropriation of someone else's insight and the annoying style may not be completely unrelated. One reason why I find this style annoying is that it gives me the strong impression of someone who is optimizing for sounding good. This sort of style -- punchy sentences, not too much complexity in how they relate to one another, the impression of degree of emphasis on every sentence -- feels like a public speaking style to me, and when I see someone writing this way I can't shake the feeling that someone is trying to manipulate me, to oversimplfy things to make them more likely to lodge in the brain, etc. And stealing other people's ideas and pretending they're your own is also a thing people do when they are optimizing for sounding good. (Obviously everything in this footnote is super-handwavy and unfair.)
In case anyone is in doubt about abstractapplic's accusation, I've checked. The relevant passage is near the end of section 3 of the chapter entitled "Spontaneity"; in my copy it's on page 88. I'm not sure "almost verbatim" is quite right, but the overall claim being made is the same, "fried mermaid" and "fish" are both there, and "will desperately try to think up something original" is taken verbatim from Johnstone.
One can't put a price on glory.
Wow, that was incredibly close.
I think simon and aphyer deserve extra credit for noticing the "implicit age variable" thing.
There are a few things in the list that I would say differently, which I mention not because the versions in the post are _wrong_ but because if you're using a crib-sheet like this then you might get confused when other people say it differently:
- I say "grad f", "div f", "curl f" for , , . I more often say "del" than "nabla" and for the Laplacian I would likely say either "del squared f" or "Laplacian of f".
- I pronounce "cos" as "coss" not as "coz".
- For derivatives I will say "dash" at least as often as "prime".
The selection of things in the list feels kinda strange (if it was mostly produced by GPT-4 then that may be why) -- if the goal is to teach you how to say various things then some of the entries aren't really pulling their weight (e.g., the one about the z-score, or the example of how to read out loud an explicit matrix transpose, when we've already been told how to say "transpose" and how to read out the numbers in a matrix). It feels as if whoever-or-whatever generated the list sometimes forgot whether they were making a list of bits of mathematical notation that you might not know how to say out loud or a list of things in early undergraduate mathematics that you might not know about.
It always makes me just a little bit sad when I see Heron's formula for the area of a triangle. Not because there's anything wrong with it or because it isn't a beautiful formula -- but because it's a special case of something even nicer. If you have a cyclic quadrilateral with sides then (writing ) its area is . Heron's formula is just the special case where two vertices coincide so . The more general formula (due to Brahmagupta) is also more symmetrical and at least as easy to remember.
With rather little confidence, I estimate for turtles A-J respectively:
22.93, 18.91, 25.47, 21.54, 17.79, 7.24, 30.36, 20.40, 24.25, 20.69 lb
Justification, such as it is:
The first thing I notice on eyeballing some histograms is that we seem to have three different distributions here: one normal-ish with weights < 10lb, one maybe lognormal-ish with weights > 20lb, and a sharp spike at exactly 20.4lb. Looking at some turtles with weight 20.4lb, it becomes apparent that 6-shell-segment turtles are special; they all have no wrinkles, green colour, no fangs, normal nostrils, no misc abnormalities, and a weight of 20.4lb. So that takes care of Harold. Then the small/large distinction seems to go along with (gray, fangs) versus (not-gray, no fangs). Among the fanged gray turtles, I didn't find any obvious sign of relationships between weight and anything other than number of shell segments, but there there's a clear linear relationship. Variability of weight doesn't seem interestingly dependent on anything. Residuals of the model a + b*segs look plausibly normal. So that takes care of Flint. The other pets are all green or grayish-green so I'll ignore the greenish-gray ones. These look like different populations again, though not so drastically different. Within each population it looks as if there's a plausibly-linear dependence of weight on the various quantitative features; nostrils seem irrelevant; no obvious sign of interactions or nonlinearities. The coefficients of wrinkles and segments are very close to a 1:2 ratio and I was tempted to force that in the name of model simplicity, but I decided not to. The coefficient of misc abs is very close to 1 and I was tempted to force that too but again decided not to. Given the estimated mean, the residuals now look pretty normally distributed -- the skewness seems to be an artefact of the distribution of parameters -- with stddev plausibly looking like a + b*mean. The same goes for the grayish-green turtles, but with different coefficients everywhere (except that the misc abs coeff looks like 1 lb/abnormality again). Finally, if we have a normally distributed estimate of a turtle's weight then the expected monetary loss is minimized ifwe estimate mu + 1.221*sigma.
I assume
that there's a more principled generation process, which on past form will probably involve rolling variable numbers of dice with variable numbers of sides, but I didn't try to identify it.
I will be moderately unsurprised if
it turns out that there are subtle interactions that I completely missed that would enable us to predict some of the turtles' weights with much better accuracy. I haven't looked very hard for such things. In particular, although I found no sign that nostril size relates to anything else it wouldn't be very surprising if it turns out that it does. Though it might not! Not everything you can measure actually turns out to be relevant! Oh, and I also saw some hints of interactions among the green turtles between scar-count and the numbers of wrinkles and shell segments, though my brief attempts to follow that up didn't go anywhere useful.
Tools used: Python, Pandas, statsmodels, matplotlib+seaborn. I haven't so far seen evidence that this would benefit much from
fancier models like random forests etc.