Posts
Comments
For a long time, I used to wonder what causes people to consistently mispronounce certain words even when they are exposed to many people pronouncing them correctly. (which mostly applies to people speaking in a non-native language, e.g. people from continental Europe speaking English)
Some examples that I’ve heard from different people around me over the years:
- Saying “rectangel” instead of “rectangle”
- Saying “pre-purr” (like prefer, but with a p) instead of “prepare”
- Saying something like, uhh, “devil-oupaw” instead of “developer”
- Saying “leech” instead of “league”
- Saying “immu-table” instead of “immutable”
- Saying "cyurrently" instead of "currently"
I did, of course, understand that if you only read a word, particularly in English where pronunciations are all over the place and often unpredictable, you may end up with a wrong assumption of how it's pronounced. This happened to me quite a lot[1]. But then, once I did hear someone pronounce it, I usually quickly learned my lesson and adapted the correct way of saying it. But still I've seen all these other people stick to their very unusual pronunciations anyway. What's up with that?[2] Naturally, it was always too awkward for me to ask them directly, so I never found out.
Recently, however, I got a rather uncomfortable insight into how this happens when a friend pointed out that I was pronouncing "dude" incorrectly, and have apparently done so for all my life, without anyone ever informing me about it, and without me noticing it.
So, as I learned now, "dude" is pronounced "dood" or "dewd". Whereas I used to say "dyood" (similar to duke). And while I found some evidence that dyood is not completely made up, it still seems to be very unusual, and something people notice when I say it.
Hence I now have the, or at least one, answer to my age-old question of how this happens. So, how did I never realize? Basically, I did realize that some people said "dood", and just took that as one of two possible ways of pronouncing that word. Kind of, like, the overly American way, or something a super chill surfer bro might say. Whenever people said "dood" (which, in my defense, didn't happen all that often in my presence[3]) I had this subtle internal reaction of wondering why they suddenly saw the need to switch to such a heavy accent for a single word.
I never quite realized that practically everyone said "dood" and I was the only "dyood" person.
So, yeah, I guess it was a bit of a trapped prior and it took some well-directed evidence to lift me out of that valley. And maybe the same is the case for many of the other people out there who are consistently mispronouncing very particular words.
But, admittedly, I still don't wanna be the one to point it out to them.
And when I lie awake at night, I wonder which other words I may be mispronouncing with nobody daring to tell me about it.
- ^
e.g., for some time I thought "biased" was pronounced "bee-ased". Or that "sesame" was pronounced "see-same". Whoops. And to this day I have a hard time remembering how "suite" is pronounced.
- ^
Of course one part of the explanation is survivorship bias. I'm much less likely to witness the cases where someone quickly corrects their wrong pronunciation upon hearing it correctly. Maybe 95% of cases end up in this bucket that remains invisible to me. But still, I found the remaining 5% rather mysterious.
- ^
Maybe they were intimidated by my confident "dyood"s I threw left and right.
or can read interview transcripts in much less time than listening to a podcast would take.
This always baffles me. :) Guess I'm both a slow reader and a fast listener, but for me audio allows for easily 3x as much speed as reading.
So what made you change your mind?
It's interesting how two years later, the "buy an expert's time" suggestion is almost outdated. There are still situations where it makes sense, but probably in the majority of situations any SOTA LLM will do a perfectly fine job giving useful feedback on exercises in math or language learning.
Thanks for the post!
The puzzle does not include any question or prompt. What does "try it out" mean exactly? I suppose it means "figure out how the notation works", or am I missing something? (I didn't read the rest to not get spoiled)
I guess a related pattern is the symmetric case where people talk past each other because both sides are afraid their arguments won't get heard, so they both focus on repeating their arguments and nobody really listens (or maybe they do, but not in a way that convinces the other person they really got their argument). So there, too, I agree with your advice - taking a step back and repeating the other person's viewpoint seems like the best way out of this.
Some further examples:
- Past me might have said: Apple products are "worse" because they are overpriced status symbols
- Many claims in politics, say "we should raise the minimum wage because it helps workers"
- We shouldn't use nuclear power because it's not really "renewable"
- When AI lab CEOs warn of AI x risk we can dismiss that because they might just want to build hype
- AI cannot be intelligent, or dangerous, because it's just matrix multiplications
- One shouldn't own a cat because it's an unnatural way for a cat to live
- Pretty much any any-benefit mindset that makes it into an argument rather than purely existing in a person's behavior
It certainly depends on who's arguing. I agree that some sources online see this trade-off and end up on the side of not using flags after some deliberation, and I think that's perfectly fine. But this describes only a subset of cases, and my impression is that very often (and certainly in the cases I experienced personally) it is not even acknowledged that usability, or anything else, may also be a concern that should inform the decision.
(I admit though that "perpetuates colonialism" is a spin that goes beyond "it's not a 1:1 mapping" and is more convincing to me)
This makes me wonder, how could an AI figure out whether it had conscious experience? I always used to assume that from first person perspective it's clear when you're conscious. But this is kind of circular reasoning as it assumes you have a "perspective" and are able to ponder the question. Now what does a, say, reasoning model do? If there is consciousness, how will it ever know? Does it have to solve the "easy" problem of consciousness first and apply the answer to itself?
In no particular order, because interestingness is multi-dimensional and they are probably all to some degree on my personal interesting Pareto frontier:
- We're not as 3-dimensional as we think
- Replacing binary questions with "under which circumstances"
- Almost everything is causally linked, saying "A has no effect on B" is almost always wrong (unless you very deliberately search for A and B that fundamentally cannot be causally linked). If you ran a study with a bazillion subjects for long enough, practically anything you can measure would reach statistical significance
- Many disagreements are just disagreements about labels ("LLMs are not truly intelligent", "Free will does not exist") and can be easily resolved / worked around once you realize this (see also)
- Selection biases of all kind
- Intentionality bias, it's easy to explain human behavior with supposed intentions, but there is much more randomness and ignorance everywhere than we think
- Extrapolations tend to work locally, but extrapolating further into the future very often gets things wrong; kind of obvious, applies to e.g. resource shortages ("we'll run out of X and then there won't be any X anymore!"), but also Covid (I kind of assumed Covid cases would just exponentially climb until everything went to shit, and forgot to take into account that people would get afraid and change their behavior on a societal scale, at least somewhat, and politicians would eventually do things, even if later than I would), and somewhat AI (we likely won't just "suddenly" end up with a flawless superintelligence)
- "If only I had more time/money/whatever" style thinking is often misguided, as often when people say/think this, the sentence continues with "then I could spend that time/money/whatever in other/more ways than currently", meaning as soon as you get more of X, you would immediately want to spend it, so you'll never sustainably end up in a state of "more X". So better get used to X being limited and having to make trade-offs and decisions on how to use that limited resource rather than daydreaming about a hypothetical world of "more X". (This does not mean you shouldn't think about ways to increase X, but you should probably distance yourself from thinking about a world in which X is not limited)
- Taleb's Extremistan vs Mediocristan model
- +1 to Minimalism that lsusr already mentioned
- The mindblowing weirdness of very high-dimensional spaces
- Life is basically an ongoing coordination problem between your past/present/future selves
- The realization that we're not smart enough to be true consequentialists, i.e. consequentialism is somewhat self-defeating
- The teleportation paradox, and thinking about a future world in which a) teleportation is just a necessity to be successful in society (and/or there is just social pressure, e.g. all your friends do it and you get excluded from doing cool things if you don't join in) and b) anyone having teleported before having convincing memories of having gone through teleportation and coming out on the other side. In such a world, anyone with worries about teleportation would basically be screwed. Not sure if I should believe in any kind of continuity of consciousness, but that certainly feels like a thing. So I'd probably prefer not to be forced to give that up just because the societal trajectory happens to lead through ubiquitous teleportation.
Random thought: maybe (at least pre-reasoning-models) LLMs are RLHF'd to be "competent" in a way that makes them less curious & excitable, which greatly reduces their chance of coming up with (and recognizing) any real breakthroughs. I would expect though that for reasoning models such limitations will necessarily disappear and they'll be much more likely to produce novel insights. Still, scaffolding and lack of context and agency can be a serious bottleneck.
Interestingly, the text to speech conversion of the "Text does not equal text" section is another very concrete example of this:
- The TTS AI summarizes the "Hi!" ASCII art picture as "Vertical lines arranged in a grid with minor variations". I deliberately added an alt text to that image, describing what can be seen, and I expected that this alt text would be used for TTS - but seemingly that is not the case, and instead some AI describes the image in isolation. If I were to describe that image without any further context, I would probably mention that it says "Hi!", but I grant that describing it as "Vertical lines arranged in a grid with minor variations" would also be a fair description.
- the "| | | |↵|-| | |↵| | | o" string is read out as "dash O". I would have expected the AI to just read that out in full, character by character. Which probably is an example of me falsely taking my intention as a given. There are probably many conceivable cases where it's actually better for the AI to not read out cryptic strings character by character (e.g. when your text contains some hash or very long URL). So maybe it can't really know that this particular case is an exception.
But what you're probably not aware of is that 0.8% of the US population ends up dieing due to intentional homicide
That is an insane statistic. According to a bit of googling this indeed seems plausible, but would still be interested in your source if you can provide it.
Downvoted for 3 reasons:
- The style strikes me as very AI-written. Maybe it isn't - but the very repetitive structure looks exactly like the type of text I tend to get out of ChatGPT much of the time. Which makes it very hard to read.
- There are many highly superficial claims here without much reasoning to back them up. Many claims of what AGI "would" do without elaboration. "AGI approaches challenges as problems to be solved, not battles to be won." - first, why? Second, how does this help us when the best way to solve the problem involves getting rid of humans?
- Lastly, I don't get the feeling this post engages with the most common AI safety arguments at all. Neither does it with evidence from recent AI developments. How do you expect "international agreements" with any teeth in the current arms race? When we don't even get national or state level agreements. While Bing/Sydney was not an AGI, it clearly showed that much of what this post dismisses as anthropocentric projections is realistic, and, currently, maybe even the default of what we can expect of AGI as long as it's LLM-based. And even if you dismiss LLMs and think of more "Bostromian" AGIs, that still leaves you with instrumental convergence, which blows too many holes into this piece to leave anything of much substance.
Or as a possible more concrete prompt if preferred: "Create a cost benefit analysis for EU directive 2019/904, which demands that bottle caps of all plastic bottles are to remain attached to the bottles, with the intention of reducing littering and protecting sea life.
Output:
-
key costs and benefits table
-
economic cost for the beverage industry to make the transition
-
expected change in littering, total over first 5 years
-
QALYs lost or gained for consumers throughout the first 5 years"
In the EU there's some recent regulation about bottle caps being attached to bottles, to prevent littering. (this-is-fine.jpg)
Can you let the app come up with a good way to estimate the cost benefit ratio of this piece of regulation? E.g. (environmental?) benefit vs (economic? QALY?) cost/drawbacks, or something like that. I think coming up with good metrics to quantify here is almost as interesting as the estimate itself.
I have the vague impression that this is true for me as well, and I remember having made that same claim (that spontaneous conversations at conferences seem maybe most valuable) to a friend when traveling home from an EAGx. My personal best guess: planned conversations are usually 30 minutes long, and while there is some interest based filtering going on, there's usually no guarantees you vibe well with the person. Spontaneous encounters however have pretty variable length, so the ones where you're not vibing will just be over naturally quickly, whereas the better connections will last longer. So my typical "spontaneous encounter minute" tends to be more enjoyable than my typical "planned 1-1 minute". But hard to say how this transfers to instrumental value.
I made a somewhat similar point in a post earlier this year, but much more superficial and less technical. So it was nice to read your deeper exploration of the topic.
Did this already happen? :)
Almost two years after writing this post, this is still a concept I encounter relatively often. Maybe less so in myself, as, I like to think, I have sufficiently internalized the idea to not fall into the "fake alternative trap" anymore very often. But occasionally this comes up in conversations with others, when they're making plans, or we're organizing something together.
With some distance, and also based on some of the comments, I think there is room for improvement:
- the Gym membership example is a tricky one, as "getting a gym membership to go to the gym" is, for many people, also kind of a fake option, as they get the membership and pay for it, but still end up not going to the gym anyway. That example works for people who are more likely to go to the gym than to work out at home. But if you would in expectation exercise no more at the gym than you would at home, then paying a gym membership is not helpful.
- Maybe an example that applies to more people would be studying at the (university) library vs studying at home? The former works better for many. So studying at home would potentially be a fake alternative. Just because you could in principle study for 10 hours a day at home, doesn't mean you actually end up doing that.
- I was and still am a bit unhappy about the "Option A - Option B - Do nothing" diagram. Somehow it's harder to read than its simplicity would suggest.
- The AI generated title image doesn't really convey the idea of of the post. But back then, AI image generation was still more limited than today, and it was difficult enough to even get that image to look acceptable.
But besides that, I think it holds up. It's a relevant concept, "fake alternatives" seems like a good handle to represent it, and the post is short and focused.
For people who like guided meditations: there's a small YouTube channel providing a bunch of secular AI-generated guided meditations of various lengths and topics. More are to come, and the creator (whom I know) is happy about suggestions. Three examples:
They are also available in podcast form here.
I wouldn't say these meditations are necessarily better or worse than any others, but they're free and provide some variety. Personally, I avoid apps like Waking Up and Headspace due to both their imho outrageous pricing model and their surprising degree of monotony. Insight Timer is a good alternative, but the quality varies a lot and I keep running into overly spiritual content there. Plus there's obviously thousands and thousands of guided meditations on YouTube, but there too it's hit and miss. So personally I'm happy about this extra source of a good-enough-for-me standard.
Also, in case you ever wanted to hear a guided meditation on any particular subject or in any particular style, I guess you can contact the YouTube channel directly, or tell me and I'll forward your request.
I'm a bit torn regarding the "predicting how others react to what you say or do, and adjust accordingly" part. On the one hand this is very normal and human and makes sense. It's kind of predictive empathy in a way. On the other hand, thinking so very explicitly about it and trying to steer your behavior in a way so as to get the desired reaction out of another person also feels a bit manipulative and inauthentic. If I knew another person would think that way and plan exactly how they interacted with me, I would find that quite off-putting. But maybe the solution is just "don't overdo it", and/or "only use it in ways the other person would likely consent to" (such as avoiding to accidentally say something hurtful).
My take on this is that patching the more "obvious" types of jailbreaking and obfuscation already makes a difference and is probably worth it (as long as it comes at no notable cost to the general usefulness of the system). Sure, some people will put in the effort to find other ways, but the harder it is, and the fewer little moments of success you have when first trying it, the fewer people will get into it. Of course one could argue that the worst outcomes come from the most highly motivated bad actors, and they surely won't be deterred by such measures. But I think even for them there may be some path dependencies involved where they only ended up in their position because over the years, while interacting with LLMs, they ended up running into a bunch of just ready enough jailbreaking scenarios that kept their interest up. Of course that's an empirical question though.
Some other comments already discussed the issue that often neither A nor B are necessarily correct. I'd like to add that there are many cases where the truth, if existent in any meaningful way, depends on many hidden variables, and hence A may be true in some circumstances, and B in some other circumstances, and it's a mistake to look for "the one static answer". Of course the question "when are A or B correct?" / "What does it depend on?" are similarly hard questions. But it's possible that this different framing can already help, as inquiring why the two sides believe what they believe can sometimes uncover these hidden variables, and it becomes apparent that the two sides' "why"s are not always opposite sides of a single axis.
An argument against may be that for some people there's probably a risk of getting addicted / losing control. I'm not familiar with to what degree it's possible to predict such tendencies in advance, but for some people that risk may well outweigh any benefits of arbitrate opportunities or improvements to their calibration.
Note from the future: I asked a bunch of LLMs for Terry Pratchett quotes on the human stomach, and while there's no guarantee any of them are actual non-hallucinated quotes (in different conversations I got many different ones while no single one came up twice), I think they're all pretty good:
"All he knew was that his stomach had just started investigating some of the more revolutionary options available to it."
"The stomach is smarter than the mind, which is why it likes to make all the important decisions."
"His stomach was making the kind of noises that normally precede the arrival of a self-propelled meal."
"His stomach felt like it was trying to digest a live weasel while attempting to escape through his boots."
"The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it. And the trouble with having an empty stomach is that it wants food all the time."
“The stomach is an essential part of the nervous system. It tells the brain what it wants far more clearly than the brain manages to tell it.”
"The human stomach is an amazing thing. It can stretch to accommodate all sorts of things. In theory, anyway. It just doesn’t appreciate it when you try to prove it."
Bonus Example: The Game Codenames
There’s a nice board + online game called Codenames. The basic idea is: you have two teams, each team split into two roles, the spymaster and the operatives. All players see an array of 25 cards with a single word on each of them. Everybody sees the words, but only the spymasters see the color of these cards. They can be blue or red, for the two teams, or white for neutral. The teams then take turns. Each time, the spymaster tries to come up with a single freely chosen word that would then allow their operatives to select, by association to that word, as large as possible a number of cards of their team’s color. The spymaster hence communicates that word, as well as the number of cards to be associated with that word, to the rest of their team. The operatives then discuss amongst each other which cards are most likely to fit that provided word[1].
I’ve played this game with a number of people, and noticed that many seem to play this in “forward” mode: spymasters often just try to find some plausible word that matches some of their team’s cards, almost as if they were trying to solve the problem: if somebody saw what I saw, they should agree this word makes sense. Whereas the better question would be: which word, if my team heard it, would make them choose the cards that have our team’s color? And the operatives on the other hand usually just check plainly which cards fit this word best? But almost nobody asks themselves if the selection of cards I’ve picked now really is the one the spymaster had in mind, would they have picked the word that they did?
To name a concrete example of the latter point, let’s say the spymaster said the word “transportation” and the number 2, so you know you’re looking for exactly two cards with some word on them that relates to transportation. And after looking at all available cards, there are three candidates: “wheel”, “windshield” and “boat”. Forward reasoning would allow basically any 2 out of these 3 cards, so you basically had to guess. But with inverse reasoning you would at least notice that, if “wheel” and “windshield” were the two words the spymaster was hinting at, they would most certainly have used “car” rather than “transportation”. But as they did not, in fact, choose “car”, you can be pretty sure that “boat” should be among your selection, so you can at least be pretty sure about that one word.
Of course one explanation for all of this may be that Codenames is, after all, just a game, and doing all this inverse reasoning is a bit effortful. Still it made me realize how rarely people naturally go into “inverse mode”, even in such a toy setting where it would be comparably clean and easy to apply.
- ^
I guess explaining the rules of a game is another problem that can be approached in forward or inverse ways. The forward way would just be to explain the rules in whichever way seems reasonable to you as someone familiar with the game. Whereas the inverse way would be to think about how you can best explain things in a way such that somebody who has no clue about the game will quickly get an idea of what’s going on. I certainly tried to do the latter, but, ehhh, who knows if I succeeded.
Without having thought much about it, I would think that it's a) pretty addictive and b) "scales well". Many forms of consumption have some ~natural limit, e.g. you can only eat so much food, going to the movies or concerts or whatever takes some energy and you probably wouldn't want to do this every day. Even addictive activities like smoking tend to have at least somewhat of a cap on how much you spend on it. Whereas gambling (which sports betting probably basically is to many people) potentially can just eat up all your savings if you let it.
So it would at least seem that it has much more potential to be catastrophic for individuals with low self control, even though that's a different story than the average effect on household investment, I guess.
While much of this can surely happen to varying degrees, I think an important aspect in music is also recognition (listening to the same great song you know and like many times with some anticipation), as well as sharing your appreciation of certain songs with others. E.g. when hosting parties, I usually try to create a playlist where for each guest there are a few songs in there that they will recognize and be happy to hear, because it has some connection to both of us. Similarly, couples often have this meme of "this is our song!", which throws them back into nostalgic memories of how they first met.
None of this is to disagree with the post though. I mostly just wanted to point out that novelty and "personal fit" are just two important aspects in any person's music listening experience, and I think it's unlikely these two aspects will dominate the future of music that much.
I once had kind of the opposite experience: I was at a friend's place, and we watched the recording of a System of a Down concert from a festival that we both had considered attending but didn't. I thought it was terrific and was quite disappointed not to have attended in person. He however got to the conclusion that the whole thing was so full of flaws that he was glad he hadn't wasted money on a ticket.
Just like you, I was baffled, and to be honest just kind of assumed he was just trying to signal his high standards or something but surely didn't actually mean that.
Given that he was quite the musician himself, playing multiple instruments, and I'm quite the opposite, I now for the first time seriously consider whether he really did dislike that concert as much as he said.
I appreciate your perspective, and I would agree there's something to it. I would at first vaguely claim that it depends a lot on the individual situation whether it's wise to be wary of people's insecurities and go out of one's way to not do any harm, or to challenge (or just ignore) these insecurities instead. One thing I've mentioned in the post is the situation of a community builder interacting with new people, e.g. during EA or lesswrong meetups. For such scenarios I would still defend the view that it's a good choice to be very careful not to throw people into uncomfortable situations. Not only because that's instrumentally suboptimal, but also because you're in a position of authority and have some responsibility not to e.g. push people to do something against their will.
However, when you're dealing with people you know well, or even with strangers but on eye level, then there's much more wiggle room, and you can definitely make the case that it's the better policy to not broadly avoid uncomfortable situations for others.
Thanks for sharing your thoughts and experience, and that first link indeed goes exactly in the direction I was thinking.
I think in hindsight I would adjust the tone of my post a bit away from "we're generally bad at thinking in 3D" and more towards "this is a particular skill that many people probably don't have as you can get through the vast majority of life without it", or something like that. I mostly find this distinction between "pseudo 3D" (as in us interacting mostly with surfaces that happen to be placed in a 3D environment, but very rarely, if ever, with actual volumes) and "real 3D" interesting, as it's probably rather easy to overlook.
I find your first point particularly interesting - I always thought that weights are quite hard to estimate and intuit. I mean of course it's quite doable to roughly assess whether one would be able to, say, carry an object or not. But when somebody shows me a random object and I'm supposed to guess the weight, I'm easily off by a factor of 2+, which is much different from e.g. distances (and rather in line with areas and volumes).
That github link yields a 404. Is it just an issue with the link itself, or did something change about the dataset being public?
Indeed! I think I remember having read that a while ago. A different phrasing I like to use is "Do you have a favorite movie?", because many people actually do and then are happy to share it, and if they don't, they naturally fall back on something like "No, but I recently watched X and it was great" or so.
Good point. I guess one could come up with examples that have less of this inefficiency but still are "computationally unkind". Although in the end, there's probably some correlation between these concepts anyway. So thanks for adding that. 👌
I would add 3) at the start of an event, everyone is asked to state their hopes and expectations about the event. While it's certainly useful to reflect on these things, I (embarassingly?) often in such situations don't even have any concrete hopes or expectations and am rather in "let's see what happens" mode. I still think it's fair to ask this question, as it can provide very benefitial feedback for the organizer, but they should at least be aware that a) this can be quite stressful for some participants, and b) many of the responses may be "made up" on the fly, rather than statements backed by a sufficient level of reflection. Of course just being honest there and saying "I don't have any expectations yet and just thought the title of the event sounded interesting" is probably the best option, but I think 10-years-ago-me would probably not have been confident enough to say that, and instead made up some vague plausible sounding claims that had a higher chance of signaling "I've got my shit together and definitely thought deeply about why I'm attending this event beforehand".
I think it's a fair point. To maybe clarify a bit though, while potentially strawmanning your point a bit, my intention with the post was not so much to claim "the solution to all social problems is that sufficiently-assertive people should understand the weaknesses of insufficiently-assertive people and make sure to behave in ways that don't cause them any discomfort", but rather I wanted to try to shed some light on situations that for a long time I found confusing and frustrating, without being fully aware of what caused that perceived friction. So I certainly agree that one solution to these situations can be to "tutor the insufficiently-assertive". But still, such people will always exist in this world, and if you're, say, a community builder who frequently interacts with new people, then it can still be valuable to be aware of these traps.
Thanks a lot for the write-up, very interesting and a good resource to get back to for future workshops.
I would expect that they fare much better with a text representation. I'm not too familiar with how multimodality works exactly, but kind of assume that "vision" works very differently from our intuitive understanding of it. When we are asked such a question, we look at the image and start scanning it with the problem in mind. Whereas transformers seem like they just have some rather vague "conceptual summary" of the image available, with many details, but maybe not all for any possible question, and then have to work with that very limited representation. Maybe somebody more knowledgeable can comment on how accurate that is. And whether we can expect scaling to eventually just basically solve this problem, or some different mitigation will be needed.
Maybe I accidentally overpromised here :D this code is just an expression, namely 1.0000000001 ** 175000000000, which, as wolframalpha agrees, yields 3.98e7.
One crucial question in understanding and predicting the learning process, and ultimately the behavior, of modern neural networks, is that of the shape of their loss landscapes. What does this extremely high dimensional landscape look like? Does training generally tend to find minima? Do minima even exist? Is it predictable what type of minima (or regions of lower loss) are found during training? What role does initial randomization play? Are there specific types of basins in the landscape that are qualitatively different from others, that we might care about for safety reasons?
First, let’s just briefly think about very high dimensional spaces. One somewhat obvious observation is that they are absolutely vast. With each added dimension, the volume of the available space increases exponentially. Intuitively we tend to think of 3-dimensional spaces, and often apply this visual/spatial intuition to our understanding of loss landscapes. But this can be extremely misleading. Parameter spaces are utterly incredibly vast to a degree that our brain can hardly fathom. Take GPT3 for instance. It has 175 billion parameters, or dimensions. Let’s assume somewhat arbitrarily that all parameters end up in a range of [-0.5, 0.5], i.e. live in a 175-billion-dimensional unit cube around the origin of that space (as this is not the case, the real parameter space is actually even much, much larger, but bear with me). Even though every single axis only varies by 1 – let’s just for the sake of it interpret this as “1 meter” – even just taking the diagonal from one corner to the opposite one in this high-dimensional cube, you would get a length of ~420km. So if, hypothetically, you were sitting in the middle of this high dimensional unit cube, you could easily touch every single wall with your hand. But nonetheless, all the corners would be more than 200km distant from you.
This may be mind boggling, but is it relevant? I think it is. Take this realization for instance: if you have two minima in this high dimensional space, but one is just a tiny bit “flatter” than the other (meaning the second derivatives overall are a bit closer to 0), then the attractor basin of this flatter minimum is vastly larger than that of the other minimum. This is because the flatness implies a larger radius, and the volume depends exponentially on that radius. So, at 175 billion dimensions, even a microscopically larger radius means an overwhelmingly larger volume. If, for instance, one minimum’s attractor basin has a radius that is just 0.00000001% larger than that of the other minimum, then its volume will be roughly 40 million times larger (if my Javascript code to calculate this is accurate enough, that is). And this is only for GPT3, which is almost 4 years old by now.
The parameter space is just ridiculously large, so it becomes really crucial how the search process through it works and where it lands. It may be that somewhere in this vast space, there are indeed attractor basins that correspond to minima that we find extremely undesirable – certain capable optimizers perhaps, that have situational awareness and deceptive tendencies. If they do exist, what could we possibly tell about them? Maybe these minima have huge attractor basins that are reliably found eventually (maybe once we switch to a different network architecture, or find some adjustment to gradient descent, or reach a certain model size, or whatever), which would of course be bad news. Or maybe these attractor basins are so vanishingly small that we basically don’t have to care about them at all, because all the computer & search capacity of humanity over the next million years would have an almost 0 chance of ever stumbling onto these regions. Maybe they are even so small that they are numerically unstable, and even if your search process through some incredible cosmic coincidence happens to start right in such a basin, the first SGD step would immediately jump out of it due to the limitations of numerical accuracy on the hardware we’re using.
So, what can we actually tell at this point about the nature of high dimensional loss landscapes? While reading up on this topic, one thing that constantly came up is the fact that, the more dimensions you have, the lower the relative number of minima becomes compared to saddle points. Meaning that whenever the training process appears to slow down and it looks like it found some local minimum, it’s actually overwhelmingly likely that what it actually found is a saddle point, hence the training process never halts but keeps moving through parameter space, even if the loss doesn't change that much. Do local minima exist at all? I guess it depends on the function the neural network is learning to approximate. Maybe some loss landscapes exist where the loss can just get asymptotically closer to some minimum (such as 0), without ever reaching it. And probably other loss landscapes exist where you actually have a global minimum, as well as several local ones.
Some people argue that you probably have no minima at all, because with each added dimension it becomes less and less likely that a given point is a minimum (because not only does the first derivative of a point have to be 0 for it to be a minimum, also all the second derivatives need to be in on it, and all be positive). This sounds compelling, but given that the space itself also grows exponentially with each dimension, we also have overwhelmingly more points to choose from. If you e.g. look at n-dimensional Perlin Noise, its absolute number of local minima within an n-dimensional cube of constant side length actually increases with each added dimension. However, the relative number of local minima compared to the available space still decreases, so it becomes harder and harder to find them.
I’ll keep it at that. This is already not much of a "quick" take. Basically, more research is needed, as my literature review on this subject yielded way more questions than answers, and many of the claims people made in their blog posts, articles and sometimes even papers seemed to be more intuitive / common-sensical or generalized from maybe-not-that-easy-to-validly-generalize-from research.
One thing I’m sure about however is that almost any explanation of how (stochastic) gradient descent works, that uses 3D landscapes for intuitive visualizations, is misleading in many ways. Maybe it is the best we have, but imho all such explainers should come with huge asterisks, explaining that the rules in very high dimensional spaces may look much different than our naive “oh look at that nice valley over there, let’s walk down to its minimum!” understanding, that happens to work well in three dimensions.
So how did it work out for you?
That seems like a rather uncharitable take. Even if you're mad at the company, would you (at least (~falsely) assuming this all may indeed be standard practice and not as scandalous as it turned out to be) really be willing to pay millions of dollars for the right to e.g. say more critical things on Twitter, that in most cases extremely few people will even care about? I'm not sure if greed is the best framing here.
(Of course the situation is a bit different for AI safety researchers in particular, but even then, there's not that much actual AI (safety) related intel that even Daniel was able to share that the world really needs to know about; most of the criticism OpenAI is dealing with now is on this meta NDA/equity level)
I would assume ChatGPT gets much better at answering such questions if you add to the initial prompt (or system prompt) to eg think carefully before answering. Which makes me wonder whether "ChatGPT is (not) intelligent" even is a meaningful statement at all, given how vastly different personalities (and intelligences) it can emulate, based on context/prompting alone. Probably a somewhat more meaningful question would be what the "maximum intelligence" is that ChatGPT can emulate, which can be very different from its standard form.
Just to note your last paragraph reminds me of Stuart Russel's approach to AI alignment in Human Compatible. And I agree this sounds like a reasonable starting point.
Thanks for the post, I find this unique style really refreshing.
I would add to it that there's even an "alignment problem" on the individual level. A single human in different circumstances and at different times can have quite different, sometimes incompatible values, preferences and priorities. And even at any given moment their values may be internally inconsistent and contradictory. So this problem exists on many different levels. We haven't "solved ethics", humanity disagrees about everything, even individual humans disagree with themselves, and now we're suddenly racing towards a point where we need to give AI a definite idea of what is good & acceptable.
Aren't LLMs already capable of two very different kinds of search? Firstly, their whole deal is predicting the next token - which is a kind of search. They're evaluation all the tokens at every step, and in the end choose the most probable seeming one. Secondly, across-token search when prompted accordingly. Say "Please come up with 10 options for X, then rate them all according to Y, and select the best option" is something that current LLMs can perform very reliably - whether or not "within token search" exists as well. But then again, one might of course argue that search happening within a single forward pass, and maybe even a type of search that "emerged " via SGD rather than being hard baked into the architecture, would be particularly interesting/important/dangerous. We just shouldn't make the mistake of assuming that this would be the only type of search that's relevant.
I think across-token search via prompting already has the potential to lead to the AGI like problems that we associate with mesa optimizers. Evidently the technology is not quite there yet because PoCs like AutoGPT basically don't quite work, so far. But conditional on AGI being developed in the next few years, it would seem very likely to me that this kind of search would be the one that enables it, rather than some hidden "O(1)" search deeply within the network itself.
Edit: I should of course add a "thanks for the post" and mention that I enjoyed reading it, and it made some very useful points!
Great post! Two thoughts that came to mind while reading it:
- the post mostly discussed search happening directly within the network, e.g. within a single forward pass; but what can also happen e.g. in the case of LLMs is that search happens across token-generation rather than within. E.g. you could give ChatGPT a chess constellation and then ask it to list all the valid moves, and then check which move would lead to which state, and if that state looks better than the last one. This would be search depth 1 of course, but still a form of search. In practice it may be difficult because ChatGPT likes to give messages only of a certain length, so it probably stops prematurely if the search space gets too big, but still, search most definitely takes place in this case.
- somewhat of a project proposal, ignoring my previous point and getting back to "search within a single forward pass of the network": let's assume we can "intelligent design" our way to a neural network that actually does implement some kind of small search to solve a problem. So we know the NN is on some pretty optimal solution for the problem it solves. What does (S)GD look like at or very near to this point? Would it stay close to this optimum, or maybe instantly diverge away, e.g. because the optimum's attractor basin is so unimaginably tiny in weight space that it's just numerically highly unstable? If the latter (and if this finding indeed generalizes meaningfully), then one could assume that even though search "exists" in parameter space, it's impractical to ever be reached via SGD due to the unfriendly shape of the search space.
Thanks a lot! Appreciated, I've adjusted the post accordingly.