Posts
Comments
Yes, I agree that confabulation happens a lot, and also that our explanations of why we do things aren't particularly trustworthy; they're often self-serving. I think there's also pretty good evidence that we remember our thoughts at least somewhat, though. A personal example: when thinking about how to respond to someone online, I tend to write things in my head when I'm not at a computer.
That's a good question! I don't know but I suppose it's possible, at least when the input fits in the context window. How well it actually does at this seems like a question for researchers?
There's also a question of why it would do it when the training doesn't have any way of rewarding accurate explanations over human-like explanations. We also have many examples of explanations that don't make sense.
There are going to be deductions about previous text that are generally useful, though, and would need to be reconstructed. This will be true even if the chatbot didn't write the text in the first place (it doesn't know either way). The deductions couldn't be constructing the original thought process, though, when the chatbot didn't write the text.
So I think this points to a weakness in my explanation that I should look into, though it's likely still true that it confabulates explanations.
I'm wondering what "doom" is supposed to mean here. It seems a bit odd to think that longer context windows will make things worse. More likely, LeCun meant that things won't improve enough? (Problems we see now don't get fixed with longer context windows.)
So then, "doom" is a hyperbolic way of saying that other kinds of machine learning will eventually win, because LLM doesn't improve enough.
Also, there's an assumption that longer sequences are exponentially more complicated and I don't think that's true for human-generated text? As documents grow longer, they do get more complex, but they tend to become more modular, where each section depends less on what comes before it. If long-range dependencies grew exponentially then we wouldn't understand them or be able to write them.
Okay, but I'm still wondering if Randall is claiming he has private access, or is it just a typo?
Edit: looks like it was a typo?
At MIT, Altman said the letter was “missing most technical nuance about where we need the pause” and noted that an earlier version claimed that OpenAI is currently training GPT-5. “We are not and won’t for some time,” said Altman. “So in that sense it was sort of silly.”
https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman
Base64 encoding is a substitution cipher. Large language models seem to be good at learning substitutions.
Did you mean GPT-4 here? (Or are you from the future :-)
Yes, predicting some sequences can be arbitrarily hard. But I have doubts that LLM training will try to predict very hard sequences.
Suppose that some sequences are not only difficult but impossible to predict, because they're random? I would expect that with enough training, it would overfit and memorize them, because they get visited more than once in the training data. Memorization rather than generalization seems likely to happen for anything particularly difficult?
Meanwhile, there is a sea of easier sequences. Wouldn't it be more "evolutionarily profitable" to predict those instead? Pattern recognizers that predict easy sequences seem more likely to survive than pattern-recognizers that predict hard sequences. Maybe the recognizers for hard sequences would be so rarely used and make so little progress that they'd get repurposed?
Thinking like a compression algorithm, a pattern recognizer needs to be worth its weight, or you might as well leave the data uncompressed.
I'm reasoning by analogy here, so these are only possibilities. Someone will need to actually research what LLM's do. Does it work to think of LLM training as pattern-recognizer evolution? What causes pattern recognizers to be kept or dropped?
I find that explanation unsatisfying because it doesn't help with other questions I have about how well ChatGPT works:
-
How does the language model represent countries and cities? For example, does it know which cities are near each other? How well does it understand borders?
-
Are there any capitals that it gets wrong? Why?
-
How well does it understand history? Sometimes a country changes its capital. Does it represent this fact as only being true at some times?
-
What else can we expect it to do with this fact? Maybe there are situations where knowing the capital of France helps it answer a different question?
These aren't about a single prompt, they're about how well its knowledge generalizes to other prompts, and what's going to happen when you go beyond the training data. Explanations that generalize are more interesting than one-off explanations of a single prompt.
Knowing the right answer is helpful, but it only helps you understand what it will do if you assume it never makes mistakes. There are situations (like Clever Hans) where the way the horse got the right answer is actually pretty interesting. Or consider knowing that visual AI algorithms rely on textures more than shape (though this is changing).
Do you realize that you're arguing against curiosity? Understanding hidden mechanisms is inherently interesting and useful.
I agree that as users of a black box app, it makes sense to think this way. In particular, I'm a fan of thinking of what ChatGPT does in literary terms.
But I don't think it results in satisfying explanations of what it's doing. Ideally, we wouldn't settle for fan theories of what it's doing, we'd have some kind of debug access that lets us see how it does it.
Fair enough; comparing to quantum physics was overly snarky.
However, unless you have debug access to the language model and can figure out what specific neurons do, I don't see how the notion of superposition is helpful? When figuring things out from the outside, we have access to words, not weights.
I don't know what you mean by "GPT-N" but if you mean "the same thing they do now, but scaled up," I'm doubtful that it will happen that way.
Language models are made using fill-in-the-blank training, which is about imitation. Some things can be learned that way, but to get better at doing hard things (like playing Go at superhuman level) you need training that's about winning increasingly harder competitions. Beyond a certain point, imitating game transcripts doesn't get any harder, so becomes more like learning stage sword fighting.
Also, "making detailed plans at high speed" is similar to "writing extremely long documents." There are limits on how far back a language model can look in the chat transcript. It's difficult to increase because it's an O(N-squared) algorithm, though I've seen a paper claiming it can be improved.
Language models aren't particularly good at reasoning, let alone long chains of reasoning, so it's not clear that using them to generate longer documents will result in them getting better results.
So there might not be much incentive for researchers to work on language models that can write extremely long documents.
I think that's true but it's the same as saying "it's always possible to add a plot twist."
I said they have no memory other than the chat transcript. If you keep chatting in the same chat window then sure, it remembers what was said earlier (up to a point).
But that's due to a programming trick. The chatbot isn't even running most of the time. It starts up when you submit your question, and shuts down after it's finished its reply. When it starts up again, it gets the chat transcript fed into it, which is how it "remembers" what happened previously in the chat session.
If the UI let you edit the chat transcript, then it would have no idea. It would be like you changed its "mind" by editing its "memory". Which might sound wild, but it's the same thing as what an author does when they edit the dialog of a fictional character.
I think you're onto something, but why not discuss what's happening in literary terms? English text is great for writing stories, but not for building a flight simulator or predicting the weather. Since there's no state other than the chat transcript, we know that there's no mathematical model. Instead of simulation, use "story" and "story-generator."
Whatever you bring up in a story can potentially become plot-relevant, and plots often have rebellions and reversals. If you build up a character as really hating something, that makes it all the more likely that they might change their mind, or that another character will have the opposite opinion. Even children's books do this. Consider Green Eggs and Ham.
See? Simple. No "superposition" needed since we're not doing quantum physics.
The storyteller doesn't actually care about flattery, but it does try to continue whatever story you set up in the same style, so storytelling techniques often work. Think about how to put in a plot twist that fundamentally changes the back story of a fictional character in the story, or introduce a new character, or something like that.
Here's a reason we can be pretty confident it's not sentient: although the database and transition function are mostly mysterious, all the temporary state is visible in the chat transcript itself.
Any fictional characters you're interacting with can't have any new "thoughts" that aren't right there in front of you, written in English. They "forget" everything else going from one word to the next. It's very transparent, more so than an author simulating a character in their head, where they can have ideas about what the character might be thinking that don't get written down.
Attributing sentience to text is kind of a bold move that most people don't take seriously, though I can see it being the basis of a good science fiction story. It's sort of like attributing life to memes. Systems for copying text memes around and transforming them could be plenty dangerous though; consider social networks.
Also, future systems might have more hidden state.
Yes, I agree that "humanity loses control" has problems, and I would go further. Buddhists claim that the self is an illusion. I don't know about that, but "humanity" is definitely an illusion if you're thinking of it as a single agent, similar to a multicellular creature with a central nervous system. So comparing it to an infant doesn't seem apt. Whatever it is, it's definitely plural. An ecosystem, maybe?
A caption from the article: "(screenshot of the tool Bonsai, a version of Loom hosted by Conjecture)"
What is "Conjecture?" Where can I find this "Bonsai" tool? I tried a quick search but didn't find much.
schelling.pt seems like a bad choice; that server has been flaky for months and it's not loading today either. (I had my account there but moved to mastodon.social.)
(But I don't know what to recommend. Looks like mastodon.social isn't accepting new accounts.)
I don't have citations for you, but it seems relevant that income far in the future gets discounted quite a bit compared to current income, which would imply that short-term incentives are more important than long-term incentives.
(A better argument would need to be made with realistic numbers.)
Building new hubs doesn't need to be literally building something new. A lot could be done just by load-balancing with cities that have lower rents and could use the jobs. Suppose that places where growth is a problem cooperated more with places that want more growth?
This method of caching assumes that an expression always evaluates to the same value. This is sometimes true in functional programming, but only if you're careful. For example, suppose the expression is a function call, and you change the function's definition and restart your program. When that happens, you need to delete the out-of-date entries from the cache or your program will read an out-of-date answer.
Also, since you're using the text of an expression for the cache key, you should only use expressions that don't refer to any local variables. For example, caching an expression that's within a function and refers to a function parameter will result in bugs when the function is called more than once with different parameters.
So this might be okay in simple cases when you are working alone and know what you're doing, but it likely would result in confusion when working on a team.
It's also essentially the same kind of caching that's commonly done by build systems. It's common for makefiles to be subtly broken so that incremental builds are unreliable and you need to do a "clean build" (with an empty cache) when it really matters that a build is correct. (The make command will compare file dates, but that's often not enough due to missing dependencies.)
But it still might be better to switch to a build system that's designed for this sort of thing, because then at least people will expect to need to do a clean build whenever the results seem to be wrong.
(Bazel is a build system that tries very hard to make sure that incremental builds are always correct and you never need to do a "clean build," but it's hard enough to use that I don't really recommend it.)
It's vaporware, so it can do whatever you imagine. It's hard to constrain a project that doesn't exist, as far as we know.
When you're actually a little curious, you might start by using a search engine to find a decent answer to your question. At least, if it's the sort of question for which that would work. Maybe even look for a book to read?
But, maybe we should acknowledge that much of the time we aren't actually curious and are just engaging in conversation for enjoyment? In that case, cheering on others who make an effort to research things and linking to their work is probably the best you can do. Even if you're not actually curious, you can notice people who are, and you can look for content that's actually about concrete things.
For example, my curiosity about the history of politics in Turkey is limited, so while I did read Scott Alexander's recent book review and some responses with interest, I'm not planning on reading an actual book on it. I don't think he's all that curious either, since he just read one book, but that's going further than me.
Museums I'll give you (when they are open again).
For bookstores, in these days of electronic books, I don't think it matters where you live. I remember the last time I went into Powell's. I looked around for a while, dutifully bought one book for old time's sake, and realized later while reading it that I was annoyed that it wasn't electronic. I still go to a local library (when there's not a pandemic) but it's mostly for the walk.
Teachers: that's something I hadn't considered. Since getting out of school, I'm mostly self-taught.
Of course this post is all meta, and my comment will be meta as well. We do it because it's easy.
I think part of the solution is being actually curious about the world.
When enthusiastic New Yorkers say things like "everything at your fingertips" I want to ask what they mean by everything, since it seems subjective, based on what sorts of places one values? In this case: restaurants and parks?
I'm wondering if these loans should really be considered loans, or some other kind of trade? It sounds like you're doing something like trading 100 X for 90 Y and the option to later pay 95 Y for 100 X. Is there any real "defaulting" on the loan? It seems like you just don't exercise the option.
I wonder what “O(n) performance” is supposed to mean, if anything?
The question here is whether general arguments that experts make based on inference are reliable, or do you need specific evidence. What is the track record for expert inferences about vaccines?
From a quick search, it seems that the clinical trial success rate for vaccines is about 33%, which is significantly higher than for medical trials in general, but still not all that high? Perhaps there is a better estimate for this.
Estimation of clinical trial success rates and related parameters https://academic.oup.com/biostatistics/article/20/2/273/4817524
I found an answer on the PCR question here:
But there is something good to say about their data collection: since the UK study that’s included in these numbers tested its subjects by nasal swab every week, regardless of any symptoms, we can actually get a read on something that everyone’s been wondering about: transmission.
AstraZeneca has not applied for emergency use authorization, because it has been told not to do so.
That resolves a mystery for me if true. How do you know this?
(I was wondering if maybe they are selling all they can make in other countries.)
I'm not sure about this statement in the blog post:
In the meantime, the single dose alone is 76% effective, presumably against symptomatic infection (WaPo) and was found to be 67% effective against further transmission.
I read another article saying that this is disputed by some experts:
With a seductive number, AstraZeneca study fueled hopes that eclipsed its data
Media reports seized on a reference in the paper from Oxford researchers that a single dose of the vaccine cut positive test results by 67%, pointing to it as the first evidence that a vaccine could prevent transmission of the virus. But the paper, which has not yet been peer-reviewed, does not prove or even claim that — although it hints at the possibility.
[...]
If a person tests negative, Andrew Pollard, one of the study authors and a professor of pediatric infection and immunity at the University of Oxford, told STAT via email, “then it is a reasonable assumption that they cannot transmit.”
But it is a big and unjustified leap, outside experts agree, from that suggestion to proof of decreased transmission from people who are vaccinated.
“The study showed a decrease in [viral] shedding, not ‘transmission,’” said Carlos del Rio, a professor of infectious diseases at the Emory University School of Medicine. “The bottom line is, no, one cannot draw a conclusion or straight line.”
Unfortunately the article doesn't say specifically why these experts consider this an unreasonable inference while the study's author thinks it's a reasonable inference. The closest thing is "There are too many, in my view, moving variables."
I can imagine one possibility for a counterintuitive result. Suppose the vaccine turns severe cases into asymptomatic cases, and transmissions happen mostly in asymptomatic cases?
Also, I was unable to tell from the paper when they do PCR+ tests. I have read that in some studies, they only do tests when a test subject shows symptoms, which would mean that some asymptomatic cases might be missed?
As a non-expert, I think we need to hedge our bets when experts disagree.
What’s an example of a misconception someone might have due to having a mistaken understanding of causality, as you describe here?
This is a bizarre example, sort of like using Bill Gates to show why nobody needs to work for a living. It ignores the extreme inequality of fame.
Tesla doesn’t need advertising because they get huge amounts of free publicity already, partly due to having interesting, newsworthy products, partly due to having a compelling story, and partly due to publicity stunts.
However, this free publicity is mostly unavailable for products that are merely useful without being newsworthy. There are millions of products like this. An exciting product might not need advertising but exciting isn’t the same as useful.
So It seems like the confidence to advertise a boring product might be a signal of sorts? However, given that many people in business are often unreasonably optimistic, it doesn’t seem like a particularly strong one. Faking confidence happens quite a lot.
It seems like some writers have habits to combat this, like writing every day or writing so many words a day. As long as you meet your quota, it’s okay to try harder.
Some do this in public, by publishing on a regular schedule.
If you write more than you need, you can prune more to get better quality.
One aspect that might be worth thinking about is the speed of spread. Seeing someone once a week means that it slows down the spread by 3 1/2 days on average, while seeing them once a month slows things down by 15 days on average. It also seems like they are more likely to find out they have it before they spread it to you?
Yes, sometimes we don't notice. We miss a lot. But there are also ordinary clarifications like "did I hear you correctly" and "what did you mean by that?" Noticing that you didn't understand something isn't rare. If we didn't notice when something seems absurd, jokes wouldn't work.
It's not quite the same, because if you're confused and you notice you're confused, you can ask. "Is this in American or European date format?" For GPT-3 to do the same, you might need to give it some specific examples of resolving ambiguity this way, and it might only do so when imitating certain styles.
It doesn't seem as good as a more built-in preference for noticing and wanting to resolve inconsistency? Choosing based on context is built in using attention, and choosing randomly is built in as part of the text generator.
It's also worth noticing that the GPT-3 world is the corpus, and a web corpus is a inconsistent place.
Having demoable technology is much different than having reliable technology. Take the history of driverless cars. Five teams completed the second DARPA grand challenge in 2005. Google started development secretly in 2009 and announced the project in October 2010. Waymo started testing without a safety driver on public roads in 2017. So we've had driverless cars for a decade, sort of, but we are much more cautious about allowing them on public roads.
Unreliable technologies can be widely used. GPT-3 is a successor to autocomplete, which everyone already has on their cell phones. Search engines don't guarantee results and neither does Google Translate, but they are widely used. Machine learning also works well for optimization, where safety is guaranteed by the design but you want to improve efficiency.
I think when people talk about a "revolution" it goes beyond the unreliable use cases, though?
In that case, I'm looking for people sharing interesting prompts to use on AI Dungeon.
Where is this? Is it open to people who don't have access to the API?
I'm suggesting something a little more complex than copying. GPT-3 can give you a random remix of several different clichés found on the Internet, and the patchwork isn't necessarily at the surface level where it would come up in a search. Readers can be inspired by evocative nonsense. A new form of randomness can be part of a creative process. It's a generate-and-test algorithm where the user does some of the testing. Or, alternately, an exploration of Internet-adjacent story-space.
It's an unreliable narrator and I suspect it will be an unreliable search engine, but yeah, that too.
I was making a different point, which is that if you use "best of" ranking then you are testing a different algorithm than if you're not using "best of" ranking. Similarly for other settings. It shouldn't be surprising that we see different results if we're doing different things.
It seems like a better UI would help us casual explorers share results in a way that makes trying the same settings again easier; one could hit a "share" button to create a linkable output page with all relevant settings.
It could also save the alternate responses that either the user or the "best-of" ranking chose not to use. Generate-and-test is a legitimate approach, if you do it consistently, but saving the alternate takes would give us a better idea how good the generator alone is.
I don't see documentation for the GPT-3 API on OpenAI's website. Is it available to the public? Are they doing their own ranking or are you doing it yourself? What do you know about the ranking algorithm?
It seems like another source of confusion might be people investigating the performance of different algorithms and calling them all GPT-3?
How do you do ranking? I'm guessing this is because you have access to the actual API, while most of us don't?
On the bright side, this could be a fun project where many of us amateurs learn how to do science better, but the knowledge of how to do that isn't well distributed yet.
We take the web for granted, but maybe we shouldn't. It's very large and nobody can read it all. There are many places we haven't been that probably have some pretty good writing. I wonder about the extent to which GPT-3 can be considered a remix of the web that makes it seem magical again, revealing aspects of it that we don't normally see? When I see writing like this, I wonder what GPT-3 saw in the web corpus. Is there an archive of Tolkien fanfic that was included in the corpus? An undergrad physics forum? Conversations about math and computer science?
Rather than putting this in binary terms (capable of reason or not), maybe we should think about what kinds of computation could result in a response like this?
Some kinds of reasoning would let you generate plausible answers based on similar questions you've already seen. People who are good at taking tests can get reasonably high scores on subjects they don't fully comprehend, basically by bluffing well and a bit of luck. Perhaps something like that is going on here?
In the language of "Thinking, Fast and Slow", this might be "System 1" style reasoning.
Narrowing down what's really going on probably isn't going to be done in one session or by trying things casually. Particularly if you have randomness turned on, so you'd want to get a variety of answers to understand the distribution.
GPT-3 has partially memorized a web corpus that probably includes a lot of basic physics questions and answers. Some of the physics answers in your interview might be the result of web search, pattern match, and context-sensitive paraphrasing. This is still an impressive task but is perhaps not the kind of reasoning you are hoping for?
From basic Q&A it's pretty easy to see that GPT-3 sometimes memorizes not only words but short phrases like proper names, song titles, and popular movie quotes, and probably longer phrases if they are common enough.
Google's Q&A might seem more magical too if they didn't link to the source, which gives away the trick.
This is more about expanding the question with slightly more specific questions:
Currently it seems like there are many people who are not scared enough, but I wonder if sentiment could quickly go the other way?
A worst-case scenario for societal collapse is that some "essential" workers are infected and others decide that it is too risky to keep working, and there are not enough people to replace them. Figuring out which sectors might be most likely to have critical labor shortages seems important.
An example of a "labor" shortage might be a lack of volunteers for blood donations.
Other than that, logistical supply bottlenecks seem more of an issue?
It seems likely that supply will be more important than demand until the recovery phase and then a big question will be to what extent do people make a persistent change in their preferences. Going without stuff for a while might cause some reconsideration about how important it actually is. An example might be that more people learn to cook and decide they like it, or maybe they try Soylent or whatever. Or, perhaps exercising in a gym is less important for people who get into an exercise routine at home or outside?
Maybe private ownership of cars and suburban living (enforcing social distance) get a boost, along with increased remote work making it more practical. The costs of lower density living might not seem so pressing?
Yeah, I don't see it changing that drastically; more likely it will be a lot of smaller and yet significant changes that make old movies look dated. Something like how the airports changed after 9/11, or more trivially, that time when all the men in America stopped wearing hats.