LessWrong 2.0 Reader
View: New · Old · Topnext page (older posts) →
next page (older posts) →
I'm confused, why does that make the term no longer useful? There's still a large distinction between companies focusing on developing AGI (OpenAI, Anthropic, etc.) vs those focusing on more 'mundane' advancements (Stability, Black Forest, the majority of ML research results). Though I do disagree that it was 'only' used to distinguish them from narrow AI.
I agree 'AGI' has become an increasingly vague term, but that's because it is a useful distinction and so certain groups use it to hype.
We should talk more about specific cognitive capabilities, but that isn't stopped by us using the term AGI, it is stopped by not having people analyzing whether X is an important capability for risk or capability for stopping risk.
You seem overly anchored on COT as the only scaffolding system in the near-mid future (2-5 years). While I'm uncertain what specific architectures will emerge, the space of possible augmentations (memory systems, tool use, multi-agent interactions, etc.) seems vastly larger than current COT implementations.
COT (and particularly the extension tree of thoughts) seems like the strongest of those to me, probably because I can see an analogy to Solomonoff induction -> AIXI. I am curious whether you have some particular more sophisticated memory system in mind?
My point is that these are all things that might work, but there is no strong reason to think they will - particularly to the extent of being all that we need. AI progress is usually on the scale of decades and often comes from unexpected places (though for the main line, ~always involving a neural net in some capacity).
Something like that seems like it would be a MVP of "actually try and get an LLM to come up with something significantly economically valuable. I expect that the lack of this type of experiment existing is because major AI labs feel like that would be choosing to exploit while there are still many gains to be made from exploring further architectural and scaffolding-esque improvements.
I find this kind of hard to swallow - a huge number of people are using and researching LLMs, I suspect that if something like this "just works" we would know by now. I mean, it would certainly win a lot of acclaim for the first group to pull it off, so the incentives seem sufficient - and it doesn't seem that hard to pursue this in parallel to basic research on LLMs. Plus, the two investments are synergistic; for example, one would probably learn about the limitations of current models by pursuing this line. Maybe Anthropic is too small and focused to try it, but GDM could easily spin off a team.
Where you say "Certainly LLMs should be useful tools for coding, but perhaps not in a qualitatively different way than the internet is a useful tool for coding, and the internet didn't rapidly set off a singularity in coding speed.", I find this to be untrue both in terms of the impact of the internet (while it did not cause a short takeoff, it did dramatically increase the amount of new programmers and the effective transfer of information between them. I expect without it we would see computers having <20% of their current economic impact), and in terms of the current and expected future impact of LLM's (LLM's simply are widely used by smart/capable programmers. I trust them to evaluate if it is noticeably better than StackOverflow/the rest of the internet).
I expect LLMs to offer significant advantages above the internet. I am simply pointing out that not every positive feedback loop is a singularity. I expect great coding assistants (essentially excellent autocomplete) but not drop-in replacements for software engineers any time soon. This is one factor that will increase the pace of AI research somewhat, but also Moore's law is running out, which will definitely slow the pace. Not sure which one wins out directionally.
cole-wyeth on My model of what is going on with LLMsI think the central claim is plausible, and would very much like to find out I'm in a world where AGI is decades away instead of years. We might be ready by then.
Me too!
If I am reading this correctly, there are two specific tests you mention:
1) GPT-5 level models come out on schedule (as @Julian Bradshaw [LW · GW] noted, we are still well within the expected timeframe based on trends to this point)
See my response to his comment - I think its not so clear that projecting those trends invalidates my model, but it really depends on whether GPT-5 is actually a qualitative upgrade comparable to the previous steps, which we do not know yet.
2) LLMs or agents built on LLMs do something "important" in some field of science, math, or writing
I would add on test 2 that neither have almost all humans. We don't have a clear explanation for why some humans have much more of this capability than others, and yet all the human brains are running on similar hardware and software. This suggests the number of additional insights needed to boost us from "can't do novel important things" to "can do" may be as small as zero, though I don't think it is actually zero. In any case, I am hesitant to embrace a test for AGI that a large majority of humans fail.
This seems about right, but there are two points to keep in mind.
a) It is more surprising that LLMs can't do anything important because their knowledge far surpasses any humans, which indicates that there is some kind of cognitive function qualitatively missing.
b) I think that about the bottom 30% (very rough estimate) of humans in developed nations are essentially un-agentic. The kind of major discoveries and creations I pointed to mostly come from the top 1%. However, I think that in the middle of that range there are still plenty of people capable of knowledge work. I don't see LLMs managing the sort of project that would take a mediocre mid-level employee a week or month. So there's a gap here, even between LLMs and ordinary humans. I am not as certain about this as I am about the stronger test, but it lines up with my experience with DeepResearch - I asked it for a literature review of my field and it had pretty serious problems that would have made it unusable, despite requiring ~no knowledge creation (I can email you an annotated copy if you're interested).
In practical terms, suppose this summer OpenAI releases GPT-5-o4, and by winter it's the lead author on a theoretical physics or pure math paper (or at least the main contributor - legal considerations about personhood and IP might stop people from calling AI the author). How would that affect your thinking?
Assuming the results of the paper are true (everyone would check) and at least somewhat novel/interesting (~sufficient for the journal to be credible) this would completely change my mind. As I said, it is a crux.
rvnnt on rvnnt's ShortformA potentially somewhat important thing which I haven't seen discussed:
(This looks like a decisionmaker is not the beneficiary -type of situation.)
Why does that matter?
It has implications for modeling decisionmakers, interpreting their words, and for how to interact with them.[1]
If we are in a gradual-takeoff world[2], then we should perhaps not be too surprised to see the wealthy and powerful push for AI-related policies that make them more wealthy and powerful, while a majority of humans become disempowered and starve to death (or live in destitution, or get put down with viruses or robotic armies, or whatever). (OTOH, I'm not sure if that possibility can be planned/prepared for, so maybe that's irrelevant, actually?)
For example: we maybe should not expect decisionmakers to take risks from AI seriously until they realize those risks include a high probability of "I, personally, will die". As another example: when people like JD Vance output rhetoric like "[AI] is not going to replace human beings. It will never replace human beings" [LW · GW], we should perhaps not just infer that "Vance does not believe in AGI", but instead also assign some probability to hypotheses like "Vance thinks AGI will in fact replace lots of human beings, just not him personally; and he maybe does not believe in ASI, or imagines he will be able to control ASI". ↩︎
Here I'll define "gradual takeoff" very loosely as "a world in which there is a >1 year window during which it is possible to replace >90% of human labor, before the first ASI comes into existence". ↩︎
Wow, crazy timing for the GPT-5 announcement! I'll come back to that, but first the dates that you helpfully collected:
It's not clear to me that this timeline points in the direction you are arguing. Exponentially increasing time between "step" improvements in models would mean that progress rapidly slows to the scale of decades. In practice this would probably look like a new paradigm with more low-hanging fruit overtaking or extending transformers.
I think your point is valid in the sense that things were already slowing down by GPT-3 -> GPT-4, which makes my original statement at least potentially misleading. However, research and compute investment have also been ramping up drastically - I don't know by exactly how much, but I would guess nearly an order of magnitude? So the wait times here may not really be comparable.
Anyway, this whole speculative discussion will soon (?) be washed out when we actually see GPT-5. The announcement is perhaps a weak update against my position, but really the thing to watch is whether it is a qualitative improvement on the scale of previous GPT-N -> GPT-(N+1). If it is, then you are right that progress has not slowed down much. My standard is whether it starts doing anything important.
gunnar_zarncke on Gunnar_Zarncke's ShortformYes. And either way it would be a decent outcome. Unless people come to the wrong conclusions about what the problem is, e.g. "it's the companies fault."
lblack on Proof idea: SLT to AITI do use programs that give probabilistic outputs here. See claim 1 and the setup section.
I am fairly sure that there is a version of Solomonoff induction where the programs themselves output probabilities, and it's equivalent in the limit to the version where programs output binary answers. I think it's called 'stochastic' Solomonoff induction, or something like that.
I hadn't actually realised how load-bearing this might be for the proof until you pointed it out though. Thank you.
Simplified the solomonoff prior is the distribution you get when you take a uniform distribution over all strings and feed them to a turing machine.
Since the outputs are also strings: What happens if we iterate this? What is the stationary distribution? Is there even one? The fixed points will be quines, programs that copy their source code to the output. But how are they weighted? By their length? Presumably you can also have quine-cycles of programs that generate each other in turn, in a manner reminiscent metagenesis. Do these quine cycles capture all probability mass or does some diverge?
Very grateful for answers and literature suggestions.
The time and space bounded SI is then approximately optimal in the sense that its total error compared to this efficient predictor, as measured by KL-divergence from the predictions made by P∗, will be ≤|P∗| bits summed across all data points.[7]
I think that classical Solomonoff induction gives zero posterior to any program with less than perfect prediction record? I can see why this works for Solomonoff with unbounded description length, this is solved by the DH(h) term you mention above.
But for bounded Solomonoff you have to allow some non-perfect programs to stay in the posterior, or you might be left with no candidates at all?
Is there an easy way to deal with this?
This is not a problem if the programs are giving probabilistic outputs, but that's a different class of programs than used in classical Solomonoff induction.
When you start a new chat, you reset the memory, if I understand it correctly. Maybe you should do that once in a while. Then you may need to explain stuff again, but maybe it gives you a new perspective? Or you could write the background story in a text file, and copy-paste it to each new chat.
Could the LLM accidentally reinforce negative thought patterns or make unhelpful suggestions?
I am not an expert, but I think that LLMs are prone to agreeing with the user, so if you keep posting negative thought patterns, there is a risk that LLM will reflect them back to you.
What if the LLM gives advice that contradicts what my therapist says? How would I know what to do?
Trust the therapist, I guess? And maybe bring it up in the next session, kinda "I got an advice to do X, what is your opinion on that?".
What is the risk of becoming too dependent on the LLM, and how can I check for that?
No idea. People are different; things that are harmless for 99 may hurt 1. I don't know you.
Are there specific prompts or ways of talking to the LLM that would make it safer or more helpful for this kind of use?
Just guessing here, but maybe specify its role (not just "you are a therapist", but specifically e.g. "a Rogerian therapist"), and specify the goal you want to achieve ("your purpose is to help me get better in my everyday life" or something).
Maybe at the end ask LLM for a summary of things that you should tell your human therapist? Something like "now I need to switch to another therapist, prepare notes for him so that he can continue the therapy".