LessWrong 2.0 Reader

I appreciate that, though it seems since yesterday my post may have been downvoted even more. I wouldn't mind as much except nobody has explained why when I bothered putting in the effort. I could think maybe it's because of the clickbait-y title, or on account of the fact that it's a YouTube video meant to convey important info about AI to, like, normies in a mainstream way, and is therefore assumed to be of super low quality.

Yet that'd be in spite of facts that:
1. I clarified this is from theoretical physicist and science communicator who's trying to inform the public in an approachable way, which is something I figure others on LessWrong could appreciate.
2. I have now summarized the details as best as I can so that others on LessWrong don't need to bother with digesting the info through a medium they don't prefer, in spite of the fact that having that available in a multimedia format is more preferred by others, so it could recognized how it's constructive there are multiple options.

I suspect part of it might just be a latent preference on LessWrong for the sort of lengthy blog posts in a style they're accustomed to, which is valid, but a tendency to presume the same sort of info they like being exposed to but delivered in a different way means it must be lower quality. That could be a bias I might question, though it's fair enough if others just disagree. I'm just hoping you can offer insight into whether I should keep bothering with the effort of posts like this because I'm the one who's off here, or others just have superficial reactions. 

@Nathan Helm-Burger [LW · GW] I know this was a month ago but I've also been working with Goodfire batches a lot and I have some stuff that can help now.

https://github.com/keenanpepper/langchain-goodfire

What I've been doing for the most part is creating a langfire client with an InMemoryRateLimiter, then just starting off all my requests in a big parallel batch and doing asyncio.gather().

None of the labs would be doing undirected drift. That wouldn't yield improvement for exactly the reasons you suggest.

In the absence of a ground truth quality/correctness signal, optimizing for coherence works. This can give prettier answers (in the way that averaged faces are prettier) but this is limited. The inference time scaling equivalent would be a branching sampling approach that searches for especially preferred token sequences rather than the current greedy sampling approach. Optimising for idea level coherence can improve model thinking to some extent.

For improving raw intelligence significantly, ground truth is necessary. That's available in STEM domains, computer programming tasks being the most accessible. One can imagine grounding hard engineering the same way with a good mechanical/electrical simulation package. TLDR:train for test-time performance.

Then just cross your fingers and hope for transfer learning into softer domains.

For softer domains, ground truth is still accessible via tests on humans (EG:optimise for user approval). This will eventually yield super-persuaders that get thumbs up from users. Persuasion performance is trainable but maybe not a wise thing to train for.

As to actually improving some soft domain skill like "write better english prose" that's not easy to optimise directly as you've observed.

>Safety is limited to refusals, notably including refusals for medical or legal advice. Have they deliberately restricted those abilities to avoid lawsuits or to limit public perceptions of expertise being overtaken rapidly by AI? 

I think it's been well over a year since I've had an issue with getting an LLM to give me medical advice, including GPT-4o and other SOTA models like Claude 3.5/7, Grok 3 and Gemini 2.0 Pro. I seem to recall that the original GPT-4 would occasionally refuse, but could be coaxed into it. 

I am a doctor, and I tend to include that information either in model memory or in a prompt (mostly to encourage the LLM to assume background knowledge and ability to interpret facts). Even without it, my impression is that most models simply append a "consult a human doctor" boilerplate disclaimer instead of refusing. 

I would be rather annoyed if GPT 4.5 was a reversion in that regard, as I find LLMs immensely useful for quick checks on topics I'm personally unfamiliar with (and while hallucinations happen, they're quite rare now, especially with search, reasoning and grounding). I don't think OAI or other AI companies have faced any significant amount of litigation from either people who received bad advice, or doctors afraid of losing a job. 

I'm curious about whether anyone has had any issues in that regard, though I'd expect not. 

GPT-4.5 API price for input tokens is $75/1M to GPT-4o's $2.5/1M, and knowledge cutoff is Oct 2023, the same for both. This strongly suggests that GPT-4.5 is compute optimal while GPT-4o is overtrained^[1], so there might be a future overtrained variant of GPT-4.5 in the works.

The mystery of the base model for o3 remains unresolved, but unchanged knowledge cutoff negates one of the arguments against it being GPT-4.5^[2], while the plot of ARC-AGI success rate depending on the size of the problem continues to argue in favor (o3 maintains good performance for much larger puzzles than o1 does).

I had a couple-year long obsession with evolution simulators. And that hobby convinced me, that longevity is a pretty risky thing. Most of dominant species in the simulations went extinct not because of new species emerging and conflicting, but because of longevity causing problems. 

1) Reduces amount of resources available to the new generation. That increases competition, and this kind of competition is evolutionary force which encourages being a long-living line, because it is easier to accumulate experience and use it later. Creates a feedback loop with increasing longevity. (Some analogue of this can be followed when looking at land properties - young people already have more problems than previous generations) 1.5) if there is no conflict netween generations at first, then overpopulation might lead to the same endpoint. 2) Reduces the frequency of new generations. Thus reduces the adaptation rate of the population. As a result, the whole population is less strong. Smaller crisis (caused by environment or other species) can have stronger effects, because damage can outpace the recovery by birth.

Looking at forest fires stats you can see the same picture. Forests that have regular fires and diversity of ages are quickly recovering. But if a forest did not have regular fires, then eventually a fire will appear and it will be more damaging, with a chance of completely eradicating life in the area.

Longevity for the sake of it is a personal egoistic trait dictated by instincts, and it might not be healthy for the human species as a whole.

The very short answer is that the people with the most experience in alignment research (Eliezer and Nate Soares) say that without an AI pause lasting many decades the alignment project is essentially hopeless because there is not enough time. Sure, it is possible the alignment project succeeds in time, but the probability is really low.

Eliezer has said that AIs based on the deep-learning paradigm are probably particularly hard to align, so it would probably help to get a ban or a long pause on that paradigm even if research in other paradigms continues, but good luck getting even that because almost all of the value currently being provided by AI-based services are based on deep-learning AIs.

One would think that it would be reassuring to know that the people running the labs are really smart and obviously want to survive (and have their children survive) but it is only reassuring before one listens to what they say and reads what they write about their plans on how to prevent human extinction and other catastrophic risks. (The plans are all quite inadequate.)

Patching security problems in big old organizations involves problems that go a lot beyond "looking at code and changing it", especially if aiming for a "strong" solution like formal verification.

TL;DR: Political problems, code that makes no sense, problems that would be easy to fix even with a simple LLM that isn't specialized on improving security.

The best public resource I know is about this is Recoding America.

Some examples iirc:

1. Not having a clear primary key to identify people with.
2. Having a website (a form) that theoretically works but doesn't run on any browser that people actually use.
3. Having a security component which is supposed to catch fake submissions of forms, but is way more likely to catch real submissions (it is imo net negative).

I also learned some surprising things from working on fixing/rewriting a major bank in Israel. I can't share such juicy stories as Recoding America publicly, but here are some that I can:

1. "written in kobol" is maybe ~1% of the problem and imo not an interesting pain point to focus on
2. Many systems are microservices (much harder to define the expected functionality of an async system)
   1. Different parts of the bank are written using different technologies
3. An example problem you'd find in the code is "representing amounts of money using javascript floats"
   1. It's not complicated to fix, technically
   2. But people might say "we're used to using floats, why change it?"
   3. Making changes might mean they'll have to run manual tests, so they'll be against it
   4. Some teams just don't like having others touch their code, and maybe some care about their job security
4. Another example easy thing to fix is "use enums"
   1. I've seen more than one conversations where "agreeing to use an enum" was a big political debate
5. Sometimes, to understand a system, I'd do a combination of "looking at the code", "looking at the docs (word documents with the design)", and "talking to one or more of the people in charge of the system" (sometimes different people have different parts of the picture)

[written with the hope that orgs trying to patch security problems will do well]

I roughly agree with that timeline, although I also emphasize the range over a point estimate; we need to be prepared for shorter as well as longer timelines, since there are so many ways to get faster as well as slower advances than we'd expect.

One factor in my estimates after EAG is the practical difficulties of scripting for agents. It is reportedly even more frustrating to "debug" a sequence of prompts and model calls that malfunctions stochastically. I do expect automated processes similar to software improving its error-handling and debugging, but all of this will take time.

I also wonder what your definition of "AGI" is; that's often a deciding factor in timelines, since people mean so many different things. My definition is "can reason about anything, which requires nontrivial continuous learning", which I take to be the original definition; that in turn requires some amount of autonomy for self-directed learning. See "Real AGI" [LW · GW]. But there are lots of other definitions in play now too, and I don't remember what yours is.

Not yet sorry we are working on it