LessWrong 2.0 Reader

I want to push harder on Q33: "Isn't goal agnosticism pretty fragile? Aren't there strong pressures pushing anything tool-like towards more direct agency?"

In particular, the answer: "Being unable to specify a sufficiently precise goal to get your desired behavior out of an optimizer isn't merely dangerous, it's useless!" seems true to some degree, but incomplete.  Let's use a specific hypothetical of a stock-trading company employing an AI system to maximize profits.  They want the system to be agentic because this takes the humans out of the loop on actually getting profits, but also understand that there is a risk that the system will discover unexpected/undesired methods of achieving its goals like insider trading.  There are a couple of core problems:

1. Externalized Cost: if the system can cover its tracks well enough that the company doesn't suffer any legal consequences for its illegal behavior, then the effects of insider trading on the market are "somebody else's problem."
2. Irreversible Mistake: if the company is overly optimistic about their ability to control their system, doesn't understand the risks, etc. then they might use it despite regretting this decision later.  On a large scale, this might be self-correcting if some companies have problems with AI agents and this gives the latter a bad reputation, but that assumes there are lots of small problems before a big one.

At first I thought this was a tutorial on how to catch a talented liar, and it didn’t seem that accurate. As I read, I realized that this is a tutorial on how to create common knowledge between yourself and a bad liar that you know they are lying, even if they are very stupid. This is also an interesting task, and I appreciate your insight on how to approach it.

Good point. But I think the real game changer will be self-modification tech, not longevity tech. In that case we won't have a "slow adaptation" problem, but we'll have a "fast adaptation in weird directions" problem which is probably worse.

I appreciate that, though it seems since yesterday my post may have been downvoted even more. I wouldn't mind as much except nobody has explained why when I bothered putting in the effort. I could think maybe it's because of the clickbait-y title, or on account of the fact that it's a YouTube video meant to convey important info about AI to, like, normies in a mainstream way, and is therefore assumed to be of super low quality.

Yet that'd be in spite of facts that:
1. I clarified this is from theoretical physicist and science communicator who's trying to inform the public in an approachable way, which is something I figure others on LessWrong could appreciate.
2. I have now summarized the details as best as I can so that others on LessWrong don't need to bother with digesting the info through a medium they don't prefer, in spite of the fact that having that available in a multimedia format is more preferred by others, so it could recognized how it's constructive there are multiple options.

I suspect part of it might just be a latent preference on LessWrong for the sort of lengthy blog posts in a style they're accustomed to, which is valid, but a tendency to presume the same sort of info they like being exposed to but delivered in a different way means it must be lower quality. That could be a bias I might question, though it's fair enough if others just disagree. I'm just hoping you can offer insight into whether I should keep bothering with the effort of posts like this because I'm the one who's off here, or others just have superficial reactions. 

@Nathan Helm-Burger [LW · GW] I know this was a month ago but I've also been working with Goodfire batches a lot and I have some stuff that can help now.

https://github.com/keenanpepper/langchain-goodfire

What I've been doing for the most part is creating a langfire client with an InMemoryRateLimiter, then just starting off all my requests in a big parallel batch and doing asyncio.gather().

None of the labs would be doing undirected drift. That wouldn't yield improvement for exactly the reasons you suggest.

In the absence of a ground truth quality/correctness signal, optimizing for coherence works. This can give prettier answers (in the way that averaged faces are prettier) but this is limited. The inference time scaling equivalent would be a branching sampling approach that searches for especially preferred token sequences rather than the current greedy sampling approach. Optimising for idea level coherence can improve model thinking to some extent.

For improving raw intelligence significantly, ground truth is necessary. That's available in STEM domains, computer programming tasks being the most accessible. One can imagine grounding hard engineering the same way with a good mechanical/electrical simulation package. TLDR:train for test-time performance.

Then just cross your fingers and hope for transfer learning into softer domains.

For softer domains, ground truth is still accessible via tests on humans (EG:optimise for user approval). This will eventually yield super-persuaders that get thumbs up from users. Persuasion performance is trainable but maybe not a wise thing to train for.

As to actually improving some soft domain skill like "write better english prose" that's not easy to optimise directly as you've observed.

>Safety is limited to refusals, notably including refusals for medical or legal advice. Have they deliberately restricted those abilities to avoid lawsuits or to limit public perceptions of expertise being overtaken rapidly by AI? 

I think it's been well over a year since I've had an issue with getting an LLM to give me medical advice, including GPT-4o and other SOTA models like Claude 3.5/7, Grok 3 and Gemini 2.0 Pro. I seem to recall that the original GPT-4 would occasionally refuse, but could be coaxed into it. 

I am a doctor, and I tend to include that information either in model memory or in a prompt (mostly to encourage the LLM to assume background knowledge and ability to interpret facts). Even without it, my impression is that most models simply append a "consult a human doctor" boilerplate disclaimer instead of refusing. 

I would be rather annoyed if GPT 4.5 was a reversion in that regard, as I find LLMs immensely useful for quick checks on topics I'm personally unfamiliar with (and while hallucinations happen, they're quite rare now, especially with search, reasoning and grounding). I don't think OAI or other AI companies have faced any significant amount of litigation from either people who received bad advice, or doctors afraid of losing a job. 

I'm curious about whether anyone has had any issues in that regard, though I'd expect not. 

GPT-4.5 API price for input tokens is $75/1M to GPT-4o's $2.5/1M, and knowledge cutoff is Oct 2023, the same for both. This strongly suggests that GPT-4.5 is compute optimal while GPT-4o is overtrained^[1], so there might be a future overtrained variant of GPT-4.5 in the works.

The mystery of the base model for o3 remains unresolved, but unchanged knowledge cutoff negates one of the arguments against it being GPT-4.5^[2], while the plot of ARC-AGI success rate depending on the size of the problem continues to argue in favor (o3 maintains good performance for much larger puzzles than o1 does).

I had a couple-year long obsession with evolution simulators. And that hobby convinced me, that longevity is a pretty risky thing. Most of dominant species in the simulations went extinct not because of new species emerging and conflicting, but because of longevity causing problems. 

1) Reduces amount of resources available to the new generation. That increases competition, and this kind of competition is evolutionary force which encourages being a long-living line, because it is easier to accumulate experience and use it later. Creates a feedback loop with increasing longevity. (Some analogue of this can be followed when looking at land properties - young people already have more problems than previous generations) 1.5) if there is no conflict netween generations at first, then overpopulation might lead to the same endpoint. 2) Reduces the frequency of new generations. Thus reduces the adaptation rate of the population. As a result, the whole population is less strong. Smaller crisis (caused by environment or other species) can have stronger effects, because damage can outpace the recovery by birth.

Looking at forest fires stats you can see the same picture. Forests that have regular fires and diversity of ages are quickly recovering. But if a forest did not have regular fires, then eventually a fire will appear and it will be more damaging, with a chance of completely eradicating life in the area.

Longevity for the sake of it is a personal egoistic trait dictated by instincts, and it might not be healthy for the human species as a whole.

The very short answer is that the people with the most experience in alignment research (Eliezer and Nate Soares) say that without an AI pause lasting many decades the alignment project is essentially hopeless because there is not enough time. Sure, it is possible the alignment project succeeds in time, but the probability is really low.

Eliezer has said that AIs based on the deep-learning paradigm are probably particularly hard to align, so it would probably help to get a ban or a long pause on that paradigm even if research in other paradigms continues, but good luck getting even that because almost all of the value currently being provided by AI-based services are based on deep-learning AIs.

One would think that it would be reassuring to know that the people running the labs are really smart and obviously want to survive (and have their children survive) but it is only reassuring before one listens to what they say and reads what they write about their plans on how to prevent human extinction and other catastrophic risks. (The plans are all quite inadequate.)