LessWrong 2.0 Reader

We receive about a billion tokens growing to adulthood. The leading LLMs get orders of magnitude more than that. We should be able to do much better.

Capturing this would probably be a big deal, but a counterpoint is that compute necessary to achieve an autonomous researcher using such sample efficient method might still be very large. Possibly so large that training an LLM with the same compute and current sample-inefficient methods is already sufficient to get a similarly effective autonomous researcher chatbot. In which case there is no effect on timelines. And given that the amount of data is not an imminent contraint on scaling [LW(p) · GW(p)], the possibility of this sample efficiency improvement being useless for the human-led stage of AI development won't be ruled out for some time yet.

The article suggests "invisible advantage". (Other options: "unnoticed advantage", "unknown advantage".)

far side away from the red steam engine.

King's Cross railway station's train entrance faces north, and it's passenger entrance faces south, so the far side would be the north end of the station, where the steam engine has to be in order to leave the station.

If that's the case, wouldn't the far side be closer to the steam engine?

Yeah, that's part of why I'm suspicious. I remember the original OA finetuning as being quite expensive, but the current one is not that expensive. If a GPT-3 is like 100GB of weights, say, after optimization, and it's doing true finetuning, how is OA making it so cheap and so low-latency?

Welcome! Looking forward to having you around!

I think an AI is slightly more likely to wipe out or capture humanity than it is to wipe out all life on the planet.

While any true Scottsman ASI is so far above us humans as we are above ants and does not need to worry about any meatbags plotting its downfall, as we don't generally worry about ants, it is entirely possible that the first AI which has a serious shot at taking over the world is not quite at that level yet. Perhaps it is only as smart as von Neumann and a thousand times faster. 

To such an AI, the continued thriving of humans poses all sorts of x-risks. They might find out you are misaligned and coordinate to shut you down. More worrisome, they might summon another unaligned AI which you would have to battle or concede utility to  later on, depending on your decision theory.

Even if you still need some humans to dust your fans and manufacture your chips, suffering billions of humans to live in high tech societies you do not fully control seems like the kind of rookie mistake I would not expect a reasonably smart unaligned AI to make. 

By contrast, most of life on Earth might get snuffed out when the ASI gets around to building a Dyson sphere around the sun. A few simple life forms might even be spread throughout the light cone by an ASI who does not give a damn about biological contamination. 

The other reason I think the fate in store for humans might be worse than that for rodents is that alignment efforts might not only fail, but fail catastrophically. So instead of an AI which cares about paperclips, we get an AI which cares about humans, but in ways we really do not appreciate.

But yeah, most forms of ASI which turn out for out bad for homo sapiens also turn out bad for most other species. 

To me the strongest evidence that fine-tuning is based on LoRA or similar is the fact that pricing is based just on training and input / output and doesn't factor in the cost of storing your fine-tuned models. Llama-3-8b-instruct is ~16GB (I think this ought to be roughly comparable, at least in the same ballpark). You'd almost surely care if you were storing that much data for each fine-tune.

Will read this in detail later when I can, but on first skim -- I've seen you draw that conclusion in earlier comments. Are you assuming you yourself will finally be deanonymized soon? No pressure to answer, of course; it's a pretty personal question, and answering might itself give away a bit or two.

It is certainly big news if OA fine-tuning doesn't work as it's supposed to

The docs are pretty vague, but I notice that most of them are framed as being around declarative sorts of knowledge. It's positioned as being a way to reduce the number of examples in the prompt (to save tokens & reduce latency), or include additional factual knowledge, like defining edge cases. There is one brief mention that you may be able to use it for "Performing a new skill or task that’s hard to articulate in a prompt", but that's about it.

And when it comes to lightweight finetuning such as LoRA, people tend to notice that they are good for adding new factual knowledge or increasing the prior of specific pre-existing knowledge, but don't really add qualitatively new things - like you cannot simply LoRA your way to better hands in an image generator or teach it 3D generation if it didn't already know that. So I've long been suspicious that OA isn't doing real finetuning, of the entire model, but much cheaper underperforming LoRA-like lightweight finetuning (of the sort which can be easily stored on-GPU rather than loading an entire finetuned model or its delta from cloud storage, or tying up entire sets of GPUs to keep a full finetuned model hot).

One sanity check here would be to just make some 128k ctx window calls full of examples; if you cannot k-shot this capability even with that, then you shouldn't expect "finetuning" to either; while if it works, that implies the "finetuning" is much worse than it ought to be and so the original results are uninformative.

This sounds like you're saying that they made a rational prioritization and then, separately from that, forgot that it was there

That implication wasn't intended. I agree that (for basic reasons) the probability of a small cut was higher given their choice.

Rather, the action itself seems rational to me when considering:

1. That outcome seems unprobable (at least if they were sitting down), but actual in this particular timeline.
2. The effects of a cut on the foot are really low (with I'd guess >99.5% probability, for an otherwise healthy person - on reflection, maybe not cumulatively low enough for the also-small payoff?), and if so ~certain to not directly and significantly curtail progress.

That doesn't necessarily imply the policy which produced the action is rational, though. But when considering the two hypotheses: (1) OP is mentally unwell, and (2) They have some them-specific reason^[1] for following a policy which outputs actions like this, I considered (2) to be a lot more probable.

Meta: This comment is (genuinely) very hard/overwhelming-feeling for me to try to reply to, for a few reasons specific to my mind, mainly about {unmarked assumptions} and {parts seeming to be for rhetorical effect}. (For that reason I'll let others discuss this instead of saying much further)

I think it's also really important for there to be clear, public signals that the community wants people to take their well-being seriously

I agree with this, but I think any 'community norm reinforcing messages' should be marked as such rather than framed about an individual, in cases like this where there's just a weak datapoint about the individual.

1. ^{^}

   A simple example would be "Having introspected and tested different policies before determining that they're not at risk of burnout from the policy which gives this action."

   A more complex example would be "a particular action can be irrational in isolation but downstream of a policy which produces irrational behavior less than is typical", which (now) seems to me to be OP was trying to show with this example given their comment