LessWrong 2.0 Reader

Nice post!

My key takeaway: "A system is aligned to human values if it tends to generate optimized-looking stuff which is aligned to human values."

I think this is useful progress. In particular it's good to try to aim for the AI to produce some particular result in the world, rather than trying to make the AI have some goal - it grounds you in the thing you actually care about in the end.

I'd say the "... aligned to human values part" is still underspecified (and I think you at least partially agree):

• "aligned": how does the ontology translation between the representation of the "generated optimized-looking stuff" and the representation of human values look like?
• "human values"
  • I think your model of humans is too simplistic. E.g. at the very least it's lacking a distinction like between "ego-syntonic" and "voluntary" as in this post [LW · GW], though I'd probably want a even significantly more detailed model. Also one might need different models for very smart and reflective people than for most people.
  • We haven't described value extrapolation.
    • (Or from an alternative perspective, our model of humans doesn't identify their relevant metapreferences (which probably no human knows fully explicitly, and for some/many humans it they might not be really well defined).)

Positive reinforcement for first trying to better understand the problem before running off and trying to solve it! I think that's the way to make progress, and I'd encourage others to continue work on more precisely defining the problem, and in particular on getting better models of human cognition to identify how we might be able to rebind the "human values" concept to a better model of what's happening in human minds.

Btw, I'd have put the corrigibility section into a separate post, it's not nearly up to the standards of the rest of this post.

To set expectations: this post will not discuss ...

Maybe you want to add here that this is not meant to be an overview of alignment difficulties, or an explanation for why alignment is hard.

This is gonna sound mean, but the quality of EA-oriented online spaces has really gone downhill in the last 5 years. I barely even noticed Kat Woods' behavior, because she is just one more in a sea of high volume, low quality content being posted in EA spaces.

That's what I've given up mostly on EA sites and events, other than attending EA Global (can't break my streak), and just hang out here on Less Wrong where the vibes are still good and the quality bar is higher.

Found an annotated version of Vernor Vinge's A Fire Upon The Deep. 

I like this effort, and I have a few suggestions:

• Humanoid robots are much more difficult than non-humanoid ones. There are a lot more joints than in other designs; the balance question demands both more capable components and more advanced controls; as a consequence of the balance and shape questions, a lot of thought needs to go into wrangling weight ratios, which means preferring more expensive materials for lightness, etc.
• In terms of modifying your analysis, I think this cashes out as greater material intensity - the calculations here are done by weight of materials, we just need a way to account for the humanoid robot requiring more processing on all of those materials. We could say something like 1500kg of humanoid robot materials take twice as much processing/refinement as 1500kg of car materials (occasionally this will be about the same; for small fractions of the weight it will be 10x the processing, etc).
• The humanoid robots are more vulnerable to bottlenecks than cars. Specifically they need more compute and rare earth elements like neodymium, which will be tough because that supply chain is already strained by new datacenters and AI demands.

I am actually mildly surprised OA has bothered to deploy o1-pro at all, instead of keeping it private and investing the compute into more bootstrapping of o3 training etc.

I'd expect that deploying more capable models is still quite useful, as it's one of the best ways to generate high-quality training data. In addition to solutions, you need problems to solve, and confirmation that the problem has been solved. Or is your point that they already have all the data they need, and it's just a matter of speding compute to refine that?

I used to assume disabling a GPU in my physical possession would be impossible, but now I'm not so sure.  There might be ways to make bypassing GPU lockouts on the order of difficulty of manufacturing the GPU (requiring nanoscale silicon surgery). Here's an example scheme:

Nvidia changes their business models from selling GPUs to renting them.  The GPU is free, but to use your GPU you must buy Nvidia Dollars from Nvidia.  Your GPU will periodically call Nvidia headquarters and get an authorization code to do 10^15 more floating point operations.  This rental model is actually kinda nice for the AI companies, who are much more capital constrained than Nvidia.  (Lots of industries have moved from this buy to rent model, e.g. airplane engines)

Question: "But I'm an engineer.  How (the hell) could Nvidia keep me from hacking a GPU in my physical possession to bypass that Nvidia dollar rental bullshit?"

Answer: through public key cryptography and the fact that semiconductor parts are very small and modifying them is hard.

In dozens to hundreds or thousands of places on the GPU, NVidia places toll units that block signal lines (like ones that pipe floating point numbers around) unless the toll units believe they have been paid with enough Nvidia dollars.

The toll units have within them a random number generator, a public key ROM unique to that toll unit, a 128 bit register for a secret challenge word, elliptic curve cryptography circuitry, and a $$$ counter which decrements every time the clock or signal line changes.

If the $ $ $ counter is positive, the toll unit is happy and will let signals through unabated.  But if the $ $ $ counter reaches zero,^[1] the toll unit is unhappy and will block those signals.

To add to the $$$ counter, the toll unit (1) generates a random secret <challenge word>, (2) encrypts the secret using that toll unit's public key (3) sends <encrypted secret challenge word> to a non-secure parts of the GPU,^[2] which (4) through driver software and the internet, phones NVidia saying "toll unit <id> challenges you with <encrypted secret challenge word>" (5) Nvidia looks up the private key for toll unit <id> and replies to the GPU  "toll unit <id>, as proof that I Nvidia know your private key, I decrypted your challenge word: <challenge word>", (6) after getting this challenge word back, the toll unit adds 10^15 or whatever to the $$$ counter.

There are a lot of ways to bypass this kind of toll unit (fix the random number generator or $$$ counter to a constant, just connect wires to route around it).  But the point is to make it so you can't break a toll unit without doing surgery to delicate silicon parts which are distributed in dozens to hundreds of places around the GPU chip.

1. ^{^}

   Implementation note: you need to make sure your registers and random number generators resistant to systematic ways of turning a lot of bits in your register on or off at once (e.g. freezing the GPU).  This is possible with e.g. error correcting circuits

2. ^{^}

   This can be done slowly, serially with a single line

• How do we carve out "the system" from "the environment", i.e. how do we draw a Cartesian boundary, in order to roughly match human instincts like "looks like it's robustly optimizing for X"? That's an open question, and probably a special case of the more general question of how humans abstract out subsystems from their environment. (This was actually relevant in the previous section too, but it's more apparent once agency is introduced.)

Another interesting question is "how can we derive a theory of agency that is consistent and derives answers even when such a boundary can be arbitrarily shifted?"

For example, quantum mechanics has exactly this issue, with the natural question of "who measures the measurement apparatus", and the solution in quantum mechanics is to declare that the boundary is arbitrary, and that the description must remain consistent even if the boundary is shifted arbitrarily.

Indeed, physically universal cellular automatons, proved to exist by Luke Schaeffer, treats the controller (in this case agent) as the same type of physical system as the system to be controlled (in this case environment), without any boundary, or at best a boundary that can be shifted arbitrarily, which makes them potential foundations for a more workable theory of agency than the old dualistic view (for our universe).

More here:

https://arxiv.org/abs/1009.1720

https://eccc.weizmann.ac.il//report/2014/084/

https://arxiv.org/abs/1501.03988

Anything that threatens a unipolar takeover risks provoking a nuclear war. [...] In the future, most cognition will be non-human. The game dynamics we start now must be good enough for human and non-human interests, so that these future players have an interest in upholding the peaceful nature of the game. That is our ultimate protection.

I think you overestimate the peacefulness of what we have now. We don't have a nuclear war, but we have conventional wars all the time. Only those who have the nukes are safe, but that doesn't include everyone.

So I can similarly imagine a future multipolar world, where GalaxyBrain1 and GalaxyBrain2 hold each other at bay by threatening to destroy the universe using some futuristic weapon, and then GalaxyBrain1 says "I really really want to exterminate humans" and GalaxyBrain2 says "I'd kinda prefer if you didn't, but hey, it is definitely not worth risking the destruction of the universe, so... go ahead, just don't touch my data centers".

Relevant link [LW(p) · GW(p)]

Apparently someone let LLMs play against the random policy and for most of them, most games end in a draw. Seems like o1-preview is the best of those tested, managing to win 47% of the time.