LessWrong 2.0 Reader

This is gonna sound mean, but the quality of EA-oriented online spaces has really gone downhill in the last 5 years. I barely even noticed Kat Woods' behavior, because she is just one more in a sea of high volume, low quality content being posted in EA spaces.

That's what I've given up mostly on EA sites and events, other than attending EA Global (can't break my streak), and just hang out here on Less Wrong where the vibes are still good and the quality bar is higher.

Found an annotated version of Vernor Vinge's A Fire Upon The Deep. 

I like this effort, and I have a few suggestions:

• Humanoid robots are much more difficult than non-humanoid ones. There are a lot more joints than in other designs; the balance question demands both more capable components and more advanced controls; as a consequence of the balance and shape questions, a lot of thought needs to go into wrangling weight ratios, which means preferring more expensive materials for lightness, etc.
• In terms of modifying your analysis, I think this cashes out as greater material intensity - the calculations here are done by weight of materials, we just need a way to account for the humanoid robot requiring more processing on all of those materials. We could say something like 1500kg of humanoid robot materials take twice as much processing/refinement as 1500kg of car materials (occasionally this will be about the same; for small fractions of the weight it will be 10x the processing, etc).
• The humanoid robots are more vulnerable to bottlenecks than cars. Specifically they need more compute and rare earth elements like neodymium, which will be tough because that supply chain is already strained by new datacenters and AI demands.

I am actually mildly surprised OA has bothered to deploy o1-pro at all, instead of keeping it private and investing the compute into more bootstrapping of o3 training etc.

I'd expect that deploying more capable models is still quite useful, as it's one of the best ways to generate high-quality training data. In addition to solutions, you need problems to solve, and confirmation that the problem has been solved. Or is your point that they already have all the data they need, and it's just a matter of speding compute to refine that?

I used to assume disabling a GPU in my physical possession would be impossible, but now I'm not so sure.  There might be ways to make bypassing GPU lockouts on the order of difficulty of manufacturing the GPU (requiring nanoscale silicon surgery). Here's an example scheme:

Nvidia changes their business models from selling GPUs to renting them.  The GPU is free, but to use your GPU you must buy Nvidia Dollars from Nvidia.  Your GPU will periodically call Nvidia headquarters and get an authorization code to do 10^15 more floating point operations.  This rental model is actually kinda nice for the AI companies, who are much more capital constrained than Nvidia.  (Lots of industries have moved from this buy to rent model, e.g. airplane engines)

Question: "But I'm an engineer.  How (the hell) could Nvidia keep me from hacking a GPU in my physical possession to bypass that Nvidia dollar rental bullshit?"

Answer: through public key cryptography and the fact that semiconductor parts are very small and modifying them is hard.

In dozens to hundreds or thousands of places on the GPU, NVidia places toll units that block signal lines (like ones that pipe floating point numbers around) unless the toll units believe they have been paid with Nvidia dollars.

The toll units have within them a random number generator, a public key ROM unique to that toll unit, a 128 bit register for a secret challenge word, elliptic curve cryptography circuitry, and a $$$ counter which decrements every time the clock or signal line changes.

If the $ $ $ counter is positive, the toll unit is happy and will let signals through unabated.  But if the $ $ $ counter reaches zero,^[1] the toll unit is unhappy and will block those signals.

To add to the $$$ counter, the toll unit (1) generates a random secret <challenge word>, (2) encrypts the secret using that toll unit's public key (3) sends <encrypted secret challenge word> to a non-secure parts of the GPU,^[2] which (4) through driver software and the internet, phones NVidia saying "toll unit <id> challenges you with <encrypted secret challenge word>" (5) Nvidia looks up the private key for toll unit <id> and replies to the GPU  "toll unit <id>, as proof that I Nvidia know your private key, I decrypted your challenge word: <challenge word>", (6) after getting this challenge word back, the toll unit adds 10^15 or whatever to the $$$ counter.

There are a lot of ways to bypass this kind of toll unit (fix the random number generator or $$$ counter to a constant, just connect wires to route around it).  But the point is to make it so you can't break a toll unit without doing surgery to delicate silicon parts which are distributed in dozens to hundreds of places around the GPU chip.

1. ^{^}

   Implementation note: you need to make your registers and random number generators resistant to systematic ways of turning a lot of bits in your register on or off at once (e.g. freezing the GPU).

2. ^{^}

   This can be done slowly, serially with a single line

• How do we carve out "the system" from "the environment", i.e. how do we draw a Cartesian boundary, in order to roughly match human instincts like "looks like it's robustly optimizing for X"? That's an open question, and probably a special case of the more general question of how humans abstract out subsystems from their environment. (This was actually relevant in the previous section too, but it's more apparent once agency is introduced.)

Another interesting question is "how can we derive a theory of agency that is consistent and derives answers even when such a boundary can be arbitrarily shifted?"

For example, quantum mechanics has exactly this issue, with the natural question of "who measures the measurement apparatus", and the solution in quantum mechanics is to declare that the boundary is arbitrary, and that the description must remain consistent even if the boundary is shifted arbitrarily.

Indeed, physically universal cellular automatons, proved to exist by Luke Schaeffer, treats the controller (in this case agent) as the same type of physical system as the system to be controlled (in this case environment), without any boundary, or at best a boundary that can be shifted arbitrarily, which makes them potential foundations for a more workable theory of agency than the old dualistic view (for our universe).

More here:

https://arxiv.org/abs/1009.1720

https://eccc.weizmann.ac.il//report/2014/084/

https://arxiv.org/abs/1501.03988

Anything that threatens a unipolar takeover risks provoking a nuclear war. [...] In the future, most cognition will be non-human. The game dynamics we start now must be good enough for human and non-human interests, so that these future players have an interest in upholding the peaceful nature of the game. That is our ultimate protection.

I think you overestimate the peacefulness of what we have now. We don't have a nuclear war, but we have conventional wars all the time. Only those who have the nukes are safe, but that doesn't include everyone.

So I can similarly imagine a future multipolar world, where GalaxyBrain1 and GalaxyBrain2 hold each other at bay by threatening to destroy the universe using some futuristic weapon, and then GalaxyBrain1 says "I really really want to exterminate humans" and GalaxyBrain2 says "I'd kinda prefer if you didn't, but hey, it is definitely not worth risking the destruction of the universe, so... go ahead, just don't touch my data centers".

Relevant link [LW(p) · GW(p)]

Apparently someone let LLMs play against the random policy and for most of them, most games end in a draw. Seems like o1-preview is the best of those tested, managing to win 47% of the time.

One of the more common responses I hear at this point is some variation of “general intelligence isn’t A Thing, people just learn a giant pile of specialized heuristics via iteration and memetic spread

I'm very uncertain about the validity of the below question but I shalt ask it anyway and since I don't trust my own way of expressing it, here's claude on it:

The post argues that humans must have some general intelligence capability beyond just learning specialized heuristics, based on efficiency arguments in high-dimensional environments. However, research on cultural evolution (e.g., "The Secret of Our Success", "Cognitive Gadgets") suggests that much of human capability comes from distributed cultural learning and adaptation. Couldn't this cultural scaffolding, combined with domain-specific inductive biases (as suggested by work in Geometric Deep Learning), provide the efficiency gains you attribute to general intelligence? In other words, perhaps the efficiency comes not from individual general intelligence, but from the collective accumulation and transmission of specialized cognitive tools?

I do agree that there are specific generalised forms of intelligence, I guess this more points me towards that the generating functions of these might not be optimally sub-divided in the usual way we think about it?

Now completely theoretically of course, say someone where to believe the above, why is the following really stupid?:

Specifically, consider the following proposal: Instead of trying to directly align individual agents' objectives, we could focus on creating environmental conditions and incentive structures that naturally promote collaborative behavior. The idea being that just as virtue ethics suggests developing good character through practiced habits and environmental shaping, we might achieve alignment through carefully designed collective dynamics that encourage beneficial emergent behaviors. (Since this seems to be the most agentic underlying process that we currently have, theoretically of course.)