LessWrong 2.0 Reader

Not on sci-hub or Anna's Archive, so I'm just going off the abstract and summary here; would love a PDF if anyone has one.

If you email the authors they will probably send you the full article.

Thanks for the support. I'll try and work a bit more on my first post in the coming days and I hope it will be up soon.

Some rough notes from Michael Aird's workshop on project selection in AI safety. 

Tl;dr how to do better projects? 

• Backchain to identify projects.
• Get early feedback, iterate quickly
• Find a niche

On backchaining projects from theories of change

• Identify a "variable of interest"  (e.g., the likelihood that big labs detect scheming).
• Explain how this variable connects to end goals (e.g. AI safety).
• Assess how projects affect this variable
• Red-team these. Ask people to red team these. 

On seeking feedback, iteration. 

• Be nimble. Empirical. Iterate. 80/20 things
• Ask explicitly for negative feedback. People often hesitate to criticise, so make it socially acceptable to do so
• Get high-quality feedback. Ask "the best person who still has time for you". 

On testing fit

• Forward-chain from your skills, available opportunities, career goals.
• "Speedrun" projects. Write papers with hypothetical data and decide whether they'd be interesting. If not then move on to something else.
• Don't settle for "pretty good". Try to find something that feels "amazing" to do, e.g. because you're growing a lot / making a lot of progress. 

Other points

On developing a career

• "T-shaped" model of skills; very deep in one thing and have workable knowledge of other things
• Aim for depth first. Become "world-class" at something. This ensures you get the best possible feedback at your niche and gives you a value proposition within larger organization. After that, you can broaden your scope. 

Product-oriented vs field-building research

• Some research is 'product oriented', i.e. the output is intended to be used directly by somebody else
• Other research is 'field building', e.g. giving a proof of concept, or demonstrating the importance of something. You (and your skills / knowledge) are the product. 

A specific process to quickly update towards doing better research. 

1. Write “Career & goals 2-pager”
2. Solicit ideas from mentor, experts, decision-makers (esp important!)
3. Spend ~1h learning about each plausible idea (very brief). Think about impact, tractability, alignment with career goals, personal fit, theory of change
4. “Speedrun” the best idea (10-15h). (Consider using dummy data! What would the result look like?)
5. Get feedback on that, reflect, iterate.
6. Repeat steps 1-5 as necessary. If done well, steps 1-5 only take a few days! Either keep going (if you feel good), or switch to different topic (if you don't). 

The usual story about where rank > 1 density matrices come from is when your subsystem is entangled with an environment that you can't observe. 

The simplest example is to take a Bell state, say 

|00> + |11>  (obviously I'm ignoring normalization) and imagine you only have access to the first qubit; how should you represent this state? Precisely because it's entangled, we know that there is no |Psi> in 1-qubit space that will work. The trace method alluded to in the post is to form the (rank-1) density matrix of the Bell state, and then "trace out" the second system; if you think of the density matrix as living in M_2 tensor M_2, this means applying the trace operator just to the right side of the tensor, i.e. mapping matrix units E_ij tensor E_kl to delta_kl E_ij and then extending by linearity.

You can check that for this example you get the (normalized) 2x2 identity matrix. 

You can think of this tracing out process as a quantum version of marginalization. To get a feel for it intuitively, it's useful to consider the following: suppose you are given access to an endless supply of one-of-a-Bell-pair qubits, and you make repeated measurements, what will you see? 

It's pretty clear that if you measure in the standard basis, you'll have a 50/50 chance of measuring |0> or |1>. This is the sort of thing a first-timer might pattern match to an equal superposition but that's not correct, no matter what basis you measure in you'll obtain 50/50--this is because 

conceptually: you're measuring half an entangled state so the whole point is it can't yield a given state with certainly under measurement

mathematically: the Bell state can be written as 

|xx> + |yy> for any orthogonal states |x>, |y> , there's nothing special about the standard basis. So in any measurement, the entangled state has an equal chance of both being x and both being y, so sometime who can only see one qubit will see an equal chance of x and of y

formalism-ly (?): the rule for calculating measurement probabilities, |<x, y>|^2 = <x|(|y><y|)|x> where y is your state, and x is the state whose probability after measurement you wish to know, generalizes obviously to <x|rho|x> for any density matrix; in our case rho is a multiple of the identity, and all states are norm 1, so all potential measurement outcomes yield the same probability.

The point about the Lindbladian is that it's pretty generic for rank-1 states to evolve to higher rank mixed states; it's basically the same idea as decoherence: you entangle with the rest of the world, but then lose track of all the precise degrees of freedom, so you only really see a small subsystem of a large entangled state. 

Indeed it's true a given high rank density matrix can have multiple purifications--rank-1 states of which it is the traced out part corresponding to one subsystem--but that's to be expected, in this point of view: if we had perfect knowledge of the whole system, including everything our subsystem had ever become entangled with, we'd use a regular, pure state. The use of a mixed, higher rank density matrix corresponds to our loss of information to the "environment". And yes, the rank of the density matrix is related to the minimal dimension of the Hilbert space needed for an environment to purify your density matrix.

Current take on the implications of "GPT-4b micro": Very powerful, very cool, ~zero progress to AGI, ~zero existential risk. Cheers.

First, the gist of it appears to be:

OpenAI’s new model, called GPT-4b micro, was trained to suggest ways to re-engineer the protein factors to increase their function. According to OpenAI, researchers used the model’s suggestions to change two of the Yamanaka factors to be more than 50 times as effective—at least according to some preliminary measures.

The model was trained on examples of protein sequences from many species, as well as information on which proteins tend to interact with one another. [...] Once Retro scientists were given the model, they tried to steer it to suggest possible redesigns of the Yamanaka proteins. The prompting tactic used is similar to the “few-shot” method, in which a user queries a chatbot by providing a series of examples with answers, followed by an example for the bot to respond to.

Crucially, if the reporting is accurate, this not an agent. The model did not engage in autonomous open-ended research. Rather, humans guessed that if a specific model is fine-tuned on a specific dataset, the gradient descent would chisel into it the functionality that would allow it to produce groundbreaking results in the corresponding domain. As far as AGI-ness goes, this is functionally similar to AlphaFold 2; as far as agency goes, it's at most at the level of o1.

To speculate on what happened: Perhaps GPT-4b ("b" = "bio"?) is based on some distillation of an o-series model, say o3. o3's internals contain a lot of advanced machinery for mathematical reasoning. What this result shows, then, is that the protein-factors problem is in some sense a "shallow" mathematical problem that could be easily solved if you think about it the right way. Finding the right way to think about it is itself highly challenging, however – a problem teams of brilliant people have failed to crack – yet deep learning allowed to automate this search and crack it.

This trick likely generalizes. There may be many problems in the world that could be cracked this way^[1]: those that are secretly "mathematically shallow" in this manner, and for which you can get a clean-enough fine-tuning dataset.

... Which is to say, this almost certainly doesn't cover social manipulation/scheming (no clean dataset), and likely doesn't cover AI R&D (too messy/open-ended, although I can see being wrong about this). (Edit: And if it Just Worked given any sorta-related sorta-okay fine-tuning dataset, the o-series would've likely generalized to arbitrary domains out-of-the-box, since the pretraining is effectively this dataset for everything. Yet it doesn't.)

It's also not entirely valid to call that "innovative AI", any more than it was valid to call AlphaFold 2 that. It's an innovative way of leveraging gradient descent for specific scientific challenges, by fine-tuning a model pre-trained in a specific way. But it's not the AI itself picking a field to innovate and then producing the innovation; it's humans finding a niche which this class of highly specific (and highly advanced) tools can fill.

It's not the type of thing that can get out of human control.

So, to restate: By itself, this seems like a straightforwardly good type of AI progress. Zero existential risk or movement in the direction of existential risks, tons of scientific and technological utility.

Indeed, on the contrary: if that's the sort of thing OpenAI employees are excited about nowadays, and what their recent buzz about AGI in 2025-2026 and innovative AI and imminent Singularity was all about, that seems like quite a relief.

1. ^{^}

   And indeed, OpenAI-associated vagueposting suggests there's some other domain in which they'd recently produced a similar result.

Why do you need to have all feature descriptions at the outset? Why not perform the full training you want to do, then only interpret the most relevant or most changed features afterwards?

My sense is political staffers and politicians aren't that great at predicting their future epistemic states this way, and so you won't get great answers for this question. I do think it's a really important one to model!

If items are only available at "gouged" rates, then this will make them more expensive. That is, this tax will fall only on people in the emergency zone, and specifically those who are desperate enough to buy goods at these elevated costs. Since demand is very inelastic under these circumstances, the tax burden will fall almost entirely on the consumer.

Another approach might be to temporarily raise taxes everywhere except the emergency zone on these goods. For example, if bottled water falls under a temporary excise tax during a hurricane everywhere except the hurricane zone, that incentivizes sellers to bring bottled water to the hurricane victims.

Your argument about corporate secrets is sufficient to change my mind on activist patent trolling being a productive strategy against AI X-risk.

The part about funding would need to be solved with philanthropy. I don't believe that org exists, but I don't see why it couldn't.

I'm still curious whether there are other cases in which activist patent trolling can be a good option, such as animal welfare, chemistry, public health, or geoengineering (ie fracking).

I added in two more tags (Redwood Research and AI Risk Skepticism), but I don't know how to properly format them. As it stands, there are extra lines before both of my entries. I am unsure of how to remove them.