LessWrong 2.0 Reader

I purposefully use these terms vaguely since my concepts about them are in fact vague. E.g., when I say “alignment” I am referring to something roughly like “the AI wants what we want.” But what is “wanting,” and what does it mean for something far more powerful to conceptualize that wanting in a similar way, and what might wanting mean as a collective, and so on? All of these questions are very core to what it means for an AI system to be “aligned,” yet I don’t have satisfying or precise answers for any of them. So it seems more natural to me, at this stage of scientific understanding, to simply own that—not to pretend I'm speaking more rigorously than I in fact am, but also not to pretend I know more than I in fact do. 

The goal, of course, is to eventually understand minds well enough to be more precise. But before we get there, most precision will likely be misguided—formalizing the wrong thing, or missing the key relationships, or what have you. And I think this does more harm than good, as it causes us to collectively misplace where the remaining confusion lives, when locating our confusion is (imo) one of the major bottlenecks to solving alignment.  

In practice, it just requires hardware with limited functionality and physical security — hardware security modules exist.

An HSM-analogue for ML would be a piece of hardware that can have model weights loaded into its nonvolatile memory, can perform inference, but doesn't provide a way to get the weights out. (If it's secure enough against physical attack, it could also be used to run closed models on a user's premises, etc.; there might be a market for that.)

Given that the US taxes income everywhere in the world for citizens, I am pretty sure you could at least offset any donations from your US taxes. And then my best guess is you could get out of the same taxes in Australia given various tax treaties, but I am not confident. I would ask Claude or ChatGPT for a rough guidance.

Software only singularity is a singularity driven by just AI R&D on a basically fixed hardware base. As in, can you singularity using only a fixed datacenter (with no additional compute over time) just by improving algorithms? See also here.

This isn't directly talking about the outcomes from this.

You can get a singularity via hardware+software where the AIs are also accelerating the hardware supply chain such that you can use more FLOP to train AIs and you can run more copies. (Analogously to the hyperexponential progress throughout human history seemingly driven by higher population sizes, see here.)

We're probably more skeptical than AI companies and less skeptical than the general lesswrong crowd. We have made such a list and thought about it in a moderate amount of detail.

My current sense is that automated AI safety/alignment research to reach a reasonable exit condition seems hard, but substantially via the mechanism that achieving the desirable exit condition is an objectively very difficult research tasks for which our best ideas aren't amazing, and less so via the mechanism that its impossible to get the relevant AIs to help (including that I don't think it is impossible to get these AIs to help a bunch if they are scheming / egregiously misaligned).

Concretely, I'd target the exit condition of producing AIs (or potentially emulated human minds) which:

• Are capable and (philosophically) wise enough to be capable of obsoleting almost all humans including top human researchers. Note this doesn't (clearly) require such AIs to be wildly superhuman.
• And which are aligned enough that we're happy to defer to them on tricky hard to check questions like "how should we proceed from here given our preferences (feel free to ask us questions to clarify our preferences)"

The basic hope would be to first rule out threat models that might make behavioral testing misleading (ruling out at least scheming, potentially reward hacking) and then constructing good enough behavioral tests of wisdom and iterating against these in ways that seem unlikely to substantially goodhart.

(Such AIs wouldn't be CEV aligned and we wouldn't be happy totally defering to them on what to do with the future.)

If we had these AIs, there are many options. We could ask them to figure out what to do overall, ask them to do their own research into aligning much smarter models, we could try to run them much faster to achieve effective superintelligence, and we could try having them control somewhat smarter models.

I think the research needed to reach this exit condition is hard and we don't have that many amazing ideas. An alterative exit condition I consider is achieving sufficient coordination and non-proliferation to avoid building much-smarter-than-human AIs (really, whatever AIs we can't control) for a long time (e.g. >10 years) and then figuring something else out.

My current shitty and potentially hard-to-understand list of semi-plausible paths to this "aligned and deferable human replacement" exit is:

• Try to build as many diverse model organism of scheming test beds as possible. We can try to build these test beds with this AI itself as a subject and with weaker AIs. (It gets increasingly hard to make model organisms experiments with smarter AIs that understand we might be running experiments and thus they might try to mess with our experiments.) Test a variety of countermeasures in this setting and try to build a robust empirical science of scheming misalignment. E.g., test high level interpretability strategies where we try to analyze what AIs are thinking. Or test simple detection methods like probes.
• Develop AIs which are very dumb within a forward pass, but which are very good at using natural language reasoning such that they are competitive with our current systems. Demonstrate that these AIs are very unlikely to be scheming due to insufficient capacity outside of natural language (if we monitor their chains of thought). After ruling out scheming, solve other problems which seem notably easier.
• Pursue a very different AI design which is much more modular and more hand constructed (as in, more GOFAI style). This can involve usage of many small and dumb neural components, but needs to be sufficiently interpretable in aggregate which might be hard. This can be done by having the AIs apply huge amounts of labor.
• High-level and top-down interp trying to understand core objective-pursuing aspects of the AI. (E.g., what is the high level flowchart of how the AI makes decisions in non-obvious cases and what low level mechanisms could drive misalignment.)
• Develop somewhat principled approaches that marginally reduce the probability of scheming (e.g. various types of RAT/LAT, applying massive reductions in the amount of situational awareness throughout training and making this performant enough, adjusting the architecture in ways that make scheming less likely such as by replacing opaque reasoning with CoT)
• Do a huge drive toward making literal EMs. This probably requires a huge amount of capital and is really hard, but I could imagine AIs helping a huge amount.
• Reliable prediction + ELK via empirical route (Likely incomprehensible to people without context, sorry)
• ARC theory
• Probes which detect disobedience or some other scalable control using white box methods. (This is spooky to use as it doesn't guarantee the actual policy is aligned...)
• Trying to build brain-like AI and various approximations of EMs
• Pursue >3 other moonshots in parallel. None of the other moonshots I'm aware of seem super promising, but maybe. (Things like: deep understanding via bottom up interpretability, very good science of deep learning, something like Davidad's proposal, generic agent foundations.)

These are roughly ranked in goodness from my perspective.

Are you able to report the median AGI timeline for ~all METR employees? Or are you just saying that the "more than half" is how many responded to the survey question?

The main problem of nutritional research is that it's hard to get people to eat controlled diets. I don't think the key problem is about sourcing ingredients. 

I only just got the email about this so I have not yet spent much time evaluating this, but how much work on your end would it be to become a GDR (AU tax efficient charity)? Can you accept donations of stocks. Is anyone familiar with the tax implications of donating to a US charity as a dual tax resident of the US and Australia?

I'm also interested in naming something 'Lady Eriape Was Innocent' if I end up deciding to donate but do not want to let that influence my actual donation decision. 

Egyptians felling all their trees and turning their environment into a desert feels quite similar to fossil fuels. 

(Though the "fail to live up to potential" problems are probably mostly indirect, see here [LW(p) · GW(p)].)