I think the key missing piece you’re pointing at (making sure that our interpretability tools etc actually tell us something alignment-relevant) is one of the big things going on in model organisms of misalignment (iirc there’s a step that’s like ‘ok, but if we do interpret ability/control/etc at the model organism does that help?’). Ideally this type of work, or something close to it, could become more common // provide ‘evals for our evals’ // expand in scope and application beyond deep deception.

If that happened, it seems like it would fit the bill here.

Does that seem true to you?

Humans as social animals have a strong instinctual bias towards trust of con-specifics in prosperous times.  Which makes sense from a game theoretic strengthen-the-tribe perspective.  But I think that leaves us, as a collectively dumb mob of naked apes, entirely lacking a sensible level of paranoia in the building ASI that has no existential need for pro-social behavior.

The one salve I have for hopelessness is that perhaps the Universe will be boringly deterministic and 'samey' enough that ASI will find it entertaining to have agentic humans wandering around doing their mildly unpredictable thing.  Although maybe it will prefer to manufacture higher levels of drama (not good for our happiness)

Mathematical reasoning might be specifically conducive to language invention because our ability to automatically verify reasoning means that we can potentially get lots of training data. The reason I expect the invented language to be “intelligible” is that it is coupled (albeit with some slack) to automatic verification.

It was a very frustrating conversation to listen to, because Wolfram really hasn't engaged his curiosity and done the reading on AI-kill-everyoneism.  So we just got a torturous number of unnecessary and oblique diversions from Wolfram who didn't provide any substantive foil to Eliezer

I'd really like to find Yudkowsky debates with better prepared AI optimists prepared to try and counter his points.  Do any exist?

I'm surprised the legion of doom is so quiet?

I've only read a little bit of Martin Gardner, but he might be the Matt Levine of recreational math.

When I imagine models inventing a language my imagination is something like Shinichi Mochizuki's Inter-universal Teichmüller theory invented for his supposed proof of abc conjecture. It is clearly something like mathematical English and you could say it is "quite intelligible" compared to "neuralese", but at the end, it is not very intelligible.

The Metaphysical Structure of Pearl's Theory of Time

Epistemic status: metaphysics

I was reading Factored Space Models (previously, Finite Factored Sets) and was trying to understand in what sense it was a Theory of Time.

Scott Garrabrant says [LW · GW] "[The Pearlian Theory of Time] ... is the best thing to happen to our understanding of time since Einstein". I read Pearl's book on Causality^[1], and while there's math, this metaphysical connection that Scott seems to make isn't really explicated. Timeless Causality [LW · GW] and Timeless Physics [LW · GW] is the only place I saw this view explained explicitly, but not at the level of math / language used in Pearl's book.

Here is my attempt at explicitly writing down what all of these views are pointing at (in a more rigorous language)—the core of the Pearlian Theory of Time, and in what sense FSM shares the same structure.

Causality leave a shadow of conditional independence relationships over the observational distribution. Here's an explanation providing the core intuition:

1. Suppose you represent the ground truth structure of [causality / determination] of the world via a Structural Causal Model over some variables, a very reasonable choice. Then, as you go down the Pearlian Rung: SCM $\to$^[2] Causal Bayes Net $\to$^[3] Bayes Net, theorems guarantee that the Bayes Net is still Markovian wrt the observational distribution.
   1. (Read Timeless Causality [LW · GW] for an intuitive example.)
2. Causal Discovery then (at least in this example) reduces to inferring the equation assignment directions of the SCM, given only the observational distribution.
3. The earlier result guarantees that all you have to do is find a Bayes Net that is Markovian wrt the observational distribution. Alongside the faithfulness assumption, this thus reduces to finding a Bayes Net structure G whose set of independencies (implied by d-separation) are identical to that of P (or, finding the Perfect Map of a distribution [LW · GW]^[4]).
4. Then, at least some of the edges of the Perfect Map will have its directions nailed down by the conditional independence relations.

The metaphysical claim is that, this direction is the definition of time^[5], morally so, based on the intuition provided by the example above.

So, the Pearlian Theory of Time is the claim that Time is the partial order over the variables of a Bayes Net corresponding to the perfect map of a distribution.

Abstracting away, the structure of any Theory of Time is then to:

1. find a mathematical structure [in the Pearlian Theory of Time, a Bayes Net]
2. ... that has gadgets [d-separation]
3. ... that are, in some sense, "equivalent" [soundness & completeness] to the conditional independence relations of the distribution the structure is modeling
4. ... while containing a notion of order [parenthood relationship of nodes in a Bayes Net]
5. ... while this order induced from the gadget coinciding to that of d-separation [trivially so here, because we're talking about Bayes Nets and d-separation] such that it captures the earlier example which provided the core intuition behind our Theory of Time.

This is exactly what Factored Space Model does:

1. find a mathematical structure [Factored Space Model]
2. ... that has gadgets [structural independence]
3. ... that are, in some sense, "equivalent" [soundness & completeness] to the conditional independence relations of the distribution the structure is modeling
4. ... while containing a notion of order [preorder relation induced by the subset relationship of the History]
5. ... while this order induced from the gadget coinciding to that of d-separation [by a theorem of FSM] such that it captures the earlier example which provided the core intuition behind our Theory of Time.
6. while, additionally, generalizing the scope of our Theory of Time from [variables that appear in the Bayes Net] to [any variables defined over the factored space].

... thus justifying calling FSM a Theory of Time in the same spirit that Pearlian Causal Discovery is a Theory of Time.

1. ^{^}

   Chapter 2, specifically, which is about Causal Discovery. All the other chapters are mostly irrelevant for this purpose.

2. ^{^}

   By (1) making a graph with edge direction corresponding to equation assignment direction, (2) pushforwarding uncertainties to endogenous variables, and (3) letting interventional distributions be defined by the truncated factorization formula.

3. ^{^}

   By (1) forgetting the causal semantics, i.e. no longer associating the graph with all the interventional distributions, and only the no intervention observational distribution.

4. ^{^}

   This shortform answers this question I had.

5. ^{^}

   Pearl comes very close. In his Temporal Bias Conjecture (2.8.2):

   "In most natural phenomenon, the physical time coincides with at least one statistical time."

   (where statistical time refers to the aforementioned direction.)

   But doesn't go as far as this ought to be the definition of Time.

A theory of how alignment research should work

(cross-posted from danielfilan.com)

Epistemic status:

• I listened to the Dwarkesh episode with Gwern and started attempting to think about life, the universe, and everything
• less than an hour of thought has gone into this post
• that said, it comes from a background of me thinking for a while about how the field of AI alignment should relate to agent foundations research

Maybe obvious to everyone but me, or totally wrong (this doesn't really grapple with the challenges of working in a domain where an intelligent being might be working against you), but:

• we currently don't know how to make super-smart computers that do our will
  • this is not just a problem of having a design that is not feasible to implement: we do not even have a sense of what the design would be
  • I'm trying to somewhat abstract over intent alignment vs control approaches, but am mostly thinking about intent alignment
  • I have not thought that much about societal/systemic risks very much, and this post doesn't really address them.
• ideally we would figure out how to do this
• the closest traction that we have: deep learning seems to work well in practice, altho our theoretical knowledge of why it works so well or how capabilities are implemented is lagging
• how should we proceed? Well:
  • thinking about theory alone has not been practical
  • probably we need to look at things that exhibit alignment-related phenomena and understand them, and that will help us develop the requisite theory
    • said things are probably neural networks
  • there are two ways we can look at neural networks: their behaviour, and their implementation.
  • looking at behaviour is conceptually straightforward, and valuable, and being done
  • looking at their implementation is less obvious
  • what we need is tooling that lets us see relevant things about how neural networks are working
  • such tools (e.g. SAEs) are not impossible to create, but it is not obvious that their outputs tell us quantities that are actually of interest
  • in order to discipline the creation of such tools, we should demand that they help us understand models in ways that matter
    • see Stephen Casper's engineer's interpretability sequence [? · GW], Jason Gross on compact proofs
  • once we get such tools, we should be trying to use them to understand alignment-relevant phenomena, to build up our theory of what we want out of alignment and how it might be implemented
    • this is also a thing that looking at the external behaviour of models in alignment-relevant contexts should be doing
• so should we be just doing totally empirical things? No.
  • firstly, we need to be disciplined along the way by making sure that we are looking at settings that are in fact relevant to the alignment problem, when we do our behavioural analysis and benchmark our interpretability tools. This involves having a model of what situations are in fact alignment-relevant, what problems we will face as models get smarter, etc
  • secondly, once we have the building blocks for theory, ideally we will put them together and make some actual theorems like "in such-and-such situations models will never become deceptive" (where 'deceptive' has been satisfactorily operationalized in a way that suffices to derive good outcomes from no deception and relatively benign humans)
• I'm imagining the above as being analogous to an imagined history of statistical mechanics (people who know this history or who have read "inventing temperature" should let me know if I'm totally wrong about it):
  • first we have steam engines etc
  • then we figure out that 'temperature' and 'entropy' are relevant things to track for making the engines run
  • then we relate temperature, entropy, and pressure
  • then we get a good theory of thermodynamics
  • then we develop statistical mechanics
• exceptions to "theory without empiricism doesn't work":
  • thinking about deceptive mesa-optimization
  • RLHF failures [LW · GW]
  • CIRL analysis
• lesson of above: theory does seem to help us analyze some issues and raise possibilities

why most perfect algorithms that recreate a strawberry on the molecular level destroy the planet as well.

Phrased like this, the answer that comes to mind is "Well, this requires at least a few decades' worth of advances in materials science and nanotechnology and such, plus a lot of expensive equipment that doesn't exist today, and e.g. if you want this to happen with high probability, you need to be sure that civilization isn't wrecked by nuclear war or other threats in upcoming decades, so if you come up with a way of taking over the world that has higher certainty than leaving humanity to its own devices, then that becomes the best plan."  Classic instrumental convergence, in other words.