I'm surprised the legion of doom is so quiet?

I've only read a little bit of Martin Gardner, but he might be the Matt Levine of recreational math.

When I imagine models inventing a language my imagination is something like Shinichi Mochizuki's Inter-universal Teichmüller theory invented for his supposed proof of abc conjecture. It is clearly something like mathematical English and you could say it is "quite intelligible" compared to "neuralese", but at the end, it is not very intelligible.

The Metaphysical Structure of Pearl's Theory of Time

Epistemic status: metaphysics

I was reading Factored Space Models (previously, Finite Factored Sets) and was trying to understand in what sense it was a Theory of Time.

Scott Garrabrant says [LW · GW] "[The Pearlian Theory of Time] ... is the best thing to happen to our understanding of time since Einstein". I read Pearl's book on Causality^[1], and while there's math, this metaphysical connection that Scott seems to make isn't really explicated. Timeless Causality [LW · GW] and Timeless Physics [LW · GW] is the only place I saw this view explained explicitly, but not at the level of math / language used in Pearl's book.

Here is my attempt at explicitly writing down what all of these views are pointing at (in a more rigorous language)—the core of the Pearlian Theory of Time, and in what sense FSM shares the same structure.

Causality leave a shadow of conditional independence relationships over the observational distribution. Here's an explanation providing the core intuition:

1. Suppose you represent the ground truth structure of [causality / determination] of the world via a Structural Causal Model over some variables, a very reasonable choice. Then, as you go down the Pearlian Rung: SCM $\to$^[2] Causal Bayes Net $\to$^[3] Bayes Net, theorems guarantee that the Bayes Net is still Markovian wrt the observational distribution.
   1. (Read Timeless Causality [LW · GW] for an intuitive example.)
2. Causal Discovery then (at least in this example) reduces to inferring the equation assignment directions of the SCM, given only the observational distribution.
3. The earlier result guarantees that all you have to do is find a Bayes Net that is Markovian wrt the observational distribution. Alongside the faithfulness assumption, this thus reduces to finding a Bayes Net structure G whose set of independencies (implied by d-separation) are identical to that of P (or, finding the Perfect Map of a distribution [LW · GW]^[4]).
4. Then, at least some of the edges of the Perfect Map will have its directions nailed down by the conditional independence relations.

The metaphysical claim is that, this direction is the definition of time^[5], morally so, based on the intuition provided by the example above.

So, the Pearlian Theory of Time is the claim that Time is the partial order over the variables of a Bayes Net corresponding to the perfect map of a distribution.

Abstracting away, the structure of any Theory of Time is then to:

1. find a mathematical structure [in the Pearlian Theory of Time, a Bayes Net]
2. ... that has gadgets [d-separation]
3. ... that are, in some sense, "equivalent" [soundness & completeness] to the conditional independence relations of the distribution the structure is modeling
4. ... while containing a notion of order [parenthood relationship of nodes in a Bayes Net]
5. ... while this order induced from the gadget coinciding to that of d-separation [trivially so here, because we're talking about Bayes Nets and d-separation] such that it captures the earlier example which provided the core intuition behind our Theory of Time.

This is exactly what Factored Space Model does:

1. find a mathematical structure [Factored Space Model]
2. ... that has gadgets [structural independence]
3. ... that are, in some sense, "equivalent" [soundness & completeness] to the conditional independence relations of the distribution the structure is modeling
4. ... while containing a notion of order [preorder relation induced by the subset relationship of the History]
5. ... while this order induced from the gadget coinciding to that of d-separation [by a theorem of FSM] such that it captures the earlier example which provided the core intuition behind our Theory of Time.
6. while, additionally, generalizing the scope of our Theory of Time from [variables that appear in the Bayes Net] to [any variables defined over the factored space].

... thus justifying calling FSM a Theory of Time in the same spirit that Pearlian Causal Discovery is a Theory of Time.

1. ^{^}

   Chapter 2, specifically, which is about Causal Discovery. All the other chapters are mostly irrelevant for this purpose.

2. ^{^}

   By (1) making a graph with edge direction corresponding to equation assignment direction, (2) pushforwarding uncertainties to endogenous variables, and (3) letting interventional distributions be defined by the truncated factorization formula.

3. ^{^}

   By (1) forgetting the causal semantics, i.e. no longer associating the graph with all the interventional distributions, and only the no intervention observational distribution.

4. ^{^}

   This shortform answers this question I had.

5. ^{^}

   Pearl comes very close. In his Temporal Bias Conjecture (2.8.2):

   "In most natural phenomenon, the physical time coincides with at least one statistical time."

   (where statistical time refers to the aforementioned direction.)

   But doesn't go as far as this ought to be the definition of Time.

A theory of how alignment research should work

(cross-posted from danielfilan.com)

Epistemic status:

• I listened to the Dwarkesh episode with Gwern and started attempting to think about life, the universe, and everything
• less than an hour of thought has gone into this post
• that said, it comes from a background of me thinking for a while about how the field of AI alignment should relate to agent foundations research

Maybe obvious to everyone but me, or totally wrong (this doesn't really grapple with the challenges of working in a domain where an intelligent being might be working against you), but:

• we currently don't know how to make super-smart computers that do our will
  • this is not just a problem of having a design that is not feasible to implement: we do not even have a sense of what the design would be
  • I'm trying to somewhat abstract over intent alignment vs control approaches, but am mostly thinking about intent alignment
  • I have not thought that much about societal/systemic risks very much, and this post doesn't really address them.
• ideally we would figure out how to do this
• the closest traction that we have: deep learning seems to work well in practice, altho our theoretical knowledge of why it works so well or how capabilities are implemented is lagging
• how should we proceed? Well:
  • thinking about theory alone has not been practical
  • probably we need to look at things that exhibit alignment-related phenomena and understand them, and that will help us develop the requisite theory
    • said things are probably neural networks
  • there are two ways we can look at neural networks: their behaviour, and their implementation.
  • looking at behaviour is conceptually straightforward, and valuable, and being done
  • looking at their implementation is less obvious
  • what we need is tooling that lets us see relevant things about how neural networks are working
  • such tools (e.g. SAEs) are not impossible to create, but it is not obvious that their outputs tell us quantities that are actually of interest
  • in order to discipline the creation of such tools, we should demand that they help us understand models in ways that matter
    • see Stephen Casper's engineer's interpretability sequence [? · GW], Jason Gross on compact proofs
  • once we get such tools, we should be trying to use them to understand alignment-relevant phenomena, to build up our theory of what we want out of alignment and how it might be implemented
    • this is also a thing that looking at the external behaviour of models in alignment-relevant contexts should be doing
• so should we be just doing totally empirical things? No.
  • firstly, we need to be disciplined along the way by making sure that we are looking at settings that are in fact relevant to the alignment problem, when we do our behavioural analysis and benchmark our interpretability tools. This involves having a model of what situations are in fact alignment-relevant, what problems we will face as models get smarter, etc
  • secondly, once we have the building blocks for theory, ideally we will put them together and make some actual theorems like "in such-and-such situations models will never become deceptive" (where 'deceptive' has been satisfactorily operationalized in a way that suffices to derive good outcomes from no deception and relatively benign humans)
• I'm imagining the above as being analogous to an imagined history of statistical mechanics (people who know this history or who have read "inventing temperature" should let me know if I'm totally wrong about it):
  • first we have steam engines etc
  • then we figure out that 'temperature' and 'entropy' are relevant things to track for making the engines run
  • then we relate temperature, entropy, and pressure
  • then we get a good theory of thermodynamics
  • then we develop statistical mechanics
• exceptions to "theory without empiricism doesn't work":
  • thinking about deceptive mesa-optimization
  • RLHF failures [LW · GW]
  • CIRL analysis
• lesson of above: theory does seem to help us analyze some issues and raise possibilities

why most perfect algorithms that recreate a strawberry on the molecular level destroy the planet as well.

Phrased like this, the answer that comes to mind is "Well, this requires at least a few decades' worth of advances in materials science and nanotechnology and such, plus a lot of expensive equipment that doesn't exist today, and e.g. if you want this to happen with high probability, you need to be sure that civilization isn't wrecked by nuclear war or other threats in upcoming decades, so if you come up with a way of taking over the world that has higher certainty than leaving humanity to its own devices, then that becomes the best plan."  Classic instrumental convergence, in other words.

Let's expand on this line of argument and look at your example of bee waggle-dances. You question whether the abstractions represented by the various dances are natural. I agree! Using a Cartesian frame that treats bees and humans as separate agents, not part of Nature, they are not Natural Abstractions. With an Embedded frame they are a Natural Abstraction for anyone seeking to understand bees, but in a trivial way. As you say, "one of the systems explicitly values and works towards understanding the abstractions the other system is using".

Also, the meter is not a natural abstraction, which we can see by observing other cultures using yards, cubits, and stadia. If we re-ran cultural evolution, we'd expect to see different measurements of distance chosen. The Natural Abstraction isn't the meter, it's Distance. Related concepts like relative distance are also Natural Abstractions. If we re-ran cultural evolution, we would still think that trees are taller than grass.

I'm not a bee expect, but Wikipedia says:

In the case of Apis mellifera ligustica, the round dance is performed until the resource is about 10 meters away from the hive, transitional dances are performed when the resource is at a distance of 20 to 30 meters away from the hive, and finally, when it is located at distances greater than 40 meters from the hive, the waggle dance is performed

The dance doesn't actually mean "greater than 40 meters", because bees don't use the metric system. There is some distance, the Waggle Distance, where bees switch from a transitional dance to a waggle dance. Claude says, with low confidence, that the Waggle Distance varies based on energy expenditure. In strong winds, the Waggle Distance goes down.

Humans also have ways of communicating energy expenditure or effort. I don't know enough about bees or humans to know if there is a shared abstraction of Effort here. It may be that the Waggle Distance is bee-specific. And that's an important limitation on the NAH, it says, as you quote, "there exist abstractions which are natural", but I think we should also believe the Artificial Abstraction Hypothesis that says that there exist abstractions which are not natural.

This confusion is on display in the discussion around My AI Model Delta Compared To Yudkowsky [LW · GW], where Yudkowsky is quoted as apparently rejecting the NAH:

The AI does not think like you do, the AI doesn't have thoughts built up from the same concepts you use, it is utterly alien on a staggering scale. Nobody knows what the hell GPT-3 is thinking, not only because the matrices are opaque, but because the stuff within that opaque container is, very likely, incredibly alien - nothing that would translate well into comprehensible human thinking, even if we could see past the giant wall of floating-point numbers to what lay behind.

But then in a comment on that post he appears to partially endorse the NAH:

I think that the AI's internal ontology is liable to have some noticeable alignments to human ontology w/r/t the purely predictive aspects of the natural world; it wouldn't surprise me to find distinct thoughts in there about electrons.

But also endorses the AAH:

As the internal ontology takes on any reflective aspects, parts of the representation that mix with facts about the AI's internals, I expect to find much larger differences -- not just that the AI has a different concept boundary around "easy to understand", say, but that it maybe doesn't have any such internal notion as "easy to understand" at all, because easiness isn't in the environment and the AI doesn't have any such thing as "effort".

I agree that I wouldn't want to lean on the sweet-spot-by-default version of this, and I agree that the example is less strong than I thought it was. I still think there might be safety gains to be had from blocking higher level reflection if you can do it without damaging lower level reflection. I don't think that requires a task where the AI doesn't try and fail and re-evaluate - it just requires that the re-evalution never climbs above a certain level in the stack.

There's such a thing as being pathologically persistent, and such a thing as being pathologically flaky. It doesn't seem too hard to train a model that will be pathologically persistent in some domains while remaining functional in others. A lot of my current uncertainty is bound up in how robust these boundaries are going to have to be.

we've checked for various forms of funny business and our tools would notice if it was happening.

I think it's a high bar due to the nearest unblocked strategy problem and alienness.

I agree that when AGI R&D starts to 2x or 5x due to AI automating much of the process, that's when we need the slowdown/pause)

If you start stopping proliferation when you're a year away from some runaway thing, then everyone has the tech that's one year away from the thing. That makes it more impossible that no one will do the remaining research, compared to if the tech everyone has is 5 or 20 years away from the thing.

There's also a comic series with explicitly this premise, unfortunately this is a major plot point so revealing it will be a spoiler: