Posts

Finite Factored Sets: Applications 2021-08-31T21:19:03.570Z
Finite Factored Sets: Inferring Time 2021-08-31T21:18:36.189Z
Finite Factored Sets: Polynomials and Probability 2021-08-17T21:53:03.269Z
Finite Factored Sets: Conditional Orthogonality 2021-07-09T06:01:46.747Z
Finite Factored Sets: LW transcript with running commentary 2021-06-27T16:02:06.063Z
Finite Factored Sets: Orthogonality and Time 2021-06-10T01:22:34.040Z
Finite Factored Sets: Introduction and Factorizations 2021-06-04T17:41:34.827Z
Finite Factored Sets 2021-05-23T20:52:48.575Z
Saving Time 2021-05-18T20:11:14.651Z
2021 New Year Optimization Puzzles 2020-12-31T08:22:29.951Z
Teacher's Password: The LessWrong Mystery Hunt Team 2020-12-04T00:04:42.900Z
Time in Cartesian Frames 2020-11-11T20:25:19.400Z
Eight Definitions of Observability 2020-11-10T23:37:07.827Z
Committing, Assuming, Externalizing, and Internalizing 2020-11-09T16:59:01.525Z
Additive and Multiplicative Subagents 2020-11-06T14:26:36.298Z
Sub-Sums and Sub-Tensors 2020-11-05T18:06:44.421Z
Multiplicative Operations on Cartesian Frames 2020-11-03T19:27:15.489Z
Subagents of Cartesian Frames 2020-11-02T22:02:52.605Z
Functors and Coarse Worlds 2020-10-30T15:19:22.976Z
Controllables and Observables, Revisited 2020-10-29T16:38:14.878Z
What are good election betting opportunities? 2020-10-29T07:19:10.409Z
Biextensional Equivalence 2020-10-28T14:07:38.289Z
Additive Operations on Cartesian Frames 2020-10-26T15:12:14.556Z
Introduction to Cartesian Frames 2020-10-22T13:00:00.000Z
Puzzle Games 2020-09-27T21:14:13.979Z
What Would I Do? Self-prediction in Simple Algorithms 2020-07-20T04:27:25.490Z
Does Agent-like Behavior Imply Agent-like Architecture? 2019-08-23T02:01:09.651Z
Intentional Bucket Errors 2019-08-22T20:02:11.357Z
Risks from Learned Optimization: Conclusion and Related Work 2019-06-07T19:53:51.660Z
Deceptive Alignment 2019-06-05T20:16:28.651Z
The Inner Alignment Problem 2019-06-04T01:20:35.538Z
Conditions for Mesa-Optimization 2019-06-01T20:52:19.461Z
Risks from Learned Optimization: Introduction 2019-05-31T23:44:53.703Z
Yes Requires the Possibility of No 2019-05-17T22:39:32.879Z
Thoughts on Human Models 2019-02-21T09:10:43.943Z
Epistemic Tenure 2019-02-18T22:56:03.158Z
How the MtG Color Wheel Explains AI Safety 2019-02-15T23:42:59.637Z
How does Gradient Descent Interact with Goodhart? 2019-02-02T00:14:51.673Z
Formal Open Problem in Decision Theory 2018-11-29T03:25:46.134Z
The Ubiquitous Converse Lawvere Problem 2018-11-29T03:16:16.453Z
Hyperreal Brouwer 2018-11-29T03:15:23.650Z
Fixed Point Discussion 2018-11-24T20:53:39.545Z
Iteration Fixed Point Exercises 2018-11-22T00:35:09.885Z
Diagonalization Fixed Point Exercises 2018-11-18T00:31:19.683Z
Topological Fixed Point Exercises 2018-11-17T01:40:06.342Z
Fixed Point Exercises 2018-11-17T01:39:50.233Z
Embedded Agency (full-text version) 2018-11-15T19:49:29.455Z
Embedded Curiosities 2018-11-08T14:19:32.546Z
Subsystem Alignment 2018-11-06T16:16:45.656Z
Robust Delegation 2018-11-04T16:38:38.750Z

Comments

Comment by Scott Garrabrant on Is MIRI's reading list up to date? · 2021-09-14T00:45:27.036Z · LW · GW

I guess I am partially trying to reject the frame of "contributing to all the different research directions that MIRI is pursuing." 

Ask not what you can do for agent foundations - ask what agent foundations can do for you, in your own personal quest to figure out how to save the world.

Comment by Scott Garrabrant on Is MIRI's reading list up to date? · 2021-09-14T00:38:07.262Z · LW · GW

Most of the intelligence.org website has been updated very little over the last 4 years, and is thus at least a little out of date. The MIRI reading list in particular hasn't really been updated since before I started at MIRI in 2015.

I can summarize (what I think are the generators of) the MIRI reading list as:

1) If you want to build a philosophically solid reductionist understanding of anything, you should probably start by learning math (theoretical CS is part of math).

2) If you want to work in a preparadigmatic field, focus on breadth and foundations.

3) If the thing you want to understand is about how minds work, things related to epistemics (logic, probability, information theory, topology), optimization (machine learning, game theory, calculus, decision theory), and algorithms maybe should get a little more attention. 

(When you also bring in category theory because you want reductionism, which means you need to be consistently viewing the same thing at multiple different levels, this leads to basically the same conclusion as 1+2: learn all fields of math.)

4) Here is our best guess at the best textbook on every subject.

I personally basically haven't read any of those books, so I don't have much to say about point 4, but I believe that Nate learned a bunch of math from many of these exact books.

I stand behind points 1-3, as conditional statements, so if your goal (or your primary instrumental subgoal) is to build a philosophically solid reductionist understanding of how minds work, maybe you should read some of those books. 

Note that this advice doesn't have anything to do with MIRI, except in so far as I, and some others at MIRI, are reasonably well described as "trying to build a philosophically solid reductionist understanding of how minds work." 

If your plan is to spend 2 years learning all the math, and then reaching out to MIRI saying "I did my homework, can I have a job?" I recommend instead discussing this plan with an actual person at MIRI first. 

If on the other hand, it feels obvious to you that you want a philosophically solid reductionist understanding of how minds work, enough that you would still want it if MIRI announced tomorrow that it was pivoting to focus on global poverty, then I recommend you start by learning some math, and I don't have any better advice about what math to learn than what is on that webpage.

Comment by Scott Garrabrant on Countable Factored Spaces · 2021-09-09T18:44:24.908Z · LW · GW

Note that the title is misleading. This is really countable dimension factored spaces, which is much better, since it allows for the possibility of something kind of like continuous time, where between any two points in time, you can specify a time strictly between them.

Comment by Scott Garrabrant on Finite Factored Sets: Polynomials and Probability · 2021-09-09T02:20:12.522Z · LW · GW

Fixed, Thanks.

Comment by Scott Garrabrant on Finite Factored Sets: Inferring Time · 2021-09-06T20:13:06.957Z · LW · GW

Yeah, also note that the history of  given  is not actually a well defined concept. There is only the history of  given  for . You could define it to be the union of all of those, but that would not actually be used in the definition of orthogonality. In this case , and  are all independent of choice of , but in general, you should be careful about that.

Comment by Scott Garrabrant on Finite Factored Sets: Inferring Time · 2021-09-06T10:54:04.712Z · LW · GW

I think that works, I didn't look very hard. Yore histories of X given Y and V given Y are wrong, but it doesn't change the conclusion.

Comment by Scott Garrabrant on Finite Factored Sets: Orthogonality and Time · 2021-08-31T21:45:10.495Z · LW · GW

I could do that. I think it wouldn't be useful, and wouldn't generalize to sub partitions.

Comment by Scott Garrabrant on Finite Factored Sets: Orthogonality and Time · 2021-08-31T21:42:07.970Z · LW · GW

fixed, thanks.

Comment by Scott Garrabrant on Garrabrant and Shah on human modeling in AGI · 2021-08-18T05:17:06.622Z · LW · GW

I don't know, the negation of the first thing? A system that can freely model humans, or at least perform computation indistinguishable from modeling humans.

Comment by Scott Garrabrant on Power vs Precision · 2021-08-18T02:14:39.937Z · LW · GW

I claim that conversations about good fuzzy abstract clusters are both difficult, and especially in the comparative advantage of LessWrong. I claim that LW was basically founded on analogy, specifically the analogy between human cognition and agent foundations/AI. The rest of the world is not very precise, and that is a point for LW having a precision comparative advantage, but the rest of the world seems to also be lacking abstraction skill.

The claim that precision requires purity leads to the conclusion that you have to choose between precision and abstraction, and it also leads to the conclusion that precision is fragile, and thus that you should expect a priori for there to less of it naturally, but it is otherwise symmetric between precision and abstraction. It is only saying that we should pick a lane, not which lane it should be.

I claim that when I look at science academia, I see a bunch of precision directed in inefficient ways, together with a general failure to generalize across pre-defined academic fields, and even more of a failure to generalize out of the field of "science" and into the real world.

Warning: I am mostly conflating abstraction with analogy with lack of precision above, and maybe the point is that we need precise analogies.

Comment by Scott Garrabrant on Finite Factored Sets: Orthogonality and Time · 2021-07-09T01:47:09.644Z · LW · GW

Fixed, Thanks.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-07-07T22:02:44.768Z · LW · GW
  1. Given a finite set  of cardinality , find a computable upper bound on the largest finite factored set model that is combinatorially different from all smaller finite factored set models. (We say that two FFS models are combinatorially different if they say the same thing about the emptiness of all boolean combinations of histories and conditional histories of partitions of .) (Such an upper bound must exist because there are only finitely many combinatorially distinct FFS models, but a computable upper bound, would tell us that temporal inference is computable.)
  2. Prove the fundamental theorem for finite dimensional factored sets. (Seems likely combinatorial-ish, but I am not sure.)
  3. Figure out how to write a program to do efficient temporal inference on small examples. (I suspect this requires a bunch of combinatorial insights. By default this is very intractable, but we might be able to use math to make it easier.)
  4. Axiomatize complete consistent orthogonality databases (consistent means consistent with some model, complete means has an opinion on every possible conditional orthogonality) (To start, is it the case that compositional semigraphoid axioms already work?)

If by "pure" you mean "not related to history/orthogonality/time," then no, the structure is simple, and I don't have much to ask about it.

Comment by Scott Garrabrant on A simple example of conditional orthogonality in finite factored sets · 2021-07-06T23:45:22.571Z · LW · GW

Yeah, this is the point that orthogonality is a stronger notion than just all values being mutually compatible. Any x1 and x2 values are mutually compatible, but we don't call them orthogonal. This is similar to how we don't want to say that x1 and (the level sets of) x1+x2 are compatible. 

The coordinate system has a collection of surgeries, you can take a point and change the x1 value without changing the other values. When you condition on E, that surgery is no longer well defined. However the surgery of only changing the x4 value is still well defined, and the surgery of changing x1 x2 and x3 simultaneously is still well defined (provided you change them to something compatible with E).

We could define a surgery that says that when you increase x1, you decrease x2 by the same amount, but that is a new surgery that we invented, not one that comes from the original coordinate system.

Comment by Scott Garrabrant on A simple example of conditional orthogonality in finite factored sets · 2021-07-06T18:13:59.332Z · LW · GW

Thanks for writing this.

On the finiteness point, I conjecture that "finite dimensional" (|B| is finite) is sufficient for all of my results so far, although some of my proofs actually use "finite" (|S| is finite). The example with real numbers is still finite dimensional, so I don't expect any problems. 

Comment by Scott Garrabrant on Finite Factored Sets: Introduction and Factorizations · 2021-06-20T22:01:58.516Z · LW · GW

Fixed, Thanks.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-06-05T02:45:20.444Z · LW · GW

Sure, if you want to send me an email and propose some times, we could set up a half hour chat. (You could also wait until I post all the math over the next couple weeks.)

Comment by Scott Garrabrant on Finite Factored Sets · 2021-06-01T07:17:27.220Z · LW · GW

Looks like you copied it wrong. Your B only has one 4.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-28T16:52:27.864Z · LW · GW

I have not thought much about applying to things other than finite sets. (I looked at infinite sets enough to know there is nontrivial work to be done.) I do think it is good that you are thinking about it, but I don't have any promises that it will work out.

What I meant when I think that this can be done in a categorical way is that I think I can define a nice symmetric monodical category of finite factored sets such that things like orthogonality can be given nice categorical definitions. (I see why this was a confusing thing to say.)

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-28T16:45:08.113Z · LW · GW

If I understand correctly, that definition is not the same. In particular, it would say that you can get nontrivial factorizations of a 5 element set: {{{0,1},{2,3,4}},{{0,2,4},{1,3}}}.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-28T16:39:02.392Z · LW · GW

When I prove it, I prove and use (a slight notational variation on) these two lemmas.

  1. If , then  for all .
  2. .

(These are also the two lemmas that I have said elsewhere in the comments look suspiciously like entropy.)

These are not trivial to prove, but they might help.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-27T18:44:27.074Z · LW · GW

I think that you are pointing out that you might get a bunch of false positives in your step 1 after you let a thermodynamical system run for a long time, but they are are only approximate false positives.  

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-27T16:51:24.151Z · LW · GW

I think my model has macro states. In game of life, if you take the entire grid at time t, that will have full history regardless of t. It is only when you look at the macro states (individual cells) that my time increases with game of life time.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-27T16:49:10.163Z · LW · GW

As for entropy, here is a cute observation (with unclear connection to my framework): whenever you take two independent coin flips (with probabilities not 0,1, or 1/2), their xor will always be high entropy than either of the individual coin flips. 

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-27T16:43:08.629Z · LW · GW

Wait, I misunderstood, I was just thinking about the game of life combinatorially, and I think you were thinking about temporal inference from statistics. The reversible cellular automaton story is a lot nicer than you'd think.

if you take a general reversible cellular automaton (critters for concreteness), and have a distribution over computations in general position in which initial conditions cells are independent, the cells may not be independent at future time steps.

If all of the initial probabilities are 1/2, you will stay in the uniform distribution, but if the probabilities are in general position, things will change, and time 0 will be special because of the independence between cells.

There will be other events at later times that will be independent, but those later time events will just represent "what was the state at time 0."

For a concrete example consider the reversible cellular automaton that just has 2 cells, X and Y, and each time step it keeps X constant and replaces Y with X xor Y. 

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-27T00:38:00.674Z · LW · GW

Yep, there is an obnoxious number of factorizations of a large game of life computation, and they all give different definitions of "before."

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-26T22:00:04.291Z · LW · GW

I don't have a great answer, which isn't a great sign.

I think the scientist can infer things like. "algorithms reasoning about the situation are more likely to know X but not Y than they are to know Y but not X, because reasonable processes for learning Y tend to learn learn enough information to determine X, but then forget some of that information." But why should I think of that as time?

I think the scientist can infer things like "If I were able to factor the world into variables, and draw a DAG (without determinism) that is consistent with the distribution with no spurious independencies (including in deterministic functions of the variables), and X and Y happen to be variables in that DAG, then there will be a path from X to Y."

The scientist can infer that if Z is orthogonal to Y, then Z is also orthogonal to X, where this is important because Z is orthogonal to Y can be thought of as saying that Z is useless for learning about Y. (and importantly a version of useless for learning that is closed under common refinement, so if you collect a bunch of different Z orthogonal to Y, you can safely combine them, and the combination will be orthogonal to Y.)

This doesn't seem to get at why we want to call it before. Hmm.

Maybe I should just list a bunch of reasons why it feels like time to me (in no particular order):

  1. It seems like it gets a very reasonable answer in the Game of Life example
  2. Prior to this theory, I thought that it made sense to think of time as a closure property on orthogonality, and this definition of time is exactly that closure property on orthogonality, where X is weakly before Y if whenever Z is orthogonal to Y, Z is also orthogonal to X. (where the definition of orthogonality is justified with the fundamental theorem.)
  3. If Y is a refinement of X, then Y cannot be strictly before X. (I notice that I don't have a thing to say about why this feels like time to me, and indeed it feels like it is in direct opposition to your "doesn't agree with what can be computed from what," but it does agree with the way I feel like I want to intuitively describe time in the stories told in the "Saving Time" post.) (I guess one thing I can say is that as an agent learns over time, we think of the agent as collecting information, so later=more information makes sense.)
  4. History looks a lot like a non-quantitative version of entropy, where instead of thinking of how much randomness goes into a variable, we think of which randomness goes into the variable. There are lemmas towards proving the semigraphoid axioms which look like theorems about entropy modified to replace sums/expectations with unions. Then, "after" exactly corresponds to "greater entropy" in this analogy.
  5. If I imagine X and Z being computed independently, and Y as being computed from X and Z, it will say that X is before Y, which feels right to me (and indeed this property is basically the definition. It seems like my time is maybe the unique thing that gets the right answer on this simple story and also treats variables with the same info content as the same.)
  6. We can convert a Pearlian DAG to a FFS, and under this conversion, d-seperation is sent to conditional orthogonality, and paths between nodes are sent to time. (on the questions Pearl knows how to ask. We also generalize the definition to all variables)
Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-26T17:50:11.551Z · LW · GW

I partially agree, which is partially why I am saying time rather than causality.

I still feel like there is an ontological disagreement in that it feels like you are objecting to saying the physical thing that is Alice's knowledge is (not) before the physical thing that is Bob's knowledge.

In my ontology:
1) the information content of Alice's knowledge is before the information content of Bob's knowledge. (I am curios if this part is controversial.)

and then,

2) there is in some sense no more to say about the physical thing that is e.g. Alice's knowledge beyond the information content.

So, I am not just saying Alice is before Bob, I am also saying e.g. Alice is before Alice+Bob, and I can't disentangle these statements because Alice+Bob=Bob.

I am not sure what to say about the second example. I am somewhat rejecting the dynamics. "Alice travels back in time" is another way of saying that the high level FFS time disagrees with the standard physical time, which is true. The "high level" here is pointing to the fact that we are only looking at the part of Alice's brain that is about the envelopes, and thus talking about coarser variables than e.g. Alice's entire brain state in physical time. And if we are in the ontology where we are only looking at the information content, taking a high level version of a variable is the kind of thing that can change its temporal properties, since you get an entirely new variable.

I suspect most of the disagreement is in the sort of "variable nonrealism" of reducing the physical thing that is Alice's knowledge to its information content?

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-26T17:06:45.886Z · LW · GW

Now on OEIS: https://oeis.org/A338681

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-26T15:47:00.722Z · LW · GW

 is the event you are conditioning on, so the thing you should expect is that , which does indeed hold.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-25T16:27:37.078Z · LW · GW

I think I (at least locally) endorse this view, and I think it is also a good pointer to what  seems to me to be the largest crux between the my theory of time and Pearl's theory of time.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-25T04:46:27.616Z · LW · GW

Hmm, I am not sure what to say about the fundamental theorem, because I am not really understanding the confusion. Is there something less motivating about the fundamental theorem, than the analogous theorem about d-seperation being equivalent to conditional independence in all distributions comparable with the DAG?

Maybe this helps? (probably not): I am mostly imagining interacting with only a single distributions in the class, and the claim about independence in all probability distributions comparable with the structure can be replaced with instead independence in a general position probability distribution comparable with the structure.

I am not thinking of it as related to a maximum entropy argument.

The point about SEMs having more structure that I am ignoring is correct. I think that the largest philosophical difference between my framework and Pearlian one is that I am denying realism of anything beyond the "apparently identical." Another way to think about it is that I am denying realism about there being anything about the variables beyond their information. All of my definitions are only based on the information content of the variables, and so, for example, if you have two variables that are deterministic copies of each other, they will have all the same temporal properties, while in an SEM, they could be different. The surprising thing is that even without intervention data, this variable non-realism allows us to define and infer something that looks a lot like time.

I have a lot of uncertainty about learning algorithms. On the surface, it looks like my structure just has so much to check, and is going to have a hard time being practical, but I could see it going either way. Especially if you imagine it as a minor modification to graph learning, where maybe you don't consider all re-factorizations, but you do consider e.g. taking a pair of nodes and replacing one if them with the XOR.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T21:49:00.901Z · LW · GW

Makes sense. I think a bit of my naming and presentation was biased by being so surprised by the not on OEIS fact.

I think I disagree about the bipartite graph thing. I think it only feels more natural when comparing to Pearl. The talk frames everything in comparison to Pearl, but I think if you are not looking at Pearl, I think graphs don’t feel like the right representation here. Comparing to Pearl is obviously super important, and maybe the first introduction should just be about the path from Pearl to FFS, but once we are working within the FFS ontology, graphs feel not useful. One crux might be about how I am excited for directions that are not temporal inference from statistical data.

My guess is that if I were putting a lot of work into a very long introduction for e.g. the structure learning community, I might start the way you are emphasizing, but then eventually convert to throwing all the graphs away.

(The paper draft I have basically only ever mentions Pearl/graphs for motivation at the beginning and in the applications section.)

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T19:30:37.783Z · LW · GW

I was originally using the name Time Cube, but my internal PR center wouldn't let me follow through with that :)

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T19:19:48.104Z · LW · GW

Thanks Paul, this seems really helpful.

As for the name I feel like "FFS" is a good name for the analog of "DAG", which also doesn't communicate that much of the intuition, but maybe doesn't make as much sense for name of the framework.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T17:47:50.483Z · LW · GW

Here is a more concrete example of me using FFS the way I intend them to be used outside of the inference problem. (This is one specific application, but maybe it shows how I intend the concepts to be manipulated).

I can give an example of embedded observation maybe, but it will have to come after a more formal definition of observation (This is observation of a variable, rather than the observation of an event above):

Definition: Given a FFS , and , which are partitions of , where , we say  observes  relative to W if:
1) ,

2)  can be expressed in the form , and 

3) .

(This should all be interpreted combinatorially, not probabilistically.)

The intuition of what is going on here is that to observe an event, you are being promised that you 1) do not change whether the event holds, and 3) do not change anything that matters in the case where that event does not hold. Then, to observe a variable, you can basically 2) split yourself up into different fragments of your policy, where each policy fragment observes a different value of that variable. (This whole thing is very updateless.)

Example 1: (non-observation)
An agent  does not observe a coinflip , and chooses to raise either his left or right hand. Our FFS  is given by ,  and . (I am abusing notation here slightly by conflating  with the partition you get on  by projecting onto the  coordinate.) Then W is the discrete partition on .

In this example, we do not have observation. Proof: A only has two parts, so if we express A as a common refinement of 2 partitions, at least one of these two partitions must be equal to A. However, A is not orthogonal to W given H and A is not orthogonal to W given T. (). Thus we must violate condition 3.

Example 2: (observation)

An agent  does observe a coinflip , and chooses to raise either his left or right hand. We can think of  as actually choosing a policy that is a function from  to , where the two character string in the parts in  are the result of H followed by the result of T.

Our FFS  is given by ,  and , where  represents what the agent would do seeing heads, and  represents what the agent word do given seeing tails. . We also have a partition representing what the agent actually does , where  and  are each four element sets in the obvious way. We will then say , so W does not get to see what  would have done, it only gets to see the coin flip and what actually did.

Now I will prove that  observes  relative to  in this example. First, , and , so we get the first condition, . We will break up A in the obvious way set up in the problem for condition 2, so it suffices now to show that , (and it will follow symmetrically that .)

Im not going to go through the details, but , while , which are disjoint. The important thing here is that  doesn't care about  in worlds in which  holds. 

Discussion:

So largely I am sharing this to give an example for how you can manipulate FFS combinatorially, and how you can use this to say things that you might otherwise want to say using graphs, Granted, you could also say the above things using graphs, but now you can say more things, because you are not restricted to the nodes you choose, you can ask the same combinatorial question about any of the other partitions, The benefit is largely about not being dependent on our choice of variables.

It is interesting to try to translate this definition of observation to transparent Newcomb or counterfactual mugging, and see how some of the orthogonalities are violated, and thus it does not count as an observation.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T17:45:00.595Z · LW · GW

I'll try. My way of thinking doesn't use the examples, so I have to generate them for communication.

I can give an example of embedded observation maybe, but it will have to come after a more formal definition of observation (This is observation of a variable, rather than the observation of an event above):

Definition: Given a FFS , and , which are partitions of , where , we say  observes  relative to W if:
1) ,

2)  can be expressed in the form , and 

3) .

(This should all be interpreted combinatorially, not probabilistically.)

The intuition of what is going on here is that to observe an event, you are being promised that you 1) do not change whether the event holds, and 3) do not change anything that matters in the case where that event does not hold. Then, to observe a variable, you can basically 2) split yourself up into different fragments of your policy, where each policy fragment observes a different value of that variable. (This whole thing is very updateless.)

Example 1 (non-observation)
An agent  does not observe a coinflip , and chooses to raise either his left or right hand. Our FFS  is given by ,  and . (I am abusing notation here slightly by conflating  with the partition you get on  by projecting onto the  coordinate.) Then W is the discrete partition on .

In this example, we do not have observation. Proof: A only has two parts, so if we express A as a common refinement of 2 partitions, at least one of these two partitions must be equal to A. However, A is not orthogonal to W given H and A is not orthogonal to W given T. (). Thus we must violate condition 3.

Example 2: (observation)

An agent  does observe a coinflip , and chooses to raise either his left or right hand. We can think of  as actually choosing a policy that is a function from  to , where the two character string in the parts in  are the result of H followed by the result of T.

Our FFS  is given by ,  and , where  represents what the agent would do seeing heads, and  represents what the agent word do given seeing tails. . We also have a partition representing what the agent actually does , where  and  are each four element sets in the obvious way. We will then say , so W does not get to see what  would have done, it only gets to see the coin flip and what actually did.

Now I will prove that  observes  relative to  in this example. First, , and , so we get the first condition, . We will break up A in the obvious way set up in the problem for condition 2, so it suffices now to show that , (and it will follow symmetrically that .)

Im not going to go through the details, but , while , which are disjoint. The important thing here is that  doesn't care about  in worlds in which  holds. 

Discussion:

So largely I am sharing this to give an example for how you can manipulate FFS combinatorially, and how you can use this to say things that you might otherwise want to say using graphs, Granted, you could also say the above things using graphs, but now you can say more things, because you are not restricted to the nodes you choose, you can ask the same combinatorial question about any of the other partitions, The benefit is largely about not being dependent on our choice of variables.

It is interesting to try to translate this definition of observation to transparent Newcomb or counterfactual mugging, and see how some of the orthogonalities are violated, and thus it does not count as an observation.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T16:35:24.657Z · LW · GW

Hmm, first I want to point out that the talk here sort of has natural boundaries around inference, but I also want to work in a larger frame that uses FFS for stuff other than inference.

If I focus on the inference question, one of the natural questions that I answer is where I talk about grue/bleen in the talk. 

I think for inference, it makes the most sense to think about FFS relative to Pearl. We have this problem with looking at smoking/tar/cancer, which is what if we carved into variables the wrong way. What if instead of tar/cancer, we had a variable for "How much bad stuff is in your body?" and "What is the ratio of tar to cancer?" The point is that our choice of variables both matters for the Pearlian framework, and is not empirically observable. I am trying to do all the good stuff in Pearl without the dependence on the variables

Indeed, I think the largest crux between FFS and Pearl is something about variable realism. To FFS, there is no realism to a variable beyond its information content, so it doesn't make sense to have two variables X, X' with the same information, but different temporal properties. Pearl's ontology, on the other hand, has these graphs with variables and edges that say "who listens to whom," which sets us up to be able to have e.g. a copy function from X to X', and an arrow from X to Y, which makes us want to say X is before Y, but X' is not.

For the more general uses of FFS, which are not about inference, my answer is something like "the same kind of stuff as Cartesian frames." e.g. specifying embedded observations. (A partition  observes a subset  relative to a high level world model  if  and . Notice the first condition is violated by transparent Newcomb, and the second condition is violated by counterfactual mugging. (The symbols here should be read as combinatorial properties, there are no probabilities involved.)) 

I want to be able to tell the stories like in the Saving Time post, where there are abstract versions of things that are temporally related.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T16:11:48.089Z · LW · GW

I are note sure what you are asking (indeed I am not sure if you are responding to me or cousin_it.)

One thing that I think is going on is that I use "factorization" in two places. Once when I say Pearl is using factorization data, and once where I say we are inferring a FFS. I think this is a coincidence. "Factorization" is just a really general and useful concept.

So the carving into A and B and C is a factorization of the world into variables, but it is not the kind of factorization that shows up in the FFS, because disjoint factors should be independent in the FFS.

As for why to switch to this framework, the main reason (to me) is that it has many of the advantages of Pearl with also being able to talk about some variables being coarse abstract versions of other variables. This is largely because I am interested in embedded agency applications. 

Another reason is that we can't tell a compelling story about where the variables came from in the Pearlian story. Another reason is that sometimes we can infer time where Pearl cannot. 

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T16:00:52.544Z · LW · GW

I think that the answers to both the concern about 7 elements, and the desire to have questions depend of previous questions come out of thinking about FFS models, rather than FFS.

If you want to have 7 elements in , that just means you will probably have more than 7 elements in .

If I want to model a situation where some questions I ask depend on other questions, I can just make a big FFS that asks all the questions, and have the model hide some of the answers. 

For example, Let's say I flip a biased coin, and then if heads I roll a biased 6 sided die, and if tails I roll a biased 10 sided die. There are 16 outcomes in 

I can build a 3 dimensional factored set 2x6x10, which I will imagine as sitting on my table with height 2. heads is on the bottom, and tails is on the top.  will then merge together the rows on the bottom, and the columns on the top, so it will look a little like the game Jenga.

In this way, I am imagining there is some hidden data about each world in which I get heads and roll the 6 sided die, which is the answer to the question "what would have happened if I rolled the 10 sided die. Adding in all this counterfactual data gives a latent structure of 120 possible worlds, even though we can only distinguish 16 possible worlds.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T15:49:35.109Z · LW · GW

Yep, this all seems like a good way of thinking about it.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T15:48:28.102Z · LW · GW

Hmm, I doubt the last paragraph about sets of partitions is going to be valuable, bet the eigenspace thinking might be useful. 

Note that I gave my thoughts about how to deal with the uniform distribution over 4 elements in the thread responding to cousin_it.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T15:44:59.504Z · LW · GW

So we are allowing S to have more than 4 elements (although we dont need that in this case), so it is not just looking at a small number of factorizations of a 4 element set. This is because we want an FFS model, not just a factorization of the sample space.

If you factor in a different way, X will not be before Y, but if you do this it will not be the case that X is orthogonal to X XOR Y. The theorem in this example is saying that X being orthogonal to X XOR Y implies that X is before Y.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T15:38:37.019Z · LW · GW

Ok, makes sense. I think you are just pointing out that when I am saying "general position," that is relative to a given structure, like FFS or DAG or symmetric FFS.

If you have a probability distribution, it might be well modeled by a DAG, or a weaker condition is that it is well modeled by a FFS, or an even weaker condition is that it is well modeled by a SFFS (symmetric finite factored set). 

We have a version of the fundamental theorem for DAGs and d-seperation, we have a version of the fundamental theorem for FFS and conditional orthogonality, and we might get a version of the fundamental theorem for SFFS and whatever corresponds to conditional independence in that world.

However, I claim that even if we can extend to a fundamental theorem for SFFS, I still want to think of the independences in a SFFS as having different sources. There are the independences coming from orthogonality, and there are there the independences coming from symmetry (or symmetry together with orthogonality.

In this world, orthogonality won't be as inferable because it will only be a subset of independence, but it will still be an important concept. This is similar to what I think will happen when we go to the infinite dimensional factored sets case.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T15:25:15.287Z · LW · GW

It looks like X and V are independent binary variables with different probabilities in general position, and Y is defined to be X XOR V. (and thus V=X XOR Y).

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T03:25:07.689Z · LW · GW

Yeah, you are right. I will change it. Thanks.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T01:46:48.626Z · LW · GW

I don't understand what conspiracy is required here.

X being orthogonal to X XOR Y implies X is before Y, we don't get the converse.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T01:13:07.524Z · LW · GW

The swapping within a factor allows for considering rational probabilities to be in general position, and the swapping factors allows IID samples to be considered in general position. I think this is an awesome research direction to go in, but it will make the story more complicated, since will not be able to depend on the fundamental theorem, since we are allowing for a new source of independence that is not orthogonality. (I want to keep calling the independence that comes from disjoint factors orthogonality, and not use "orthogonality" to describe the new independences that come from the principle of indifference.)

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-24T01:06:16.273Z · LW · GW

So you should probably not work with probabilities equal to 1/2 in this framework, unless you are doing so for a specific reason. Just like in Pearlian causality, we are mostly talking about probabilities in general position. I have some ideas about how to deal with probability 1/2 (Have a FFS, together with a group of symmetry constraints, which could swap factors, or swap parts within a factor), but that is outside of the scope of what I am doing here.

To give more detail, the uniform distribution on four elements does not satisfy the compositional semigraphoid axioms, since if we take X, Y, Z to be the three partitions into two parts of size two, X is independent with Y and X is independent with Z, but X is not independent with the common refinement of Y and Z. Thus, if we take the orthogonality database generated by this probability distribution, you will find that it is not satisfied by any models.

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-23T23:44:26.985Z · LW · GW

If you look at the draft edits for this sequence that is still awaiting approval on OEIS, you'll find some formulas. 

Comment by Scott Garrabrant on Finite Factored Sets · 2021-05-23T22:51:00.295Z · LW · GW

Nope, we have , but not . That breaks the symmetry.