Posts

Time in Cartesian Frames 2020-11-11T20:25:19.400Z
Eight Definitions of Observability 2020-11-10T23:37:07.827Z
Committing, Assuming, Externalizing, and Internalizing 2020-11-09T16:59:01.525Z
Additive and Multiplicative Subagents 2020-11-06T14:26:36.298Z
Sub-Sums and Sub-Tensors 2020-11-05T18:06:44.421Z
Multiplicative Operations on Cartesian Frames 2020-11-03T19:27:15.489Z
Subagents of Cartesian Frames 2020-11-02T22:02:52.605Z
Functors and Coarse Worlds 2020-10-30T15:19:22.976Z
Controllables and Observables, Revisited 2020-10-29T16:38:14.878Z
What are good election betting opportunities? 2020-10-29T07:19:10.409Z
Biextensional Equivalence 2020-10-28T14:07:38.289Z
Additive Operations on Cartesian Frames 2020-10-26T15:12:14.556Z
Introduction to Cartesian Frames 2020-10-22T13:00:00.000Z
Puzzle Games 2020-09-27T21:14:13.979Z
What Would I Do? Self-prediction in Simple Algorithms 2020-07-20T04:27:25.490Z
Does Agent-like Behavior Imply Agent-like Architecture? 2019-08-23T02:01:09.651Z
Intentional Bucket Errors 2019-08-22T20:02:11.357Z
Risks from Learned Optimization: Conclusion and Related Work 2019-06-07T19:53:51.660Z
Deceptive Alignment 2019-06-05T20:16:28.651Z
The Inner Alignment Problem 2019-06-04T01:20:35.538Z
Conditions for Mesa-Optimization 2019-06-01T20:52:19.461Z
Risks from Learned Optimization: Introduction 2019-05-31T23:44:53.703Z
Yes Requires the Possibility of No 2019-05-17T22:39:32.879Z
Thoughts on Human Models 2019-02-21T09:10:43.943Z
Epistemic Tenure 2019-02-18T22:56:03.158Z
How the MtG Color Wheel Explains AI Safety 2019-02-15T23:42:59.637Z
How does Gradient Descent Interact with Goodhart? 2019-02-02T00:14:51.673Z
Formal Open Problem in Decision Theory 2018-11-29T03:25:46.134Z
The Ubiquitous Converse Lawvere Problem 2018-11-29T03:16:16.453Z
Hyperreal Brouwer 2018-11-29T03:15:23.650Z
Fixed Point Discussion 2018-11-24T20:53:39.545Z
Iteration Fixed Point Exercises 2018-11-22T00:35:09.885Z
Diagonalization Fixed Point Exercises 2018-11-18T00:31:19.683Z
Topological Fixed Point Exercises 2018-11-17T01:40:06.342Z
Fixed Point Exercises 2018-11-17T01:39:50.233Z
Embedded Agency (full-text version) 2018-11-15T19:49:29.455Z
Embedded Curiosities 2018-11-08T14:19:32.546Z
Subsystem Alignment 2018-11-06T16:16:45.656Z
Robust Delegation 2018-11-04T16:38:38.750Z
Embedded World-Models 2018-11-02T16:07:20.946Z
Decision Theory 2018-10-31T18:41:58.230Z
Embedded Agents 2018-10-29T19:53:02.064Z
(A -> B) -> A 2018-09-11T22:38:19.866Z
History of the Development of Logical Induction 2018-08-29T03:15:51.889Z
Bayesian Probability is for things that are Space-like Separated from You 2018-07-10T23:47:49.130Z
Optimization Amplifies 2018-06-27T01:51:18.283Z
Counterfactual Mugging Poker Game 2018-06-13T23:34:59.360Z
On the Chatham House Rule 2018-06-13T21:41:05.057Z
Prisoners' Dilemma with Costs to Modeling 2018-06-05T04:51:30.700Z
The Chromatic Number of the Plane is at Least 5 - Aubrey de Grey 2018-04-11T18:19:50.419Z

Comments

Comment by scott-garrabrant on Puzzle Games · 2020-11-17T05:11:48.407Z · LW · GW

I've only played it for a few hours, but I think it is also Tier-2-according-to-Scott.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-12T06:03:55.180Z · LW · GW

Fixed, thanks.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:34:19.106Z · LW · GW


Category-theory-first approaches

I am in general not especially proficient in category theory, and I think that the whole framework could be rewritten from the ground up by someone who is more proficient in category theory than me, and be made much better in the process.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:34:02.377Z · LW · GW

Time and coarse world models

I feel like the partial observability I get from taking a coarsening of the world and saying an agent has observations in that coarsening is similar to the partial observability I get when saying an agent learns something at a specific time. In particular, these two things seem similar enough to me that one might be able to unify the two definitions, and in the process reveal new things about them.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:33:45.509Z · LW · GW

Computational complexity

A random open question I am curious about, but doesn't seem that important: Is the existence of a morphism between Cartesian frames NP-complete?

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:33:29.153Z · LW · GW

Formalizing time

I think that much of the meat of what I want Cartesian frames to do is connected to time, and I have only really touched the surface of that. I think that there is a lot more to say about time, and I think there are options we have about how to think about time in Cartesian frames. The one I presented is my favorite at the moment, but I am uncertain.

For example, one might want to think about an agent, and the collection of pairs of partitions  and  of , such that the agent has a (multiplicative?) subagent that could choose , while observing . This collection of pairs is closed under coarsening in both arguments, and so one could talk about a sort of Pareto frontier of how refined you can make  given  or vice versa. I think this Pareto frontier looks a lot like time.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:33:13.377Z · LW · GW

Logical uncertainty

There is a sense in which Cartesian frames is a very updateless ontology, and thus I am concerned about how to make it play nicely with logical uncertainty. Indeed, Cartesian frames are basically assuming that we have a set of possible worlds, which is assuming that we have objects that are the possible world that are not realized. Logical uncertainty does not do well with this assumption. Extending Cartesian frames to connect up with logical uncertainty is a major open problem.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:32:55.991Z · LW · GW

Logical time

In agent simulates predictor, I am given a proof that I output a certain action, and then I must make a choice. In making this choice, I am determining whether or not I am given that proof in the first place. Further, the proof must in some sense compress my deliberation, or I would not be able to comprehend it. Thus, I feel that there are some details of the proof that are not "true inputs" for me.

I want to say that my deciding what I would do if I saw a proof is "earlier" than the proof according to some generalized notion of causality, or earlier in "logical time." I want to say that the only way to make the agent-simulates-predictor set-up make sense is to have the full proof itself not be a true input for me. I think that Cartesian frames is a step towards making continuous the notion of inputs and outputs, and so could help our thinking around this problem.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:32:37.131Z · LW · GW

Subagents

I think that our current ability to talk about agents contained within other agents is pretty limited, and Cartesian frames is a significant step forward on that. It would not surprise me if this could help with fixing our ontology around subsystem alignment. It could also help with our ontology around reasoning about committees, subcommittees, and members.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:32:18.871Z · LW · GW

Preferences and goals

It might be interesting to put on top of this theory something that is dealing more with utilities, or something similar. Since this theory is basically a calculus of what agents could do, it seems likely that we could say interesting things by putting on top of it analysis of what agents should do.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:31:58.549Z · LW · GW

Generalizing observability

Observables can clearly be extended to infinite partitions, and maybe further to a sigma algebra or something similar. One might want to also think of  and  as sigma algebras.

Observables can also be extended to talk about separating two subsets of , rather than separating a subset of  from its complement. One could also talk about observables that don't allow for arbitrary functions from the observed to , but instead allow for some restricted class such as continuous or Kakutani functions.

Such restricted classes might make more sense when using this more general notion of observables, or it might be possible to entirely construct these classes from this notion of observables.

This could allow the theory to encompass game theory, since you could have two agents which choose a probabilistic strategy, while knowing the probabilistic strategy chosen by the other player.

Comment by scott-garrabrant on Time in Cartesian Frames · 2020-11-11T20:31:36.174Z · LW · GW

Frames that are partitions into rectangles

I think that there might be significantly more that can be said about Cartesian frames that are a "partition into rectangles" than can be said about Cartesian frames in general.

By a "partition into rectangles," I mean a Cartesian frame  such that if , then . In particular, this assumption is saying something to the effect of "the level of description of this world is refined enough to play nicely with the factorization into  and ."

Comment by scott-garrabrant on Eight Definitions of Observability · 2020-11-11T20:29:33.970Z · LW · GW

Seems right, except I don't use the word "product" for the multiplicative definition. 

I don't have much to say about the internalizing-externalizing definition philosophically. One thing to say is that I think the condition that  observes  is a weaker notion of observability, that might actually agree with philosophical intuition more, and the internalizing-externalizing definition might be easier to interpret if you are thinking in terms of this condition.

Comment by scott-garrabrant on Biextensional Equivalence · 2020-11-11T19:31:04.462Z · LW · GW

Fixed. Thanks.

Comment by scott-garrabrant on Committing, Assuming, Externalizing, and Internalizing · 2020-11-10T20:14:49.421Z · LW · GW

Yep, Thanks.

Comment by scott-garrabrant on Subagents of Cartesian Frames · 2020-11-05T18:52:30.051Z · LW · GW

Yep.

There is a single morphism from  to  for every world in , so  means all of these morphism factor through 

A morphism from  to  is basically a column of  and a morphism from  to  is basically an row in , all of whose entries are in , and these compose to the morphism corresponding to the entry where this column meets this row.

Thus  if and only if when you delete all rows not entirely in , the resulting matrix has image .

I think this equivalent to what you said. I just wrote it out myself because that was the easiest way for me to verify what you said.

Comment by scott-garrabrant on Multiplicative Operations on Cartesian Frames · 2020-11-05T18:41:00.968Z · LW · GW

I am not sure, but I think that the answer is that you can't say anything interesting with just , but can maybe say interesting things with  and , which I am about to introduce.  In the post that just went up is the relationship between one of the components and a sub-sum, and  is the relationship between one of the components and a sub-tensor.  is the transitive closure of  and .

I think that if , then there is a nice morphism from  to , and if , there is a set of nice morphisms from  to  but in some degenerate cases that set is empty, which is how I constructed a counter example in my other comment.

Comment by scott-garrabrant on Multiplicative Operations on Cartesian Frames · 2020-11-05T18:28:03.664Z · LW · GW

I believe, if  and , then ,   and . Thus , but there is no morphism from 0 to 

Comment by scott-garrabrant on Multiplicative Operations on Cartesian Frames · 2020-11-05T00:26:06.704Z · LW · GW

I haven't put time into thinking about most of your comments yet, but I'm pretty sure the answer to this is yes.

EDIT: Oh, I just realized it wasn't a question.

Comment by scott-garrabrant on Multiplicative Operations on Cartesian Frames · 2020-11-04T19:13:58.671Z · LW · GW

Yep, fixed, thanks.

Comment by scott-garrabrant on Additive Operations on Cartesian Frames · 2020-10-30T20:58:01.368Z · LW · GW

Yep. Fixed. Thanks.

Comment by scott-garrabrant on What are good election betting opportunities? · 2020-10-30T18:38:37.950Z · LW · GW

No, you can't go negative. The "cash" is actually cash (modulo a delay and a 5% withdraw fee). If you withdraw or lose, you won't be able to sell you shares from this market, but there is no risk of debt.

Comment by scott-garrabrant on AI risk hub in Singapore? · 2020-10-30T07:49:16.092Z · LW · GW

The number of bits of evidence is 

Basically, if you view the probability as a fraction, , then a positive bit of evidence doubles  while a negative bit of evidence doubles .

You can imagine the ruler: 3% 6% 11% 20% 33% 50% 67% 80% 89% 94% 97%. Each is one more bit than the last. For probabilities near 0, it is roughly counting doublings, for probabilities near 1, it is counting doublings of the complement.

Comment by scott-garrabrant on AI risk hub in Singapore? · 2020-10-30T07:40:00.662Z · LW · GW

Yeah, I had this in mind I said I'm not sure if I would endorse this if I thought about it more. I am still uncertain.

Comment by scott-garrabrant on What are good election betting opportunities? · 2020-10-30T07:35:12.070Z · LW · GW

I think Predictit pays on arbitrage according to the assumption that at most one bucket resolves to yes, but not according to the assumption exactly one bucket resolves yes, so that 850 is in the hypothetical world where none of the 16 buckets resolve to yes.

Comment by scott-garrabrant on Biextensional Equivalence · 2020-10-30T00:13:00.077Z · LW · GW

You might be right, I am not sure.

It looks to me like it satisfies the definition on wikipedia, which does not require that the morphism  is unique, only that it exists.

Comment by scott-garrabrant on AI risk hub in Singapore? · 2020-10-29T21:03:39.460Z · LW · GW

"I tentatively guess that if Singapore were to become a thriving hub for AI risk reduction, this would reduce AI risk by 16%."

The units on this claim seem bad. There is a big difference between 50%  34% and 99%  83%. I'm not sure if I would endorse this if I thought about it more, but maybe good units for a claim like the number of bits of evidence the update is equivalent to. Going from 50% to 33% would be the same as going from 99% to 98% (1 bit).

Comment by scott-garrabrant on Biextensional Equivalence · 2020-10-29T20:44:00.325Z · LW · GW

I think the right way to think about biextensional collapse categorically is as a reflector.

Comment by scott-garrabrant on Controllables and Observables, Revisited · 2020-10-29T20:39:58.151Z · LW · GW

You can also duplicate rows in , and then add columns, so you can get things like . There are infinitely many biextensional Cartesian frames over  with morphism to , with arbitrarily large dimensions.

Comment by scott-garrabrant on Controllables and Observables, Revisited · 2020-10-29T20:31:53.087Z · LW · GW

 is wrong. You can see it has Ensurables that  does not have.

Comment by scott-garrabrant on What are good election betting opportunities? · 2020-10-29T16:21:40.376Z · LW · GW

Predictit promises that only one will be yes and immediately gives you back the liquid money when you arbitrage.

Comment by scott-garrabrant on What are good election betting opportunities? · 2020-10-29T16:20:19.848Z · LW · GW

There are caps on predictit. You can’t put more than 850 dollars on any contract.

Comment by scott-garrabrant on What are good election betting opportunities? · 2020-10-29T07:45:09.508Z · LW · GW

You can turn 10 dollars into ~100 dollars on predictit right now, with no risk, and you immediately get the money to make other bets on predictit (although it takes 30 days to turn it back into cash, and the process takes a while). Here are the steps:

  1. Go here, and add up all the numbers in the rightmost column under "Best Offer" for "Buy No"
  2. Confirm that the sum is at most 14.89. It is 14.80 at the time I am writing this. If it is more than 14.89, you can't make (much) money here. The amount you make is roughly 8 dollars per cent of difference with 14.90.
  3. Make an account on predictit, and deposit the minimum amount of 10 dollars. (You can skip this if you already have a predictit account)
  4. Buy 10 shares of No for each of the 16 buckets in the above market.
  5. Confirm that your cash is now strictly greater than 10 dollars.
  6. Repeat step 4 until you have bought 858 shares of each bucket. As you have more cash, you can increase the number of shares you can buy at a time.
  7. On the top right of this market, there will be an underlined number under "Max. Payout." Click on that number to see a pop-up with a chart showing (in the rightmost column) how much extra money you will get in each world. If any of those numbers is greater than a dollar, repeatedly buy another No share of that world, until all the numbers are less than a dollar.
  8. Do the same for this market. (This one is at 14.88 at the time I am writing this)
  9. You now have ~100 dollars of predictit money. You can take it out for a 5% fee, and a 30 day delay, or you can bet in other predictit markets.
Comment by scott-garrabrant on Puzzle Games · 2020-10-25T23:40:46.684Z · LW · GW

I have now beaten A Monster's Expedition. Here are some extra challenges:

  1. It is possible to ride a raft off of the map.
  2. It is possible (at least at the time I did it) to put logs in a configuration that the game does not know how to handle and crash the game. This will also make the game unplayable (even for other save files) unless you go in and edit your save file.
Comment by scott-garrabrant on Introduction to Cartesian Frames · 2020-10-23T03:00:02.740Z · LW · GW

coincidence

Comment by scott-garrabrant on Introduction to Cartesian Frames · 2020-10-22T18:08:21.220Z · LW · GW

Your suspected answer right.

Comment by scott-garrabrant on Introduction to Cartesian Frames · 2020-10-22T16:46:19.538Z · LW · GW

Yep. Fixed. Thanks.

Comment by scott-garrabrant on Puzzle Games · 2020-09-28T17:44:23.564Z · LW · GW

I took a look at marbles puzzle, and it seems really good

Comment by scott-garrabrant on Puzzle Games · 2020-09-28T17:43:41.054Z · LW · GW

The Swapper section violates spoiler policy

Comment by scott-garrabrant on Puzzle Games · 2020-09-28T17:43:00.673Z · LW · GW

This comment violates spoiler policy

Comment by scott-garrabrant on Puzzle Games · 2020-09-28T17:41:12.519Z · LW · GW

Your comments on Baba violate spoiler policy

Comment by scott-garrabrant on Puzzle Games · 2020-09-28T17:39:03.808Z · LW · GW

Yes

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T23:40:14.858Z · LW · GW

You're going to want a fine tip dry erase pen to play TLP. (or you could get the PDF version and use annotations, but a major selling point for me was the lack of screen time.)

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T23:11:03.979Z · LW · GW

I think everything but the first sentence here should be in a spoiler box, which you can make by typing ">!" at the start of a line.

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T22:46:38.370Z · LW · GW

Actually, on second thought, I don't like you named a game in the parenthetical about Tier 1 and Tier 2

It doesn't feel much like a spoiler, but I want the spoiler policy to feel like a white list.

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T22:45:27.289Z · LW · GW

I have played a very small amount of Antichamber.

Very minor spoilers for Anti-chamber: 

I haven't played much of Antichamber, but I have seen it recommended a bunch. I think that this is a place where my aesthetic differs from others. I dont think I like this kind of game nearly as much, because (I believe) there is a lot of stuff that I would call guessing the developer's password. It is something like I feel like I have to try things without knowing whether it would work, and have the game tell me whether they work in a way I could not deduce myself.

I have played through Gorogoa

Minor spoilers for Gorogoa:

I think Gorogoa made me feel like I had to try things without knowing whether they would work, and let the game tell me whether or not they worked, in a way I could not deduce myself (even with enough time to consider all the options). This seems pretty bad according to my aesthetics. I feel like I have not pin-pointed the thing, because a lot of good games could also be described this way, but in a way that is good. I think it feels good when it feels like I am doing science, and Gorogoa did not feel like science. It felt like in Gorogoa I was exploring a space made by man rather than by nature? IDK

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T22:30:17.689Z · LW · GW

I think this is okay. 

"some of the above do not match central examples of puzzle games that you give in the post" made me pause and think about whether it follows the spoiler rules, but I think it is okay, since you are averaging over 9 games.

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T21:25:10.036Z · LW · GW

There are not many good deduction puzzle games, partially because deduction puzzles are often individual puzzles, rather than collections into games. 

One type of deduction puzzle I especially like is slitherlink.

gmpuzzles.com is a great source for high quality deduction puzzles

puzz.link has a database of many free puzzles.

Simon Tantham's Puzzles is another source to look at. (This one also has a mobile app)

Comment by scott-garrabrant on Puzzle Games · 2020-09-27T21:16:22.407Z · LW · GW

A Monster's Expedition came out a couple weeks ago, and letting people know about that game is part of what inspired me to make this post.

Comment by scott-garrabrant on Some thoughts on criticism · 2020-09-23T23:21:55.841Z · LW · GW

I just noticed a thing happen in me when giving criticism. I gave criticism in a context where it seemed like appropriate to give criticism, but there was maybe not complete common knowledge of this fact. The person I was talking to then apologized for the thing I criticized. The apology made me doubt the appropriateness of the criticism.

I think this is because it felt like the apology set up a frame where my criticism was coming along with a bid for change or at least acknowledgment. 

It seems to me like it might be hard to criticize because criticism comes along with an implied bid for change, and  it is scary to make bids that might get rejected, and so it can be useful to try to set up an context where there is common knowledge that the criticism does not automatically come along with a bid. Further, it feels like apology might interfere with this context.

As concrete advice, it seems like (maybe) thanking someone for criticism can serve the same politeness role as apology, while preserving a context in criticism does not automatically feel like it comes with a bid.