LessWrong 2.0 Reader

There is a difference in external behavior only if you need to communicate knowledge about the environment and the other players explicitly. If this knowledge is already part of an agent (or rock), there is no behavior of learning it, and so no explicit dependence on its observation. Yet still there is a difference in how one should interact with such decision-making algorithms.

I think this describes minds/models better (there are things they've learned long ago in obscure ways and now just know) than learning that establishes explicit dependence of actions on observed knowledge in behavior (which is more like in-context learning).

This is extremely informative, especially the bit about the holdout set. I think it'd reassure a lot of people about the FrontierMath's validity to know more here. Have you used it to assess GPT-4o or o1? If so, how, and what were the results?

On the spectrum from stochastic parrot to general reasoner I'm at 70%. We're definitely closer to a general reasoner than a parrot. 

I don't have a clear answer as to what I expect the outcome to be. I was a physics major and I wish there were less discrete jumps in physics. Special relativity and general relativity seem like giant jumps in terms of their difficulty to derive and there aren't any intermediate theories. When this paper comes out we'll probably be saying something the AI is 50% between being being able to derive this law and a conceptually harder one. 

With respect to something analogous to Newtonian Mechanics I think that it does heavily depend on what kind of information can be observed. If a model can directly observe the equivalents of forces and acceleration than I believe most current models could derive it. If a model can only observe the equivalent of distances between objects and the equivalents of corresponding times and has to derive a second-order relationship from that I suspect that only o3 could do that. In six months, I believe that all frontier models will be able to do that. 

Given that Terrence Tao described o1 as a mediocre graduate student it probably won't be long until frontier models are actually contributing to research and that will be the most valuable feedback. I say all this with a lot of uncertainty and if I'm wrong this project will prove that. Likewise there's going to be a long period of time where some people will insist that AI can do legitimate automated R&D and others who insist that it can't. At that point this will be a useful test to argue one way or another. 

It would be interesting to vary the amount of information an AI is given until can derive the whole set of equations. For example, see if it can solve for the Maxwell equation given the other 3 and the ability to perform experiments or can it solve for the dynamic version of the equations given only the static ones and the ability to perform experiments.    

I can't make any confident claims or promises in my personal capacity right now, but my best guess is that we will make sure this new benchmark stays entirely private and under Epoch's control, to the extent this is feasible for us. However, I want to emphasize that by saying this, I'm not making a public commitment on behalf of Epoch.

Well, I'd sure like to know whether you are planning to give the dataset to OpenAI or any other frontier companies! It might influence my opinion of whether this work is net positive or net negative.

Here's how I'd deal with those examples:

Theory X: Jesus will come again: Presumably this theory assigns some probability mass >0 to observing Jesus tomorrow, whereas theory Y assigns ~0. If jesus is not observed tomorrow, that's a small amount of evidence for theory Y and a small amount of evidence against theory X. So you can say that theory X has been partially falsified. Repeat this enough times, and then you can say theory X has been fully falsified, or close enough. (Your credence in theory X will never drop to 0 probably, but that's fine, that's also true of all sorts of physical theories in good standing e.g. all the major theories of cosmology and cognitive science, which allow for tiny probabilities of arbitrary sequences of experiences happening in e.g. Boltzmann Brains)

With the sky color example:
My way of thinking about falsifiability is, we say two theories are falsifiable relative to each other if there is evidence we expect to encounter that will distinguish them / cause us to shift our relative credence in them. 

In the case of Theory Z, there is an implicit theory Z2 which is "NOT blah blah, and therefore the sky could be green or not green." (Presumably that's what you are holding in the back of your mind as the alternative to Z, when you imagine updating for or against Z on the basis of seeing blue sky, and decide that you wouldn't?) Because the theory Z3 "NOT blah blah and therefore the sky is blue" would be confirmed by seeing a blue sky, and if somehow you were splitting your credence between Z and Z3, then you would decrease your credence in Z if you saw a blue sky.

Great post, I was inner simulating [LW · GW] you posting something about bad vibes from people given your multiagent model of the mind and baggage which comes with it, and here it is.  

More technical definition of "fairness" here is that environment doesn't distinguish between algorithms with same policies, i.e. mappings <prior, observation_history> -> action? I think it captures difference between CooperateBot and FairBot.

As I understand, "fairness" was invented as responce to statement that it's rational to two-box and Omega just rewards irrationality.

So far, I have trouble because it lacks some form of spacial structure, and the algorithm feels too random to build meaningful connections btw different cards

Hmm, I think that after doing a lot of anki, my brain kind of formed it's own spatial structure [LW · GW], but I don't think this happens to everyone.

I just use basic card type for math with some latex. Here are some examples: 

I find that doing fancy card types is kind of like premature optimization. Doing the reviews is the most important part. On the other hand, it's really important that the cards themselves are written well. This essay contained my most refined views on card creation. Some other nice ones are the 20 rules of knowledge formulation and How to write good prompts: using spaced repetition to create understanding. Hope this answer helped!

What distinguishes a cooperate-rock from an agent that cooperates in coordination with others is the decision-making algorithm. Facts about this algorithm also govern the way outcome can be known in advance or explained in hindsight, how for a cooperate-rock it's always "cooperate", while for a coordinated agent it depends on how others reason, on their decision-making algorithms.

So in the same way that Newcomblike problems are the norm [LW · GW], so is the "unfair" interaction with decision-making algorithms. I think it's just a very technical assumption that doesn't make sense conceptually and shouldn't be framed as "unfairness".