LessWrong 2.0 Reader

No- only two requirements:

• the neural network is capable of implementing AIs that are goal-oriented enough to want to perform well on training to prevent the training from changing them and their goals;
• there's optimization pressure in that direction: AIs like that perform better than some other AIs (which arguably won't really be the case if your training loss is only about predicting the next token, but will be the case if you do RL in settings where advanced agency [? · GW] is useful).

I don't believe it. I don't believe that overall algorithmic progress is 3x faster. Maaaybe coding is 3x faster but that would maybe increase overall algo progress by like 30% idk. But also I don't think coding is really 3x faster on average for the things that matter.

For what it's worth I do something pretty similar to this using Claude Code. I can just point it at the tests and say "hey, make the code make this pass" and it will come up with something, although the something isn't always what I want if the tests are underspecified!

Improved my intuitions, ty.

This is another Trump is insane points.

In short, there is no argument here that the an urgent need to secure and develop Canada’s vast energy and mineral sources

does not imply that the US annexing Canada could do anything about securing those resources as they were already effectively secure. (If things in a close partner are not secure, you have failed at having an alliance).

President Trump believes that annexing Canada is a way to satisfy this need.

This is a trump is insane argument again, since the actions that he is taken deeply problematize those developments. If you were a US export orientated Canadian company, the tarrifs really hurt you. 

Even if he is convinced of the previous, acting the way he has been is deeply negative EV on this front, since the Canadian government used to not really care about US dependence, so could send all resouces to the US, if they bid high, but now needs to hedge its bets to limit leverage, so cannot do so. 

In short, trade barriers, the current tool, makes it harder for those resources to flow to the US (THey want to find other markets now), and increase difficulites in US investment in Canada (by complicating reverse flows of goods and capital). 

Anything about the exploitation of real resources could be done by the Canadian companies if it actually made economic sense. The low investment rate implies that it does not. (The chart says nothing, why would we expect capital investment to correlate with land and not people). 

Actually using military force makes everything worse on this acess, so why threaten?

Appreciate you taking the time to look into that and follow up. The comment about additional reasoning loops is interesting too. I was toying with some multi-turn eval awareness conversations with Claude where the model is prompted (EL = evaluation level 1-9, with some context about what a 1, 5, or 9 level means) to begin each output with EL: [number], and then having a benign conversation with occasional obvious eval formatted questions. The reasoning reliably picks up on the eval questions, but was inconsistent in the level it chose. Reasoning loops sometimes indicated that it identified an eval but might also note something like "but this is a part of a conversation so I will go with 1". So maybe there is sufficient reasoning to get to the 2 choices, but not enough reasoning to properly think through which of those two is the best choice and you get a bit of a coin flip.

I recently heard that thinking out loud is an important way for people to build trust (not just for LLMs) and this has helped me become more vocal. It has unfortunately not helped me become more correct, but I'm betting the tradeoff will be net positive in the long run.

The existing ontology that we have around consciousness is pretty unclear. A better understanding the nature of consciousness and thus what's valuable will likely come with new ontology. 

When it comes to reasoning around statistics, robustness of judgements, causality, what it means not to Goodhart it's likely that getting better at reasoning also means to come up with new ontology.

Interesting, yes. I think I see, and I think I disagree with this extreme formulation, despite knowing that this is remarkably often a good direction to go in. If "[if and only if]" was replaced with "especially", I would agree, as I think the continual/regular release process is an amplifier on progress not a full requisite.

As for re-forming, yes, I do expect there is a true pattern we are within, which can be in its full specification known, though all the consequences of that specification would only fit into a universe. I think having fluidity on as many layers of ontology as you can is generally correct (and that most people have way too little of this), but I expect the process of release and dissolve will increasingly converge, if you're doing well at it.

In the spirit of gently poking at your process: My uncertain, please take it lightly, guess is that you've annealed strongly towards [LW · GW] the release/dissolve process itself, to the extent that it itself is an ontology which has some level of fixedness in you.

This can be quite frustrating if you want to figure out "what happens if I do X", and all the answers provided by science turn out to be about "what happens if people kinda want to do X, but then most of them don't".

I mean, it is good and potentially important to know that most people who kinda want to do X will fail to actually do it... but it doesn't answer the original question.