LessWrong 2.0 Reader

Interesting! Graziano's Attention Schema Theory is also basically the same: he proposes consciousness to be found in our models of our own attention, and that these models evolved to help control attention. To be clear, though, it's not the mere fact of modelling or controlling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that's what explains our intuitions about phenomenal consciousness.

Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and skeptical of fish and arthropods.

Something interesting here is that a part of why AI companies won't want to use agents is because their capabilities are good enough that being very reckless with them might actually cause small-scale misalignment issues, and if that's truly a big part of the problem in getting companies to adopt AI agents, this is good news for our future:

https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/?commentId=qEkRqHtSJfoDA7zJX [LW · GW]

Diplomacy AI by Meta [LW · GW] is a clear example of how adding LLMs can improve narrow game playing. Most multiplayer games with communication will benefit in the same way.

I don't understand your perspective. Many forms of warfighting required soldiers to move towards danger. Your basic Roman infantry charge is running towards people who want to stab you. Some video games can be seen as a safe super-stimulus for warfighting and require moving towards danger. If an innate preference for warfighting stopped being reproductively fit in the 1800s that won't have a large genetic effect by the 2000s.

The bible is large and includes both pro-war and anti-war passages. I agree that a preference for warfighting is not universal, but innate preferences need not be universal. Some are attracted to men, some are attracted to women, some both, some neither; this does not mean that sexual attraction is not an innate preference. The naive evo-psych perspective implies that warfighting is more often an innate preference for young males, and not for women, very young children, and elders. Innate preferences can be deactivated or inverted by reality, as in the turn against warfighting in post-WW1 Europe, or post-Vietnam USA. Or they can be activated.

Without making any normative arguments: if you're in a position (industrially and technologically) to disassemble the sun at all, or build something like a Dyson swarm, then it's probably not too difficult to build an artificial system to light the Earth in such a way as to mimic the sun, and make it look and feel nearly identical to biological humans living on the surface, using less than a billionth of the sun's normal total light output. The details of tides might be tricky, but probably not out of reach.

Nice post! I think these are good criticisms that don't justify the title. Points 1 through 4 are all (specific, plausible) examples of ways we may interpret the activation space incorrectly. This is worth keeping in mind, and I agree that just looking at the activation space of a single layer isn't enough, but it still seems like a very good place to start. 

A layer's activation is a relatively simple space, constructed by the model, that contains all the information that the model needs to make its prediction. This makes it a great place to look if you're trying to understand how the model's thinking.

Yes, many people will have problems with the Archimedes' axiom because it implies that everything has a price (that any good option can be probability-diluted enough that a mediocre is chosen instead), and people don't take it kindly when you tell "you absolutely must have a trade-off between value A and value B"  - especially if they really don't have a trade-off, but also if they don't want to admit or consciously estimate it.

Thankfully, that VNM property is not that critical for rational decision-making because we can simply use surreal numbers instead.

One possible real-world example (with integer-valued $U_1$ for deterministic outcomes) would be a parent whose top priority is minimizing the number of their children who die within the parent's lifetime, with the rest of their utility function being secondary.

Wouldn't work well since in real world outcomes are non-deterministic; given that, minimizing expected number is accomplished by simply having zero children.

I see, thanks for sharing!

The third term in that. Though it was in a somewhat different context related to the weight partitioning project mentioned in the last paragraph, not SAE training.

Yes, brittle in hyperparameters. It was also just very painful to train in general. I wouldn't straightforwardly extrapolate our experience to a standard SAE setup though, we had a lot of other things going on in that optimisation. 

Did you use something like $L_{SAE}$ as described here ? By brittle do you mean w.r.t the sparsity penality (and other hyperparameters)?