LessWrong 2.0 Reader

This is really cool work!!

In other experiments we've run (not presented here), the MSP is not well-represented in the final layer but is instead spread out amongst earlier layers. We think this occurs because in general there are groups of belief states that are degenerate in the sense that they have the same next-token distribution. In that case, the formalism presented in this post says that even though the distinction between those states must be represented in the transformers internal, the transformer is able to lose those distinctions for the purpose of predicting the next token (in the local sense), which occurs most directly right before the unembedding.

Would be interested to see analyses where you show how an MSP is spread out amongst earlier layers.

Presumably, if the model does not discard intermediate results, something like concatenating residual stream vectors from different layers and then linearly correlating with the ground truth belief-state-over-HMM-states vector extracts the same kind of structure you see when looking at the final layer. Maybe even with the same model you analyze, the structure will be crisper if you project the full concatenated-over-layers resid stream, if there is noise in the final layer and the same features are represented more cleanly in earlier layers?

In cases where redundant information is discarded at some point, this is a harder problem of course.

And some further personal comments: https://aleph.se/andart2/personal/thoughts-at-the-end-of-an-era/

Still working on setting it up, once I have the details I'll announce them (e.g. pricing and whatnot).

I'm aiming to have childcare available in some form for the full 9-day LessOnline-to-Summer-Camp-to-Manifest period. I'm excited for folks to come with their full families.

If you wanna talk about the humanity(ies), well I looked up Chief Vision Officer of AISI Adam Russel, and he has an interesting profile.

Russell completed a Bachelor of Arts in Cultural Anthropology from Duke University, and an M.Phil. and a D.Phil. in Social Anthropology from University of Oxford, where he was a Rhodes Scholar.^[2] He played with the Oxford University RFC for four varsity matches and also worked with the United States national rugby union team, and worked as High Performance director for the United States women's national rugby union team in the 2014 and 2017 Women's Rugby World Cups.^[3]

Russell was in the industry, where he was a senior scientist and principal investigator on a wide range of human performance and social science research projects and provided strategic assessments for a number of different government organizations.^[2][4] Russell joined Intelligence Advanced Research Projects Activity (IARPA) as a program manager.^[2][4] He developed and managed a number of high-risk, high-payoff research projects for the Office of the Director of National Intelligence.^[2] Russell joined DARPA as a program manager in July 2015.^[2][4] His work there focused on new experimental platforms and tools to facilitate discovery, quantification and "big validation" of fundamental measures in social science, behavioral science and human performance.^[2]

In 2022, secretary Xavier Becerra selected Russell to serve as the acting deputy director for the Advanced Research Projects Agency for Health (ARPA-H), effective June 6. In this role, Russell leads the process to stand up ARPA-H.^[5]

Hmm he's done a lot of macho human-enhancement-adjacent stuff. I wonder if there were some centaurists involved here.

• I previously noted a lot of research projects in neurotech research in DoD funding awards [LW · GW]. I'm making a connection between this and a joke I heard recently on a navy seals podcast. "The guys often ask what they can do to deal with drones. So you start showing them how to work the jammer devices, or net guns, and their eyes glaze over, it's not what they wanted, they're disappointed. They're thinking like, 'no... how can I deal with it. Myself.' "
• So even though alignment-by-merger is kinda obviously not going to work (you'd have to reverse-engineer two vats of inscrutable matrices, instead of one. And the fleshy pink one wasn't designed to be read from and can only be read on a neuron-by-neuron level after being plastinated (which also kills it). AGI alignment is something that a neuralink cannot solve.), it's conceivable that it's an especially popular line of thought among military/sports types.

Otherwise, this kinda lines up with my confessions on manhattan projects for AGI. You arguably need an anthropologist to make decisions about what 'aligned' means. I don't know if you really need one (a philosophically inclined decision theorist, likely to already be involved already, would be enough for me) but I wouldn't be surprised to see an anthropologist appointed in the most serious projects.

How's the childcare situation looking? Last I heard it wasn't clear and the organizers were seeing how much interest there was in it.

Your POS system exports data that your inventory software imports and uses. But I strongly suspect that this is often not possible in practice.

 

This sounds like exactly the sort of problem that a business might pay for a solution to, particularly if there is one particular pair of POS system / inventory software that is widely used in the industry in question, where those pieces of software don't natively play well together.

From the latest Conversations with Tyler interview of Peter Thiel

I feel like Thiel misrepresents Bostrom here. He doesn’t really want a centralized world government or think that’s "a set of things that make sense and that are good". He’s forced into a world surveillance not because it’s good but because it’s the only alternative he sees to dangerous ASI being deployed.

I wouldn’t say he’s optimistic about human nature. In fact it’s almost the very opposite. He thinks that we’re doomed by our nature to create that which will destroy us.

The Daily Nous (a relatively 'popular' academic philosophy blog) managed to get a non-statement out of Oxford:

Oxford University has taken the difficult decision to close the Future of Humanity Institute, a research centre in the Faculty of Philosophy. The Institute has made an important contribution to the study of the future of humanity, for which we would like to thank and recognise the research team. Researchers elsewhere across Oxford University are likely to continue to work on this emerging field.

I would say that the closest to FHI at Oxford right now would probably be Global Priorities Institute (GPI). A lot of these papers would've made just as much sense coming out of FHI. (Might be worth considering how GPI apparently seems to have navigated Oxford better.)

EA has an extraordinary bad image right now, thanks largely to FTX. EA is not a good association to have in any context other than its base.

I suspect the pushback from within NIST has more to do with the fact that their budget has been cut to pay for this and very valuable projects put into indefinite suspension, for a cause that basically no one there supports.