LessWrong 2.0 Reader

A lot of AI safety seems to assume that humans are safer than they are, and that producing software that operates within a specification is harder than it is.  It's nice to see this paper moving towards integrating actual safety analysis (the remark about collapsing bridges was a breath of fresh air), instead of general demands that 'the AI always do as humans say'!

A human intelligence placed in charge of a nation state can kill 7 logs of humans and still be remembered heroically.  An AI system placed in charge of a utopian reshaping of the society of a major country with a 'keep the deaths within 6 logs' guideline that it can actually stay within would be an improvement on the status quo.

If safety people are saying 'we cant build AI systems that could make people feel bad, and we definitely can't build systems that kill people' their demand for perfection is in conflict with improvement.  

I suspect that major AI alignment failure will come from 'we put the human in charge, and human error led to the model doing bad'.  The industrial/aviation safety community now rightly views 'pilot error' as a lazy way of ending an analysis and avoiding making the engineering changes to the system that the accident conditions demand.

My raw and mostly confused/snarky comments as I was going through the paper can be found here (third section).

Cleaner version: this is not a technical agenda. This is not something that would elicit interesting research questions from a technical alignment researcher. There are however interesting claims:

• what a safe system ought to be like; it proposes three scales describing its reliability;
• how far up the scales we should aim for at minimum;
• how low on the scales currently large deployed models are.

While it positions a variety of technical agendas (mainly those of the co-authors) on the scales, the paper does not advocate for a particular approach, only the broad direction of "here are the properties we would like to have". Uncharitably, it's a reformulation of the problem.

The scales can be useful to compare the agenda that belong to the "let's prove that the system adheres to this specification" family. It makes no claims over what the specification entails, nor failure modes of various (combinations of) levels.

I appreciate this paper as a gateway to the related agendas and relevant literature, but I'm not enthusiastic about it.

I see. Thanks for the clarification!

Response to clarifying question:

Yes. The Duke has learned the hard way that his architects' guesses as to how much their projects will end up costing are consistently worse than useless; if you want to optimize on cost as well as impossibility, that's another thing you'll have to deduce from the record of finished projects.

Second, Lomborg’s 2024 paper finds evidence that women time their births to just before a wage growth plateau. The evidence it gives again comes from IVF failures. Women who were planning to have a birth, but never succeed, have much flatter wage growth after their planned birth year, even though they didn’t actually have any kids. So the divergence between childrearing mothers and non-childbearing mothers shows up even in this placebo case when neither group actually had kids. Therefore, the event study is overstating the earnings impact of childbirth.

Couldn't it also be that the women in question plan their career based on the expectation to have children and this is what leads to the plateau? In that case it seems like it would be incorrect to interpret these results as evidence against a child penalty, as it's merely that the child penalty affects women regardless of whether they have the children. To check, I think you should ask the study participants why their career plateaued then.

There's a soft patch around 5 and 6. Why is testability important? It's a charactersitic of science, but science assumes an external world. It's not a characteristic of philosophy -- good explanation is enough in philosophy, and the general posit of some sort of external world does explanatory work. And it's separate from the specific posit that the external world is knowable in some particular way.

Finally, I get to give one a try! I'll edit this post with my analysis and strategy. But first, a clarifying question - are the new plans supposed to be lacking costs?

Having a large set of small 2-10 minutes task on the screen may thus feel (incorrectly) overwhelming. 

The size of a task on the screen is a leaky abstraction (of it's length in time).

This is a valuable insight and makes reading this whole post worth it for me. And the obvious thought for how to correct this error is to attatch time-estimates for small tasks and convert them into a time-period view on a calendar. That way, it feels like "oh, I need 20 minutes to do all my chores today, better set a pomdoro" instead of "I have 20 things to do! 20 is big!"

Will have to try this. TEST: It doesn't look that big, though I'm including starting steps of longer term tasks. Hmm, this doesn't feel that bad, thought maybe that's the endorphins from deciding to test this talking. 

BTW I gave a strong upvote because I want to see more rationality related content on LW. Otherwise I would've given a normal upvote, or maybe not even that. Nevertheless, that still means this post gets a strong upvote.
 

There's a tiny possibility he may have influenced my thinking. I did spend 6 months editing him, among others for a documentary.

That looks interesting, will read :) Thanks.