LessWrong 2.0 Reader

Must humans obey the Axiom of Irrelevant Alternatives?

If someone picks option A from options A, B, C, then they'll pick option A from options A and B. Roughly speaking, whether you prefer option A or B is independent of whether I offer you an irrelevant option C. This is an axiom of rationality, which is treated more fundamental than VNM, but should humans follow this? Maybe not.

Maybe humans are the negotiation between various "subagents", and many bargaining solutions (e.g. Kalai–Smorodinsky) violate IIA. We can use insight to decompose humans into subagents.

Let's suppose you pick A from {A,B,C} and B from {A,B} where:

• A = Walk with your friend
• B = Dinner party
• C = Stay home alone

This feel like something I can imagine.

The subagent explanation of these choices are that I am composed of two agents: the introvert and the extrovert. The introvert has preferences C > A > B and the extrovert has the opposite preferences B > A > C.

When the possible options are A and B, then the KS bargaining solution between the introvert and the extrovert will be B. At least, if the introvert has more "weight". But when the option space expands to include C, then the bargaining solution might shift to B. Intuitively, the "fair" solution is one where neither bargainer is sacrificing significantly more than the other.

Want to just give a quick take on this $450 o1-style model: https://novasky-ai.github.io/posts/sky-t1/

I think this matches a pattern we see a lot throughout the history of human engineering. Once a thing is known to be possible, and rough clues about how it was done are known (especially if many people get to play around with the product), then it won't be long until some other group figures out how to replicate a shoddy version of the new tech. And from there, usually (if there's market for it) improvements can steadily cause the shoddy version to catch up to close to the original in performance.

When we apply this lesson to AGI, we should assume that a similar sort of thing will happen if some company develops AGI and shows it off to the world. Especially if they give hints about how they did it, and if they let users interact with it. The question then is, how long until the world produces a '$450' knock-off version of the AGI?

This is super relevant for governance. You can't assume that everyone who makes a knock-off will be taking the same security precautions as the original inventors. If the thing blocking the AGI from self-improving is the disciplined restraint, government oversight, and security mindset of the original inventors... well, don't count on those things. If the knock-off AGI is good enough to self-improve, it's future versions won't be second-rate for long. Choosing not to assign the AGI to making stronger AGI is an alignment tax. Defectors will defect, and gain great power thereby.

We need a plan that covers this possibility. This is not definitely the path the future will take, but it is a plausible path.

I enjoyed reading this post; thank you for writing it. LessWrong has an allergy to basically every category Marx is a member of - "armchair" philosophers, socialist theorists, pop humanities idols - in my view, all entirely unjustified.

I had no idea Marx's forecast of utopia was explicitly based on extrapolating the gains from automation; I take your word for it somewhat, but from being passingly familiar with his work, I have a hunch you may be overselling his naivete.

Unfortunately, since the main psychological barrier to humans solving the technical alignment problem at present is not altruistic intentions, but raw cognitive intelligence, any meta-alignment scheme that proposes to succeed today has far more work cut out for it than just ensuring AGI-builders are accounting for risk to the best of their ability. It has to make the best of their ability good enough. That involves, at the very minimum, an intensive selection program for geniuses who are then placed in a carefully incentives-aligned research environment [LW(p) · GW(p)], and probably human intelligence enhancement.

I think the single most important point is: Keep a paradigm with human-legible CoT. Most other points are downstream of that. If it is legible, it is possible and more likely to notice that it is not faithful and to build monitoring on top. It might be the single simple You Get About Five Words [LW · GW] thing that might make it into regulation.

I am continually surprised how many people go out of their way not to conclude something like this.

(Self review) I stand by this essay, and in particular I like having this essay to point to as an example of why some organizations are not holding the idiot ball quite as much as people might assume. This essay is somewhat self defense? I work like this most of the time these days.

Followup work on how to better juggle balls is useful, and basically leads into an existing field of management. If One Day Sooner is unusual startup mode, Never Drop A Ball is a very normal middle and end stage of many organizations, and for good reasons. It's also a genuinely superior way for many groups to work. (Consider a hospital emergency room. Dr. House going deep into one patient's medical minutia is not as good as making sure that zero people have unsterilized and unbandaged bleeding wounds.) Having a shorter pointer is useful, though probably this could be made shorter and serve as a somewhat better pointer.

Followup on when and how to set balls down would be useful. Someone else should write that, I'm rubbish at it =P

Yeah, the thing the 'scaling extrapolation' view doesn't take into account is that as soon as radical speed-ups to algorithmic research are made possible by AI R&D agents, suddenly the trendlines for algorithmic progress should be projected to steepen. How much and for how long before slow-downs are hit? That's unclear. I think there is at least some substantial probability that no slow-downs are hit before full AGI, and some smaller but still considerable probability that the improvement cycle rushes forward at high speed past that point to ASI.

This should be assumed to potentially involve dramatic gains in both peak capabilities, and in efficiency and speed of training and inference. If so, then compute governance becomes completely irrelevant for blocking creation of dangerously powerful AI. It can still help put limits on the amount of inference used. Why? Because no matter how efficient the AI is, if you have more compute you have more parallel copies (and can run them faster up to the limits of the system, which is probably somewhere between 100x to 1000x human thought speed).

If we are going to head this off, we need new governance methods, and soon. Maybe really really soon, like, before the end of 2025. Hopefully we have until more like 2028, but we can't count on that for sure.

I have very little faith in current governments to implement and enforce policies that are more complex than things on the order of governance compute and chip export controls. Much less to do so within the short timeframes we are facing.

I think the conclusion this points towards is that we need new forms of governance. Not to replace existing governments, but to complement them. Voluntary mutual inspection contracts with privacy-respecting technology using AI inspectors. Something of that sort.

Here's some recent evidence of compute thresholds not being reliable: https://novasky-ai.github.io/posts/sky-t1/

Here's some self-links to some of my thoughts on this (I recommend reading the posts these comments are on as well):

https://www.lesswrong.com/posts/DvHokvyr2cZiWJ55y/2-skim-the-manual-intelligent-voluntary-cooperation?commentId=BBjpfYXWywb2RKjz5 [LW(p) · GW(p)]

https://www.lesswrong.com/posts/FEcw6JQ8surwxvRfr/human-takeover-might-be-worse-than-ai-takeover?commentId=uSPR9svtuBaSCoJ5P [LW(p) · GW(p)]

https://www.lesswrong.com/posts/tdrK7r4QA3ifbt2Ty/is-ai-alignment-enough?commentId=An6L68WETg3zCQrHT [LW(p) · GW(p)]

(Self review) I stand by this essay and think more people should read it, though they don't need to read it deeply. 

I think some people knew this kind of work and so this serves as a pointer to "yeah, that thing we did at my last company" and some people did not realize this was an option. Making people aware of potentially exciting options they could choose in life is (in my opinion) a good use of an essay. In my ideal world everyone would read something describing the One Day Sooner mindset as they were choosing their first careers so they could have it in mind as a possible trait jobs could have. Is that trait positive or negative? Depends on the person! 

I don't know if it's LessWrong Best Of material, but given the number of people who work in this manner that work in the community I think it's good to have some term for it in the water supply. 

Best when paired with Never Drop A Ball.

apologies for the issue with the neuronpedia link. it's now been resolved.

You make a very good point—there are different ways to create contrasts in information that are quickly recognized, and that’s how I’ve come to understand humor. The faster the brain recognizes the information, the higher the chance it triggers a sense of pleasure, or perhaps falls somewhere along a spectrum of pleasure tied to recognizing patterns or resolving contrasts.

I also agree that many jokes can reduce authority. They signal that someone is not a threat, that they lower themselves to gain acceptance, which is often done by people who aren’t strong or authoritative and who use humor as a way to find their space. On the other hand, humor can also be used by authoritative figures to reinforce their power—when others laugh, it confirms that they don’t perceive the person as a threat. Some studies even suggest that chimpanzee laughter might be linked to this dynamic.

In my case, I would describe myself as someone who, in many ways, felt “weak” and used humor to create space for myself. I wasn’t in a position to demand authority outright. I had to teach skills like first aid in a very short amount of time, and I found that humor as positive reinforcement was much more effective than relying on negative reinforcement.

It’s a complex topic, isn’t it? There are so many variables in humor, but this is the perspective I’ve been able to develop so far: humor as something that operates on a spectrum of pleasure derived from the rapid recognition of information.

Additionally, I plan to share some stories about how humor has helped me stay attentive and better use what a class or learning environment offers in a more constructive way.