Posts
Comments
That's a choice, though. AGI could, for example, look like a powerful actor in its own right, with its own completely nonhuman drives and priorities, and a total disinterest in being directed in the sort of way you'd normally associate with a "resource".
My claim is that the incentives AGI creates are quite similar to the resource curse, not that it would literally behave like a resource. But:
If by "intent alignment" you mean AGIs or ASIs taking orders from humans, and presumably specifically the humans who "own" them, or are in charge of the "powerful actors", or form some human social elite, then it seems as though your concerns very much argue that that's not the right kind of alignment to be going for.
My default is that powerful actors will do their best to build systems that do what they ask them to do (ie they will not pursue aligning systems with human values).
The field points towards this: alignment efforts are primarily focused on controlling systems. I don't think this is inherently a bad thing, but it results in the incentives I'm concerned about. I've not seen great work on defining human values, creating a value set a system could follow, and forcing them to follow it in a way that couldn't be overridden by its creators. Anthropic's Constitutional AI may be a counter-example.
The incentives point towards this as well. A system that is aligned to refuse efforts that could lead resource/power/capital concentration would be difficult to sell to corporations who are likely to pursue this.
These (here, here, and here) definitions are roughly what I am describing as intent alignment.
Glad you enjoyed it!
Could you elaborate on your last paragraph? Presuming a state overrides its economic incentives (ie establishes a robust post-AGI welfare system), I'd like to see how you think the selection pressures would take hold.
For what it's worth, I don't think "utopian communism" and/or a world without human agency are good. I concur with Rudolf entirely here -- those outcomes miss agency what has so far been an core part of the human experience. I want dynamism to exist, though I'm still working on if/how I think we could achieve that. I'll save that for a future post.
I appreciate this concern, but I disagree. An incognito google search of "intelligence curse" didn't yield anything using this phrase on the front page except for this LessWrong post. Adding quotes around it or searching for the full phrase ("the intelligence curse") showed this post as the first result.
A quick twitter search in recent shows the phrase "the intelligence curse" before this post:
- In 24 tweets in total
- With the most recent tweet on Dec 21, 2024
- Before that, in a tweet from August 30, 2023
- In 10 tweets since 2020
- And all other mentions pre-2015
In short, I don't think this is a common phrase and expect that this would be the most understood usage.
I agree that this could be a popular phrase because of future political salience. I expect that the idea that being intelligence is a curse would not be confused with this anymore than saying having resources are a curse (referring to wealthy people being unhappy) confuses people with the resource curse.
I think "the intelligence resource curse" would be hard for people to remember. I'm open to considering different names that are catchy or easy to remember.
I agree. To add an example: the US government's 2021 expanded child tax credit lifted 3.7 million children out of poverty, a near 50% reduction. Moreover, according to the NBER's initial assessment: "First, payments strongly reduced food insufficiency: the initial payments led to a 7.5 percentage point (25 percent) decline in food insufficiency among low-income households with children. Second, the effects on food insufficiency are concentrated among families with 2019 pre-tax incomes below $35,000".
Despite this, Congress failed to renew the program. Predictably, child poverty spiked the following year. I don't have an estimate for how many lives this cost, but it's greater than zero.