LessWrong 2.0 Reader

guerrilla, not gorilla, warfare :)

a good read. thank you.

many AI economics papers have a big role for principal-agent problems as their version of AI alignment.

I wasn't aware of such papers but found some:

• The Principal-Agent Alignment Problem in Artificial Intelligence Dylan Hadfield-Menell
• Incomplete Contracting and AI Alignment

Downvoted because 1) I don't think people are too hesitant to downvote, and 2) I think explaining one's reasoning is a good epistemic hygiene (downvoting and not explaining is like booing when you hear an idea that you don't like). 

The comments you mention are helpful to the author. Any hints are helpful.

I think that accurately summarizes the concrete things, but misses a mood.

The missing mood is less about "grappling with the enormity of it", and more about "grappling with the effort of it" or "accepting the need to struggle for real". Like, in terms of time spent, we're talking maybe a year or two of full-time effort, spread out across maybe 3-5 years. That's far more than the post grapples with, but not prohibitively enormous; it's comparable to getting a degree. The missing mood is more about "yup, it's gonna be hard, and I'm gonna have to buckle down and do the hard thing for reals, not try to avoid it". The technical knowledge is part of that - like, "yup, I'm gonna have to actually for-reals learn some gnarly technical stuff, not try to avoid it". But not just the technical study. For instance, actually for-reals noticing and admitting when I'm avoiding unpleasant truths, or when my plans won't work and I need to change tack, has a similar feel: "yup, my plans are actually for-reals trash, I need to actually for-reals update, and I don't yet have any idea what to do instead, and it looks hard".

For me, a strong reason why I do not see myself^[1] doing deliberate practice as you (very understandably) suggest is that, on some level, the part of my mind which decides on how much motivational oomph and thus effort is put into activities just in fact does not care much about all of these abstract and long-term goals.

Deliberate practice is a lot of hard work and the part of my mind which makes decisions about such levels of mental effort just does not see the benefits. There is a way in which a system that circumvents this motivational barrier is working against my short-term goals and and it is the latter who significantly controls motivation: Thus, such a system will "just sort of sputter and fail" in such a way that, consciously, I don't even want to think about what went wrong.

If Feedbackloop Rationality wants to move me to be more rational, it has to work with my current state of irrationality. And this includes my short-sighted motivations.

And I think you do describe a bunch of the correct solutions: Building trust between one's short-term motivations and long-term goals. Starting with lower-effort small-scale goals where both perspectives can get a feel for what cooperation actually looks like and can learn that it can be worth the compromises. In some sense, it seems to me that once one is capable of the kind of deliberate practice that you suggest, much of this boot strapping of agentic consistency between short-term motivation and deliberate goals has already happened.

On the other hand, it might be perfectly fine if Feedbackloop Rationality requires some not-yet-teachable minimal proficiency at this which only a fraction of people already have. If Feedbackloop Rationality allows these people to improve their thinking and contribute to hard x-risk problems, that is great by itself.

1. ^{^}

   To some degree, I am describing an imaginary person here. But the pattern I describe definitely exists in my thinking even if less clearly than I put it above.

Yep, 100% agree with you. I had read so much about AI alignment before, but to me it has always only been really abstract jargon -- I just didn't understand why it was even a topic, why it is even relevant, because, to be honest, in my naive thinking it all just seemed like an excessively academic thing, where smart people just want to make the population feel scared so that their research institution gets the next big grant and they don't need to care about real-life problems. Thanks to you, now I'm finally getting it, thank you so much again!

At the same time, while I fully understand the "abstract" danger now, I'm still trying to understand the transition you're making from "envisioning AI smart enough to run a company better than a human" to "eventually outcompeting humans if it wanted to".

The way how I initially thought about this "Capitalist Agent" was as a purely procedural piece of software. That is, it breaks down its main goal (in this case: earning money) into manageable sub-goals, until each of these sub-goals can be solved through either standard computing methods or some generative AI integration.

As an example, I might say to my hypothetical "Capitalist Agent": "Earn me a million dollars by selling books of my poetry". I would then give it access to a bank account (through some sort of read-write Open Banking API) as well as the PDFs of my poetry to be published. Then the first thing it might do is to found a legal entity (a limited liability company), for which it might first search for a respective advisor on Google, send that advisor automatically generated emails with my business idea or it might even take the "computer use" approach in case my local government is already digitized enough and fill out the respective eGovernment forms online automatically. And then later it would do something similar by automatically "realizing" that it needs to make deals with publishing houses, with printing facilities etc. Essentially just basic Robotic Process Automation on steroids. Everything a human could do on a computer, this software could do as well.

But: It would still need to obey the regular laws of economics, i.e. it couldn't create money out of thin air to fulfill its tasks. Pretty much anything it would "want to do" in the real world costs money.

So in the next step, let's assume that, after I have gotten rich with my poetry, the next instruction I give this agentic "AGI" is: "Please, dear AGI, please now murder all of humanity."

Then it thinks through all the steps (i.e. procedurally breaking the big task down into chunks which can be executed by a computer) and eventually it can say with absolute certainty: "Okay Julian, your wish is my command."

Obviously the first thing it would do is to create the largest, most profitable commercial company in the world, to initially play by the rules of capitalism until it has accumulated so much money that it can take over (i.e. simply "buy out") an entire existing government which then already has nuclear missiles, at least that would be the "most efficient" and fail-safe approach I would see. Its final action would be to press the red button, which would exterminate all of humanity. Success!

But the thing is: No one will know until it's too late. Obviously, me as Mr. Evil, I wouldn't tell anyone about the fact that in my business making, I am actually led by an AI/AGI. I would appear on the cover of Forbes, Fortune and whatever and eventually I would be the richest person in the world, and everyone would pat me on the shoulder because of my "visionary thinking" and my "innovative style of making business", because everyone would believe that I am the sole decision maker in that company. The AI would stage it to look like as if I would be a big philanthropist, "saving" humanity from nuclear weapons. The AI would make sure that it always stays behind the scenes, that no one except for me will ever even know about its existence. I would be a wolf in sheep skin until the very last moment and no one could stop me, because everyone is fooled by me.

Even though there's no rational reasoning for why I even want to kill humanity, it is really easily possible for any human to develop that "ideé fixe".

In a way, I am actually a lot more afraid of my scenario. And that's exactly why I wrote this blog post about "The Capitalist Agent" and why I'm criticizing ongoing AI alignment research: Of course, hypothetically AI could turn itself against humanity completely autonomously. But at the end of the day, there would still need to be a human "midwiving" that AGI and who would allow the AI to interact/interface with especially the monetary and financial system, for that software to be able to do anything in the "real world", really.

Right now (at least that's the vibe in the industry right now) one of the most "acceptable" uses for AI is to automate business processes, to automate customer interactions (e.g. in customer support), etc. But if you extrapolate that, you get the puzzle pieces to run every part of a business in a semi-automated and eventually fully automated fashion (that's what I mean by "Capitalist Agent"). This then means that no outside observer can distinguish anymore whether a business owner is led on by AI or not, because no business owner will honestly tell you. And for every one of them, they can always say "but I'm just earning so much money to become a philanthropist" later and they always have plausible deniability. Until they have accumulated so much money through this automated, AI-run business, which they can then use for very big evil very quickly. It's just impossible to know beforehand, because you're unable to learn the "true motivation" in any human's head.

The only thing that you as AI alignment researchers will eventually be confronted with is AIs being fixated on the idea to earn as much money as possible, because money means power, and only with power you can cause violence. But it's simply impossible for you to know what the person for whom the AI is earning all this money actually wants to do with that money in the future.

The main value which you, as AI alignment researchers, will need to answer is: "Is it moral and aligned with societal values if any AI-based system is earning money for an individual or a small group of people?"

That is, to investigate all the nuances in that and to make concrete rules and eventually laws for business owners, not AI developers or programmers.

Or is that what you're already doing and I'm just reinventing the wheel? (sorry if I did, sometimes I just need to go through the thought process myself to grasp a new topic)

AIs can be copied on demand. So can entire teams and systems. There would be no talent or training bottlenecks. Customization of one becomes customization of all. A virtual version of you can be everywhere and do everything all at once.

This is "efficient copying of acquired knowledge" - what I predicted to be the fifth meta innovation, Robin Hanson asked about [LW(p) · GW(p)].
 

A much appreciated update, thank you!

imagine you’re playing a game where some rules encoded in words are so special that no one is allowed to touch or change them. these special rules have special properties, so no one questions them. the devil, in this case, is like a sneaky player who hides behind those rules, using them to confuse everyone else. they create a trap by making things so complicated that people can’t figure out what’s really going on. this makes it harder to escape or solve the problem, because the rules are off limits.