I'm a little confused by this post, in that it seems to be a little all over the place and comes off as a general "how things weren't so good, but are better now and there's a happy end" story. It says that it's about agency, but several of the problems involved (e.g. there being two factions with contradictory goals in the original group, the fact that the purchasing negotiations were complex) have no obvious connection to people being agenty.

I also have only a rough guess of what's meant by "agenty" in the first place, which might contribute to my confusion. I think this post could benefit if it explicitly gave a definition for the term and more clearly linked the various parts of the story with that definition. There are already some parts where the connection to agency is made quite clearly - e.g. "Today I know that if an agenty person has to write bylaws and they don't have experience, they go off and read about how to write bylaws" - but they're currently the exception rather than the norm.

I'm also not entirely convinced that "agency" is the best possible way of characterizing the events in question. For instance:

I appreciate the very few people who came to all of the meetings, and the people who actually put down their money and committed who didn't come to meetings. Even the people who just did a little, took on a risk that other people didn't, they did a lot more than the people who did nothing.

This sounds to me like "some of the people were more motivated by this goal than others" could be both a more accurate and a more useful description than "some were more agenty than others". More useful in the sense that the first description allows for the possibility of asking questions like "Might these people work harder if they were more motivated? Could they be more motivated? Is this lack of motivation a sign of the fact that they're not so strongly on board with this anyway, suggesting troubles later on?" and then taking different kinds of action based on the answers you get. The second description feels more likely to just make one shrug their shoulders and think "eh, they're just not sufficiently agenty, that's too bad". Or to put it in other words, the "agent" characterization sounds like an error model of people rather than a bug model.

So I think that this post would also benefit from not just defining "agenty", but also saying a few words about why we should expect this to be a useful and meaningful concept.

I find the following emotions are often associated with me being insufficiently "agenty":

• Wistfulness. An example.

• Exasperation. (Especially exasperation that someone else is not being agenty, about something that I could just as easily take over and get done myself.)

• Ugh-field sensations.

• Creative "stuckness". Talking to a beta reader almost always clears up this problem for me inside of fifteen to twenty minutes even if the beta doesn't actually have anything to say, and I still don't instantly grab one and start yammering when I feel it.

• Non-strategic "but I don't know how to do X!" (This is sometimes useful strategically, though.)

Often, if people fail to be agenty in the context that you are interested in, they are simply saving their energies to be agenty in another.

As to 95% failure rate: It might have been for the best that many of those projects did not proceed. Just because you've started something doesn't mean you should finish it.

I don't think it's just people being stupid. I mean, if I try to become a PC, I will die. (I will abandon my support structures, become unable to sustain effort before building new ones, get horribly sick, and die.) Many people have big losses on failure (internal or external), like responsibility to family.

Still, since you're a PC who knows lots of PCs: how do people, in practice, go about things like "At the age of 12, Badass Example left her village in East Timor to join the French Resistance. After assassinating Hitler, she founded the Donner Party and was elected President of Belgium."? I don't think you can just look up "How to locate and join a secret group" on eHow.

Perhaps I missed some previous required-reading, but... what exactly do you mean by "agenty", "agentiness", etc.?

(Also, what does "PC" refer to in this context?)

Edit: This?

Would you consider removing the last paragraph of this post? It reads like an overt "look at all the high-status people I know! I'm high-status too!" bid and jars with the rest of the post, which is significantly better.

Hypothesis: One reason there aren't many agenty people is that a lot of parents find that agenty children are more of a challenge to their authority than they want and/or take more resources of various sorts than they feel they can afford.

An even stronger criticism of AGI, both in agent and tool forms, is that a general intelligence is unlikely to be developed for economic reasons: specialzied AIs will always be more competitive.

Economic reasoning cuts many ways. Consider the trivial point known as Amdahl's law: speedups are always limited by the slowest serial component. (I've pointed this out before but less explicitly.)

Humans do not increase their speed even if specialized AIs are increasing their speed arbitrarily. Therefore, a human+specialized-AI system's performance asymptotically approaches the limit where the specialized-AI part takes zero time and the human part takes 100% of the time. The moment an AGI even slightly outperforms a human at using the specialized-AI, the same economic reasons you were counting on as your salvation suddenly turn on you and drive the replacement of any humans in the loop.

Since humans are a known fixed quantity, if an AGI can be improved - even if at all times it is strictly inferior to a specialized AI at the latter's specialization - then eventually an AGI+specialized-AI system will outperform a human+specialized-AI system barring exotic unproven assumptions about asymptotic limits.

(What human is in the loop on high frequency trading? Who was in the loop when Knight Capital's market maker was losing hundreds of millions of dollars? The answer is that no one was in the loop because humans in the loop would not have been economically competitive. That's fine when it's 'just' hundreds of millions of dollars at stake and companies can decide to take the risk for themselves or not - but the stakes can change, externalities can increase.)

I think there's a bit of a chicken-and-egg problem when you're not much of an agent yet, and you haven't accomplished anything interesting under your own steam, so it doesn't really even seem worthwhile to plan anything out. (Another failure mode: it does seem worthwhile to plan things out, but only because you haven't yet noticed that you rarely work on any of your plans.) Probably it makes sense to debug what's making you ineffective and build up a track record before tackling anything really big (see success spirals).

If you're one of those people who makes plans but never works on them, it might be a good idea to start being very distrustful of yourself whenever you say you're going to do something. Concrete example: Maybe there's some topic you've been intending to study for a while. If you were distrustful of yourself, instead of just continuing to intend to study it, you might block out some time during your week, then set up reminders on your cell phone, rules for when you can skip your study period, rules for when you're allowed to abandon the project entirely, etc.

The problem with these behavior regulation devices is that building them takes a large activation energy. Here are my solutions for this so far:

• Jot down ideas for a device whenever you have them, even if you aren't going to implement them immediately.

• Wait until you're in an especially energetic or inspired mood, then take advantage of it and implement a few devices (or debug ones that failed).

• Have the devices come in to effect a while after you've finished building them (ex: build your device in the evening and have it activate the next morning).

• Consume stimulants like coffee or kratom.

In my experience, after using such devices for a while I no longer needed them as much.

Of course, there are other components to being an agent, e.g. fearlessness. The modern world is pretty safe, but evolution calibrated us for a world that was much more dangerous, especially with regard to social blunders.

I'm wondering if what you call agency and what EY calls PC (or, in extreme/fictional cases, heroic responsibility) is what the rest of the (English-speaking) world calls initiative/motivation/perseverance?

Right after WWII John Holt was finishing out his U.S. Navy tour of duty on the west coast. In Never Too Late he tells this story about his favorite band at the time, the Woody Herman Herd, who he had listened to only on records:

They had been playing on the East Coast, and one of the many reasons I was eager to get out of the Navy was so that I could go hear them. Just as I was getting close to the date of my discharge, I heard terrible news -- the Herman band was going to come to the West Coast to play for a couple of months, and was then going to break up. I was going to miss them! I would never hear them! I was such a timid and conventional young man that it never occurred to me, not for a second, that I might stay out on the West Coast, arrange to get discharged there, see something of California and the Northwest, and hear Woody Herman in the process. But no, my home was in the East, and when the war ended I had to go home.

(Luckily he did manage to schedule his trip back east so that he crossed paths with the Herd in Chicago.)

As you say, getting things to happen takes a lot of time and effort, which is one of the things I learned when working on group projects in school. I think that most people, when they realize just how much effort it takes to Get Stuff Done, usually end up saying "screw it, there are other things I'd rather be doing" and go on doing whatever it is that they were doing before.

You are affirming the consequent and also overgeneralizing.

I argued that 'some economically valuable uses of AGI are replacing humans' (disproving Szabo's core argument that "AGI can always be outperformed at a specific task by a specialized-AI, therefore, there are no economically valuable uses of AGI").

That is not the same thing as 'all replacements of humans are economically valuable uses of AGI' for which 'non-AGI HFTs replacing humans' serves as a disproof (but so would cars or machines, for that matter).

Never heard of mayors and judges and ship captains officiating marriages?

ETA: And also James Randi of the JREF has also officiated at a wedding

For instance, this Divia lady you mention and her husband even changed their last name to 'Eden', the name of the earthly paradise in the Bible, and married in a ceremony officiated by Yudkowsky.

A marriage ceremony, officiated by an available elder of some sort. Name changes. Wow, what sort of crazy culture or subculture would do that kind of thing? Oh, right. Most of them, in some shape or form. This was actually a hat tip towards normality, not the reverse. Like celebrating Christmas with family and friends without actually believing in a Christ.

Your ranting is nonsense. Get some perspective.

First, I'd like to ask why you didn't reply directly to my previous comment, and instead started an entirely separate top-level comment. I hope your motive wasn't less than honorable, like hoping that I wouldn't notice and people would infer that I was tacitly admitting I was refuted? Hopefully I'm just being paranoid and you were careless about posting your comment or something.

But you didn't provide an argument for AGI being more effective at replacing humans at these tasks rather than more and more specialized AIs.

The claim "AGI will be more effective at replacing humans in using specialized-AIs" was assumed in my argument, and also not criticized by Szabo, who thinks his argument works even granting the existence of such AGI:

Even if there was such a thing as a "general intelligence" the specialized machines would soundly beat it in the marketplace. It would be very far from a close contest.

Great piece, Shannon. Brings to mind a couple of things.

What you call "agency" is, in Landmartian, "being cause in the matter," being "at cause," "taking a stand," and acting "consistently with that stand."

This is distinguished from being caught in a "racket," defined as a persistent complaint combined with a fixed way of being. Someone caught in a racket does not take responsibility for things as they are, but rather sets up stories that express being a victim of circumstances or others. The generic alternative is to accept responsibility, as a stand, not as a "truth."

That's been oft-misunderstood. I am responsible for, say, the WTC attack, as a stand, not as a fact. If I'm responsible, it means that I can look at my life as missing something that might make a difference, as full of possibilities.

In any case, most people, most of the time, are not at cause, we are simply reacting.

Then, if we actually take responsibility, beyond merely saying a few words, we act in accordance with that, which includes making mistakes, picking ourselves up and acting again, varying behavior as necessary to find a path to fulfillment.

A conversation I've had is "How many people does it take to transform society?"

The answer I've generally come up with is two. It's amazingly difficult to find two. Maybe that's just my racket, but your story shows how two can sometimes find more, if more are required to realize a stand. Two is where it starts. At least one of the two must be willing to be at cause, and able to stand there.

You're referring the Turing Test as a criterion and accusing us of anthroporphizing AI?

I think that an AI might become intelligent enough to destroy the human species, and still not be able to pass the Turing Test. Same way that we don't need to mimic whales or apes before being able to kill them.

It's not us who are anthroporphizing AI, it's you who're antroporphizing "intelligence that rivals and eventually surpasses the human intelligence both in magnitude and scope".

Your argument is that humans will eventually become the limiting factor in economic systems thus AGIs will be needed to replace them.

Good strawmanning. Very subtle.

gwern: The moment an AGI even slightly outperforms a human at using the specialized-AI, the same economic reasons you were counting on as your salvation suddenly turn on you and drive the replacement of any humans in the loop.

That's the key part. The specialized High-freq-trading software won't be replaced, but the humans who use that specialized software will eventually be replaced if someone figures out how to make an AGI that can think about all the relevant variables and can be scaled to go faster and better than a human.

Wow, way to miss the point and not respond to the argument - you know, the stuff that is not in parentheses.

(And anyway, how exactly am I supposed to give an example where AGI use is driven by economic pressures to surpass human performance, when AGI doesn't yet exist?)

So, even though you didn't clearly contest any of the premises nor the reasoning, let's assume that the second paragraph is a rebuttal to premises (1:) and/or (2:) of the grandparent.

An AGI is not something a bunch of nerds can cook up in their basement in their spare time.

I contest this premise, and I'm really wondering where you'd think that up. As technology progresses, we've noticed that it gets easier and easier to do stuff that was previously only possible for massive organizations.

Examples include, well, anything involving computers (since computers were first something only massive organizations could possess, until a bunch of nerds cooked it up in their basement), creating new software in the first place, creating videogames, publishing original research, running automated data-miners, creating new hardware gadgets, creating software that emulates hardware devices, validating formal mathematical proofs, running computer simulations...

...I could probably go on for a while, but I'm being warned that this should be enough to point at the right empirical cluster. Basically, we have lots of evidence saying that new-stuff-that-can-only-be-done-by-large-organizations can eventually be done by smaller groups, and not much that sets AGI apart as a particular exception other than the current perceived level of difficulty.

If an AGI will be always less effective than its contemporary specialized AIs, people will be unwilling to put their money, time and effor in it.

I just pointed out how economic reasoning can justify an AGI which is outperformed at any specific task by a specialized-AI. I'm not even an economist and it's a trivial argument, yet - there it is.

Even if one had a formal proof that AGIs must always be outperformed, that still would not show that AGIs will not be worth developing. You need a far more impressive argument covering all economic possibilities, especially since software & AI techniques are so economically valuable these days with no sign of interest letting up so handwaving arguments look implausible.

(I would be deeply amused to see a libertarian like Nick Szabo try to do such a thing because it runs so contrary to cherished libertarian beliefs about the value of local knowledge or the uselessness of elites or the weakness of theory, though I know he won't.)

anthropomorphization (the AGI), a exceptionally evil superhuman adversary (UFAI as Satan or the Antichrist)

Oddly, it seems to me that anthropomorphization is what makes people think AGI is perfectly safe.

Some future technology we currently have no idea how to develop will do X ("A miracle occurs").

Yeah, you treat the concept of new technologies (even though we experience new technologies every single year) on the same level as 'miracles' (which we've never experienced at all). I get that.

And I've seen lots of religious people argue thusly: "You believe in 'electrons' and 'quarks' that you've never seen with your own eyes, and I believe in angels and demons that I've never seen either. Therefore your 'scientific' ideas are just as faith-inspired as mine."

If we're to throw guilt-by-perceived association around, then I think that your criticism of LW-ideas are typically- religious. You're following the typical argument of the religious, where you try to claim all belief in things unseen is equally reasonable, all expectations of the future are equally reasonable, and hence "see, you're also a religion after all".

I think I'll have to revise my position - you are really not saying anything worth hearing.,.,

Maybe I'm attributing malice where a more likely explanation exists, but a policy which seems deliberately designed to incentivize groupthink appears to be more consistent with a cult rather than "a community blog devoted to refining the art of human rationality".

It's supposed to prevent people from feeding the trolls.

The system says: "Replies to downvoted comments are discouraged. You don't have the requisite 5 Karma points to proceed."

I don't follow. My comment http://lesswrong.com/lw/f53/now_i_appreciate_agency/7q56 was not at any point in the negative, much less the -5 or whatever that would cause the new karma penalty thing to kick in.

Still, that doesn't imply that AGI will be economically viable unless you show that humans will still be the limiting factor after every human skill that can be transfered to a specialized AI has been trasfered.

If every human skill has been transferred, including that of employing or combining specialized-AIs, then in what sense do the groups of specialized-AIs not then comprise an AGI?

This argument would seem to reduce you to confronting a dilemma: if every human skill has been transferred to specialized-AIs, then a complex of specialized-AIs by definition now forms an AGI which outperforms all humans; if not every human skill has been transferred, such as employing specialized-AIs, then there is the very large economic niche for AGIs which I have identified with my Amdahl's law argument. So either there exist AGI which outperform all humans, or there exists economic pressure for AGI.

A group of specialized AIs doesn't need to have shared goals or shared representations of the world. A group of interacting specialized AIs would be certainly be a complex system that will likely exhibit unanticipated behavior, but this doesn't mean that it will be an agent (an antropic model created in economy to model the behavior of humans).

I don't think this is a meaningful reply, or perhaps it's just question-begging.

If having a coherent goal is the point of the human in the loop, then you are quietly ignoring the hypothetical given that 'every human skill has been transferred' and your points are irrelevant. If having a coherent goal is not what the human is supposed to be doing, well, every agent can be considered to 'exhibit unanticipated behavior' from the point of view of its constituent parts (what human behavior would you anticipate from a single hair cell?), and it doesn't matter what the behavior of the complex system is - just that there is behavior. We can even layer on evolutionary concerns here: these complex systems will be selected upon and only the ones that act like agents will survive and spread!

Assuming that AI technology will necessarily lead to a super-intelligent but essentially human-like mind is anthropomorphization in the same sense that the gods of traditional religions are anthropomorphizations of complex and poorly understood (at the time) phenomena such as the weather or biological cycles or ecology.

Yeah, whatever.

Arguing against 'necessarily leading to a super-intelligent but essentially human-like mind' is a big part of Eliezer and LW's AI paradigm in general going back to the earliest writings & motivation for SIAI & LW, one of our perennial criticisms of mainstream SF, AI writers, and 'machine ethics' writers in particular, and a key reason for the perpetual interest by LWers in unusual models of intelligence like AIXI or in exotic kinds of decision theories.

If you've failed to realize this so profoundly that you can seriously write the above - accusing LW of naive religious-style anthropomorphizing! - all I can conclude is that you either are very dense or have not read much material.

"but still no AGI or commercial nuclear fusion, despite these having constantly been predicted to be in the next 25 years for the last 60 years.

Please clarify this plainly for me: Are you saying these technologies will NEVER be developed? Not in 25 year, nor in 100 years, nor in 500 years, nor in 10,000 years?

Is your whole disagreement a matter of timescales -- whether it is likely to happen to happen within our lifetimes or not?

Because if so, then there are a lot of us here who likewise don't expect to see AGI in our lifetimes.

If you're not saying "It will NEVER happen" then please specify a date by which time you'd assigning Probability > 50% of these technologies to have happened.

But until then, again your whole argument seems to be "it hasn't happened yet, so it will never happen."

Looks like you've completely missed the point of SIAI and massively misunderstand AI theory.

It seems to me like you have not even remotely the right order of magnitude of an idea of just how immense the laziness of some programmers can get. And the lazier programmers get, the more they try to write programs that do all their own work for them.

The ultimate achievement of the lazy programmer is to write a one-time program that will anticipate future needs to write programs, and write programs that can better anticipate such future needs and thus better write programs that meet the need, ad infinitum, without any further intervention from said programmer.

SIAI actually agrees that the above is probably not the most economically sensible thing to do and that it is not what most AIs, or even AGIs, developed in the near future will look like. However, SIAI is also aware that some people will, despite this, still want to be the ultimate lazy programmer and write the ultimate recursively self-modifying AI. No reasonable amount of reasonable arguments will change this fact.

Therefore, something must be done to prevent those AIs they will create from exterminating us. SIAI, in no small part through the work of Yudkowsky, has concluded that the best method of achieving this is currently through FAI research, and that eventually the only solution might be to make a Friendly self-modifying AGI before anyone else makes a non-Friendly one, so that the FAI has an unfair advantage and can outsmart any future non-friendly AGIs.

If you want to avoid being logically rude, you will contest either the premises (1: Some people will attempt to make the ultimate AGI. 2: One of them will eventually succeed. 3: Ultimate AGIs are Accidentally Unfriendly by default.) or some element in the chain of reasoning above. If you fail to do so, then the grandparent comment is understating how much you're missing the point and sidetracking the discussion.