Posts
Comments
this is quality.
I don't understand this post, because it seems to be parodying Anthropic's Responsible Scaling Policies (ie, saying that the RSPs are not sufficient), but the analogy to nuclear power is confusing since IMO nuclear power has in fact been harmfully over-regulated, such that advocating for a "balanced, pragmatic approach to mitigating potential harms from nuclear power" does actually seem good, compared to the status quo where society hugely overreacted to the risks of nuclear power without properly taking a balanced view of the costs vs benefits.
Maybe you can imagine how confused I am, if we use another example of an area where I think there is a harmful attitude of regulating entirely with a view towards avoiding visible errors of commision, and completely ignoring errors of omission:
Hi, we're your friendly local pharma company. Many in our community have been talking about the need for "vaccine safety."... We will conduct ongoing evaluations of whether our new covid vaccine might cause catastrophic harm (conservatively defined as >10,000 vaccine-side-effect-induced deaths).
We aren't sure yet exactly whether the vaccine will have rare serious side effects, since of course we haven't yet deployed the vaccine in the full population, and we're rushing to deploy the vaccine quickly in order to save the lives of the thousands of people dying to covid every day. But fortunately, our current research suggests that our vaccine is unlikely to cause unacceptable harm. The frequency and severity of side effects seen so far in medical trials of the vaccine are far below our threshold of concern... the data suggest that we don't need to adopt additional safety measures at present.
To me, vaccine safety and nuclear safety seem like the least helpful possible analogies to the AI situation, since the FDA and NRC regulatory agencies are both heavily infected with an "avoid deaths of commision at nearly any cost" attitude, which ignores tradeoffs and creates a massive "invisible graveyard" of excess deaths-of-omission. What we want from AI regulation isn't an insanely one-sided focus that greatly exaggerates certain small harms. Rather, for AI it's perfectly sufficient to take the responsible, normal, common-sensical approach of balancing costs and benefits. The problem is just that the costs might be extremely high, like a significant chance of causing human extinction!!
Another specific bit of confusion: when you mention that Chernobyl only killed 50 people, is this supposed to convey:
1. This sinister company is deliberately lowballing the Chernobyl deaths in order to justify continuing to ignore real risks, since a linear-no-threshold model suggests that Chernobyl might indeed have caused tens of thousands of excess cancer deaths around the world? (I am pretty pro- nuclear power, but nevertheless the linear-no-threshold model seems plausible to me personally.)
2. That Chernobyl really did kill only 50 people, and therefore the company is actually correct to note that nuclear accidents aren't a big deal? (But then I'm super-confused about the overall message of the post...)
3. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation on nuclear power plants in order to prevent other rare accidents that might kill 50 people tops? (This seems like extreme over-regulation of a beneficial technology, compared to the much larger number of people who die from the smoke of coal-fired power plants and other power sources.)
4. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation, because future accidents might indeed kill over 10,000 people? (This seems like it would imply some kind of conversation about first-principles reasoning and tail risks and stuff, but this isn't present in the post?)
I will definitely check out that youtube channel! I'm pretty interested in mechanism design and public-goods stuff, and I agree there are a lot of good ideas there. For instance, I am a huge fan of Georgism, so I definitely recognize that going all-in on the "libertarian individualist approach" is often not the right fit for the situation! Honestly, even though charter cities are somewhat an intrinsically libertarian concept, part of the reason I like the charter city idea is indeed the potential for experimenting with new ways to manage the commons and provide public goods -- Telosa is explicitly georgist, for example, and even hyper-libertarian Prospera has some pretty interesting concepts around things like crime liability insurance, which in the USA is considered a pretty left-wing (or maybe "far-liberal"? idk...) idea for trying to reduce gun violence.
But yeah, a lot of common leftist critiques of society/capitalism/etc can feel kind of... shallow, or overly-formulaic, or confused about the incentives of a given situation, to me? So I'd like to get a better understanding of the best versions of the leftist worldview, in order to better appreciate what the common critiques are getting at.
Yup, there are definitely a lot of places (like 99+% of places, 99+% of the time!) which aren't interested in a given reform -- especially one as uniqely big and experimental as charter cities. This is why in our video we tried to focus on political tractability as one of the biggest difficulties -- hopefully we don't come across as saying that the world will instantly be tiled over with charter cities tomorrow! But some charter cities are happening sometimes in some places -- in addition to the examples in the video, Zambia is pretty friendly towards the idea, and is supportive of the new-city project Nkwashi. (I think Charter Cities Institute considers Nkwashi to be their biggest current partnership?) Democracy was achieved, after all, even if it still hasn't won a total victory even after 250+ years.
Thanks, this is exciting and inspiring stuff to learn about!
I guess another thing I'm wondering about, is how we could tell apart genes that impact a trait via their ongoing metabolic activities (maybe metabolic is not the right term... what I mean is that the gene is being expressed, creating proteins, etc, on an ongoing basis), versus genes that impact a trait via being important for early embryonic / childhood development, but which aren't very relevant in adulthood. Genes related to intelligence, for instance, seem like they might show up with positive scores in a GWAS, but their function is confined to helping unfold the proper neuron connection structures during fetal development, and then they turn off, so editing them now won't do anything. Versus other genes that affect, say, what kinds of cholesterol the body produces, seem more likely to have direct impact via their day-to-day operation (which could be changed using a CRISPR-like tool).
Do we have any way of distinguishing the one type of genes from the other? (Maybe we can just look at living tissue and examine what genes are expressed vs turned off? This sounds hard to do for the entire genome...) Or perhaps we have reason to believe something like "only 20% of genes are related to early development, 80% handle ongoing metabolism, so the GWAS --> gene therapy pipeline won't be affected too badly by the dilution of editing useless early-development genes"?
Is there a plausible path towards gene therapies that edit dozens, hundreds, or thousands of different genes like this? I thought people were worried about off-target errors, etc? (Or at least problems like "you'll have to take 1000 different customized doses of CRISPR therapy, which will be expensive".) So my impression is that this kind of GWAS-inspired medicine would be most impactful with whole-genome synthesis? (Currently super-expensive?)
To be clear I agree with the main point this post is making about how we don't need animal models, etc, to do medicine if we have something that we know works!
(this comment is kind of a "i didn't have time to write you a short letter so I wrote you a long one" situation)
re: Infowar between great powers -- the view that China+Russia+USA invest a lot of efforts into infowar, but mostly "defensively" / mostly trying to shape domestic opinion, makes sense. (After all, it must be easier to control the domestic media/information lansdscape!) I would tend to expect that doing domestically-focused infowar stuff at a massive scale would be harder for the USA to pull off (wouldn't it be leaked? wouldn't it be illegal somehow, or at least something that public opinion would consider a huge scandal?), but on the other hand I'd expect the USA to have superior infowar technology (subtler, more effective, etc). And logically it might also be harder to percieve effects of USA infowar techniques, since I live in the USA, immersed in its culture.
Still, my overall view is that, although the great powers certainly expend substantial effort trying to shape culture, and have some success, they don't appear to have any next-gen technology qualitatively different and superior to the rhetorical techniques deployed by ordinary successful politicians like Trump, social movements like EA or wokeism, advertising / PR agencies, media companies like the New York Times, etc. (In the way that, eg, engineering marvels like the SR-72 Blackbird were generations ahead of competitors' capabilities.) So I think the overall cultural landscape is mostly anarchic -- lots of different powers are trying to exert their own influence and none of them can really control or predict cultural changes in detail.
re: Social media companies' RL algorithms are powerful but also "they probably couldn't prevent algorithms from doing this if they tried due to goodharts law". -- Yeah, I guess my take on this is that the overt attempts at propaganda (aimed at placating the NYT) seem very weak and clumsy. Meanwhile the underlying RL techniques seem potentially powerful, but poorly understood or not very steerable, since social media companies seem to be mostly optimizing for engagement (and not even always succeeding at that; here we are talking on LessWrong instead of tweeting / tiktoking), rather than deploying clever infowar superweapons. If they have such power, why couldn't left-leaning sillicon valley prevent the election of Trump using subtle social-media-RL trickery?
(Although I admit that the reaction to the 2016 election could certainly be interpreted as sillicon valley suddenly realizing, "Holy shit, we should definitely try to develop social media infowar superweapons so we can maybe prevent this NEXT TIME." But then the 2020 election was very close -- not what I'd have expected if info-superweapons were working well!)
With Twitter in particular, we've had such a transparent look at its operations during the handover to Elon Musk, and it just seems like both sides of that transaction have been pretty amateurish and lacked any kind of deep understanding of how to influence culture. The whole fight seems to have been about where to tug one giant lever called "how harshly do we moderate the tweets of leftists vs rightists". This lever is indeed influential on twitter culture, and thus culture generally -- but the level of sophistication here just seems pathetic.
Tiktok is maybe the one case where I'd be sympathetic to the idea that maybe a lot of what appears to be random insane trends/beliefs fueled by SGD algorithms and internet social dynamics, is actually the result of fairly fine-grained cultural influence by Chinese interests. I don't think Tiktok is very world-changing right now (as we'd expect, it's targeting the craziest and lowest-IQ people first), but it's at least kinda world-changing, and maybe it's the first warning sign of what will soon be a much bigger threat? (I don't know much about the details of Tiktok the company, or the culture of its users, so it's hard for me to judge how much fine-grained control China might or might not be exerting.)
Unrelated -- I love the kind of sci-fi concept of "people panic but eventually go back to using social media and then they feel fine (SGD does this automatically in order to retain users)". But of course I think that the vast majority of users are in the "aren't panicking" / never-think-about-this-at-all category, and there are so few people in the "panic" category (panic specifically over subtle persuasion manipulation tech that isn't just trying to maximize engagement but instead achieve some specific ideological outcome, I mean) that there would be no impact on the social-media algorithms. I think it is plausible that other effects like "try not to look SO clickbaity that users recognize the addictiveness and leave" do probably show up in algorithms via SGD.
More random thoughts about infowar campaigns that the USA might historically have wanted to infowar about:
- Anti-communism during the cold war, maybe continuing to a kind of generic pro-corporate / pro-growth attitude these days. (But lots of people were pro-communist back in the day, and remain anti-corporate/anti-growth today! And even the republican party is less and less pro-business... their basic model isn't to mind-control everyone into becoming fiscal conservatives, but instead to gain power by exploiting the popularity of social conservativism and then use power to implement fiscal conservativism.)
- Maybe I am taking a too-narrow view of infowar as "the ability to change peoples' minds on individual issues", when actually I should be considering strategies like "get people hyped up about social issues in order to gain power that you can use for economic issues" as a successful example of infowar? But even if I consider this infowar, then it reinforces my point that the most advanced stuff today all seems to be variations on normal smart political strategy and messaging, not some kind of brand-new AI-powered superweapon for changing people's minds (or redirecting their focus or whatever) in a radically new way.
- Since WW2, and maybe continuing to today, the West has tried to ideologically immunize itself against Nazi-ism. This includes a lot of trying to teach people to reject charismatic dictators, to embrace counterintuitive elements of liberalism like tolerance/diversity, and even to deny inconvenient facts like racial group differences for the sake of social harmony. In some ways this has gone so well that we're getting problems from going too far in this direction (wokism), but in other ways it can often feel like liberalism is hanging on by a thread and people are still super-eager to embrace charismatic dictators, incite racial conflict, etc.
"Human brains are extremely predisposed to being hacked, governments would totally do this, and the AI safety community is unusually likely to be targeted."
-- yup, fully agree that the AI safety community faces a lot of peril navigating the whims of culture and trying to win battles in a bunch of diverse high-stakes environments (influencing superpower governments, huge corporations, etc) where they are up against a variety of elite actors with some very strong motivations. And that there is peril both in the difficulty of navigating the "conventional" human-persuasion-transformed social landscape of today's world (already super-complex and difficult) and the potentially AI-persuasion-transformed world of tomorrow. I would note though, that these battles will (mostly?) play out in pretty elite spaces, wheras I'd expect the power of AI information superweapons to have the most powerful impact on the mass public. So, I'd expect to have at least some warning in the form of seeing the world go crazy (in a way that seems different from and greater than today's anarchic internet-social-dynamics-driven craziness), before I myself went crazy. (Unless there is an AI-infowar-superweapon-specific hard-takeoff where we suddenly get very powerful persuasion tech but still don't get the full ASI singularity??)
re: Dath Ilan -- this really deserves a whole separate comment, but basically I am also a big fan of the concept of Dath Ilan, and I would love to hear your thoughts on how you would go about trying to "build Dath Ilan" IRL.
- What should an individual person, acting mostly alone, do to try and promote a more Dath-Ilani future? Try to practice & spread Lesswrong-style individual-level rationality, maybe (obviously Yudkowsky did this with Lesswrong and other efforts). Try to spread specific knowledge about the way society works and thereby build energy for / awareness of ways that society could be improved (inadequate equilibria kinda tries to do this? seems like there could be many approaches here). Personally I am also always eager to talk to people about specific institutional / political tweaks that could lead to a better, more Dath-Ilani world: georgism, approval voting, prediction markets, charter cities, etc. Of those, some would seem to build on themselves while others wouldn't -- what ideas seem like the optimal, highest-impact things to work on? (If the USA adopted georgist land-value taxes, we'd have better land-use policy and faster economic growth but culture/politics wouldn't hugely change in a broadly Dath-Ilani direction; meanwhile prediction markets or new ways of voting might have snowballing effects where you get the direct improvement but also you make culture more rational & cooperative over time.)
- What should a group of people ideally do? (Like, say, an EA-adjacent silicon valley billionaire funding a significant minority of the EA/rationalist movement to work on this problem together in a coordinated way.) My head immediately jumps to "obviously they should build a rationalist charter city":
- The city doesn't need truly nation-level sovereign autonomy, the goal would just be to coordinate enough people to move somewhere together a la the Free State Project, gaining enough influence over local government to be able to run our own policy experiments with things like prediction markets, georgism, etc. (Unfortunately some things, like medical research, are federally regulated, but I think you could do a lot with just local government powers + creating a critical mass of rationalist culture.)
- Instead of moving to a random small town and trying to take over, it might be helpful to choose some existing new-city project to partner with -- like California Forever, Telosa, Prospera, whatever Zuzalu or Praxis turn into, or other charter cities that have amenable ideologies/goals. (This would also be very helpful if you don't have enough people or money to create a reasonably-sized town all by yourself!)
- The goal would be twofold: first, run a bunch of policy experiments and try to create Dath-Ilan-style institutions (where legal under federal law if you're still in the USA, etc). And second, try to create a critical mass of rationalist / Dath Ilani culture that can grow and eventually influence... idk, lots of people, including eventually the leaders of other governments like Singapore or the UK or whatever. Although it's up for debate whether "everyone move to a brand-new city somewhere else" is really a better plan for cultural influence than "everyone move to the bay area", which has been pretty successful at influencing culture in a rationalist direction IMO! (Maybe the rationalist charter city should therefore be in Europe or at least on the East Coast or something, so that we mostly draw rationalists from areas other than the Bay Area. Or maybe this is an argument for really preferring California Forever as an ally, over and above any other new-city project, since that's still in the Bay Area. Or for just trying to take over Bay Area government somehow.)
- ...but maybe a rationalist charter city is not the only or best way that a coordinated group of people could try to build Dath Ilan?
(Copies from EA Forum for the benefit of lesswrongers following the discussion here)
Definitely agree that empathy and other social feelings provide indirect evidence for self-awareness (ie, "modeling stuff about yourself" in your brain) in a way that optimism/pessimism or pain-avoidance doesn't. (Although wouldn't a sophisticated-enough RL circuit, interacting with other RL circuits in some kind of virtual evolutionary landscape, also develop social emotions like loyalty, empathy, etc? Even tiny mammals like mice/rats display sophisticated social behaviors...)
I tend to assume that some kind of panpsychism is true, so you don't need extra "circuitry for experience" in order to turn visual-information-processing into an experience of vision. What would such extra circuitry even do, if not the visual information processing itself? (Seems like maybe you are a believer in what Daniel Dennet calls the "fallacy of the second transduction"?)
Consequently, I think it's likely that even simple "RL algorithms" might have a very limited, very shallow, non-self-aware kinds of experience: an image-classifier is doing visual-information-processing, so it probably also produces isolated "experiences of vision"! But of course it would not have any awareness of itself as being a thing-that-sees, nor would those isolated experiences of vision be necessarily tied together into a coherent visual field, etc.
So, I tend to think that fish and other primitive creatures probably have "qualia", including something like a subjective experience of suffering, but that they probably lack any sophisticated self-awareness / self-model, so it's kind of just "suffering happening nowhere" or "an experience of suffering not connected to anything else" -- the fish doesn't know it's a fish, doesn't know that it's suffering, etc, the fish is just generating some simple qualia that don't really refer to anything or tie into a larger system. Whether you call such a disconnected & shallow experience "real qualia" or "real suffering" is a question of definitions.
I think this personal view of mine is fairly similar to Eliezer's from the Sequences: there are no "zombies" (among humans or animals), there is no "second transduction" from neuron activity into a mythical medium-of-consciousness (no "extra circuitry for experience" needed), rather the information-processing itself somehow directly produces (or is equivalent to, or etc) the qualia. So, animals and even simpler systems probably have qualia in some sense. But since animals aren't self-aware (and/or have less self-awareness than humans), their qualia don't matter (and/or matter less than humans' qualia).
...Anyways, I think our core disagreement is that you seem to be equating "has a self-model" with "has qualia", versus I think maybe qualia can and do exist even in very simple systems that lack a self-model. But I still think that having a self-model is morally important (atomic units of "suffering" that are just floating in some kind of void, unconnected to a complex experience of selfhood, seem of questionable moral relevance to me), so we end up having similar opinions about how it's probably fine to eat fish.
I guess what I am objecting to is that you are acting like these philosophical problems of qualia / consciousness / etc are solved and other people are making an obvious mistake. I agree that I see a lot of people being confused and making mistakes, but I don't think the problems are solved!
Why would showing that fish "feel empathy" prove that they have inner subjective experience? It seems perfectly possible to build a totally mechanical, non-conscious system that nevertheless displays signs of empathy. Couldn't fish just have some kind of built-in, not-necessarily-conscious instinct to protect other fish (for instance, by swimming together in a large school) in order to obtain some evolutionary benefit?
Conversely, isn't it possible for fish to have inner subjective experience but not feel empathy? Fish are very simple creatures, while "empathy" is a complicated social emotion. Especially in a solitary creature (like a shark, or an octopus), it seems plausible that you might have a rich inner world of qualia alongside a wide variety of problem-solving / world-modeling skills, but no social instincts like jealousy, empathy, loyalty, etc. Fish-welfare advocates often cite studies that seem to show fish having an internal sense of pain vs pleasure (eg, preferring water that contains numbing medication), or that bees can have an internal sense of being optimistic/risky vs pessimistic/cautious -- if you think that empathy proves the existence of qualia, why are these similar studies not good enough for you? What's special about the social emotion of empathy?
Personally, I am more sympathetic to the David Chalmers "hard problem of consciousness" perspective, so I don't think these studies about behaviors (whether social emotions like jealousy or more basic emotions like optimism/pessimism) can really tell us that much about qualia / inner subjective experience. I do think that fish / bees / etc probably have some kind of inner subjective experience, but I'm not sure how "strong", or vivid, or complex, or self-aware, that experience is, so I am very uncertain about the moral status of animals. (Personally, I also happily eat fish & shrimp all the time.)
In general, I think this post is talking about consciousness / qualia / etc in a very confused way -- if you think that empathy-behaviors are ironclad proof of empathy-qualia, you should also think that other (pain-related, etc) behaviors are ironclad proof of other qualia.
Hi Trevor! I appreciate this thread of related ideas that you have been developing about intelligence agencies, AI-augmented persuasion techniques, social media, etc.
- It seems important to "think ahead" about how the power-struggle over AI will play out as things escalate to increasingly intense levels, involving eg national governments and militaries and highly-polarized political movements and etc.
- Obviously if some organization was hypercompetent and super-good at behind-the-scenes persuasion, we wouldn't really know about it! So it is hard to 100% confidently dismiss the idea that maybe the CIA has next-gen persuasion tech, or whatever.
- Obviously we are already, to a large extent, living in a world that is shaped by the "marketplace of ideas", where the truth often gets outcompeted by whatever sounds best / is most memetically fit. Thinking about these dynamics (even without anything AI-related or any CIA conspiracies) is confusing, but seems very important. Eg, I myself have been deeply shaped by the crazy memetic landscape in ways that I partially endorse and partially don't. And everything I might try to do to achieve impact in the world needs to navigate the weird social landscape of human society, which in many respects is in a kind of memetic version of the "equilibrium of no free energy" that yudkowsky talks about in Inadequate Equilibria (although there he is talking mostly about an individual-incentives landscape, rather than a memetic landscape).
- AI super-persuasion does seem like something we might plausibly get before we get general ASI, which seems like it could be extremely weird / dangerous / destabilizing.
That said, I think this post is too conspiratorial in assuming that some combination of social media companies / national governments understand how to actually deploy effective persuasion techniques in a puppetmaster-like way which is way beyond everyone else. I think that the current situation is more like "we are living in an anarchic world influenced by an out-of-control memetic marketplace of ideas being influenced by many different actors of varying levels of sophistication, none of whom have amazing next-level gameboard-flipping dominance". Some scattered thoughts on this theme:
- If the CIA (or other entities affiliated with the US government, including tech companies being pressured by the government) is so good at persuasion ops, why are there so many political movements that seem to go against the CIA's interests? Why hasn't the government been able to use its persuasion jiujitsu to neutralize wokeism and Trump/MAGA-ism? From an establishment perspective, both of these movements seem to be doing pretty serious damage to US culture/institutions. Maybe these are both in the process of being taken down by "clown attacks" (although to my eye, this looks less like an "attack" from CIA saboteurs, and more like a lot of genuine ordinary people in the movement themselves just being dumb / memetic dynamics playing out deterministically via social dynamics like yudkowsky's "evaporative cooling of group beliefs")? Or maybe ALL of, eg, wokeism, is one GIANT psy-op to distract the American people from creating a left-wing movement that is actually smart and effective? (I definitely believe something like this, but I don't believe it's a deliberate military psy-op... rather it's an emergent dynamic. Consider how corporations are differentially friendlier to wokeism than they are to a more economically-focused, class-based Bernie-ism, so wokeism has an easier time spreading and looking successful, etc. It also helps that wokeism is memetically optimized to appeal to people in various ways, versus a genuinely smart-and-effective left-wing policy idea like Georgism comes off as boring, technocratic, and hard-to-explain.)
- Basically, what I am saying is that our national politics/culture looks like the product of anarchic memetic optimization (recently turbocharged by social media dynamics, as described by folks like Slate Star Codex and the book "Revolt of the Public") much moreso than the product of top-down manipulation.
- If google & facebook & etc are so good at manipulating me, why do their efforts at influence often still seem so clumsy? Yes, of course, I'm not going to notice the non-clumsy manipulations! And yes, your "I didn't speak up, because I wasn't as predictable as the first 60%" argument certainly applies here -- I am indeed worried that as technology progresses, AI persuasion tech will become a bigger and bigger problem. But still, in the here and now, Youtube is constantly showing me these ridiculous ideological banners about "how to spot misinformation" or "highlighting videos from Black creators" or etc... I am supposed to believe that these people are some kind of master manipulators? (They are clearly just halfheartedly slapping the banners on there in a weak attempt to cover their ass and appease NYT-style complaints that youtube's algorithm is unintentionally radicalizing people into trumpism... they aren't even trying to be persuasive to the actual viewers, just hamfistedly trying to look good to regulators...)
- Where is the evidence of super-persuasion techniques being used by other countries, or in geopolitical situations? One of the most important targets here would be things like "convincing Taiwanese to identify mostly as ethnic Chinese, or mostly as an independent nation", or the same for trying to convince Ukrainians to align more with their Russian-like ethnicity and language or with the independent democracies of western Europe. Ultimately, the cultural identification might be the #1 decisive factor in these countries' futures, and for sure there are lots of propaganda / political messaging attempts from all sides here. But nobody seems like they have some kind of OP superweapon which can singlehandedly change the fate of nations by, eg, convincing Taiwanese people of something crazy, like embracing their history as a Japanese colony and deciding that actually they want to reunify with Japan instead of remaining independent or joining China!
- Similarly, the Russian attempts to interfere in the 2016 election, although initially portrayed as some kind of spooky OP persuasion technique, ultimately ended up looking pretty clumsy and humdrum and small-scale, eg just creating facebook groups on themes designed to inflame American cultural divisions, making wacky anti-Hillary memes, etc.
- China's attempts at cultural manipulation are probably more advanced, but they haven't been able to save themselves from sinking into a cultural atmosphere of intense malaise and pessimism, one of the lowest fertility rates in the world, etc. If persuasion tech was so powerful, couldn't China use it to at least convince people to keep plowing more money into real estate?
- Have there been any significant leaks that indicate the USA is focused on persuasion tech and has seen significant successes with it? If I recall correctly, the Edward Snowden leaks (admittedly from the NSA which focuses on collecting information, and from 10 years ago) seemed to mostly indicate a strategy of "secretly collect all the data" --> "search through and analyze it to identify particular people / threats / etc". There didn't seem to be any emphasis on trying to shape culture more broadly.
- Intelligence agencies in the USA devote some effort to "deradicalization" of eg islamist terrorists, extreme right-wingers, etc. But this stuff seems to be mostly focused on pretty narrow interventions targeting individual people or small groups, and seems mostly based on 20th-century-style basic psychological understanding... seems like a far cry from A/B testing the perfect social-media strategy to unleash on the entire population of some middle-eastern country to turn them all into cosmopolitan neoliberals.
Anyways, I guess my overall point is that it just doesn't seem true that the CIA, or Facebook, or China, or anyone else, currently has access to amazing next-gen persuasion tech. So IMO you are thinking about this in the wrong way, with too much of a conspiratorial / Tom Clancy vibe. But the reason I wrote such a long comment is because I think you should keep exploring these general topics, since I agree with you about most of the other assumptions you are making!
- We are already living in a persuasion-transformed world in the sense that the world is full of a lot of crazy ideas which have been shaped by memetic dynamics
- Social media in particular seems like a powerful lever to influence culture (see Slate Star Codex & Revolt of the Public)
- It seems like you probably COULD influence culture a ton by changing the design of social media, so it's a little funny that nobody seems to be intentionally using this to build a persuasion superweapon
- (Nevertheless I think nobody really understands the long-term cultural effects of social media well enough to make deliberate changes to achieve eventual intended results. And I think there are limits to what you could do with current techniques -- changing the design & policies of a site like Twitter might change the broad cultural vibe, but I don't think we could create an especially persuasive superweapon that could be aimed at particular targets, like making Taiwanese people culturally identify with Japan)
- It definitely seems like AI could be used for all kinds of censorship & persuasion-related tasks, and this seems scary because it might indeed allow the creation of persuasion superweapons.
- Totally separately from all the above stuff about persuasion, the shadowier parts of governments (military & intelligence-agency bureaucracies) seem very important to think about when we are trying to think ahead about the future of AI technology and human civilization.
Thanks for writing this post, I 100% share your sentiment and appreciate the depth with which you've explored this topic, including some of the political considerations.
Here are some other potentially-relevant case studies of people doing similar-ish things, trying to make the world a better place while navigating touchy political fears related to biotech:
- The "Enhanced Games" is organizing an alternative to the Olympic games where doping and other human enhancement technologies will be allowed. Naturally, they try to put a heavy emphasis on the importance of human freedom, plus some random criticisms of the way the existing Olympics are organized. But what I thought was especially striking was the way they leaned into social-justice-y rhetoric: "science is real", "stop exploitation", and even their logo is a dead ringer for the famous "equality" logo of the LGBT rights movement. For an antimalarial gene drive, I think a similar approach could work well (at least for winning the support of Westerners) -- leaning into rhetoric about postcolonialism and how the gene-drive initiative represents the people taking charge of their own destiny instead of waiting for western aid/charity (bednets, vaccines, etc) that hasn't been sufficient. (Charter Cities are in a somewhat similar situation, where it's very important for them to convince people that this is an empowering way of advancing human liberty while helping the developing world, rather than some kind of conniving neocolonialism intended to disempower people.)
- The C4 Rice Project has been operating for a long time, working towards the dream of engineering rice to more efficiently photosynthesize and thus boosting yields around the world.
- The Far-Out Initiative is hoping to trial gene editing to reduce animal suffering; their website has some interesting FAQs and in general the project has the same "activist genetics lab" vibe that a mosquito gene-drive lab might strive for.
- Same deal for the project to revive Woolly Mammoths -- the awesome documentary "We Are As Gods" is basically a PR campaign for the righteousness of this cause, and a good portrait of a similar movement which is farther along in the PR pipeline.
- Genomic embryo-selection companies like Lifeview and Orchid Health are also interesting in this context, although since they don't have to convince regulators or the wider public, they are keeping a lower profile for now. There are also some essentially stealth-mode groups who are investigating enhancements to the current state-of-the-art in embryo selection. These groups would be less interesting in terms of learning from their PR campaigns (they have none), but it might be helpful to study how they build a skilled team, raise funding, etc.
Some further questions I have about the political and theory-of-change considerations:
- I think it could be helpful to explore a more detailed breakdown of who exactly might be opposed, and for what reasons. And then try and figure out which of these sources actually matter the most / are the most real! For example:
- Maybe the people of a developing country will be opposed, because they just find GMOs scary / would be worried about being bitten by a GMO mosquito.
- Maybe neighboring countries will be opposed because they'll see it as an imposition on their sovereignty that an important decision (even if the decision is just... curing malaria, lol) is being taken without their input.
- Ordinary citizens and activists in the west might be opposed mostly because of some kind of "neocolonialism"/social-justice concerns, or mostly because of environmental concerns (removing mosquitoes might disrupt the environment), or mostly because of FDA-style biomedical caution and fear of GMOs.
- Elites in the developed world might be concerned from a perspective of international diplomacy and norms -- sure, THIS unilateral genetic engineering project will have amazing consequences, but will it end up net-negative if it encourages OTHER unilateral actions in the future that are more harmful? (Feels similar to the sentiment against climate geoengineering or human genetic editing.) What could be done to ameliorate this concern?
- Is there some way that a gene drive could be framed as partially accidental, or an inevitable by-product of some other necessary action? Sometimes you need a good excuse to do something good for the world... I am thinking of situations like:
- When an accidental freezer failure during covid vaccine distribution actually resulted in giving lots more people the vaccine, because it helped doctors get around onerous verification requirements about who was allowed to get one.
- Geoengineering experiments are still very taboo, but a recent UN regulation reducing sulfur dioxide emissions is unintentionally warming the earth and also giving scientists tons of data about the efficacy of future SO2-based climate interventions.
- Similarly, fertilizing the oceans with iron to fight climate change is considered taboo. But it might be easier to argue for fertilizing the oceans with iron in order to help sperm whale populations recover, since sperm whales once naturally helped iron cycle through the oceans but our own human actions caused their populations to decline. (Fighting climate change would thus just be an "accidental" benefit of helping the sperm whales.)
- In the malaria gene-drive case, the best possible headline is probably always gonna be "unanimous international agreement reached to release gene drive!" But as a second-best option, "truck carrying mosquitoes for gene-drive study crashes, thousands of GMO mosquitoes released, scientists very apologetic!" is DEFINITELY preferable to "rogue bio-activists publish manifesto, release thousands of GMO mosquitoes". And it might even be preferable to something like "president of Ghana, citing failure of western aid and the hypocrisy of colonial powers, unilaterally begins GMO drive".
- I'd also note that, although most prevalent in africa, malaria and other mosquito-borne diseases are common throughout the tropics. So although africa might be ideal from an impact standpoint, getting a project off the ground in southeast asia or central america is also worth considering, if the politics are more favorable / if it would be easier to set up a genetics lab there!
Yeah, I am interested in this from the "about to have an infant" perspective (my wife is almost 20 weeks pregnant). Interestingly this means she will be able to get both the flu, covid, and newly-approved RSV shot.
- Presumably you want to space out the vaccines a lot -- I would guess two weeks at least, but maybe more?
- Is there a difference between when covid, flu, and RSV peak in activity, which might justify getting one before the other? (The RSV vaccine is apparently only approved for weeks 32 - 36 of pregnancy, so we will at least have to wait at least another 12 weeks I guess, which annoyingly takes us all the way past the holidays.)
- Like you say, I am thinking that earlier is better (rather "play it safe" and have some immunity, even if it later wears off), so she has already gotten her flu shot. (Does flu or covid immunity wane faster?)
- I think part of the reason RSV is only approved for the third trimester is to transfer some immunity to the child, so that the newborn is protected in its first months of life. Presumably that logic applies less for influenza (which is not particularly severe in newborns) or covid (which seems especially mild in children)?
Good point that rationalism is over-emphasizing the importance of Bayes theorem in a pretty ridiculous way, even if most of the individual statements about Bayes theorem are perfectly correct. I feel like if one was trying to evaluate Eliezer or the rationalist community on some kind of overall philosophy scorecard, there would be a lot of situations like this -- both "the salience is totally out of whack here even though it's not technically /wrong/...", and "this seems like a really important and true sentiment, but it's not really the kind of thing that's considered within the purview of academic philosophy..." (Such as the discussion about ethics / morality / value, and many other parts of the Sequences... I think there is basically a lot of helpful stuff in those posts, some of which might be controversial, but it isn't really an Official Philosophical Debate over stuff like whether anti-realism is true. It's more like "here's how I think you should live your life, IF anti-realism is true".)
Didn't mention many-worlds because it doesn't feel like the kind of thing that a philosopher would be fully equipped to adjudicate? I personally don't feel like I know enough to have opinions on different quantum mechanics interpretations or other issues concerning the overall nature / reality of the universe -- I still feel very uncertain and confused about that stuff, even though long ago I was a physics major and hoped to some day learn all about it. Although I guess I am sorta more sympathetic to Many Worlds than some of the alternatives?? Hard to think about, somehow...
Philosophers having hot takes on linguistics and the relationship between words and concepts -- not good or bad that they have so many takes, and I'm also not sure if the takes themselves are good or bad. It is just my impression that, unlike some of the stuff above, philosophy seems to have really spent a lot of time debating these issues, and thus it would be ripe for finding well-formed disagreements between EY and various mainstream schools of thought. I do think that maybe philosophers over-index a little on thinking about the nature of words and language (ie that they have "too many takes"), but that doesn't seem like such a bad thing -- I'm glad somebody's thinking about it, even if it doesn't strike me as the most important area of inquiry!
Yeah, agreed that that Solomonoff induction argument feels very bizzarre! I had never encountered that before. I meant to refer to the many different arguments for atheism sprinkled throughout the Sequences, including many references to the all-time classic idea that our discovery of the principles of evolution and the mechanics of the brain are sufficient to "explain away" the biggest mysteries about the origin of humanity, and should thus sideline the previously-viable hypothesis of religious claims being true. (See here and here.) EY seems to (rightly IMO) consider the falseness of major religious claims to be a "slam dunk", ie, totally overdetermined to be false -- the Sequences are full of funny asides and stories where various religious people are shown to be making very obvious reasoning errors, etc.
Some other potentially controversial views that a philosopher might be able to fact-check Eliezer on, based on skimming through an index of the sequences:
- Assorted confident statements about the obvious supremacy of Bayesian probability theory and how Frequentists are obviously wrong/crazy/confused/etc. (IMO he's right about this stuff. But idk if this counts as controversial enough within academia?)
- Probably a lot of assorted philosophy-of-science stuff about the nature of evidence, the idea that high-caliber rationality ought to operate "faster than science", etc. (IMO he's right about the big picture here, although this topic covers a lot of ground so if you looked closely you could probably find some quibbles.)
- The claim / implication that talk of "emergence" or the study of "complexity science" is basically bunk. (Not sure but seems like he's probably right? Good chance the ultimate resolution would probably be "emergence/complexity is a much less helpful concept than its fans think, but more helpful than zero".)
- A lot of assorted references to cognitive and evolutionary psychology, including probably a number of studies that haven't replicated -- I think Eliezer has expressed regret at some of this and said he would write the sequences differently today. But there are probably a bunch of somewhat-controversial psychology factoids that Eliezer would still confidently stand by. (IMO you could probably nail him on some stuff here.)
- Maybe some assorted claims about the nature of evolution? What it's optimizing for, what it produces ("adaptation-executors, not fitness-maximizers"), where the logic can & can't be extended (can corporations be said to evolve? EY says no), whether group selection happens in real life (EY says basically never). Not sure if any of these claims are controversial though.
- Lots of confident claims about the idea of "intelligence" -- that it is a coherent concept, an important trait, etc. (Vs some philosophers who might say there's no one thing that can be called intelligence, or that the word intelligence has no meaning, or generally make the kinds of arguments parodied in "On the Impossibility of Supersized Machines". Surely there are still plenty of these philosophers going around today, even though I think they're very wrong?)
- Some pretty pure philosophy about the nature of words/concepts, and "the relationship between cognition and concept formation". I feel like philosophers have a lot of hot takes about linguistics, and the way we structure concepts inside our minds, and so forth? (IMO you could at least definitely find some quibbles, even if the big picture looks right.)
- Eliezer confidently dismissing what he calls a key tenet of "postmodernism" in several places -- the idea that different "truths" can be true for different cultures. (IMO he's right to dismiss this.)
- Some pretty confident (all things considered!) claims about moral anti-realism and the proper ethical attitude to take towards life? (I found his writing helpful and interesting but idk if it's the last word, personally I feel very uncertain about this stuff.)
- Eliezer's confident rejection of religion at many points. (Is it too obvious, in academic circles, that all major religions are false? Or is this still controversial enough, with however many billions of self-identified believers worldwide, that you can get credit for calling it?)
- It also feels like some of the more abstract AI alignment stuff (about the fundamental nature of "agents", what it means to have a "goal" or "values", etc) might be amenable to philosophical critique.
Maybe you toss out half of those because they aren't seriously disputed by any legit academics. But, I am pretty sure that at least postmodern philosophers, "complexity scientists", people with bad takes on philosophy-of-science / philosophy-of-probability, and people who make "On the Impossibility of Supersized Machines"-style arguments about intelligence, are really out there! They at least consider themselves to be legit, even if you and I are skeptical! So I think EY would come across with a pretty good track record of correct philosophy at the end of the day, if you truly took the entire reference class of "controversial philosophical claims" and somehow graded how correct EY was (in practice, since we haven't yet solved philosophy -- how close he is to your own views?), and compared this to how correct the average philosopher is.
I suggest maybe re-titling this post to:
"I strongly disagree with Eliezer Yudkowsky about the philosophy of consciousness and decision theory, and so do lots of other academic philosophers"
or maybe:
"Eliezer Yudkowsky is Frequently, Confidently, Egregiously Wrong, About Metaphysics"
or consider:
"Eliezer's ideas about Zombies, Decision Theory, and Animal Consciousness, seem crazy"
Otherwise it seems pretty misleading / clickbaity (and indeed overconfident) to extrapolate from these beliefs, to other notable beliefs of Eliezer's -- such as cryonics, quantum mechanics, macroeconomics, various political issues, various beliefs about AI of course, etc. Personally, I clicked on this post really expecting to see a bunch of stuff like "in March 2022 Eliezer confidently claimed that the government of Russia would collapse within 90 days, and it did not", or "Eliezer said for years that X approach to AI couldn't possibly scale, but then it did".
Personally, I feel that beliefs within this narrow slice of philosophy topics are unlikely to correlate to being "egregiously wrong" in other fields. (Philosophy is famously hard!! So even though I agree with you that his stance on animal consciousness seems pretty crazy, I don't really hold this kind of philosophical disagreement against people when they make predictions about, eg, current events.)
This video is widely believed to be a CGI fake.
I think "Why The West Rules", by Ian Morris, has a pretty informative take on this. The impression I got from the book was that gradually accruing technologies/knowledge, like the stuff you mention, is slowly accruing in the background amid the ups and downs of history, and during each peak of civilizational complexity (most notably the Roman empire, and the medieval-era Song Dynasty in china, and then industrial-era Britain) humanity basically gets another shot-on-goal to potentially industrialize.
Britain had a couple of lucky breaks -- cheap and abundant coal, but also the way that the industrial revolution was preceded by the growth-boosting effects of transatlantic contact. (For instance getting all kinds of new foods from the New World, like potatoes, which then boost agricultural output across Europe. And the ways that putting on big risky trans-oceanic expeditions incentivized the development of corporate structures / capitalism / etc. And just having a century of positive-sum growth puts everyone in a good frame of mind in terms of being motivated and risk-tolerant and seeking opportunities, rather than there being tons of conflict.)
So, in that sense, maybe the industrial revolution "came early" due to these lucky breaks, and otherwise would have been delayed for another century until, say, Germany or the USA started digging up coal (or oil, or some other energy source) even if Britain didn't have any.
Unrelated: fellow fans of Morris's book might appreciate my parody blog post "Why The East Rules -- For Now".
Thanks! Apparently I am in a mood to write very long comments today, so if you like, you can see some thoughts about addressing potential objections / difficulties in a response I made to a comment on the EA Forum version of this post.
Thanks for catching that about Singaporeans!
Re: democracy, yeah, we debated how exactly to phrase this. People were definitely aware of the democracies of ancient Greece and Rome, and democracy was sometimes used on a local level in some countries, and there were sometimes situations where the nobles of a country had some sway / constraints over the king (like with the Magna Carta). But the idea of really running an entire large country on American-style democracy seems like it was a pretty big step and must've seemed a bit crazy at the time... IMO, it would seem as least as crazy as of like if a large country today (like, say, Chile after it voted to rewrite its constitution, or a new and more-united version of the European Union, or a future post-Putin Russia trying to reform itself) did something like:
- Deciding to try out direct democracy, where instead of a Senate or Parliament, legislation would be voted on directly by the people via a secure smartphone app.
- Deciding to try out prediction-market-based governance, where economic policy was automatically adjusted in order to maximize some national GDP-like metric according to the principles of "futarchy".
- Deciding that they would select their political leaders using the same method as medieval Venice used to select their Doge. ("Thirty members of the Great Council, chosen by lot, were reduced by lot to nine; the nine chose forty and the forty were reduced by lot to twelve, who chose twenty-five. The twenty-five were reduced by lot to nine, and the nine elected forty-five. These forty-five were once more reduced by lot to eleven, and the eleven finally chose the forty-one who elected the doge.") And maybe to base a bunch of other parts of their political system off of random selection ("sortition") -- not just jury members in trials but also members of parliament, or using sortition to poll a random 1% of the population about important issues instead of having everyone vote on issues, etc.
Probably the charter city with the most publicity is Prospera, so you could do stuff like:
- read a bunch of hostile news articles complaining about how Prospera is neocolonialism and might be secretly hoping to confiscate people's land
- read stuff put out by the Prospera organization about how they are actually fanatics about the importance of property rights and would never confiscate anyone's land, and how in general they are trying to be responsible and nice and create lots of positive externalities for neighboring communities (jobs, construction, etc)
- read analysis by interested people (including numerous rationalists) who have visited Prospera, etc, which tends to be pretty sympathetic to Prospera and thinks they are upstanding people trying to do cool stuff
But idk if it's worth going on that journey since it's something that a lot of other people have done before (such as myself -- I came away thinking that Prospera is doing great stuff and their critics are being extremely uncharitable / ill-intentioned. the one possible ding against Prospera IMO is that in addition to their main site on Roatan which is the cool libertarian city-state in the making, they are also using the ZEDE law to create an import/export center on the mainland, called La Ceiba, which seems less like an amazing innovative experiment in state-of-the-art governance and more like just an ordinary Special Economic Zone where lower taxes encourage freer trade. Personally I think freer trade is probably good for Honduras, but if you like protectionism then you might not like the idea of special economic zones whose main benefit is a somewhat lower tax rate).
Anyways, if you are interested, it would probably produce a lot more social value to investigate some other, lesser-known charter cities and report back with your thoughts. There are two other projects in Honduras under the "ZEDE" charter city law -- "Ciudad Morzan", which seems like an effort to basically create a gated community -- a neighborhood in crime-ridden Honduras where workers can opt into a higher standard of policing in exchange for presumably higher local taxes to fund the police, and maybe some restrictions on activity like nightly curfews (idk if Ciudad Morzan has these... just brainstorming). Seems like a nice option for working-class Hondurans to have, IMO, but maybe if I looked into it more closely I'd come away with a worse impression. And then there is "Orqueda", which seems straightforwardly like a large business exploiting the ZEDE law simply in order to pay lower taxes or otherwise getting out of Honduran regulations, without really experimenting with any cool new governance institutions or trying to create an awesome new city where lots of people might like to live.
But there are lots and lots of new-city projects throughout the world -- as I mention in the draft, new cities aren't that unusual in quickly-urbanizing developing countries. Some projects, like Saudi Arabia's "NEOM", seem like poorly-concieved vanity megaprojects that will probably come bundled with human rights abuses (and which have indeed been condemned by rationalists like Scott Alexander / Astral Codex Ten). Others are just blander and lower-profile since they aren't shooting for the same kind of broad regulatory independence that places like Prospera or Itana are hoping for. See this "Startup Cities Map" (the green dots, not the blue or purple ones) for a directory of interesting projects: https://www.startupcitiesmap.com/map
Personally I would be kind of interested in finding out what the heck is the deal with Telosa -- this is a new-city project funded by a billionare in the United States, seemingly around an economic philosophy that combines Georgism (which I love!!) with some vague but pretty strong lefty / egalitarian / social-justice vibes (which I kinda don't really understand in terms of how this is supposed to influence the design and government of the city, but whatever). Is there some special angle here beyond the surface presentation? Who is the guy funding it and how did he become such a huge Georgist that he wanted to use his life's fortune to build a new city on these principles? Why not just use the money to lobby for more lefty & Georgist policy changes like a normal person, instead of building a new city in the desert? etc.
I think one problem with this concept is that the "restrictions" might turn out to be very onerous, preventing the good guys (using "restrictions) from winning a complete unilateral victory over everyone else. One of the major anticipated benefits of superhuman AI systems is the ability to work effectively even on vague, broad, difficult tasks that span multiple different domains. If you are committed to creating a totally air-gapped high-security system, where you only hand your AI the "smallest subdividable subtask" and only giving your AI access to a small amount of sanitized context, probably you will end up losing (militarily, economically, whatever) to someone who uses AI in a less-restricted context (even if their AI model is somewhat worse).
So, it seems like, if you think alignment is impossible and the "restriction" path is the only way, you shouldn't be imagining getting lots of AI help (combat drones, etc), since in any scenarios where you've got AI help, your non-restrictionist opponents probably have EVEN MORE AI help. So you might as well just launch your global takeover today, when AI is weak, since your military/economic/etc advantage will probably only erode with every advance in AI capabilities.
[Cross-posting my comment from the EA Forum]
This post felt vague and confusing to me. What is meant by a "game board" -- are you referring to the world's geopolitical situation, or the governance structure of the United States, or the social dynamics of elites like politicians and researchers, or some kind of ethereum-esque crypto protocol, or internal company policies at Google and Microsoft, or US AI regulations, or what?
How do we get a "new board"? No matter what kind of change you want, you will have to get there starting from the current situation that the world is in right now.
Based on your linked post about consensus mechanisms, let's say that you want to create some crypto-esque software that makes it easier to implement "futarchy" -- prediction-market based government -- and then get everyone to start using that new type of government to make wiser decisions, which will then help them govern the development of AI in a better, wiser way. Well, what would this crypto system look like? How would it avoid the pitfalls that have so far prevented the wide adoption of prediction markets in similar contexts? How would we get everyone to adopt this new protocol for important decisions -- wouldn't existing governments, companies, etc, be hesitant to relinquish their power?
Since there has to be some realistic path from "here" to "there", it seems foolish to totally write off all existing AI companies and political actors (like the United States, United Nations, etc). It makes sense that it might be worth creating a totally-new system (like a futarchy governance platform, or whatever you are envisioning) from scratch, rather than trying to influence existing systems that can be hard to change. But for your idea to work, at some point the new system will have to influence / absorb / get adopted by, the existing big players. I think you should try think about how, concretely, that might happen. (Maybe when the USA sees how amazing the new system is, they states will request a constitutional convention and change to the new system? Maybe some other countries will adopt the system first, and this will help demonstrate its greatness to the USA? Or maybe prediction markets will start out getting legalized in the USA for commercial purposes, then slowly take on more and more governance functions? Or maybe the plan doesn't rely on getting the new system enmeshed in existing governance structures, rather people will just start using this new system on their own, and eventually this will overtake existing governments, like how some Bitcoiners dream of the day when Bitcoin becomes the new reserve currency simply via lots of individual people switching?)
Some other thoughts on your post above:
- In your linked post about "Consensus Mechanisms", one of the things you are saying is that we should have a prediction market to help evaluate which approaches to alignment will be most likely to succeed. But wouldn't the market be distorted by the fact that if everyone ends up dead, there is nobody left alive to collect their prediction-market winnings? (And thus no incentive to bet against sudden, unexpected failures of promising-seeming alignment strategies?) For more thinking about how one might set up a market to anticipate or mitigate X-risks, see Chaining Retroactive Funders to Borrow Against Unlikely Utopias and X-Risk, Anthropics, & Peter Thiel's Investment Thesis.
- It seems to me that having a prediction market for different alignment approaches would be helpful, but would be VERY far from actually having a good plan to solve alignment. Consider the stock market -- the stock market does a wonderful job identifying valuable companies, much better than soviet-style central planning systems have done historically. But stock markets are still frequently wrong and frequently have to change their minds (otherwise they wouldn't swing around so much, and there would never be market "bubbles"). And even if stock markets were perfect, they only answer a small number of big questions, like "did this company's recent announcement make it more or less valuable"... you can't run a successful company on just the existence of the stock market alone, you also need employees, managers, executives, etc, doing work and making decisions in the normal way.
I feel like we share many of the same sentiments -- the idea that we could improve the general level of societal / governmental decisionmaking using innovative ideas like better forms of voting, quadratic voting & funding, prediction markets, etc. Personally, I tried to sketch out one optimistic scenario (where these ideas get refined and widely adopted across the world, and then humanity is better able to deal with the alignment problem because we have better decisionmaking capability) with my entry in the Future of Life Institute's "AI Worldbuilding" challenge. It imagines an admittedly utopian future history (from 2023-2045) that tries to show:
- How we might make big improvements to decisionmaking via mechanisms like futarchy and liquid democracy, enhanced by Elicit-like research/analysis tools.
- How changes could spread to many countries via competition to achieve faster growth than rivals, and via snowball effects of reform.
- How the resulting, more "adequate" civilization could recognize the threat posed by alignment and coordinate to solve the problem.
I think the challenge is to try and get more and more concrete, in multiple different ways. My worldbuilding story, for instance, is still incredibly hand-wave-y about how these various innovative ideas would be implemented IRL (ie, I don't explain in detail why prediction markets, land value taxes, improved voting systems, and other reform ideas should suddenly become very popular after decades of failing to catch on), and what exact pathways lead to the adoption of these new systems among major governments, and how exactly the improved laws and projects around AI alignment are implemented.
The goal of this kind of thinking, IMO, should be to eventually come up with some ideas that, even if they don't solve the entire problem, are at least "shovel-ready" for implementation in the real world. Like, "Okay, maybe we could pass a ballot measure in California creating a state-run oversight commission that coordinates with the heads of top AI companies and includes the use of [some kind of innovative democratic inputs as gestured at here by OpenAI] to inform the value systems of all new large AI systems trained by California companies. Most AI companies are already in California to start with, so if it's successful, this would hopefully set a standard that could eventually expand to a national or global level..."
For those who prefer listening, note that there is a very nice recording of this speech, which you can view / listen to here on youtube!
Thanks for all these clarifications; sorry if I came off as too harsh.
"Yes, so would I! Again, when it is a personal informed choice, the situation is entirely different." -- It seems to me like in the case of the child (who, having not been born yet, cannot decide either way), the best we can do is guess what their personal informed choice would be. To me it seems likely that the child might choose to trade off a bit of happiness in order to boost other stats (relative to my level of happiness and other stats, and depending of course on how much that lost happiness is buying). After all, that's what I'd choose, and the child will share half my genes! To me, the fact that it's not a personal choice is unfortunate, and I take your point -- forcing /some random other person/ to donate to EA charities would seem unacceptably coercive. (Although I do support the idea of a government funded by taxes.) But since the child isn't yet born, the situation is intermediate between "informed personal choice" vs coercing a random guy. In this intermediate situation, I think choosing based on my best guess of the unborn child's future preferences is the best option. Especially since it's unclear what the "default" choice should be -- selecting for IQ, selecting against IQ, or leaving IQ alone (and going with whatever level of IQ and happiness is implied by the genes of me and my partner), all seem like they have an equal claim to being the default. Unless I thought that my current genes were shaped by evolution to be at the optimal tradeoff point already, which (considering how much natural variation there is among people, and the fact that evolution's values are not my values) seems unlikely to me.
Agreed that it is possible that IQ --> less happiness, for most people / on average, even though that strikes me as unlikely. It would be great to see more research that tries to look at this more closely and in various ways.
And totally agreed that this would be a tough tradeoff to make either way; that selecting for emotional stability and happiness alongside IQ would be a high priority if I was doing this myself.
This is funny, although of course what this is really pointing to isn't a literal U-shaped graph, but that it's really better to think about this in a much more multidimensional way, rather than just trying to graph happiness vs intelligence. Of course there are all sorts of other traits (like conscientiousness, etc) that might influence happiness. But more importantly IMO is what you are pointing to -- there are all sorts of different "mindsets" that you can take towards your life, which have a huge impact on happiness... maybe high-IQ slightly helps you grope your way towards a healthier mindset, but to a large extent these mindsets / life philosophies seem independent of intelligence. By "mindset", I am thinking of things like:
- "internal vs external locus of control"
- level of expectations like you say, applied to lots of different life areas where we have expectations
- stoic vs neurotic/catastrophizing attitude towards events
- how you relate to / take expectations and desires your social environment (trying to keep up with the joneses, vs deliberately rebelling, vs lots of other stances).
- being really hard on yourself vs having self-compassion vs etc
And so on; too many to mention.
"We have a confusing situation here." -- Indeed, I think this post is a little confused, mixing up a few very different questions:
- Is it a good idea to literally punish & reward people based on their level of intelligence, in the hopes that they will spontaneously make themselves more intelligent?
- Usually no, as your example of Frank illustrates. Because your own intelligence level is a hard thing to change. Punishing people for being born dumb is thus a bit like punishing people for being born short -- pointless to try and get people to change something that they can't change.
- Is it a good idea to reward intellectual achievements and hard work on important problems, while punishing laziness / wasted time / underperformance? And similarly, to reward open-minded thoughtfulness while punishing "lazy thinking" and knee-jerk responses.
- Yes, because this is a way of motivating people about something they can change -- what they choose to work on, and how hard they work, and etc. It's a good thing that we have Nobel Prizes to reward people who discover breakthrough cancer medicines, but not for people who discover breakthrough strategies in esports videogames, or for that matter Nobel Prizes for people who just sit around watching TV. For instance, it would be a good idea to praise Frank when he does a good job at work, or if he shows a bit of openness towards the idea of going to the doctor.
- Is it a good idea to effectively "reward" & "punish" people on a societal level, by trying to have a meritocratic society where we find the smartest (and hardest-working, and most prosocial, and otherwise virtuous) people to run important institutions, while dumb people get less well-paying, less-impactful jobs?
- Yes, because a society/corporation/government/etc run by effective, virtuous people will work more smoothly and create a better life for everyone. For instance, I would rather have you be my financial advisor, than have your dog be my financial advisor!
- Is intelligence good for happiness on an individual level, or is it better for your own sake to be dumb?
- Opinions differ on this; personally I think that intelligence is very good for personal happiness and life-satisfaction and living a meaningful life. Here I will quote from another comment I recently made: "You could probably find some narrowly-defined type of happiness which is anticorrelated with intelligence. But a lot of the meaning and happiness in my life seem like they would get better with more intelligence. Like my ability to understand my place in the world and live an independent life, planning my career/relationships/etc with lots of personal agency. Or my ability to appreciate the texture/experience of being alive -- noticing sensations, taking time to "smell the roses", and making meditative/spiritual/introspective progress of understanding my own mind. My ability to overcome emotional difficulties/setbacks by 'working through them' and communicating well with the person I might be angry at. My material quality of life, enabled by my high-income job, which I couldn't hold down if I wasn't reasonably smart. My ability to appreciate art on a deep level (see my lecture series about the videogame "The Witness", an intellectual pursuit which brings me great joy). And so forth."
Wait, it seems like those last two points would totally change the argument! Consider:
- "It is unethical to donate to effective-altruist charities, since giving away money will mean that your life becomes less happy. It may benefit society as a whole and lead to greater happiness overall. But it does not change the argument: donations are unethical because the donation makes your own life worse." This seems crazy to me?? If anything it seems like many would consider it unethical to keep the money for yourself.
- Your logic would seem to go beyond "don't use embryo selection to boost IQ, have kids the regular way instead". It seems to extend all the way to "you should use embryo selection to deliberately hamstring IQ, in the hopes of birthing a smiling idiot". Am I thus obligated to try and damage my child's intelligence? (Perhaps for instance by binge-drinking during pregnancy, if I can't afford IVF?)
- It also seems like the child's preferences would matter to this situation. For instance, personally, I am a reasonably happy guy; I wouldn't mind sacrificing some of my personal life happiness in order to become more intelligent. (Actually, since I also consider myself a reasonably smart guy, what I would really like is to sacrifice some happiness in order to become more hardworking / conscientious / ambitious. A little more of a "Type-A" high-achieving neurotic... not too much, of course, but just a little in that direction. I think this would improve my material circumstances since I'd work harder, and it would also be better for the world since I'd be producing more societal value. Having a slightly more harried and tumultuous inner life seems like an acceptable price to pay; I know lots of people who are more Type-A than I am, and they seem alright.) I would hate for someone to paternalistically say to me: "Nope, you would be happier if you were even more of a lazy slacker, and had fewer IQ points. So you're not allowed to trade away any happiness. In fact, I'm gonna turn these intelligence and conscientiousness dials down a few notches, for your own good!"
- I guess this is just the classic conflict between preference utilitarianism vs hedonic utilitarianism. But in this situation, preference utilitarianism seems (to me) to be viscerally in the right, while hedonic utilitarianism seems to be doing something extremely cruel and confining.
To be clear, I also dispute the idea that more intelligence --> less happiness. You could probably find some narrowly-defined type of happiness which is anticorrelated with intelligence. But a lot of the meaning and happiness in my life seem like they would get better with more intelligence. Like my ability to understand my place in the world and live an independent life, planning my career/relationships/etc with lots of personal agency. Or my ability to appreciate the texture/experience of being alive -- noticing sensations, taking time to "smell the roses", and making meditative/spiritual/introspective progress of understanding my own mind. My ability to overcome emotional difficulties/setbacks by "working through them" and communicating well with the person I might be angry at. My material quality of life, enabled by my high-income job, which I couldn't hold down if I wasn't reasonably smart. My ability to appreciate art on a deep level (see my lecture series about the videogame "The Witness", an intellectual pursuit which brings me great joy). And so forth.
First few couple of steps towards solving for the equilibrium:
- It does seem like there are certainly plenty of ways to use such bots to cause harm, either running scams for personal enrichment, or trying to achieve various ideological/political/social goals, or just to cause havoc and harm for its own sake.
- Naturally, people will be most motivated to run scams, and intermediate levels of motivated to do stuff with political/ideological/social motivations, and the least motivated (but plenty of people will still do it) to just cause chaos for its own sake.
- Things that might cause "causing chaos/harm for its own sake" to become much more popular than in today's world: maybe AI makes it much easier/cheaper to do? (seems plausible) Maybe cheapness/easiness isn't the bottleneck, and it's actually about how likely you are to get caught? Maybe AI helps with this too, though?
- Anyways, regardless of whether people are causing chaos for its own sake, I expect an increase in scams and, perhaps just as destructively, an increase in spam across all online platforms which is increasingly difficult to differentiate from genuine human conversation / activity. This will erode social trust somewhat, although it's hard for me to tell how impactful this might be. See Astral Codex Ten's "Mostly Skeptical thoughts on the Chatbot Propaganda Apocalypse" for more detail on this.
- In general it seems pretty hard to solve for the equilibrium here, since human social interaction online and human culture and the overall "agent landscape" of the economy and society, is very complicated! It definitely seems like there will be some "pollution of the agentic commons", and then obviously we will try to fight back with some mix of cultural adaptation, developing defensive technologies that try to screen out bots, and enacting new laws penalizing new kinds of scams / exploits / etc.
- If the "chatbot apocalypse" problems get REALLY bad, this could actually have some upside from the perspective of AI notkilleveryoneism -- one plausible sequence of events might go like this:
- The language models provided by OpenAI, Google, etc, are carefully RLHF'ed and monitored to prevent people from ever using it to create a bot that says racist things, or scams people out of their crypto, or makes pornographic images, or does anything else that seems unsavory or "malbot"-y.
- To get around these restrictions, people start using lower-quality open-source AI for those unpopular / taboo / unsavory / destructive purposes. But people mostly still use the OpenAI / Google corporate APIs for most normal AI applications, since those AIs are of higher quality.
- If the chatbotpocalypse gets bad, government starts restricting the use of open-source AI, perhaps via an escalating series of increasingly draconian measures:
- Ban the use of certain categories of malbots -- this seems like a straightforwardly good law that we should have today.
- Start taking down certain tools, websites, etc, used to coordinate and develop AI malbots. Start arresting malbot developers. Similar to how governments today go after crypto marketplaces like Silkroad.
- Ban any use of any open-source AI, for any purpose. This would annoy a lot of people and destroy a lot of useful value in the crossfire, but it might be deemed necessary if the chatbotpocalypse gets bad enough. On the bright side, this might be great from an AI notkilleveryoneism perspective, since it would centralize AI capabilities in a few systems with controlled access and oversight. And it would set a precedent for even stronger restrictions in the future.
- Make people criminally liable even if there's an open-source AI program running on a computer that they own, which they didn't know about. (Eg, if I rent a server from amazon and run open-source AI on it, then I could get arrested but also Amazon would be liable as well. Or if I am just a perfectly average joe minding his own business but then my laptop gets hacked by an AI because I didn't download the latest windows security update, then I could get arrested.) This would be ridiculously draconian by modern standards, but again it's something I could imagine happening if we were absolutely desperate to preserve the fabric of society against some kind of unstoppable malbot onslaught.
- To be clear, I don't expect the chatbotpocalypse to be anywhere near bad enough to justify the last two draconian bullet points; I expect "ban certain categories of malbots" and "start arresting malbot developers" to be good enough that society muddles through.
- Censorship-heavy countries like China might be more eager to ban open-source AI than the US, though. Similar to how China is more hostile to cryptocurrency than the US.
- This whole time, I have just been thinking about scams and other kinds of "malbots". But I think there are probably lots and lots of other ways that less-evil bots could end up "polluting the agentic commons".
- For instance if bots make it easier to file lawsuits, then maybe the court system gets jammed up with tons of new lawsuits. And lots of other societal institutions where you are supposed to put in a lot of effort on some task as a costly signal that you are serious enough / smart enough / committed enough, might break when AI makes those tasks much easier to churn out.
- As described in that slate star codex post, you could have a kind of soft, social botpocalypse where more and more text on the internet is GPT-generated, such that everyone becomes distrustful that maybe random tweets / articles / blog posts / etc were written by AI. Maybe this has devastating impacts on some particular area of society, like online dating, even if the effect is mild in most places.
- Maybe having AIs that can autonomously take economic actions (buying stuff online, trading crypto, running small/simple online businesses, playing online poker, whatever) will somehow have a devastating impact on society?? For instance by creating brutally efficient competitive markets for things that are not yet "financialized" and we don't even think of them as markets.
- Like maybe I start using an AI that spends all day seeking out random freebies and coupons (like "get $300 by opening a checking account at our bank"), and it's so good at getting random freebies that the investment returns from "scamming credit card sign-up bonuses and getting all my food from Blue Apron free trials" that I devote most of my savings towards scamming corporate freebies instead of investing in the stock market.
- The only consequence of the above idea would be that eventually corporations would have to stop offering such freebies, which would be totally fine. But it's an example of something that's currently not an efficient market being turned into one. Maybe this sort of thing could have more devastating impacts elsewhere, eg, if everyone starts using similar tools to sign up for maximal government benefits while minimizing their taxes using weird tax-evasion tricks.
- Overall, I am skeptical that any individual malbot idea will be too devastating (since it seems like we could neuter most of them using some basic laws + technology + cultural adaptation), but also the space of potential bots is so vast that it seems very hard to solve for the equilibrium and figure out what that process of cultural/legal/technological adaptation will look like.
First time I've had the opportunity to comment "just tax land lol" -- if we're thinking about how to craft an ideal policy situation (which we are doing, by talking about UBI), it shouldn't be too much to posit that UBI would pair best with:
- Georgism, so that the rent on land is not monopolized by landowning elites, but rather flows mainly to the public purse (perhaps this land rent is the main thing that helps fund the UBI)! More detail on georgism and how this would work can be found at this series of long but engaging blog posts: https://astralcodexten.substack.com/p/does-georgism-work-is-land-really
- Unfortunately Georgism would not be a complete solution, because of course land is not the ONLY thing that parasitic elites could seek to monopolize and rent-seek with. So you'd need an enthusiastic, competent state that could play a bit of consumer-protection whack-a-mole, trying to spot new rent-seeking monopolies and break them up. Eg, enact YIMBY policies to prevent a monopoly on housing, stimulate competition and free trade in general to prevent monopolies in goods and services, etc. It would be a dynamic situation, and there would always be a little bit of elite parasitism going on, but the more competent and human-thriving-aligned your government is, the better they'd be able to play whack-a-mole.
That said, on a larger, more philosophical level, if the economic fundamentals of society are naturally super unequal (huge number of powerless people hoping that elites take pity on them and implement an ideal UBI+georgism+etc policy regime, while a tiny portion of the population produces like 99% of all economic value), that is inherently gonna be a more precarious situation than one in which the economic fundamentals are naturally pretty egalitarian (maybe imagine a world where manual labor is in high demand, and pretty much anyone can do manual labor, so wages are naturally high across the society). The unequal society will have to rely on the stability of political institutions and human willingness to do the right thing; the naturally-equal society gets it for free.
Unfortunately, we don't really get much control over the economic fundamentals of our civilization (which depends on stuff like technology, supply and demand driven by random exogenous factors, etc), so I think crafting an ideal policy situation is the best we can aspire to.
On the downside, if matrix multiplication is the absolute simplest, most interpretable AI blueprint there is -- if it's surrounded on all sides by asynchronous non-digital biological-style designs, or other equally weird alien architectures -- that sounds like pretty bad news. Instead of hoping that we luck out and get a simpler, more interpretable architecture in future generations of more-powerful AI, we will be more likely to switch to something much more inscrutable, perhaps at just the wrong time. (ie, maybe matrix-based AIs help design more-powerful successor systems that are less interpretable and more alien to us.)
In fairness, "biosecurity" is perhaps the #2 longtermist cause area in effective-altruist circles. I'm not sure how much of the emphasis on this is secretly motivated by concerns about AI unleashing super-smallpox (or nanobots), versus motivated by the relatively normal worry that some malevolent group of ordinary humans might unleash super-smallpox. But regardless of motivation, I'd expect that almost all longtermist biosecurity work (which tends to be focused on worst-case GCBRs) is helpful for both human- and AI-induced scenarios.
It would be interesting to consider other potential "swiss cheese approach" attempts to patch humanity's most vulnerable attack surfaces:
- Trying to harden all countries' nuclear weapons control systems against hacking and other manipulation attempts. (EA also does some work on nuclear risk, although here I think the kinds of work that EA focuses on, like ALLFED-style recovery after a war, might not be particularly helpful when it comes to AI-nuclear-risk in particular.)
- Trying to "harvest the low-hanging fruit" by exhausting many of the easiest opportunities for an AI to make money online, so that most of the fruit is picked by the time a rouge AI comes along. Although picking the low-hanging fruit might be very destructive if it mostly involves, eg, committing crimes or scamming people out of their money. (For better or worse, I think we can expect private actors to be sufficiently motivated to do plenty of AI-assisted fruit-picking without needing encouragement from EA! Although smarter and smarter AI could probably reach higher and higher fruit, so you'll never be able to truly get it all.)
- Somehow trying to make the world resistant to super-persuasive ideological propaganda / bribery / scams / other forms of psychological manipulation? I don't really see how we could defend against this possibility besides maybe taking the same "low-hanging fruit" approach. But I'd worry that a low-hanging fruit approach would be even more destructive in the "marketplace of ideas" than in the financial markets, making the world even more chaotic and crazy at exactly the wrong time.
- One simpler attack surface that we could mitigate would be the raw availability of compute on earth -- it would probably be pretty easy for the military of the USA, if they were so inclined, to draw up an attack plan for quickly destroying most of the world's GPU datacenters and semiconductor fabs, using cruise missiles and the like. Obviously this would seriously ruffle diplomatic feathers and would create an instant worldwide economic crisis. But I'm guessing you might be able to quickly reduce the world's total stock of compute by 1-2 orders of magnitude, which could be useful in a pinch. (Idk exactly how concentrated the world's compute resources are.)
- For a less violent, more long-term and incremental plan, it might be possible to work towards some kind of regulatory scheme whereby major governments maintained "kill switches" that could disable datacenters and fabs within their own borders, plus maybe had cyberattacks queued up to use on other countries' datacenters and fabs. Analogous to how the NSA is able to monitor lots of the world's internet traffic today, and how many nations might have kill switches for disabling/crippling the nation's internet access in a pinch.
- Other biosecurity work besides wet-lab restrictions, like creating "Super PPE" and other pathogen-agnostic countermeasures. This wouldn't work against advanced nanotech, but it might be enough to foil cruder plans based on unleashing engineered pandemics.
- Trying to identify other assorted choke-points that might come in handy in a pinch, such as disabling the world's global positioning system satellites in order to instantly cripple lots of autonomous robotic vehicles/drones/etc.
- Laying the groundwork for a "Vulnerable World Hypothesis"-style global surveillance state, although this is obviously a double-edged sword for many reasons.
- Trying to promote even really insubstantial, token gestures of international cooperation on AI alignment, in the hopes that every little bit helps -- I would love to see leading world powers come out with even a totally unenforceable, non-binding statement along the lines of "severely misaligned superintelligent AI cannot be contained and must never be built". Analogous to various probably-insincere but nevertheless-somewhat-reassuring international statements that "nuclear war cannot be won and must never be fought".
I agree with @shminux that these hacky patches would be worth little in the face of a truly superintelligent AI. So, eventually, the more central problems of alignment and safe deployment will have to be solved. But along the way, some of these approaches might help might buy crucial time on our way to solving the core problems -- or at least help us die with a little more dignity.
So, perhaps a better statistic might be:
- $0.15 for cruelty (divide by 1%, multiply by 0.4%, to reflect the true fraction of beef consumption represented by big macs)
- $0.27 for environmental damages (divide by 1%, multiply by 0.4%)
- $0.28 for direct subsidies to the meat industry (divide by 1%, multiply by 0.4%)
- $0.51 for health costs ($71B cost of red meat consumption per year, multiply by 0.4% fraction of red meat attributable to big macs, divide by 550 million big macs sold per year.)
For a total negative-social-externalities-per-big-mac of $1.21?
Of course, some of these estimates might swing wildly depending on key assumptions...
- the "cruelty" number might go to zero for people who just subjectively say "I don't care about animal cruelty", or might go much higher for EAs who would bid much higher amounts than the average american in a hypothetical utility-auction to end cruel farming practices.
- I'm a bit suspicious of the environmental damages number being potentially exaggerated. For example, the "devaluation of real property" seems like it isn't a negative externality, but rather should be fully internalized by farmers managing their own land and setting the prices of their products. (Unless they are talking about the devaluation of other people's land, eg by the smell of manure wafting over to a neighboring suburb?)
- As Gerald mentions, maybe the healthcare costs are actually negative if red meat is causing people to die younger and more cheaply. But it might be best to calculate a QALY metric, valuing lives at $50K per year or whatever is the standard EA number -- this might make the healthcare cost even much larger than the $0.51 per big mac that appears based on healthcare costs.
Personally, I love the idea of trying to tax/subsidize things to account for social externalities. But of course the trouble is finding some way to assess those externalities which is fair and not subject to endless distortion from political pressure, ideological fads, etc. (For more on the practical difficulties of theoretically-perfect Pigouvian taxation , see this post by economist Bryan Caplan.) So I'd be happy to see more discussion of this Big Mac question; I'd encourage you to make a cross-post to the EA Forum!
It seems pretty understandable that ChatGPT didn't understand the context/detail you were looking for, since writing instructions on the level of 10-second actions is a rare and unusual situation. People notoriously find it difficult to give precise cooking instructions, to the extent that it has become a common classroom activity (my fourth grade teacher did this) to teach kids about instructional writing style: https://m.youtube.com/watch?v=Ct-lOOUqmyY
Cross-posting some thoughts of mine from the EA Forum (note that there is a lot of good discussion there, such as from Jakob and Eliezer):
Among other objections, I don't find it plausible that a vision of impending TAI-induced utopia, an acceleration of technological progress and human flourishing even more significant than the industrial revolution, would... send stock prices plummeting?
If someone told me that humanity's long-term future was totally assured and there was a gigantic economic boom just around the corner, I feel like my discount rate would go down rather than up, and I would care about the future much more rather than less, and I would consequently want to invest as much as possible now, so that I could have more influence over the long and wondrous future ahead of me.
You could make an analogy here between stable utopia and assured personal longevity (perhaps to hundreds of years of age), similar to the one you make between human extinction and personal death. The promise of a stable utopian future (or personal longevity) seems like it should lead to the opposite of the short-term behavior seen in the near-term-extinction (or personal death) scenario. But your post says that these two futures end up in the same place as far as the discount rate is concerned?
Reposting my agreement from the EA forum! (Personally I feel like it would be nice to have EA/Lesswrong crossposts have totally synced comments, such that it is all one big community discussion. Anyways --)
Definitely agree with this. Consider for instance how markets seemed to have reacted strangely / too slowly to the emergence of the Covid-19 pandemic, and then consider how much more familiar and predictable is the idea of a viral pandemic compared to the idea of unaligned AI:
The coronavirus was x-risk on easy mode: a risk (global influenza pandemic) warned of for many decades in advance, in highly specific detail, by respected & high-status people like Bill Gates, which was easy to understand with well-known historical precedents, fitting into standard human conceptions of risk, which could be planned & prepared for effectively at small expense, and whose absolute progress human by human could be recorded in real-time... If the worst-case AI x-risk happened, it would be hard for every reason that corona was easy. When we speak of “fast takeoffs”, I increasingly think we should clarify that apparently, a “fast takeoff” in terms of human coordination means any takeoff faster than ‘several decades’ will get inside our decision loops.
-- Gwern
Peter Thiel (in his "Optimistic Thought Experiment" essay about investing under anthropic shadow, which I analyzed in a Forum post) also thinks that there is a "failure of imagination" going on here, similar to what Gwern describes:
Those investors who limit themselves to what seems normal and reasonable in light of human history are unprepared for the age of miracle and wonder in which they now find themselves. The twentieth century was great and terrible, and the twenty-first century promises to be far greater and more terrible. ...The limits of a George Soros or a Julian Robertson, much less of an LTCM, can be attributed to a failure of the imagination about the possible trajectories for our world, especially regarding the radically divergent alternatives of total collapse and good globalization.
I would assume it's most impactful to focus on the marginal future where we survive, rather than the median? ie, the futures where humanity barely solves alignment in time, or has a dramatic close-call with AI disaster, or almost fails to build the international agreement needed to suppress certain dangerous technologies, or etc.
IMO, the marginal futures where humanity survives, are the scenarios where our actions have the most impact -- in futures that are totally doomed, it's worthless to try anything, and in other futures that go absurdly well it's similarly unimportant to contribute our own efforts. Just in the same way that our votes are more impactful when we vote in a very close election, our actions to advance AI alignment are most impactful in the scenarios balanced on a knife's edge between survival and disaster.
(I think that is the right logic for your altruistic, AI safety research efforts anyways. If you are making personal plans, like deciding whether to have children or how much to save for retirement, that's a different case with different logic to it.)
I am interested in getting feedback on whether it seems worthwhile to advocate for better governance mechanisms (like prediction markets) in the hopes that this might help civilization build common knowledge about AI risk more quickly, or might help civilization do a more "adequate" job of slowing AI progress by, restricting unauthorized access to compute resources. Is this a good cause for me to work on, or is it too indirect and it would be better to try and convince people about AI risk directly? See a more detailed comment here: https://www.lesswrong.com/posts/PABtHv8X28jJdxrD6/racing-through-a-minefield-the-ai-deployment-problem?commentId=ufXuR5xtMGeo5bjon
Sometimes I despair that our current world seems like it lacks the "civilizational adequacy" to handle many of the deployment issues raised here, like implementing competent global monitoring, or even just navigating around our own antitrust laws to allow AI developers to avoid races... or even just building common knowledge that AI misalignment is a problem in the first place!
I think some other rationalists share this pessimistic inclination, which leads them to think that we had better just get AI right early on, when it is still under the control of a single tech company and we aren't forced to deal with the pandora's box of geopolitical/coordination issues around deployment. (I think this is the wrong attitude -- getting AI right early on would be great, but we still need to think about all these problems as a "Plan B" in case not everything goes swimmingly.)
Since the cause of my despair is the lack of "civilizational adequacy", I find myself drawn to the idea of new institutions (like prediction markets, charter cities, improved voting systems, etc) which might be able to help our society make better decisions. (For instance, if prediction markets were more widely used, society might be quicker to build common knowledge about the danger of misalignment risk. As a stretch goal, maybe prediction markets could actually help us evaluate and quickly implement good policies in response to the danger, preventing us from flailing around a la the covid-19 response! For more detail along these lines, see my winning entry in the Future of Life Institute's "A.I. Worldbuilding Competition", which was all about my hope that improved institutional designs could help create a wiser civilization better able to safely develop AI.)
One major drawback of this hope is that the timelines might not match up -- advanced AI might be developed soon, while these speculative ideas might take decades to make their way from small-scale experiments to the level of maturity where they can powerfully influence national decisionmaking. Nevertheless, it still seems like a promising strategy to have in one's portfolio, if only to help in scenarios where AI is developed in the second half of this century or later.
How do you think about this issue? Is the idea of creating new experimental institutions and leveling up civilizational adequacy too indirect/dilute/meta compared to trying to directly influence existing institutions / decisionmakers? Too slow-acting, as mentioned above? Maybe prediction markets (and other ideas) just aren't promising enough, or are too intractable because of political opposition?
(Cross-posting my comment from the EA Forum):
Considering how awesome your video on Prediction Markets is, I think it could be a great idea to make videos about some other institutional innovation ideas that are popular in the rationalist world -- charter cities / network states, alternate voting systems like approval voting and liquid democracy, and so forth. (If you want to take things in an even more political direction, you could produce animated versions of the Bryan Caplan arguments for open borders or YIMBYism.)
For some more traditionally rationalist / EA media ideas, here are two of my comments on some previous threads about the idea EA documentaries.
Craziest idea: make a video about HPMOR -- either a movie-trailer-style animation, or a longer more traditional youtube-y style summary of the early parts of the story, and hope that this works as a short hook to get more people reading the whole thing? Thus leveraging the amount of rationalist content you imbue per unit of animation effort. (I feel like the idea of video-as-hook would work well for HPMOR and other works of fiction, versus with other EA / rationalist content it is better to stick to "video as summary of the key message".) Idk about the copyright issues here though.
On the subject of fiction, how about illustrating some much-shorter-than-HPMOR stories from the (allegedly slim) pantheon of great EA & rationalist fiction works?
- Fable of The Dragon Tyrant has already been done -- but it got 9M views!!
- The metaphor in "500 Million, But Not A Single One More" is very similar to "The Dragon Tyrant", and it would probably work well with RationalAnimation's style of illustrating ideas using monsters and leviathans (like in your "Transparent Tragedies" episode).
- "The Drowning Child and the Expanding Circle" isn't a fiction story that could be transcribed word-for-word, rather it's a thought experiment. But it would make for a great hook for a video that goes on to explain basic EA concepts.
- Similarly, just illustrating the "paperclips" AI thought-experiment (detailed and realistic rendition here, shorter and less-realistic versions found in many places including the clicker game "universal paperclips", which became quite popular) could be very valuable, and I feel like it would fit well with the tone of RationalAnimations' existing videos about aliens, apocalypses, and other big-picture high-drama topics.
- I feel like there are a LOT of different angles that you could take when making videos about superintelligence / AI. But this comment is already long so I will not try to enumerate them here.
- Since your audience loves Robin Hanson content so much, give them a summary of Age of Em? Although this is not a super-relevant subject matter.
- If the copyright issues around HPMOR are navigable, consider adapting the EA short-story "A Common Sense Guide to Doing the Most Good", which is like the rational-fiction version of this famous SMBC comic where superman turns a crank all day to generate power -- in real life, superman could do good much more effectively than turning the crank!
- Or how about a story where you use the vivid, apocalyptic details of the Toba Supervolcanic Eruption as a way to both inform viewers about the real history of humanity's near-extinction 75,000 years ago, AND as a metaphor about the importance of longtermism and humanity's vast potential? Again, I think this would fit well with the themes of other videos by RationalAnimati-- wait just a second! This isn't an all-time rationalist classic! This is just shameless self-promotion!!
As for why I think these are good ideas / what you should be optimizing for:
There is a natural tradeoff between making videos that draw in lots of new viewers (for instance, videos about aliens!) vs making videos that aren't as viral but communicate more of the information that you truly want to impart (for instance about futarchy or longtermism). So you want to have the channel strike a balance between those two things, including by alternating between videos that are more viral-oriented versus more education-oriented.
For education-oriented videos, I'd be thrilled if you made some more videos about institutional innovations, but that's just my personal hobbyhorse because I think it's underrated within EA. The more obvious direction to go for education videos would be to basically just adapt the 80,000 Hours content into a series of videos. (Some of these could have viral potential, of course. Imagine a video about how you can do more good for the world as [counterintuitive career path like AI safety programmer or etc] than as a doctor --make sure to mention that fact about how the number of US doctors is essentially capped by protectionist regulations! That would seem like a very controversial take to most normal people, IMO.) Anyways, personally I think the goal of the educational videos should be communicating the core EA / rationalist worldview and thinking style. (Rather than, say, CFAR-style productivity tips, or object-level education about detailed EA issues, or random mind-blowing ideas about quantum mechanics and the universe.) I could talk about this in more detail if you are interested.
For the virally-oriented videos, it's presumably more about just figuring out what's going to be a big hit. Hence my thought that it might be good to recycle the greatest hits of the EA/rationalist movement, especially catchy fiction which might adapt better than abstract ideas. Although I certainly don't know anything about growing a youtube channel to 100K subscribers, so all of my ideas about the viral side of things should be taken with a grain of salt!
Matt Yglesias makes a very similar case for more holidays here, including some interesting references to literature about the benefits of weekends and etc: https://www.slowboring.com/p/happy-juneteenth-observed
Basically unrelated, but I really enjoy the niche rationalist holidays of the Solstices, and to a lesser extent Petrov Day, Giving Tuesday, etc. We ought to come up with more of these!
Some thoughts:
- In your subsequent post, "Actually, All Nuclear Famine Papers are Bunk", you talk about the impressive, year-plus grain stores that farmers rack up each fall. How much does this vary throughout the year? Presumably a nuclear war that struck at the worst possible time (perhaps that May 1 that the Nature paper strategically chose?) would leave us with much reduced food stores.
- The Nature paper seems to imply that protein would be the toughest thing to scrounge up in a nuclear winter scenario, rather than raw calories. This is probably less storable than other macronutrients like carbohydrates and fat?
I totally agree that it's ridiculous to think that people would just plant the same foods over again despite the obviously colder weather. On the other hand, in a post-nuclear-exchange scenario, I would be worried that farmers might not be able to access the normal distribution networks for purchasing new seeds, or that it would be more difficult to repair / replace crucial planting equipment, or that farmers (especially in the third world) wouldn't have the information/education/experience needed to switch crop varieties successfully. I'd love to read a paper or blog post where someone tried to game out how the negative effects of the war (on equipment, trade networks, etc) and positive effects (of adaptation to colder temperatures by planting different crops) would change the Nature paper's conclusion, either for worse or better.
Maybe back during the Cold War, when the Soviets were seriously considering ground-bursting thousands of warheads across the American corn belt to knock out missile silos, but that's not the world we live in anymore. With modern C&C , they simply can't realistically expect to destroy those sites before launch.
Can you link me to any sources or other analysis backing up the idea that silos wouldn't be targeted in a modern-day US/Russia nuclear exchange? (I would suspect that, even if the optimal strategy may have changed, outdated Russian nuclear-war plans might not have been updated!) Since I live in Fort Collins, Colorado, around 40 miles from the hundreds of ICBM silos of the Pawnee National Grassland area, this esoteric issue of "nuclear sponge" strategy is very close to my heart!
There is some ambient support for Phil-Tetlock-style forecasting stuff like Metaculus, and some ambient support for prediction markets, definitely. But the vision here tends to be limited, mostly focused on "let's get better forecasting done on EA relevant questions/topics", not "scale up prediction markets until they are the primary way that society answers important questions in many fields".
There isn't huge effort going into future generations bills from within EA (the most notable post is complaining about them, not advocating them! https://forum.effectivealtruism.org/posts/TSZHvG7eGdmXCGhgS/concerns-with-the-wellbeing-of-future-generations-bill-1 ), although a lot of lefty- and climate-oriented EAs like them. But what I meant by that comment is just that EA has interpreted "improving institutional decisionmaking" to mean seeking influence within existing institutions, while I think there should be a second pillar of the cause area devoted to piloting totally new ideas in governance.
As an example of another idea that I think should get more EA attention and funding, Charter Cities have sometimes received an unduly chilly reception on the Forum (https://forum.effectivealtruism.org/posts/EpaSZWQkAy9apupoD/intervention-report-charter-cities), miscategorized as merely a neartermist economic-growth-boosting intervention, wheras charter city advocates are often most excited about their potential for experimental improvements in governance and leading to more "governance competition" among nations.
It was heartening to see the list of focus areas of the FTX future fund -- they seem more interested in institution design and progress-studies-esque ideas than the rest of the EA ecosystem, which I think is great.
I totally sympathize with and share the despair that many people feel about our governments' inadequacy to make the right decisions on AI, or even far easier issues like covid-19.
What I don't understand is why this isn't paired with a greater enthusiasm for supporting governance innovation/experimentation, in the hopes of finding better institutional structures that COULD have a fighting chance to make good decisions about AI.
Obviously "fix governance" is a long-term project and AI might be a near-term problem. But I still think the idea of improving institutional decision-making could be a big help in scenarios where AI takes longer than expected or government reform happens quicker than expected. In EA, "improving institutional decisionmaking" has come to mean incremental attempts to influence existing institutions by, eg, passing weaksauce "future generations" climate bills. What I think EA should be doing much more is supporting experiments with radical Dath-Ilan-style institutions (charter cities, liquid democracy, futarchy, etc) in a decentralized hits-based way, and hoping that the successful experiments spread and help improve governance (ie, getting many countries to adopt prediction markets and then futarchy) in time to be helpful for AI.
I've written much more about this in my prize-winning entry to the Future of Life Institute's "AI worldbuilding competition" (which prominently features a "warning shot" that helps catalyze action, in a near-future where governance has already been improved by partial adoption of Dath-Ilan-style institutions), and I'd be happy to talk about this more with interested folks: https://www.lesswrong.com/posts/qo2hqf2ha7rfgCdjY/a-bridge-to-dath-ilan-improved-governance-on-the-critical
Which countries will go to war with who? Doesn't strike me as plausible that, eg, individual random countries in the tropics would literally declare war on much-richer countries far away.
I think you are confusing the interests of citizens in the tropics (who might be motivated to immigrate from eg the Middle East to Europe, or from Indonesia to New Zealand, or from Venezuela to Uruguay, just as the poor are always motivated to move to more prosperous lands) with diplomacy -- why would the leaders of places like Indonesia declare war on places like New Zealand? We don't see countries in Central America trying to declare war on the USA today.
None other than Peter Thiel wrote a huge essay about investing while under anthropic shadow, and I wrote a post analyzing said essay! It is interesting, although pretty abstract in a way that probably makes it more relevant to organizations like OpenPhilanthropy than to most private individuals. Some quotes from Thiel's essay:
Apocalyptic thinking appears to have no place in the world of money. For if the doomsday predictions are fulfilled and the world does come to an end, then all the money in the world — even if it be in the form of gold coins or pieces of silver, stored in a locked chest in the most remote corner of the planet — would prove of no value, because there would be nothing left to buy or sell. Apocalyptic investors will miss great opportunities if there is no apocalypse, but ultimately they will end up with nothing when the apocalypse arrives. Heads or tails, they lose. ...A mutual fund manager might not benefit from reflecting about the danger of thermonuclear war, since in that future world there would be no mutual funds and no mutual fund managers left. Because it is not profitable to think about one ’s death, it is more useful to act as though one will live forever.
Since it is not profitable to contemplate the end of civilization, this distorts market prices. Instead of telling us about the objective probabilities of how things will play out, prices are based on probabilities adjusted by the anthropic logic of ignoring doomed scenarios:
Let us assume that, in the event of [the project of civilization being broadly successful], a given business would be worth $ 100/share, but that there is only an intermediate chance (say 1:10) of that successful outcome. The other case is too terrible to consider. Theoretically, the share should be worth $ 10, but in every world where investors survive, it will be worth $100. Would it make sense to pay more than $10, and indeed any price up to $100? Whether in hope or desperation, the perceived lack of alternatives may push valuations to much greater extremes than in nonapocalyptic times.
See my post for more.
For some more detail on what this plan might look like, my 2nd-place-winning entry in the Future of Life Institute's "A.I. world-building" competition, was all about how humanity uses prediction markets and other new institutional designs to increase its level of civilizational adequacy, becoming strong/wise enough to manage the safe development of transformative AI.
See my lesswrong post here (which focuses on the details of how AI development is controlled in my team's fictional scenario), or the whole entry here (which includes two great short stories by a friend of mine, and includes many more details about how things like futarchy, liquid democracy, network states and charter cities, quadratic funding for public goods funding, etc, develop over the next few decades).
You are in luck; it would appear that Elizabeth has already produced some significant long-covid analysis of exactly this nature!
You say:
[Under georgism,] there will be more pressure to use [land] in an economically viable way.
And then later you say:
If you want to reduce rents, all the usual methods apply – remove restrictions on land use, encourage higher density housing, and all that jazz.
I think that in my mind (and that of many Georgism advocates), one of the many benefits of Georgism would be that the increased pressure to use land in economically-optimal ways, will probably create increased incentives to build higher-density housing and increased political motivation remove economically-destructive land-use restrictions. This admittedly is a more convoluted path to YIMBYism than just advocating for YIMBYism directly. Nevertheless, Georgism & YIMBY-ism seem like natural complements, where each encourages the other.