Posts
Comments
Satellites were also plausibly a very important military technology. Since the 1960s, some applications have panned out, while others haven't. Some of the things that have worked out:
- GPS satellites were designed by the air force in the 1980s for guiding precision weapons like JDAMs, and only later incidentally became integral to the world economy. They still do a great job guiding JDAMs, powering the style of "precision warfare" that has given the USA a decisive military advantage ever since 1991's first Iraq war.
- Spy satellites were very important for gathering information on enemy superpowers, tracking army movements and etc. They were especially good for helping both nations feel more confident that their counterpart was complying with arms agreements about the number of missile silos, etc. The Cuban Missile Crisis was kicked off by U-2 spy-plane flights photographing partially-assembled missiles in Cuba. For a while, planes and satellites were both in contention as the most useful spy-photography tool, but eventually even the U-2's successor, the incredible SR-71 blackbird, lost out to the greater utility of spy satellites.
- Systems for instantly detecting the characteristic gamma-ray flashes of nuclear detonations that go off anywhere in the world (I think such systems are included on GPS satellites), and giving early warning by tracking ballistic missile launches during their boost phase (the Soviet version of this system famously misfired and almost caused a nuclear war in 1983, which was fortunately forestalled by one Lieutenant colonel Stanislav Petrov) are obviously a critical part of nuclear detterence / nuclear war-fighting.
Some of the stuff that hasn't:
- The air force initially had dreams of sending soldiers into orbit, maybe even operating a military base on the moon, but could never figure out a good use for this. The Soviets even test-fired a machine-gun built into one of their Salyut space stations: "Due to the potential shaking of the station, in-orbit tests of the weapon with cosmonauts in the station were ruled out. The gun was fixed to the station in such a way that the only way to aim would have been to change the orientation of the entire station. Following the last crewed mission to the station, the gun was commanded by the ground to be fired; some sources say it was fired to depletion".
- Despite some effort in the 1980s, were were unable to figure out how to make "Star Wars" missile defence systems work anywhere near well enough to defend us against a full-scale nuclear attack.
- Fortunately we've never found out if in-orbit nuclear weapons, including fractional orbit bombardment weapons, are any use, because they were banned by the Outer Space Treaty. But nowadays maybe Russia is developing a modern space-based nuclear weapon as a tool to destroy satellites in low-earth orbit.
Overall, lots of NASA activities that developed satellite / spacecraft technology seem like they had a dual-use effect advancing various military capabilities. So it wasn't just the missiles. Of course, in retrospect, the entire human-spaceflight component of the Apollo program (spacesuits, life support systems, etc) turned out to be pretty useless from a military perspective. But even that wouldn't have been clear at the time!
Maybe other people have a very different image of meditation than I do, such that they imagine it as something much more delusional and hyperreligious? Eg, some religious people do stuff like chanting mantras, or visualizing specific images of Buddhist deities, which indeed seems pretty crazy to me.
But the kind of meditation taught by popular secular sources like Sam Harris's Waking Up app, (or that I talk about in my "Examining The Witness" youtube series about the videogame The Witness), seems to me obviously much closer to basic psychology or rationality techniques than to religious practices. Compare Sam Harris's instructions about paying attention to the contents of one's experiences, to Gendlin's idea of "Circling", or Yudkowsky's concept of "sit down and actually try to think of solutions for five minutes", or the art of "noticing confusion", or the original Feynman essay where he describes holding off on proposing solutions. So it's weird to me when people seem really skeptical of meditation and set a very high burden of proof that they wouldn't apply for other mental habits like, say, CFAR techniques.
I'm not like a meditation fanatic -- personally I don't even meditate these days, although I feel bad about not doing it since it does make my life better. (Just like how I don't exercise much anymore despite exercise making my day go better, and I feel bad about that too...) But once upon a time I just tried it for a few weeks, learned a lot of interesting stuff, etc. I would say I got some mundane life benefits out of it -- some, like exercise or good sleep, that only lasted as long as I kept up the habit. and other benefits were more like mental skills that I've retained to today. I also got some very worthwhile philosophical insights, which I talk about, albeit in a rambly way mixed in with lots of other stuff, in my aforementioned video series. I certainly wouldn't say the philosophical insights were the most important thing in my whole life, or anything like that! But maybe more skilled deeper meditation = bigger insights, hence my agnosticism on whether the more bombastic metitation-related claims are true.
So I think people should just download the Waking Up app and try meditating for like 10 mins a day for 2-3 weeks or whatever-- way less of a time commitment than watching a TV show or playing most videogames -- and see for themselves if it's useful or not, instead of debating.
Anyways. For what it's worth, I googled "billionares who pray". I found this article (https://www.beliefnet.com/entertainment/5-christian-billionaires-you-didnt-know-about.aspx), which ironically also cites Bill Gates, plus the Walton Family and some other conservative CEOs. But IMO, if you read the article you'll notice that only one of them actually mentions a daily practice of prayer. The one that does, Do Won Chang, doesn't credit it for their business success... seems like they're successful and then they just also pray a lot. For the rest, it's all vaguer stuff about how their religion gives them a general moral foundation of knowing what's right and wrong, or how God inspires them to give back to their local community, or whatever.
So, personally I'd consider this duel of first-page-google-results to be a win for meditation versus prayer, since the meditators are describing a more direct relationship between scheduling time to regularly meditate and the assorted benefits they say it brings, while the prayer people are more describing how they think it's valuable to be christian in an overall cultural sense. Although I'm sure with more effort you could find lots of assorted conservatives claiming that prayer specifically helps them with their business in some concrete way. (I'm sure there are many people who "pray" in ways that resemble meditation, or resemble Yudkowsky's sitting-down-and-trying-to-think-of-solutions-for-five-minutes-by-the-clock, and find these techniques helpful!)
IMO, probably more convincing than dueling dubious claims of business titans, is testimony from rationalist-community members who write in detail about their experiences and reasoning. Alexey Guzey's post here is interesting, as he's swung from being vocally anti-meditation, to being way more into it than I ever was. He seems to still generally have his head on straight (ie hasn't become a religious fanatic or something), and says that meditation seems to have been helpful for him in terms of getting more things done: https://guzey.com/2022-lessons/
I think there are many cases of reasonably successful people who often cite either some variety of meditation, or other self-improvement regimes / habits, as having a big impact on their success. This random article I googled cites the billionaires Ray Dalio, Marc Benioff, and Bill Gates, among others. (https://trytwello.com/ceos-that-meditate/)
Similarly you could find people (like Arnold Schwarzenegger, if I recall?) citing that adopting a more mature, stoic mindset about life was helpful to them -- Ray Dalio has this whole series of videos on "life principles" that he likes. And you could find others endorsing the importance of exercise and good sleep, or of using note-taking apps to stay organized.
I think the problem is not that meditation is ineffective, but that it's not usually a multiple-standard-deviations gamechanger (and when it is, it's probably usually a case of "counting up to zero from negative", as TsviBT calls it), and it's already a known technique. If nobody else in the world meditated or took notes or got enough sleep, you could probably stack those techniques and have a big advantage. But alas, a lot of CEOs and other top performers already know to do this stuff.
(Separately from the mundane life-improvement aspects, some meditators claim that the right kind of deep meditation can give you insight into deep philosophical problems, or the fundamental nature of conscious experience, and that this is so valuable that achieving this goal is basically the most important thing you could do in life. This might possibly even be true! But that's different from saying that meditation will give you +50 IQ points, which it won't. Kinda like how having an experience of sublime beauty while contemplating a work of art, might be life-changing, but won't give you +50 IQ points.)
It feels sorta understandable to me (albeit frustrating) that OpenPhil faces these assorted political constraints. In my view this seems to create a big unfilled niche in the rationalist ecosystem: a new, more right-coded, EA-adjacent funding organization could optimize itself for being able to enter many of those blacklisted areas with enthusiasm.
If I was a billionare, I would love to put together a kind of "completion portfolio" to complement some of OP's work. Rationality community building, macrostrategy stuff, AI-related advocacy to try and influence republican politicians, plus a big biotechnology emphasis focused on intelligence enhancement, reproductive technologies, slowing aging, cryonics, gene drives for eradicating diseases, etc. Basically it seems like there is enough edgy-but-promising stuff out there (like studying geoengineering for climate, or advocating for charter cities, or just funding oddball substack intellectuals to do their thing) that you could hope to create a kind of "alt-EA" (obviously IRL it shouldn't have EA in the name) where you batten down the hatches, accept that the media will call you an evil villain mastermind forever, and hope to create a kind of protective umbrella for all the work that can't get done elsewhere. As a bonus, you could engage more in actual politics (like having some hot takes on the US budget deficit, or on how to increase marriage & fertility rates, or whatever), in some areas that OP in its quest for center-left non-polarization can't do.
Peter Thiel already lives this life, kinda? But his model seems 1. much more secretive, and 2. less directly EA-adjacent, than what I'd try if I was a billionare.
Dustin himself talks about how he is really focused on getting more "multipolarity" to the EA landscape, by bringing in other high-net-worth funders. For all the reasons discussed, he obviously can't say "hi, somebody please start an edgier right-wing offshoot of EA!!" But it seems like a major goal that the movement should have, nonetheless.
Seems like you could potentially also run this play with a more fully-left-coded organization. The gains there would probably be smaller, since there's less "room" to OP's left than to their right. But maybe you could group together wild animal welfare, invertebrate welfare, digital minds, perhaps some David Pearce / Project Far Out-style "suffering abolition" transhumanist stuff, other mental-wellbeing stuff like the Organization for the Prevention of Intense Suffering, S-risk work, etc. Toss in some more aggressive political activism on AI (like PauseAI) and other issues (like Georgist land value taxation), and maybe some forward-looking political stuff on avoiding stable totalitarianism, regulation of future AI-enabled technologies, and how to distribute the gains from a positive / successful singularity (akin to Sam Altman's vision of UBI supported by georgist/pigouvian taxes, but more thought-through and detailed and up-to-date.)
Finding some funders to fill these niches seems like it should be a very high priority of the rationalist / EA movement. Even if the funders were relatively small at first (like say they have $10M - $100M in crypto that they are preparing to give away), I think there could be a lot of value in being "out and proud" (publicising much of their research and philosophy and grantmaking like OP, rather than being super-secretive like Peter Thiel). If a small funder manages to build a small successful "alt-EA" ecosystem on either the left or right, that might attract larger funders in time.
There are actually a number of ways that you might see a permanently stable totalitarian government arise, in addition to the simplest idea that maybe the leader never dies:
https://80000hours.org/problem-profiles/risks-of-stable-totalitarianism/
I and perhaps other LessWrongers would appreciate reading your review (of any length) of the book, since lots of us loved HPMOR, the Sequences, etc, but are collectively skeptical / on the fence about whether to dive into Project Lawful. (What's the best way to read the bizzare glowfic format? What are the main themes of the book and which did you like best? etc)
This is nice! I like seeing all the different subfields of research listed and compared; as a non-medical person I often just hear about one at a time in any given news story, which makes things confusing.
Some other things I hear about in longevity spaces:
- Senescent-cell-based theories and medicines -- what's up with these? This seems like something people were actually trying in humans; any progress, or is this a dud?
- Repurposing essentially random drugs that might have some effect on longevity -- most famously the diabetes drug metformin (although people aren't expecting a very large increase in lifespan from this, rather at best a kind of proof-of-concept), also the immunosuppresant rapamicyn. Anything promising here, or is this all small potatoes compared to more ambitious approaches like cellular reprogramming?
I enjoyed this other LessWrong post trying to investigate root causes of aging, which focuses more on macro-scale problems like atheroschlerosis (although many of these must ultimately driven by some kind of cellular-level problems like proteins getting messed up via oxidization).
Fellow Thiel fans may be interested in this post of mine called "X-Risk, Anthropics, & Peter Thiel's Investment Thesis", analyzing Thiel's old essay "The Optimistic Thought Experiment", and trying to figure out how he thinks about the intersection of markets and existential risk.
"Americans eat more fats and oils, more sugars and sweets, more grains, and more red meat; all four items that grew the most in price since 2003."
Nice to know that you can eat healthy -- fish, veggies, beans/nuts, eggs, fresh fruit, etc -- and beat inflation at the same time! (Albeit these healthier foods still probably have a higher baseline price. But maybe not for much longer!)
The linked chart actually makes red meat look fine (beef has middling inflation, and pork has actually experienced deflation), but beverages, another generally unhealthy food, are near the top: https://www.ers.usda.gov/data-products/chart-gallery/gallery/chart-detail/?chartId=76961
As to the actual subject of the post, I have to imagine that:
- housing inflation feels so much worse in superstar cities than everywhere else, so for us cosmopolitan types it's hard to believe that the national average (brought lower by cheap housing across the Rust Belt, etc) isn't way higher.
- housing inflation is being measured in a way that doesn't indicate the true severity of the economic distortion. Like you say, housing prices cause migration -- SF is not just more expensive but also much smaller, less productive, etc, than it would be with better zoning laws. So only part of the tragedy caused by restrictive housing policy, actually shows up as high housing prices. (You could say the same for health and other things -- healthcare gets more expensive, but surely that also means people forgo certain expensive-but-beneficial treatments? But maybe housing just sees more of this effect than healthcare or education.)
A thoughtful post! I think about this kind of stuff a lot, and wonder what the implications are. If we're more pessimistic about saving lives in sub-saharan africa, should we:
- promote things like lead removal (similar evidence-backed, scalable intervention as bednets, but aimed more directly at human capital)?
- promote things like charter cities (untested crazy longshot megaproject, but aimed squarely at transformative political / societal improvements)?
- switch to bednet-style lifesaving charities in South Asia, like you mention?
- keep on trucking with our original Givewell-style africa-based lifesaving charities, because even after considering all the above, the original charities still look better than any of the three ideas above?
I would love it if you cross-posted this to the EA Forum (I'm sure you'd get some more criticism there vs Lesswrong, but I think it would nevertheless be a good conversation for them to have!) https://forum.effectivealtruism.org/
Re: your point #2, there is another potential spiral where abstract concepts of "greatness" are increasingly defined in a hostile and negative way by partisans of slave morality. This might make it harder to have that "aspirational dialogue about what counts as greatness", as it gets increasingly difficult for ordinary people to even conceptualize a good version of greatness worth aspiring to. ("Why would I want to become an entrepeneur and found a company? Wouldn't that make me an evil big-corporation CEO, which has a whiff of the same flavor as stories about the violent, insatiable conquistador villans of the 1500s?")
Of course, there are also downsides when culture paints a too-rosy picture of greatness -- once upon a time, conquistators were in fact considered admirable!
Feynman is imagining lots of components being made with "hand tools", in order to cut down on the amount of specialized machinery we need. So you'd want sophisticated manipulators to use the tools, move the components, clean up bits of waste, etc. Plus of course for gathering raw resources and navigating Canadian tundra. And you'd need video cameras for the system to look at what it's doing (otherwise you'd only have feed-forward controls in many situations, which would probably cause lots of cascading errors).
I don't know how big a rasberry pi would be if it had to be hand-assembled from transistors big enough to pick up individually. So maybe it's doable!
idk, you still have to fit video cameras and complex robotic arms and wifi equipment into that 1m^3 box, even if you are doing all the AI inference somewhere else! I have a much longer comment replying to the top-level post, where I try to analyze the concept of an autofac and what an optimized autofac design would really look like. Imagining a 100% self-contained design is a pretty cool intellectual exercise, but it's hard to imagine a situation where it doesn't make sense to import the most complex components from somewhere else (at least initially, until you can make computers that don't take up 90% of your manufacturing output).
This was a very interesting post. A few scattered thoughts, as I try to take a step back and take a big-picture economic view of this idea:
What is an autofac? It is a vastly simplified economy, in the hopes that enough simplification will unlock various big gains (like gains from "automation"). Let's interpolate between the existing global economy, and Feynman's proposed 1-meter cube. It's not true that "the smallest technological system capable of physical self-reproduction is the entire economy.", since I can imagine many potential simplifications of the economy. Imagine a human economy with everything the same, but no pianos, piano manufacturers, piano instructors, etc... the world would be a little sadder without pianos, but eliminating everything piano-related would slightly simplify the economy and probably boost overall productivity. The dream of the Autofac involves many more such simplifications, of several types:
- Eliminate luxuries (like pianos) and unnecessary complexity (do we really need 1000 types of car, instead of say, 5? The existence of so many different car manufacturers and car models is an artifact of capitalist competition and consumer preferences, not a physical necessity. Similarly, do we really need more than 5 different colors of paint / types of food / etc...).
- Give up on internal production of certain highly complex products, like microchips, in order to further simplify the economy. Keep giving up on more and more categories of complex products until your remaining internal economy is simple enough that you can automate the entire thing. Hopefully, this remaining automated economy will still account for most of the mass/energy being manipulated, with only a small amount of imports (lubricants, electronics, etc) required.
- Why make such a fuss about disentangling a "fully automatable" simplified subset of the economy from a distant "home base" that exports microchips and lubricant? I don't think a self-sufficient autofac plan would ever make sense in the middle of, say, the city of Shenzhen in China, when you are already surrounded by an incredibly complex manufacturing ecosystem that can easily provide whatever inputs you need. I can think of two reasons why you might want to cleave an economy in half like this, rather than just cluster everything together in a normal Shenzhen-style agglomeration mishmash:
- If you want to industrialize a distant, undeveloped location, and the cost of shipping goods there is very high, then it makes sense to focus on producing the heaviest materials locally and importing the smallest / most complex / most value-per-kg stuff.
- If you can cut humans entirely out of the loop of the simplified half of the economy, then you don't have to import or produce any of the stuff humans need (food, housing, healthcare, etc), which is a big efficiency win. This looks especially attractive if you want to industrialize a harsh, uninhabitable location (like Baffin Island, Antarctica, the Sahara desert, the bottom of the ocean, the moon, Mars, etc), where the costs of supporting humans are higher than normal.
- Why make such a fuss about disentangling a "fully automatable" simplified subset of the economy from a distant "home base" that exports microchips and lubricant? I don't think a self-sufficient autofac plan would ever make sense in the middle of, say, the city of Shenzhen in China, when you are already surrounded by an incredibly complex manufacturing ecosystem that can easily provide whatever inputs you need. I can think of two reasons why you might want to cleave an economy in half like this, rather than just cluster everything together in a normal Shenzhen-style agglomeration mishmash:
- Take an efficency hit in order to eliminate efficiency-boosting complexity. Perhaps instead of myriad types of alloy, we could get by with just a handful. Perhaps instead of myriad types of fastener, we could just use four standard sizes of screw. Perhaps instead of lots of specialized machines, we could make many of our tools "by hand" using generalized machine-shop tools.
- But wait -- I thought we were trying to maximize economic growth? Why give up things like carbide cutting tools in favor of inferior hardened-steel? Well, the hope is that if we simplify the economy enough, it will be possible to "automate" this simplified economy, and the benefits of this automation will make up for the efficiency losses.
- Okay then, why does efficiency-imparing simplification help with automation? Couldn't our autofac machine shop just as easily produce 10,000 types of fasteners, as 4 standard screws? Especially since the autofacs are making so many things "by hand?" Feynman seems very interested in an Autofac economy based almost entirely around steel -- what's the benefit of ditching useful materials like plastic, concrete, carbide, glass, rubber, etc? I see a few potential benefits to efficiency-imparing simplifications:
- lt reduces the size/cost/complexity of the initial self-replicating system. (I think this motivation is misplaced, and we should be shooting for a much larger initial size than 1 meter cubed.)
- It reduces the engineering effort needed to design the initial self-replicating system. (This motivation is reasonable, but it interacts in interesting ways with AI.)
- By trying to minimize the use of inputs like rubber and plastic, we reduce our reliance on rare natural resources like rubber trees and oil wells, neither of which exist on Baffin Island, or the moon, or etc. (This motivation is reasonable, but it only applies to a few of the proposed simplifications.)
To me, it seems that the Autofac dream comes from a particular context -- mid-20th-century visions of space exploration -- that have unduly influenced Feynman's current concept.
- Why the emphasis on creating a very small, 1 meter cubed package size?? This is a great size for something that we are shipping to the moon on a Saturn V rocket, or landing on Mars via skycrane, or perhaps sending to a distant star system as a Von Neumann probe. But for colonizing Baffin Island or the Sahara Desert or anywhere else on earth, we can use giant containter ships to easily move a much larger amount of stuff. By increasing the minimum size of our self-replicating system, we can include lots of efficiency-boosting nice-to-haves (like different types of alloys, carbide cutting tools, lubricant factories, etc). Feynman imagines initially releasing 1000 one-meter-cubed autofacs (and then supporting them with a continual stream of inputs), but I think we should instead design a single, 1000x-size autofac (it doesn't have to be one giant structure -- rather a network of factories, resource-gathering drones, steel mills, power plants, etc), since that would allow for more efficiency-boosting complexity.
- The remaining argument for 1000 one-meter-cubed autofacs is that it would be easier to design this much-smaller, much-simpler product. This is true! I'll get back to this in a bit.
- In general, I suspect that the ideal size of the autofac system should be proportional to the amount of transportation throughput you can support to Baffin Island / Mars / wherever. Design effort aside, it would be ideal to design the largest and most complex possible autofac which would fit into your transportation budget (eg, if you can afford five container ships to Baffin Island per year, then your autofac system should be large enough to fit into five container ships.
- Cutting humans entirely out of the loop is very appealing for deep-space exploration, but less appealing for places like Baffin Island. As long as you are only relying on relatively unskilled labor (such that you aren't worried about running out of humans to import, during the final stages of the industrialization of the island when millions and millions of windmills / steel mills / etc are going up), then importing a bunch of humans to handle a small percentage of high-value, hard-to-automate tasks, is probably worth it (even though it means you now have to provide housing, food, entertainment, law enforcement, etc).
- As others have mentioned, this "compromise" vision seems similar to Tesla's dreams of robotic factory workers (in large, container-ship-sized factories that still employ some human workers) and Spacex's mars colonization plans (where you still have a few humans assembling a mostly-mechanical system of nuclear power plants and solar panels, habitable spaces, greenhouses for food, etc -- but no 1-meter cubes to be seen, since Starship can carry 100 tons at a time to Mars).
- But again, I admit that re-complexifying the economy by introducing humans, does greatly increase the design complexity and thus the design effort required at the beginning.
- The one-meter-cubed autofac seems so pleasingly universal, like maybe once we've designed it, we could deploy it in all kinds of situations! But I think it is a lot less universal than it looks.
- A Baffin-Island-Plan autofac wouldn't fare well in the Sahara desert, where you'd want to manufacture solar panels (which rely more on chemistry and unique materials) instead of simple mechanical windmills that could be built almost entirely from steel. In the sahara, you'd also have less access to iron ore in the form of exposed rock; by contrast you'd have a lot of sillica that you could use to make glass. On the moon, you'd have no atmosphere at all for wind, and extreme temperatures + vacuum conditions would probably break a lot of the machine-shop tools (eg, liquid lubricants would freeze or sublimate). Etc.
- The above point isn't a fatal problem -- just having one autofac system for deserts and another for tundra would cover plenty of use cases for industrializing the unused portions of the earth. But you'd also run into problems when you finished replicating and wanted to use all those Baffin Island autofacs to contribute back to the external, human economy. Probably it would be fine to just have the Baffin Island autofacs build wind turbines and export steel + electricity, while the desert autofacs build solar panels and export glass + electricity. But if you decided that you wanted your Baffin Island autofacs to start growing food, or manufacturing textiles, you would have a big problem. The autofacs would in some ways be more flexible than a human manufacturing economy (eg, because they are doing more things "by hand", thus could switch to producing other types of steel products very quickly), but in other ways they would be much more rigid than a human manufacturing economy (if you want anything not based on steel, it might be pretty difficult for all the autofacs to reconfigure themselves).
Design effort & AI -- if AI is good enough to replace machinists, won't it be good enough to help design an autofac?
- This post reminds me of Carl Shulman's calculations (eg, on his recent 80,000 Hours podcast appearance) about the world economy's doubling times, and how fast they could possibly get, based on analogies to biological systems.
- Feynman says that, after many years, nowadays the dream of the Autofac is finally coming within reach, because AI is now good enough to operate robotics, navigate the world, use tools, and essentially replace the human machinist in a machine shop. This seems pretty likely come true, maybe in a few years.
- But creating such a small, self-contained, simplified autofac seems like it is motivated by the desire to minimize the up-front design effort needed. If AI ever gets good enough to become a drop-in remote worker not just for machinsts, but also for engineers/designers, then design effort is no longer such a big obstacle, and many of the conclusions flip.
- Consider how a paperclip-maximising superintelligence would colonize baffin island:
- The first image that jumps to mind is one of automated factories tesselated across the terrain. I think this is correct insofar as there would be lots of repetition (the basic idea of industrial production is that you can get economies of scale, cheaply churning out many copies of the same product, when you optimize a factory for producing that product). But I don't think these would be self-replicating factories.
- A superintelligent AI could do lots of design work very quickly, and wouldn't mind handling an extremely complex economy. I would expect the overall complexity of the global economy to go way up, and the minimum size of a self-replicating system to stay very large (ie, nearly the size of the entire planetary economy), and we just end up shipping lots of stuff to Baffin Island.
- If we say that the superintelligence has to start "from scratch" on Baffin Island alone, with only a limited budget for imports, then I'd expect it to start with something that looks like the 1-meter-cubed autofac, but then continually scale up in the size and complexity of its creations over time, rather than creating thousands of identical copies of one optimized design.
- A superintelligence-run economy might actually feature much less repetition than a human industrial economy, since the AI can juggle constant design changes and iteration and customization for local conditions, rather than needing to standardize things (as humans do to allow interoperability and reduce the costs of communicating between different humans).
Okay, I will try to sum up these scattered thoughts...
- I think that the ideal autofac design for a given situation will vary a lot based on factors like:
- what resources are locally available (wind vs solar, etc)
- how expensive it is to support humans as part of the design, vs going fully automated
- how much it costs to ship things to the location (the more you can ship, the bigger and more complex your autofac should be, other things equal)
- the ultimate scale you're aspiring to industrialize, relative to the size of your initial shipments (if you want to generate maximum energy on 1 acre of land using a 100 tons of payload, you should probably just import a 100-ton nuclear reactor and call it a day, rather than waste a bunch of money trying to design a factory to build a factory to build a nuclear reactor. Wheras if you are trying to generate power over the entire surface of Mars with a 100 ton payload, it is much more important to first create a self-replicating industrial base before you eventually turn towards creating lots of power plants.
- how much it costs to design a given autofac system
- a larger, more complex autofac will cost more to design, but will be more efficient
- a more-completely-self-sufficient system (eg, including lubricant factories, or eliminating the need for humans on Baffin Island) will cost more to design, but will save on shipping costs later
- if you can use lots of already-existing designs, that will lower design costs (but it will increase complexity elsewhere, since now you have to manufacture all the 10,000 types of fasteners and alloys and etc used by today's random equipment designs)
- advanced AI might be able to help greatly reduce design costs
- A fully-automated "autofac" design wins over a more traditional human-led industrialization effort, where the upfront costs of designing the mostly-self-sufficient autofac system manage to pay for themsleves by lowering the recurring costs of importing stuff, paying employees, etc, of a human-led industrialization effort.
- Whoever decides to start the cool autofac startup, should probably spend a bunch of time considering these big-picture economic tradeoffs, trying to figure out what environment (baffin island, the sahara desert, the oceans, the moon, Mars, alpha centauri, etc) offers the most upside from an autofac-style approach (Baffin-Island-like tundra might indeed be the best and most practical spot), and what tradeoffs to make in terms of autofac size/complexity, how much and where to incorporate humans into the system, and etc.
- I would personally love to get a better sense of where the efficiencies are really coming from, that help an autofac strategy win vs a human-led industrialization strategy. Contrast the autofac plan with a more traditional effort to have workers build roads and a few factories and erect windmills all over Baffin Island to export electricity -- where are the autofac wins coming from? The autofac would seem to have some big disadvantages, like that its windmill blades will be made of heavy steel instead of efficient fiberglass. Are the gains mostly from the fact that we're not paying as many worker salaries? Or is it mostly from the fact that we're producing all our heavy materials on-site rather than having to ship them in? Or somewhere else?
1950s era computers likely couldn't handle the complex AI tasks imagined here (doing image recognition; navigating rough Baffin Island terrain, finishing parts with hand tools, etc) without taking up much more than 1 meter cubed.
Socialism / communism is about equally abstract as Georgism, and it certainly inspired a lot of people to fight! Similarly, Republican campaigns to lower corporate tax rates, cut regulations, reduce entitlement spending, etc, are pretty abstract (and often actively unpopular when people do understand them!), but have achieved some notable victories over the years. Georgism is similar to YIMBYism, which has lots of victories these days, even though YIMBYism also suffers from being more abstract than conspiracy theories with obvious villains about people "hoarding" vacant housing or chinese investors bidding up prices or whatever. Finally, Georgism itself was extremely popular once, so it clearly has the potential!! Overall, I don't think being abstract is fatal for a mass movement.
But I also don't think that we need to have some kind of epic Georgist popular revolution in order to get Georgist policies -- we can do it just by making small incremental technocratic reforms to local property tax laws -- getting local governments to use tools like ValueBase (developed by Georgist Lars Doucet) to do their property value assessments, getting reforms in a few places and then hopefully seeing success and pointing to that success to build more momentum elsewhere, etc.
As Lars Doucet tells it, the main problem with historical Georgism wasn't unpopularity (it was extremely popular then!), but just the technical infeasibility of assessing land value separate from the value of the buildings on the land. But nowadays we have machine learning tools, GIS mapping systems, satellite imagery, successful home-value-estimation companies like Zillow and Redfin, etc. So nowadays we can finally implement Georgism on a technical level, which wasn't possible in the 1890s. For more on this, see the final part of Lars's epic series of georgism posts on Astral Codex Ten: https://www.astralcodexten.com/p/does-georgism-work-part-3-can-unimproved?utm_source=url
Future readers of this post might be interested this other lesswrong post about the current state of multiplex gene editing: https://www.lesswrong.com/posts/oSy5vHvwSfnjmC7Tf/multiplex-gene-editing-where-are-we-now
Future readers of this blog post may be interested in this book-review entry at ACX, which is much more suspicious/wary/pessimistic about prion disease generally:
- They dispute the idea that having M/V or V/V genes reduces the odds of getting CJD / mad cow disease / etc.
- They imply that Britain's mad cow disease problem maybe never really went away, in the sense that "spontaneous" cases of CJD have quadrupled since the 80s, so it seems CJD is being passed around somehow?
https://www.astralcodexten.com/p/your-book-review-the-family-that
What kinds of space resources are like "mice & cheese"? I am picturing civilizations expanding to new star systems mostly for the matter and energy (turn asteroids & planets into a dyson swarm of orbiting solar panels and supercomputers on which to run trillions of emulated minds, plus constructing new probes to send onwards to new star systems).
re: the Three Body Problem books -- I think the book series imagines that alien life is much, much more common (ie, many civilizations per galaxy) than Robin Hanson imagines in his Grabby Aliens hypothesis, such that there are often new, not-yet-technologically-mature civilizations popping up nearby each other, around the same time as each other. Versus an important part of the Grabby Aliens model is the idea that the evolution of complex life is actually spectacularly rare (which makes humans seem to have evolved extremely early relative to when you might expect, which is odd, but which is then explained by some anthropic reasoning related to the expanding grabby civilizations -- all new civilizations arise "early", because by the mid-game, everything has been colonized already). If you think that the evolution of complex life on other planets is actually a very common occurrence, then there is no particular reason to put much weight on the Grabby Aliens hypothesis.
In The Three Body Problem, Earth would be wise to keep quiet so that the Trisolarians don't overheard our radio transmissions and try to come and take our nice temperate planet, with its nice regular pattern of seasons. But there is nothing Earth could do about an oncoming "grabby" civilization -- the grabby civilization is already speeding towards Earth at near-lightspeed, and wants to colonize every solar system (inhabited and uninhabited, temperate planets with regular seasons or no, etc), since it doesn't care about temperate continents, just raw matter that it can use to create dyson swarms. The grabby civilizations are already expanding as fast as possible in every direciton, coming for every star -- so there is no point trying to "hide" from them.
Energy balance situation:
- the sun continually emits around 10^26 watts of light/heat/radiation/etc.
- per some relativity math at this forum comment, it takes around 10^18 joules to accelerate 1kg to 0.99c
- so, using just one second of the sun's energy emissions, you could afford to accelerate around 10^8 kg (about the mass of very large cargo ships, and of the RMS Titanic) to 0.99c. Or if you spend 100 days' worth of solar energy instead of one second, you could accelerate about 10^15 kg, the mass of Mt. Everest, to 0.99c.
- of course then you have to slow down on the other end, which will take a lot of energy, so the final size of the von neumann probe that you can deliver to the target solar system will have to be much smaller than the Titanic or Mt Everest or whatever.
- if you go slower, at 0.8c, you can launch 10x as much mass with the same energy (and you don't have to slow down as much on the other end, so maybe your final probe is 100x bigger), but of course you arrive more slowly -- if you're travelling 10 light years, you show up 1.9 years later than the 0.99c probe. If you're travelling 100 light years, you show up 19 years later.
- which can colonize the solar system and build a dyson swarm faster -- a tiny probe that arrives as soon as possible, or a 100x larger probe that arrives with a couple years' delay? this is an open question that depends on how fast your von neuman machine can construct solar panels, automated factories, etc. Carl Shulman in a recent 80K podcast figures that a fully-automated economy pushing up against physical limits, could double itself at least as quickly as once per year. So mabye the 0.99c probe would do better over the 100 light-year distance (arriving 19 years early gives time for 19 doublings!), but not for the 10 light-year distance (the 0.99c probe would only have doubled itself twice, to 4x its initial mass, by the time the 0.8c probe shows up with 100x as much mass)
- IMO, if you are trying to rapaciously grab the universe as fast as possible (for the ultimate purpose of maximizing paperclips or whatever), probably you don't hop from nearby star to nearby star at efficient speeds like 0.8c, waiting to set up a whole new dyson sphere (which probably takes many years) at each stop. Rather, your already-completed dyson swarms are kept busy launching new probes all the time, targeting ever-more-distant stars. By the time a new dyson swarm gets finished, all the nearby stars have also been visited by probes, and are already constructing dyson swarms of their own. So you have to fire your probes not at the nearest stars, but at stars some distance further away. My intuition is that the optimal way to grab the most energy would end up favoring very fast expansion speeds, but I'm not sure. (Maybe the edge of your cosmic empire expands at 0.99c, and then you "mop up" some interior stars at more efficient speeds? But every second that you delay in capturing a star, that's a whopping 10^26 joules of energy lost!)
Yes, it does have to be fast IMO, but I think fast expansion (at least among civilizations that decide to expand much at all) is very likely.
Of course the first few starships that a civilization sends to colonize the nearest stars will probably not be going anywhere near the speed of light. (Unless it really is a paperclips-style superintelligence, perhaps.) But within a million years or so, even with relatively slow-moving ships, you have colonized thousands of solar systems, built dyson swarms around every star, have a total population in the bajilions, and have probably developed about all the technology that it is physically possible to develop. So, at some point it's plausible that you start going very close to the speed of light, because you'll certainly have enough energy + technology to do so, and because it might be desirable for a variety of reasons:
- Maybe we are trying to maximize some maximizable utility function, be that paperclips or some more human notion, and want to minimize what Nick Bostrom calls "astronomical waste".
- Maybe we fail to coordinate (via a strong central government or etc), and the race to colonize the galaxy becomes a free-for-all, rewarding the fastest and most rapacious settlers, a la Robin Hanson's "Burning the cosmic commons".
Per your own comment -- if you only colonize at 0.8c so your ships can conserve energy, you are probably actually missing out on lots and lots of energy, since you will only be able to harvest resources from about half the volume that you could grab if you traveled at closer to lightspeed!
I think part of the "calculus" being run by the AI safety folks is as follows:
-
there are certainly both some dumb ways humanity could die (ie, AI-enabled bioweapon terrorism that could have easily been prevented by some RLHF + basic checks at protein synthesis companies), as well as some very tricky, advanced ways (AI takeover by a superintelligence with a very subtle form of misalignment, using lots of brilliant deception, etc)
-
It seems like the dumber ways are generally more obvious / visible to other people (like military generals or the median voter), wheras these people are skeptical of the trickier paths (ie, not taking the prospect of agentic, superintelligent AI seriously; figuring alignment will probably continue to be easy even as AI gets smarter, not believing that you could ever use AI to do useful AI research, etc).
-
The trickier paths also seem like we might need to get a longer head start on them, think about them more carefully, etc.
-
Therefore, I (one of the rare believers in things like "deceptive misalignment is likely" or "superintelligence is possible") should work on the trickier paths; others (like the US military, or other government agencies, or whatever) will eventually recognize and patch the dumber paths.
re: your comments on Fermi paradox -- if an alien super-civilization (or alien-killing AI) is expanding in all directions at close to the speed of light (which you might expect a superintelligence to do), then you mostly don't see them coming until it's nearly too late, since the civilization is expanding almost as fast as the light emitted by the civilization. So it might look like the universe is empty, even if there's actually a couple of civilizations racing right towards you!
There is some interesting cosmological evidence that we are in fact living in a universe that will eventually be full of such civilizations; see the Robin Hanson idea of "Grabby Aliens": https://www.youtube.com/watch?v=l3whaviTqqg
[spoilers for minor details of later chapters of the book] Isn't the book at least a little self-aware about this plot hole? If I recall correctly, the book eventually reveals (rot13 from here on out)...
gung gur fcnpr cebtenz cyna jnf rffragvnyyl qrfvtarq nf n CE fghag gb qvfgenpg/zbgvingr uhznavgl, juvpu vaqrrq unq ab erny cebfcrpg bs jbexvat (gbb srj crbcyr, gbb uneq gb qbqtr zrgrbef, rirelguvat arrqf gb or 100% erhfrq, rgp, yvxr lbh fnl). Juvyr gur fcnpr cebtenz jnf unccravat, bgure yrff-choyvp rssbegf (eryngrq gb ahpyrne fhoznevarf, qvttvat haqretebhaq, rgp) ner vaqrrq batbvat, nygubhtu gur tbireazragf pregnvayl nera'g gelvat gb fnir uhaqerqf bs zvyyvbaf bs crbcyr (urapr gur qrfver sbe frperpl, V thrff).
Jung'f "fhecevfvat" nobhg gur obbx'f cybg, vf gung (fvapr Arny Fgrcurafba ernyyl jnagf gb jevgr n fgbel nobhg fcnpr, engure guna n fgbel nobhg haqretebhaq ohaxref), gur fcnpr cyna npghnyyl raqf hc fhpprrqvat ng nyy, naq vaqrrq va gur ybat eha raqf hc orvat zhpu zber vasyhragvny bire gur shgher guna gur inevbhf cerfhznoyl-zber-frafvoyr cynaf sbe haqretebhaq ershtrf.
In addition to the researchy implications for topics like deception and superpersuasion and so forth, I imagine that results like this (although, as you say, unsuprising in a technical sense) could have a huge impact on the public discussion of AI (paging @Holly_Elmore and @Joseph Miller?) -- the general public often seems to get very freaked out about privacy issues where others might learn their personal information, demographic characteristics, etc.
In fact, the way people react about privacy issues is so strong that it usually seems very overblown to me -- but it also seems plausible that the fundamental /reason/ people are so sensitive about their personal information is precisely because they want to avoid being decieved or becoming easily manipulable / persuadable / exploitable! Maybe this fear turns out to be unrealistic when it comes to credit scores and online ad-targeting and TSA no-fly lists, but AI might be a genuinely much more problematic technology with much more potential for abuses here.
So, sure, there is a threshold effect in whether you get value from bike lanes on your complex journey from point A to point G. But other people throughout the city have different threshold effects:
- Other people are starting and ending their trips from other points; some people are even starting and ending their trip entirely on Naito Parkway.
- People have a variety of different tolerances for how much they are willing to bike in streets, as you mention.
- Even people who don't like biking in streets often have some flexibility. You say that you personally are flexible, "but for the sake of argument, let's just assume that there is no flexibility". But in real life, even many people who absolutely refuse to bike in traffic might be happy to walk their bike on the sidewalk (or whatever) for a single block, in order to connect two long stretches of beautiful bike path.
When you add together a million different possible journeys across thousands of different people, each with their own threshold effects, the sum of the utility provided by each new bike lane probably ends up looking much more like a smooth continuum of incremental benefits per each new bike lane that is added to a city's network, with no killer threshold effects. This is very different from a bridge, where indeed half a bridge is not useful to any portion of the city's residents.
Therefore, I don't think "if you're going to start building [a bike lane network], you better m ake sure you have plans to finish it". Rather, I think adding random pieces of the network piecemeal (as random roads undergo construction work for other reasons, perhaps) is a totally reasonable thing for cities to do.
Another example of individual threshold effects adding up to continuous benefit from the provider's perspective: suppose I only like listening to thrash-metal songs on Spotify. Whenever Spotify adds a song from a genre I don't care for -- pop, classical, doom-metal, whatever -- it provides literally ZERO value to me! There's a huge threshold effect, where I only care when spotify adds thrash-metal songs! But of course, since everyone has different preferences, the overall effect of adding songs from Spotify's perspective is to provide an incremental benefit to the quality of their product each time they add a song. (Disclaimer: I am not actually a thrash-metal fanatic.)
Nice; Colorado recently passed a statewide law that finally does away with a similar "U+2" rule in my own town of Fort Collins (as well as other such rules in Boulder and elsewhere). To progress!
this is quality.
I don't understand this post, because it seems to be parodying Anthropic's Responsible Scaling Policies (ie, saying that the RSPs are not sufficient), but the analogy to nuclear power is confusing since IMO nuclear power has in fact been harmfully over-regulated, such that advocating for a "balanced, pragmatic approach to mitigating potential harms from nuclear power" does actually seem good, compared to the status quo where society hugely overreacted to the risks of nuclear power without properly taking a balanced view of the costs vs benefits.
Maybe you can imagine how confused I am, if we use another example of an area where I think there is a harmful attitude of regulating entirely with a view towards avoiding visible errors of commision, and completely ignoring errors of omission:
Hi, we're your friendly local pharma company. Many in our community have been talking about the need for "vaccine safety."... We will conduct ongoing evaluations of whether our new covid vaccine might cause catastrophic harm (conservatively defined as >10,000 vaccine-side-effect-induced deaths).
We aren't sure yet exactly whether the vaccine will have rare serious side effects, since of course we haven't yet deployed the vaccine in the full population, and we're rushing to deploy the vaccine quickly in order to save the lives of the thousands of people dying to covid every day. But fortunately, our current research suggests that our vaccine is unlikely to cause unacceptable harm. The frequency and severity of side effects seen so far in medical trials of the vaccine are far below our threshold of concern... the data suggest that we don't need to adopt additional safety measures at present.
To me, vaccine safety and nuclear safety seem like the least helpful possible analogies to the AI situation, since the FDA and NRC regulatory agencies are both heavily infected with an "avoid deaths of commision at nearly any cost" attitude, which ignores tradeoffs and creates a massive "invisible graveyard" of excess deaths-of-omission. What we want from AI regulation isn't an insanely one-sided focus that greatly exaggerates certain small harms. Rather, for AI it's perfectly sufficient to take the responsible, normal, common-sensical approach of balancing costs and benefits. The problem is just that the costs might be extremely high, like a significant chance of causing human extinction!!
Another specific bit of confusion: when you mention that Chernobyl only killed 50 people, is this supposed to convey:
1. This sinister company is deliberately lowballing the Chernobyl deaths in order to justify continuing to ignore real risks, since a linear-no-threshold model suggests that Chernobyl might indeed have caused tens of thousands of excess cancer deaths around the world? (I am pretty pro- nuclear power, but nevertheless the linear-no-threshold model seems plausible to me personally.)
2. That Chernobyl really did kill only 50 people, and therefore the company is actually correct to note that nuclear accidents aren't a big deal? (But then I'm super-confused about the overall message of the post...)
3. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation on nuclear power plants in order to prevent other rare accidents that might kill 50 people tops? (This seems like extreme over-regulation of a beneficial technology, compared to the much larger number of people who die from the smoke of coal-fired power plants and other power sources.)
4. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation, because future accidents might indeed kill over 10,000 people? (This seems like it would imply some kind of conversation about first-principles reasoning and tail risks and stuff, but this isn't present in the post?)
I will definitely check out that youtube channel! I'm pretty interested in mechanism design and public-goods stuff, and I agree there are a lot of good ideas there. For instance, I am a huge fan of Georgism, so I definitely recognize that going all-in on the "libertarian individualist approach" is often not the right fit for the situation! Honestly, even though charter cities are somewhat an intrinsically libertarian concept, part of the reason I like the charter city idea is indeed the potential for experimenting with new ways to manage the commons and provide public goods -- Telosa is explicitly georgist, for example, and even hyper-libertarian Prospera has some pretty interesting concepts around things like crime liability insurance, which in the USA is considered a pretty left-wing (or maybe "far-liberal"? idk...) idea for trying to reduce gun violence.
But yeah, a lot of common leftist critiques of society/capitalism/etc can feel kind of... shallow, or overly-formulaic, or confused about the incentives of a given situation, to me? So I'd like to get a better understanding of the best versions of the leftist worldview, in order to better appreciate what the common critiques are getting at.
Yup, there are definitely a lot of places (like 99+% of places, 99+% of the time!) which aren't interested in a given reform -- especially one as uniqely big and experimental as charter cities. This is why in our video we tried to focus on political tractability as one of the biggest difficulties -- hopefully we don't come across as saying that the world will instantly be tiled over with charter cities tomorrow! But some charter cities are happening sometimes in some places -- in addition to the examples in the video, Zambia is pretty friendly towards the idea, and is supportive of the new-city project Nkwashi. (I think Charter Cities Institute considers Nkwashi to be their biggest current partnership?) Democracy was achieved, after all, even if it still hasn't won a total victory even after 250+ years.
Thanks, this is exciting and inspiring stuff to learn about!
I guess another thing I'm wondering about, is how we could tell apart genes that impact a trait via their ongoing metabolic activities (maybe metabolic is not the right term... what I mean is that the gene is being expressed, creating proteins, etc, on an ongoing basis), versus genes that impact a trait via being important for early embryonic / childhood development, but which aren't very relevant in adulthood. Genes related to intelligence, for instance, seem like they might show up with positive scores in a GWAS, but their function is confined to helping unfold the proper neuron connection structures during fetal development, and then they turn off, so editing them now won't do anything. Versus other genes that affect, say, what kinds of cholesterol the body produces, seem more likely to have direct impact via their day-to-day operation (which could be changed using a CRISPR-like tool).
Do we have any way of distinguishing the one type of genes from the other? (Maybe we can just look at living tissue and examine what genes are expressed vs turned off? This sounds hard to do for the entire genome...) Or perhaps we have reason to believe something like "only 20% of genes are related to early development, 80% handle ongoing metabolism, so the GWAS --> gene therapy pipeline won't be affected too badly by the dilution of editing useless early-development genes"?
Is there a plausible path towards gene therapies that edit dozens, hundreds, or thousands of different genes like this? I thought people were worried about off-target errors, etc? (Or at least problems like "you'll have to take 1000 different customized doses of CRISPR therapy, which will be expensive".) So my impression is that this kind of GWAS-inspired medicine would be most impactful with whole-genome synthesis? (Currently super-expensive?)
To be clear I agree with the main point this post is making about how we don't need animal models, etc, to do medicine if we have something that we know works!
(this comment is kind of a "i didn't have time to write you a short letter so I wrote you a long one" situation)
re: Infowar between great powers -- the view that China+Russia+USA invest a lot of efforts into infowar, but mostly "defensively" / mostly trying to shape domestic opinion, makes sense. (After all, it must be easier to control the domestic media/information lansdscape!) I would tend to expect that doing domestically-focused infowar stuff at a massive scale would be harder for the USA to pull off (wouldn't it be leaked? wouldn't it be illegal somehow, or at least something that public opinion would consider a huge scandal?), but on the other hand I'd expect the USA to have superior infowar technology (subtler, more effective, etc). And logically it might also be harder to percieve effects of USA infowar techniques, since I live in the USA, immersed in its culture.
Still, my overall view is that, although the great powers certainly expend substantial effort trying to shape culture, and have some success, they don't appear to have any next-gen technology qualitatively different and superior to the rhetorical techniques deployed by ordinary successful politicians like Trump, social movements like EA or wokeism, advertising / PR agencies, media companies like the New York Times, etc. (In the way that, eg, engineering marvels like the SR-72 Blackbird were generations ahead of competitors' capabilities.) So I think the overall cultural landscape is mostly anarchic -- lots of different powers are trying to exert their own influence and none of them can really control or predict cultural changes in detail.
re: Social media companies' RL algorithms are powerful but also "they probably couldn't prevent algorithms from doing this if they tried due to goodharts law". -- Yeah, I guess my take on this is that the overt attempts at propaganda (aimed at placating the NYT) seem very weak and clumsy. Meanwhile the underlying RL techniques seem potentially powerful, but poorly understood or not very steerable, since social media companies seem to be mostly optimizing for engagement (and not even always succeeding at that; here we are talking on LessWrong instead of tweeting / tiktoking), rather than deploying clever infowar superweapons. If they have such power, why couldn't left-leaning sillicon valley prevent the election of Trump using subtle social-media-RL trickery?
(Although I admit that the reaction to the 2016 election could certainly be interpreted as sillicon valley suddenly realizing, "Holy shit, we should definitely try to develop social media infowar superweapons so we can maybe prevent this NEXT TIME." But then the 2020 election was very close -- not what I'd have expected if info-superweapons were working well!)
With Twitter in particular, we've had such a transparent look at its operations during the handover to Elon Musk, and it just seems like both sides of that transaction have been pretty amateurish and lacked any kind of deep understanding of how to influence culture. The whole fight seems to have been about where to tug one giant lever called "how harshly do we moderate the tweets of leftists vs rightists". This lever is indeed influential on twitter culture, and thus culture generally -- but the level of sophistication here just seems pathetic.
Tiktok is maybe the one case where I'd be sympathetic to the idea that maybe a lot of what appears to be random insane trends/beliefs fueled by SGD algorithms and internet social dynamics, is actually the result of fairly fine-grained cultural influence by Chinese interests. I don't think Tiktok is very world-changing right now (as we'd expect, it's targeting the craziest and lowest-IQ people first), but it's at least kinda world-changing, and maybe it's the first warning sign of what will soon be a much bigger threat? (I don't know much about the details of Tiktok the company, or the culture of its users, so it's hard for me to judge how much fine-grained control China might or might not be exerting.)
Unrelated -- I love the kind of sci-fi concept of "people panic but eventually go back to using social media and then they feel fine (SGD does this automatically in order to retain users)". But of course I think that the vast majority of users are in the "aren't panicking" / never-think-about-this-at-all category, and there are so few people in the "panic" category (panic specifically over subtle persuasion manipulation tech that isn't just trying to maximize engagement but instead achieve some specific ideological outcome, I mean) that there would be no impact on the social-media algorithms. I think it is plausible that other effects like "try not to look SO clickbaity that users recognize the addictiveness and leave" do probably show up in algorithms via SGD.
More random thoughts about infowar campaigns that the USA might historically have wanted to infowar about:
- Anti-communism during the cold war, maybe continuing to a kind of generic pro-corporate / pro-growth attitude these days. (But lots of people were pro-communist back in the day, and remain anti-corporate/anti-growth today! And even the republican party is less and less pro-business... their basic model isn't to mind-control everyone into becoming fiscal conservatives, but instead to gain power by exploiting the popularity of social conservativism and then use power to implement fiscal conservativism.)
- Maybe I am taking a too-narrow view of infowar as "the ability to change peoples' minds on individual issues", when actually I should be considering strategies like "get people hyped up about social issues in order to gain power that you can use for economic issues" as a successful example of infowar? But even if I consider this infowar, then it reinforces my point that the most advanced stuff today all seems to be variations on normal smart political strategy and messaging, not some kind of brand-new AI-powered superweapon for changing people's minds (or redirecting their focus or whatever) in a radically new way.
- Since WW2, and maybe continuing to today, the West has tried to ideologically immunize itself against Nazi-ism. This includes a lot of trying to teach people to reject charismatic dictators, to embrace counterintuitive elements of liberalism like tolerance/diversity, and even to deny inconvenient facts like racial group differences for the sake of social harmony. In some ways this has gone so well that we're getting problems from going too far in this direction (wokism), but in other ways it can often feel like liberalism is hanging on by a thread and people are still super-eager to embrace charismatic dictators, incite racial conflict, etc.
"Human brains are extremely predisposed to being hacked, governments would totally do this, and the AI safety community is unusually likely to be targeted."
-- yup, fully agree that the AI safety community faces a lot of peril navigating the whims of culture and trying to win battles in a bunch of diverse high-stakes environments (influencing superpower governments, huge corporations, etc) where they are up against a variety of elite actors with some very strong motivations. And that there is peril both in the difficulty of navigating the "conventional" human-persuasion-transformed social landscape of today's world (already super-complex and difficult) and the potentially AI-persuasion-transformed world of tomorrow. I would note though, that these battles will (mostly?) play out in pretty elite spaces, wheras I'd expect the power of AI information superweapons to have the most powerful impact on the mass public. So, I'd expect to have at least some warning in the form of seeing the world go crazy (in a way that seems different from and greater than today's anarchic internet-social-dynamics-driven craziness), before I myself went crazy. (Unless there is an AI-infowar-superweapon-specific hard-takeoff where we suddenly get very powerful persuasion tech but still don't get the full ASI singularity??)
re: Dath Ilan -- this really deserves a whole separate comment, but basically I am also a big fan of the concept of Dath Ilan, and I would love to hear your thoughts on how you would go about trying to "build Dath Ilan" IRL.
- What should an individual person, acting mostly alone, do to try and promote a more Dath-Ilani future? Try to practice & spread Lesswrong-style individual-level rationality, maybe (obviously Yudkowsky did this with Lesswrong and other efforts). Try to spread specific knowledge about the way society works and thereby build energy for / awareness of ways that society could be improved (inadequate equilibria kinda tries to do this? seems like there could be many approaches here). Personally I am also always eager to talk to people about specific institutional / political tweaks that could lead to a better, more Dath-Ilani world: georgism, approval voting, prediction markets, charter cities, etc. Of those, some would seem to build on themselves while others wouldn't -- what ideas seem like the optimal, highest-impact things to work on? (If the USA adopted georgist land-value taxes, we'd have better land-use policy and faster economic growth but culture/politics wouldn't hugely change in a broadly Dath-Ilani direction; meanwhile prediction markets or new ways of voting might have snowballing effects where you get the direct improvement but also you make culture more rational & cooperative over time.)
- What should a group of people ideally do? (Like, say, an EA-adjacent silicon valley billionaire funding a significant minority of the EA/rationalist movement to work on this problem together in a coordinated way.) My head immediately jumps to "obviously they should build a rationalist charter city":
- The city doesn't need truly nation-level sovereign autonomy, the goal would just be to coordinate enough people to move somewhere together a la the Free State Project, gaining enough influence over local government to be able to run our own policy experiments with things like prediction markets, georgism, etc. (Unfortunately some things, like medical research, are federally regulated, but I think you could do a lot with just local government powers + creating a critical mass of rationalist culture.)
- Instead of moving to a random small town and trying to take over, it might be helpful to choose some existing new-city project to partner with -- like California Forever, Telosa, Prospera, whatever Zuzalu or Praxis turn into, or other charter cities that have amenable ideologies/goals. (This would also be very helpful if you don't have enough people or money to create a reasonably-sized town all by yourself!)
- The goal would be twofold: first, run a bunch of policy experiments and try to create Dath-Ilan-style institutions (where legal under federal law if you're still in the USA, etc). And second, try to create a critical mass of rationalist / Dath Ilani culture that can grow and eventually influence... idk, lots of people, including eventually the leaders of other governments like Singapore or the UK or whatever. Although it's up for debate whether "everyone move to a brand-new city somewhere else" is really a better plan for cultural influence than "everyone move to the bay area", which has been pretty successful at influencing culture in a rationalist direction IMO! (Maybe the rationalist charter city should therefore be in Europe or at least on the East Coast or something, so that we mostly draw rationalists from areas other than the Bay Area. Or maybe this is an argument for really preferring California Forever as an ally, over and above any other new-city project, since that's still in the Bay Area. Or for just trying to take over Bay Area government somehow.)
- ...but maybe a rationalist charter city is not the only or best way that a coordinated group of people could try to build Dath Ilan?
(Copies from EA Forum for the benefit of lesswrongers following the discussion here)
Definitely agree that empathy and other social feelings provide indirect evidence for self-awareness (ie, "modeling stuff about yourself" in your brain) in a way that optimism/pessimism or pain-avoidance doesn't. (Although wouldn't a sophisticated-enough RL circuit, interacting with other RL circuits in some kind of virtual evolutionary landscape, also develop social emotions like loyalty, empathy, etc? Even tiny mammals like mice/rats display sophisticated social behaviors...)
I tend to assume that some kind of panpsychism is true, so you don't need extra "circuitry for experience" in order to turn visual-information-processing into an experience of vision. What would such extra circuitry even do, if not the visual information processing itself? (Seems like maybe you are a believer in what Daniel Dennet calls the "fallacy of the second transduction"?)
Consequently, I think it's likely that even simple "RL algorithms" might have a very limited, very shallow, non-self-aware kinds of experience: an image-classifier is doing visual-information-processing, so it probably also produces isolated "experiences of vision"! But of course it would not have any awareness of itself as being a thing-that-sees, nor would those isolated experiences of vision be necessarily tied together into a coherent visual field, etc.
So, I tend to think that fish and other primitive creatures probably have "qualia", including something like a subjective experience of suffering, but that they probably lack any sophisticated self-awareness / self-model, so it's kind of just "suffering happening nowhere" or "an experience of suffering not connected to anything else" -- the fish doesn't know it's a fish, doesn't know that it's suffering, etc, the fish is just generating some simple qualia that don't really refer to anything or tie into a larger system. Whether you call such a disconnected & shallow experience "real qualia" or "real suffering" is a question of definitions.
I think this personal view of mine is fairly similar to Eliezer's from the Sequences: there are no "zombies" (among humans or animals), there is no "second transduction" from neuron activity into a mythical medium-of-consciousness (no "extra circuitry for experience" needed), rather the information-processing itself somehow directly produces (or is equivalent to, or etc) the qualia. So, animals and even simpler systems probably have qualia in some sense. But since animals aren't self-aware (and/or have less self-awareness than humans), their qualia don't matter (and/or matter less than humans' qualia).
...Anyways, I think our core disagreement is that you seem to be equating "has a self-model" with "has qualia", versus I think maybe qualia can and do exist even in very simple systems that lack a self-model. But I still think that having a self-model is morally important (atomic units of "suffering" that are just floating in some kind of void, unconnected to a complex experience of selfhood, seem of questionable moral relevance to me), so we end up having similar opinions about how it's probably fine to eat fish.
I guess what I am objecting to is that you are acting like these philosophical problems of qualia / consciousness / etc are solved and other people are making an obvious mistake. I agree that I see a lot of people being confused and making mistakes, but I don't think the problems are solved!
Why would showing that fish "feel empathy" prove that they have inner subjective experience? It seems perfectly possible to build a totally mechanical, non-conscious system that nevertheless displays signs of empathy. Couldn't fish just have some kind of built-in, not-necessarily-conscious instinct to protect other fish (for instance, by swimming together in a large school) in order to obtain some evolutionary benefit?
Conversely, isn't it possible for fish to have inner subjective experience but not feel empathy? Fish are very simple creatures, while "empathy" is a complicated social emotion. Especially in a solitary creature (like a shark, or an octopus), it seems plausible that you might have a rich inner world of qualia alongside a wide variety of problem-solving / world-modeling skills, but no social instincts like jealousy, empathy, loyalty, etc. Fish-welfare advocates often cite studies that seem to show fish having an internal sense of pain vs pleasure (eg, preferring water that contains numbing medication), or that bees can have an internal sense of being optimistic/risky vs pessimistic/cautious -- if you think that empathy proves the existence of qualia, why are these similar studies not good enough for you? What's special about the social emotion of empathy?
Personally, I am more sympathetic to the David Chalmers "hard problem of consciousness" perspective, so I don't think these studies about behaviors (whether social emotions like jealousy or more basic emotions like optimism/pessimism) can really tell us that much about qualia / inner subjective experience. I do think that fish / bees / etc probably have some kind of inner subjective experience, but I'm not sure how "strong", or vivid, or complex, or self-aware, that experience is, so I am very uncertain about the moral status of animals. (Personally, I also happily eat fish & shrimp all the time.)
In general, I think this post is talking about consciousness / qualia / etc in a very confused way -- if you think that empathy-behaviors are ironclad proof of empathy-qualia, you should also think that other (pain-related, etc) behaviors are ironclad proof of other qualia.
Hi Trevor! I appreciate this thread of related ideas that you have been developing about intelligence agencies, AI-augmented persuasion techniques, social media, etc.
- It seems important to "think ahead" about how the power-struggle over AI will play out as things escalate to increasingly intense levels, involving eg national governments and militaries and highly-polarized political movements and etc.
- Obviously if some organization was hypercompetent and super-good at behind-the-scenes persuasion, we wouldn't really know about it! So it is hard to 100% confidently dismiss the idea that maybe the CIA has next-gen persuasion tech, or whatever.
- Obviously we are already, to a large extent, living in a world that is shaped by the "marketplace of ideas", where the truth often gets outcompeted by whatever sounds best / is most memetically fit. Thinking about these dynamics (even without anything AI-related or any CIA conspiracies) is confusing, but seems very important. Eg, I myself have been deeply shaped by the crazy memetic landscape in ways that I partially endorse and partially don't. And everything I might try to do to achieve impact in the world needs to navigate the weird social landscape of human society, which in many respects is in a kind of memetic version of the "equilibrium of no free energy" that yudkowsky talks about in Inadequate Equilibria (although there he is talking mostly about an individual-incentives landscape, rather than a memetic landscape).
- AI super-persuasion does seem like something we might plausibly get before we get general ASI, which seems like it could be extremely weird / dangerous / destabilizing.
That said, I think this post is too conspiratorial in assuming that some combination of social media companies / national governments understand how to actually deploy effective persuasion techniques in a puppetmaster-like way which is way beyond everyone else. I think that the current situation is more like "we are living in an anarchic world influenced by an out-of-control memetic marketplace of ideas being influenced by many different actors of varying levels of sophistication, none of whom have amazing next-level gameboard-flipping dominance". Some scattered thoughts on this theme:
- If the CIA (or other entities affiliated with the US government, including tech companies being pressured by the government) is so good at persuasion ops, why are there so many political movements that seem to go against the CIA's interests? Why hasn't the government been able to use its persuasion jiujitsu to neutralize wokeism and Trump/MAGA-ism? From an establishment perspective, both of these movements seem to be doing pretty serious damage to US culture/institutions. Maybe these are both in the process of being taken down by "clown attacks" (although to my eye, this looks less like an "attack" from CIA saboteurs, and more like a lot of genuine ordinary people in the movement themselves just being dumb / memetic dynamics playing out deterministically via social dynamics like yudkowsky's "evaporative cooling of group beliefs")? Or maybe ALL of, eg, wokeism, is one GIANT psy-op to distract the American people from creating a left-wing movement that is actually smart and effective? (I definitely believe something like this, but I don't believe it's a deliberate military psy-op... rather it's an emergent dynamic. Consider how corporations are differentially friendlier to wokeism than they are to a more economically-focused, class-based Bernie-ism, so wokeism has an easier time spreading and looking successful, etc. It also helps that wokeism is memetically optimized to appeal to people in various ways, versus a genuinely smart-and-effective left-wing policy idea like Georgism comes off as boring, technocratic, and hard-to-explain.)
- Basically, what I am saying is that our national politics/culture looks like the product of anarchic memetic optimization (recently turbocharged by social media dynamics, as described by folks like Slate Star Codex and the book "Revolt of the Public") much moreso than the product of top-down manipulation.
- If google & facebook & etc are so good at manipulating me, why do their efforts at influence often still seem so clumsy? Yes, of course, I'm not going to notice the non-clumsy manipulations! And yes, your "I didn't speak up, because I wasn't as predictable as the first 60%" argument certainly applies here -- I am indeed worried that as technology progresses, AI persuasion tech will become a bigger and bigger problem. But still, in the here and now, Youtube is constantly showing me these ridiculous ideological banners about "how to spot misinformation" or "highlighting videos from Black creators" or etc... I am supposed to believe that these people are some kind of master manipulators? (They are clearly just halfheartedly slapping the banners on there in a weak attempt to cover their ass and appease NYT-style complaints that youtube's algorithm is unintentionally radicalizing people into trumpism... they aren't even trying to be persuasive to the actual viewers, just hamfistedly trying to look good to regulators...)
- Where is the evidence of super-persuasion techniques being used by other countries, or in geopolitical situations? One of the most important targets here would be things like "convincing Taiwanese to identify mostly as ethnic Chinese, or mostly as an independent nation", or the same for trying to convince Ukrainians to align more with their Russian-like ethnicity and language or with the independent democracies of western Europe. Ultimately, the cultural identification might be the #1 decisive factor in these countries' futures, and for sure there are lots of propaganda / political messaging attempts from all sides here. But nobody seems like they have some kind of OP superweapon which can singlehandedly change the fate of nations by, eg, convincing Taiwanese people of something crazy, like embracing their history as a Japanese colony and deciding that actually they want to reunify with Japan instead of remaining independent or joining China!
- Similarly, the Russian attempts to interfere in the 2016 election, although initially portrayed as some kind of spooky OP persuasion technique, ultimately ended up looking pretty clumsy and humdrum and small-scale, eg just creating facebook groups on themes designed to inflame American cultural divisions, making wacky anti-Hillary memes, etc.
- China's attempts at cultural manipulation are probably more advanced, but they haven't been able to save themselves from sinking into a cultural atmosphere of intense malaise and pessimism, one of the lowest fertility rates in the world, etc. If persuasion tech was so powerful, couldn't China use it to at least convince people to keep plowing more money into real estate?
- Have there been any significant leaks that indicate the USA is focused on persuasion tech and has seen significant successes with it? If I recall correctly, the Edward Snowden leaks (admittedly from the NSA which focuses on collecting information, and from 10 years ago) seemed to mostly indicate a strategy of "secretly collect all the data" --> "search through and analyze it to identify particular people / threats / etc". There didn't seem to be any emphasis on trying to shape culture more broadly.
- Intelligence agencies in the USA devote some effort to "deradicalization" of eg islamist terrorists, extreme right-wingers, etc. But this stuff seems to be mostly focused on pretty narrow interventions targeting individual people or small groups, and seems mostly based on 20th-century-style basic psychological understanding... seems like a far cry from A/B testing the perfect social-media strategy to unleash on the entire population of some middle-eastern country to turn them all into cosmopolitan neoliberals.
Anyways, I guess my overall point is that it just doesn't seem true that the CIA, or Facebook, or China, or anyone else, currently has access to amazing next-gen persuasion tech. So IMO you are thinking about this in the wrong way, with too much of a conspiratorial / Tom Clancy vibe. But the reason I wrote such a long comment is because I think you should keep exploring these general topics, since I agree with you about most of the other assumptions you are making!
- We are already living in a persuasion-transformed world in the sense that the world is full of a lot of crazy ideas which have been shaped by memetic dynamics
- Social media in particular seems like a powerful lever to influence culture (see Slate Star Codex & Revolt of the Public)
- It seems like you probably COULD influence culture a ton by changing the design of social media, so it's a little funny that nobody seems to be intentionally using this to build a persuasion superweapon
- (Nevertheless I think nobody really understands the long-term cultural effects of social media well enough to make deliberate changes to achieve eventual intended results. And I think there are limits to what you could do with current techniques -- changing the design & policies of a site like Twitter might change the broad cultural vibe, but I don't think we could create an especially persuasive superweapon that could be aimed at particular targets, like making Taiwanese people culturally identify with Japan)
- It definitely seems like AI could be used for all kinds of censorship & persuasion-related tasks, and this seems scary because it might indeed allow the creation of persuasion superweapons.
- Totally separately from all the above stuff about persuasion, the shadowier parts of governments (military & intelligence-agency bureaucracies) seem very important to think about when we are trying to think ahead about the future of AI technology and human civilization.
Thanks for writing this post, I 100% share your sentiment and appreciate the depth with which you've explored this topic, including some of the political considerations.
Here are some other potentially-relevant case studies of people doing similar-ish things, trying to make the world a better place while navigating touchy political fears related to biotech:
- The "Enhanced Games" is organizing an alternative to the Olympic games where doping and other human enhancement technologies will be allowed. Naturally, they try to put a heavy emphasis on the importance of human freedom, plus some random criticisms of the way the existing Olympics are organized. But what I thought was especially striking was the way they leaned into social-justice-y rhetoric: "science is real", "stop exploitation", and even their logo is a dead ringer for the famous "equality" logo of the LGBT rights movement. For an antimalarial gene drive, I think a similar approach could work well (at least for winning the support of Westerners) -- leaning into rhetoric about postcolonialism and how the gene-drive initiative represents the people taking charge of their own destiny instead of waiting for western aid/charity (bednets, vaccines, etc) that hasn't been sufficient. (Charter Cities are in a somewhat similar situation, where it's very important for them to convince people that this is an empowering way of advancing human liberty while helping the developing world, rather than some kind of conniving neocolonialism intended to disempower people.)
- The C4 Rice Project has been operating for a long time, working towards the dream of engineering rice to more efficiently photosynthesize and thus boosting yields around the world.
- The Far-Out Initiative is hoping to trial gene editing to reduce animal suffering; their website has some interesting FAQs and in general the project has the same "activist genetics lab" vibe that a mosquito gene-drive lab might strive for.
- Same deal for the project to revive Woolly Mammoths -- the awesome documentary "We Are As Gods" is basically a PR campaign for the righteousness of this cause, and a good portrait of a similar movement which is farther along in the PR pipeline.
- Genomic embryo-selection companies like Lifeview and Orchid Health are also interesting in this context, although since they don't have to convince regulators or the wider public, they are keeping a lower profile for now. There are also some essentially stealth-mode groups who are investigating enhancements to the current state-of-the-art in embryo selection. These groups would be less interesting in terms of learning from their PR campaigns (they have none), but it might be helpful to study how they build a skilled team, raise funding, etc.
Some further questions I have about the political and theory-of-change considerations:
- I think it could be helpful to explore a more detailed breakdown of who exactly might be opposed, and for what reasons. And then try and figure out which of these sources actually matter the most / are the most real! For example:
- Maybe the people of a developing country will be opposed, because they just find GMOs scary / would be worried about being bitten by a GMO mosquito.
- Maybe neighboring countries will be opposed because they'll see it as an imposition on their sovereignty that an important decision (even if the decision is just... curing malaria, lol) is being taken without their input.
- Ordinary citizens and activists in the west might be opposed mostly because of some kind of "neocolonialism"/social-justice concerns, or mostly because of environmental concerns (removing mosquitoes might disrupt the environment), or mostly because of FDA-style biomedical caution and fear of GMOs.
- Elites in the developed world might be concerned from a perspective of international diplomacy and norms -- sure, THIS unilateral genetic engineering project will have amazing consequences, but will it end up net-negative if it encourages OTHER unilateral actions in the future that are more harmful? (Feels similar to the sentiment against climate geoengineering or human genetic editing.) What could be done to ameliorate this concern?
- Is there some way that a gene drive could be framed as partially accidental, or an inevitable by-product of some other necessary action? Sometimes you need a good excuse to do something good for the world... I am thinking of situations like:
- When an accidental freezer failure during covid vaccine distribution actually resulted in giving lots more people the vaccine, because it helped doctors get around onerous verification requirements about who was allowed to get one.
- Geoengineering experiments are still very taboo, but a recent UN regulation reducing sulfur dioxide emissions is unintentionally warming the earth and also giving scientists tons of data about the efficacy of future SO2-based climate interventions.
- Similarly, fertilizing the oceans with iron to fight climate change is considered taboo. But it might be easier to argue for fertilizing the oceans with iron in order to help sperm whale populations recover, since sperm whales once naturally helped iron cycle through the oceans but our own human actions caused their populations to decline. (Fighting climate change would thus just be an "accidental" benefit of helping the sperm whales.)
- In the malaria gene-drive case, the best possible headline is probably always gonna be "unanimous international agreement reached to release gene drive!" But as a second-best option, "truck carrying mosquitoes for gene-drive study crashes, thousands of GMO mosquitoes released, scientists very apologetic!" is DEFINITELY preferable to "rogue bio-activists publish manifesto, release thousands of GMO mosquitoes". And it might even be preferable to something like "president of Ghana, citing failure of western aid and the hypocrisy of colonial powers, unilaterally begins GMO drive".
- I'd also note that, although most prevalent in africa, malaria and other mosquito-borne diseases are common throughout the tropics. So although africa might be ideal from an impact standpoint, getting a project off the ground in southeast asia or central america is also worth considering, if the politics are more favorable / if it would be easier to set up a genetics lab there!
Yeah, I am interested in this from the "about to have an infant" perspective (my wife is almost 20 weeks pregnant). Interestingly this means she will be able to get both the flu, covid, and newly-approved RSV shot.
- Presumably you want to space out the vaccines a lot -- I would guess two weeks at least, but maybe more?
- Is there a difference between when covid, flu, and RSV peak in activity, which might justify getting one before the other? (The RSV vaccine is apparently only approved for weeks 32 - 36 of pregnancy, so we will at least have to wait at least another 12 weeks I guess, which annoyingly takes us all the way past the holidays.)
- Like you say, I am thinking that earlier is better (rather "play it safe" and have some immunity, even if it later wears off), so she has already gotten her flu shot. (Does flu or covid immunity wane faster?)
- I think part of the reason RSV is only approved for the third trimester is to transfer some immunity to the child, so that the newborn is protected in its first months of life. Presumably that logic applies less for influenza (which is not particularly severe in newborns) or covid (which seems especially mild in children)?
Good point that rationalism is over-emphasizing the importance of Bayes theorem in a pretty ridiculous way, even if most of the individual statements about Bayes theorem are perfectly correct. I feel like if one was trying to evaluate Eliezer or the rationalist community on some kind of overall philosophy scorecard, there would be a lot of situations like this -- both "the salience is totally out of whack here even though it's not technically /wrong/...", and "this seems like a really important and true sentiment, but it's not really the kind of thing that's considered within the purview of academic philosophy..." (Such as the discussion about ethics / morality / value, and many other parts of the Sequences... I think there is basically a lot of helpful stuff in those posts, some of which might be controversial, but it isn't really an Official Philosophical Debate over stuff like whether anti-realism is true. It's more like "here's how I think you should live your life, IF anti-realism is true".)
Didn't mention many-worlds because it doesn't feel like the kind of thing that a philosopher would be fully equipped to adjudicate? I personally don't feel like I know enough to have opinions on different quantum mechanics interpretations or other issues concerning the overall nature / reality of the universe -- I still feel very uncertain and confused about that stuff, even though long ago I was a physics major and hoped to some day learn all about it. Although I guess I am sorta more sympathetic to Many Worlds than some of the alternatives?? Hard to think about, somehow...
Philosophers having hot takes on linguistics and the relationship between words and concepts -- not good or bad that they have so many takes, and I'm also not sure if the takes themselves are good or bad. It is just my impression that, unlike some of the stuff above, philosophy seems to have really spent a lot of time debating these issues, and thus it would be ripe for finding well-formed disagreements between EY and various mainstream schools of thought. I do think that maybe philosophers over-index a little on thinking about the nature of words and language (ie that they have "too many takes"), but that doesn't seem like such a bad thing -- I'm glad somebody's thinking about it, even if it doesn't strike me as the most important area of inquiry!
Yeah, agreed that that Solomonoff induction argument feels very bizzarre! I had never encountered that before. I meant to refer to the many different arguments for atheism sprinkled throughout the Sequences, including many references to the all-time classic idea that our discovery of the principles of evolution and the mechanics of the brain are sufficient to "explain away" the biggest mysteries about the origin of humanity, and should thus sideline the previously-viable hypothesis of religious claims being true. (See here and here.) EY seems to (rightly IMO) consider the falseness of major religious claims to be a "slam dunk", ie, totally overdetermined to be false -- the Sequences are full of funny asides and stories where various religious people are shown to be making very obvious reasoning errors, etc.
Some other potentially controversial views that a philosopher might be able to fact-check Eliezer on, based on skimming through an index of the sequences:
- Assorted confident statements about the obvious supremacy of Bayesian probability theory and how Frequentists are obviously wrong/crazy/confused/etc. (IMO he's right about this stuff. But idk if this counts as controversial enough within academia?)
- Probably a lot of assorted philosophy-of-science stuff about the nature of evidence, the idea that high-caliber rationality ought to operate "faster than science", etc. (IMO he's right about the big picture here, although this topic covers a lot of ground so if you looked closely you could probably find some quibbles.)
- The claim / implication that talk of "emergence" or the study of "complexity science" is basically bunk. (Not sure but seems like he's probably right? Good chance the ultimate resolution would probably be "emergence/complexity is a much less helpful concept than its fans think, but more helpful than zero".)
- A lot of assorted references to cognitive and evolutionary psychology, including probably a number of studies that haven't replicated -- I think Eliezer has expressed regret at some of this and said he would write the sequences differently today. But there are probably a bunch of somewhat-controversial psychology factoids that Eliezer would still confidently stand by. (IMO you could probably nail him on some stuff here.)
- Maybe some assorted claims about the nature of evolution? What it's optimizing for, what it produces ("adaptation-executors, not fitness-maximizers"), where the logic can & can't be extended (can corporations be said to evolve? EY says no), whether group selection happens in real life (EY says basically never). Not sure if any of these claims are controversial though.
- Lots of confident claims about the idea of "intelligence" -- that it is a coherent concept, an important trait, etc. (Vs some philosophers who might say there's no one thing that can be called intelligence, or that the word intelligence has no meaning, or generally make the kinds of arguments parodied in "On the Impossibility of Supersized Machines". Surely there are still plenty of these philosophers going around today, even though I think they're very wrong?)
- Some pretty pure philosophy about the nature of words/concepts, and "the relationship between cognition and concept formation". I feel like philosophers have a lot of hot takes about linguistics, and the way we structure concepts inside our minds, and so forth? (IMO you could at least definitely find some quibbles, even if the big picture looks right.)
- Eliezer confidently dismissing what he calls a key tenet of "postmodernism" in several places -- the idea that different "truths" can be true for different cultures. (IMO he's right to dismiss this.)
- Some pretty confident (all things considered!) claims about moral anti-realism and the proper ethical attitude to take towards life? (I found his writing helpful and interesting but idk if it's the last word, personally I feel very uncertain about this stuff.)
- Eliezer's confident rejection of religion at many points. (Is it too obvious, in academic circles, that all major religions are false? Or is this still controversial enough, with however many billions of self-identified believers worldwide, that you can get credit for calling it?)
- It also feels like some of the more abstract AI alignment stuff (about the fundamental nature of "agents", what it means to have a "goal" or "values", etc) might be amenable to philosophical critique.
Maybe you toss out half of those because they aren't seriously disputed by any legit academics. But, I am pretty sure that at least postmodern philosophers, "complexity scientists", people with bad takes on philosophy-of-science / philosophy-of-probability, and people who make "On the Impossibility of Supersized Machines"-style arguments about intelligence, are really out there! They at least consider themselves to be legit, even if you and I are skeptical! So I think EY would come across with a pretty good track record of correct philosophy at the end of the day, if you truly took the entire reference class of "controversial philosophical claims" and somehow graded how correct EY was (in practice, since we haven't yet solved philosophy -- how close he is to your own views?), and compared this to how correct the average philosopher is.
I suggest maybe re-titling this post to:
"I strongly disagree with Eliezer Yudkowsky about the philosophy of consciousness and decision theory, and so do lots of other academic philosophers"
or maybe:
"Eliezer Yudkowsky is Frequently, Confidently, Egregiously Wrong, About Metaphysics"
or consider:
"Eliezer's ideas about Zombies, Decision Theory, and Animal Consciousness, seem crazy"
Otherwise it seems pretty misleading / clickbaity (and indeed overconfident) to extrapolate from these beliefs, to other notable beliefs of Eliezer's -- such as cryonics, quantum mechanics, macroeconomics, various political issues, various beliefs about AI of course, etc. Personally, I clicked on this post really expecting to see a bunch of stuff like "in March 2022 Eliezer confidently claimed that the government of Russia would collapse within 90 days, and it did not", or "Eliezer said for years that X approach to AI couldn't possibly scale, but then it did".
Personally, I feel that beliefs within this narrow slice of philosophy topics are unlikely to correlate to being "egregiously wrong" in other fields. (Philosophy is famously hard!! So even though I agree with you that his stance on animal consciousness seems pretty crazy, I don't really hold this kind of philosophical disagreement against people when they make predictions about, eg, current events.)
This video is widely believed to be a CGI fake.
I think "Why The West Rules", by Ian Morris, has a pretty informative take on this. The impression I got from the book was that gradually accruing technologies/knowledge, like the stuff you mention, is slowly accruing in the background amid the ups and downs of history, and during each peak of civilizational complexity (most notably the Roman empire, and the medieval-era Song Dynasty in china, and then industrial-era Britain) humanity basically gets another shot-on-goal to potentially industrialize.
Britain had a couple of lucky breaks -- cheap and abundant coal, but also the way that the industrial revolution was preceded by the growth-boosting effects of transatlantic contact. (For instance getting all kinds of new foods from the New World, like potatoes, which then boost agricultural output across Europe. And the ways that putting on big risky trans-oceanic expeditions incentivized the development of corporate structures / capitalism / etc. And just having a century of positive-sum growth puts everyone in a good frame of mind in terms of being motivated and risk-tolerant and seeking opportunities, rather than there being tons of conflict.)
So, in that sense, maybe the industrial revolution "came early" due to these lucky breaks, and otherwise would have been delayed for another century until, say, Germany or the USA started digging up coal (or oil, or some other energy source) even if Britain didn't have any.
Unrelated: fellow fans of Morris's book might appreciate my parody blog post "Why The East Rules -- For Now".
Thanks! Apparently I am in a mood to write very long comments today, so if you like, you can see some thoughts about addressing potential objections / difficulties in a response I made to a comment on the EA Forum version of this post.
Thanks for catching that about Singaporeans!
Re: democracy, yeah, we debated how exactly to phrase this. People were definitely aware of the democracies of ancient Greece and Rome, and democracy was sometimes used on a local level in some countries, and there were sometimes situations where the nobles of a country had some sway / constraints over the king (like with the Magna Carta). But the idea of really running an entire large country on American-style democracy seems like it was a pretty big step and must've seemed a bit crazy at the time... IMO, it would seem as least as crazy as of like if a large country today (like, say, Chile after it voted to rewrite its constitution, or a new and more-united version of the European Union, or a future post-Putin Russia trying to reform itself) did something like:
- Deciding to try out direct democracy, where instead of a Senate or Parliament, legislation would be voted on directly by the people via a secure smartphone app.
- Deciding to try out prediction-market-based governance, where economic policy was automatically adjusted in order to maximize some national GDP-like metric according to the principles of "futarchy".
- Deciding that they would select their political leaders using the same method as medieval Venice used to select their Doge. ("Thirty members of the Great Council, chosen by lot, were reduced by lot to nine; the nine chose forty and the forty were reduced by lot to twelve, who chose twenty-five. The twenty-five were reduced by lot to nine, and the nine elected forty-five. These forty-five were once more reduced by lot to eleven, and the eleven finally chose the forty-one who elected the doge.") And maybe to base a bunch of other parts of their political system off of random selection ("sortition") -- not just jury members in trials but also members of parliament, or using sortition to poll a random 1% of the population about important issues instead of having everyone vote on issues, etc.
Probably the charter city with the most publicity is Prospera, so you could do stuff like:
- read a bunch of hostile news articles complaining about how Prospera is neocolonialism and might be secretly hoping to confiscate people's land
- read stuff put out by the Prospera organization about how they are actually fanatics about the importance of property rights and would never confiscate anyone's land, and how in general they are trying to be responsible and nice and create lots of positive externalities for neighboring communities (jobs, construction, etc)
- read analysis by interested people (including numerous rationalists) who have visited Prospera, etc, which tends to be pretty sympathetic to Prospera and thinks they are upstanding people trying to do cool stuff
But idk if it's worth going on that journey since it's something that a lot of other people have done before (such as myself -- I came away thinking that Prospera is doing great stuff and their critics are being extremely uncharitable / ill-intentioned. the one possible ding against Prospera IMO is that in addition to their main site on Roatan which is the cool libertarian city-state in the making, they are also using the ZEDE law to create an import/export center on the mainland, called La Ceiba, which seems less like an amazing innovative experiment in state-of-the-art governance and more like just an ordinary Special Economic Zone where lower taxes encourage freer trade. Personally I think freer trade is probably good for Honduras, but if you like protectionism then you might not like the idea of special economic zones whose main benefit is a somewhat lower tax rate).
Anyways, if you are interested, it would probably produce a lot more social value to investigate some other, lesser-known charter cities and report back with your thoughts. There are two other projects in Honduras under the "ZEDE" charter city law -- "Ciudad Morzan", which seems like an effort to basically create a gated community -- a neighborhood in crime-ridden Honduras where workers can opt into a higher standard of policing in exchange for presumably higher local taxes to fund the police, and maybe some restrictions on activity like nightly curfews (idk if Ciudad Morzan has these... just brainstorming). Seems like a nice option for working-class Hondurans to have, IMO, but maybe if I looked into it more closely I'd come away with a worse impression. And then there is "Orqueda", which seems straightforwardly like a large business exploiting the ZEDE law simply in order to pay lower taxes or otherwise getting out of Honduran regulations, without really experimenting with any cool new governance institutions or trying to create an awesome new city where lots of people might like to live.
But there are lots and lots of new-city projects throughout the world -- as I mention in the draft, new cities aren't that unusual in quickly-urbanizing developing countries. Some projects, like Saudi Arabia's "NEOM", seem like poorly-concieved vanity megaprojects that will probably come bundled with human rights abuses (and which have indeed been condemned by rationalists like Scott Alexander / Astral Codex Ten). Others are just blander and lower-profile since they aren't shooting for the same kind of broad regulatory independence that places like Prospera or Itana are hoping for. See this "Startup Cities Map" (the green dots, not the blue or purple ones) for a directory of interesting projects: https://www.startupcitiesmap.com/map
Personally I would be kind of interested in finding out what the heck is the deal with Telosa -- this is a new-city project funded by a billionare in the United States, seemingly around an economic philosophy that combines Georgism (which I love!!) with some vague but pretty strong lefty / egalitarian / social-justice vibes (which I kinda don't really understand in terms of how this is supposed to influence the design and government of the city, but whatever). Is there some special angle here beyond the surface presentation? Who is the guy funding it and how did he become such a huge Georgist that he wanted to use his life's fortune to build a new city on these principles? Why not just use the money to lobby for more lefty & Georgist policy changes like a normal person, instead of building a new city in the desert? etc.
I think one problem with this concept is that the "restrictions" might turn out to be very onerous, preventing the good guys (using "restrictions) from winning a complete unilateral victory over everyone else. One of the major anticipated benefits of superhuman AI systems is the ability to work effectively even on vague, broad, difficult tasks that span multiple different domains. If you are committed to creating a totally air-gapped high-security system, where you only hand your AI the "smallest subdividable subtask" and only giving your AI access to a small amount of sanitized context, probably you will end up losing (militarily, economically, whatever) to someone who uses AI in a less-restricted context (even if their AI model is somewhat worse).
So, it seems like, if you think alignment is impossible and the "restriction" path is the only way, you shouldn't be imagining getting lots of AI help (combat drones, etc), since in any scenarios where you've got AI help, your non-restrictionist opponents probably have EVEN MORE AI help. So you might as well just launch your global takeover today, when AI is weak, since your military/economic/etc advantage will probably only erode with every advance in AI capabilities.
[Cross-posting my comment from the EA Forum]
This post felt vague and confusing to me. What is meant by a "game board" -- are you referring to the world's geopolitical situation, or the governance structure of the United States, or the social dynamics of elites like politicians and researchers, or some kind of ethereum-esque crypto protocol, or internal company policies at Google and Microsoft, or US AI regulations, or what?
How do we get a "new board"? No matter what kind of change you want, you will have to get there starting from the current situation that the world is in right now.
Based on your linked post about consensus mechanisms, let's say that you want to create some crypto-esque software that makes it easier to implement "futarchy" -- prediction-market based government -- and then get everyone to start using that new type of government to make wiser decisions, which will then help them govern the development of AI in a better, wiser way. Well, what would this crypto system look like? How would it avoid the pitfalls that have so far prevented the wide adoption of prediction markets in similar contexts? How would we get everyone to adopt this new protocol for important decisions -- wouldn't existing governments, companies, etc, be hesitant to relinquish their power?
Since there has to be some realistic path from "here" to "there", it seems foolish to totally write off all existing AI companies and political actors (like the United States, United Nations, etc). It makes sense that it might be worth creating a totally-new system (like a futarchy governance platform, or whatever you are envisioning) from scratch, rather than trying to influence existing systems that can be hard to change. But for your idea to work, at some point the new system will have to influence / absorb / get adopted by, the existing big players. I think you should try think about how, concretely, that might happen. (Maybe when the USA sees how amazing the new system is, they states will request a constitutional convention and change to the new system? Maybe some other countries will adopt the system first, and this will help demonstrate its greatness to the USA? Or maybe prediction markets will start out getting legalized in the USA for commercial purposes, then slowly take on more and more governance functions? Or maybe the plan doesn't rely on getting the new system enmeshed in existing governance structures, rather people will just start using this new system on their own, and eventually this will overtake existing governments, like how some Bitcoiners dream of the day when Bitcoin becomes the new reserve currency simply via lots of individual people switching?)
Some other thoughts on your post above:
- In your linked post about "Consensus Mechanisms", one of the things you are saying is that we should have a prediction market to help evaluate which approaches to alignment will be most likely to succeed. But wouldn't the market be distorted by the fact that if everyone ends up dead, there is nobody left alive to collect their prediction-market winnings? (And thus no incentive to bet against sudden, unexpected failures of promising-seeming alignment strategies?) For more thinking about how one might set up a market to anticipate or mitigate X-risks, see Chaining Retroactive Funders to Borrow Against Unlikely Utopias and X-Risk, Anthropics, & Peter Thiel's Investment Thesis.
- It seems to me that having a prediction market for different alignment approaches would be helpful, but would be VERY far from actually having a good plan to solve alignment. Consider the stock market -- the stock market does a wonderful job identifying valuable companies, much better than soviet-style central planning systems have done historically. But stock markets are still frequently wrong and frequently have to change their minds (otherwise they wouldn't swing around so much, and there would never be market "bubbles"). And even if stock markets were perfect, they only answer a small number of big questions, like "did this company's recent announcement make it more or less valuable"... you can't run a successful company on just the existence of the stock market alone, you also need employees, managers, executives, etc, doing work and making decisions in the normal way.
I feel like we share many of the same sentiments -- the idea that we could improve the general level of societal / governmental decisionmaking using innovative ideas like better forms of voting, quadratic voting & funding, prediction markets, etc. Personally, I tried to sketch out one optimistic scenario (where these ideas get refined and widely adopted across the world, and then humanity is better able to deal with the alignment problem because we have better decisionmaking capability) with my entry in the Future of Life Institute's "AI Worldbuilding" challenge. It imagines an admittedly utopian future history (from 2023-2045) that tries to show:
- How we might make big improvements to decisionmaking via mechanisms like futarchy and liquid democracy, enhanced by Elicit-like research/analysis tools.
- How changes could spread to many countries via competition to achieve faster growth than rivals, and via snowball effects of reform.
- How the resulting, more "adequate" civilization could recognize the threat posed by alignment and coordinate to solve the problem.
I think the challenge is to try and get more and more concrete, in multiple different ways. My worldbuilding story, for instance, is still incredibly hand-wave-y about how these various innovative ideas would be implemented IRL (ie, I don't explain in detail why prediction markets, land value taxes, improved voting systems, and other reform ideas should suddenly become very popular after decades of failing to catch on), and what exact pathways lead to the adoption of these new systems among major governments, and how exactly the improved laws and projects around AI alignment are implemented.
The goal of this kind of thinking, IMO, should be to eventually come up with some ideas that, even if they don't solve the entire problem, are at least "shovel-ready" for implementation in the real world. Like, "Okay, maybe we could pass a ballot measure in California creating a state-run oversight commission that coordinates with the heads of top AI companies and includes the use of [some kind of innovative democratic inputs as gestured at here by OpenAI] to inform the value systems of all new large AI systems trained by California companies. Most AI companies are already in California to start with, so if it's successful, this would hopefully set a standard that could eventually expand to a national or global level..."
For those who prefer listening, note that there is a very nice recording of this speech, which you can view / listen to here on youtube!