grey goo is unlikely

Not an expert in chemistry or biochemistry, but this post seems to basically not engage with the feasibility studies Drexler has made in Nanosystems, and makes a bunch of assertions without justification, including where Nanosystems has counterarguments. I wish more commenters would engage on the object level because I really don't have the background to, and even I see a bunch of objections. Nevertheless I'll make an attempt. I encourage OP and others to correct me where I am ignorant of some established science.

Points 1, 2, 3, 4 are not relevant to Drexlerian nanotech and seem like reasonable points for other paradigms.

Regarding 5, my understanding is that mechanosynthesis involves precise placement of individual atoms according to blueprints, thus making catalysts that selectively bind to particular molecules unnecessary.

6. no liquid
Any self-replicating nanobot must have many internal components. If the interior is not filled with water, those components will clump together and be unable to move around, because electrostatic & dispersion interactions are proportionately much stronger on a small scale. The same is true to a lesser extent for the nanobots themselves.

Vacuum is even worse. Any self-replicating cell must move material between outside and multiple compartments. Gas leakage by the transporters would be inevitable. Cellular vacuum pumps would require too much energy and may be impossible. Also, strongly binding the compounds used (eg CO2) to carriers at every step would require too much energy. ("Too much energy" means too much to be competitive with normal biological processes.)

• If your components are all fixed in place by covalent bonds, they can't clump together. 
• Nanosystems section 11.4 gives arguments against gas leakage being inevitable, and a proposed "turbomolecular pump" design.
• It's not clear why a cellular vacuum pump must have a low efficiency, and the actual work (P*V) that must be done by a vacuum pump should be well below the kcal/gram range that biological cells need to replicate.

7 is not relevant given that I'm imagining hard vacuum systems.

8. high temperatures

[...] Enzymes need to be able to change shape somewhat. Without conformational changes, enzymes can't grab their substrate well enough. Without conformational changes, there's no way to drive an unfavorable reaction with a favorable reaction, and that's necessary.

Because enzymes must be able to do conformational changes, they need to have some strong interactions and some weaker interactions that can be broken or shifted. Those weaker interactions can't hold molecules together at high temperatures. Some life can grow at 100 C but 200 C isn't possible.

If the "enzymes" are not made of proteins, their weaker interactions can be stronger than the hydrogen bonds that hold proteins together. If they're roughly twice as strong (say by using covalent bonds or twice as many hydrogen bonds), they wouldn't denature even at 200C.

Also, there is no way to later remove carbon atoms from the diamond at low temperature. How, then, would a nanobot with a diamond shell replicate?

The shell could be made of diamond panels with airtight joints. The daughter cell's internal components and membrane are manufactured inside the parent cell, then the membrane is added to the parent cell's membrane, it unfolds in an origami fashion into two membranes of original size, then the daughter cell separates.

This is a pretty obvious idea to me, so unless there's some obvious reason why something like this doesn't work, I get the feeling that the post isn't engaging with the strongest arguments.

10-13 seem pretty reasonable to me.

14. positional nanoassembly

[...] Protein-sized position sensors don't exist.

Molecular linear motors do exist, but 1 ATP (or other energy carrier) is needed for every step taken.

If you want to catalyze reactions, you need floppy enzymes. Even if you attach them to a rigid bed, they'll flop all over the place. (On a microscopic scale, normal temperatures are like a macroscopic 3d printer being shaken violently.)

It seems to me that the "step" for molecular linear motors could be an arbitrarily long distance. The moving part randomly moves about the stator, and there are ratchets every, say, 100 nm that let the moving part pass in one direction when an ATP is consumed. Then when it needs to be fixed in position, a different mechanism does that.

The "floppy enzymes" has the same solution as section 8. In chapter 13 of Nanosystems Drexler also gives three different ways this problem is solved, two of which involve molecular manipulators:

Aside from differences of scale and component properties, molecular manipulators differ from macroscale devices in that they must maintain positional accuracy despite thermal excitation. This problem can be minimized either (1) by operation at reduced temperatures, which receives no further attention here; or (2) by the use of a stiff mechanism, as described in Section 13.4.1:or (3) by use of local nonbonded contacts to align reagent devices to workpieces immediately before reaction, as discussed in Section 13.4.2.

My current impression is that this post is perfectly consistent with the problem of making a self-replicating diamondoid bacterium being easy enough that unamplified humans could do it in <500 years given good research practices, software, and perhaps some narrow AI tools like some future version of AlphaFold. It's true that Drexlerian nanotech is not really optimized for being self-replicating nanomachines, as Drexler envisions it's most useful for mass manufacturing and computing. They might be less evolvable, consume more energy, or be specialized to certain environments like the atmosphere or ocean, especially when designed by human-level intelligences. But there are also potentially huge advantages like being undigestible to biological life and viruses, being impervious to more forms of damage, or a wider range of metabolic pathways. But it requires a different argument to say that either (a) nanotech is impossible, or (b) the disadvantages of nanotech-based life forms outweigh the advantages.

If I were in the mindset I get from this post, I would have a hard time not asserting that powered flight would be impractical for airplanes larger than birds, or that modern semiconductor manufacturing were impossible due to the precision required. I would probably deny other possibilities currently thought extremely plausible, like fusion energy and immortality. Maybe the author isn't making the same mistake, but I'm doubtful nonetheless.

↑ comment by bhauth · 2023-04-18T22:21:11.366Z · LW(p) · GW(p)

Regarding 5, my understanding is that mechanosynthesis involves precise placement of individual atoms according to blueprints, thus making catalysts that selectively bind to particular molecules unnecessary.

No, that does not follow.

The shell could be made of diamond panels with airtight joints. The daughter cell's internal components and membrane are manufactured inside the parent cell, then the membrane is added to the parent cell's membrane, it unfolds in an origami fashion into two membranes of original size, then the daughter cell separates.

...for one thing, that's not airtight.

It seems to me that the "step" for molecular linear motors could be an arbitrarily long distance.

No, the steps happen by diffusion so they become slower. That's why slower muscles are more efficient.

The "floppy enzymes" has the same solution as section 8. In chapter 13 of Nanosystems Drexler also gives three different ways this problem is solved, two of which involve molecular manipulators:

see this reply [LW(p) · GW(p)]

Replies from: thomas-kwa

↑ comment by Thomas Kwa (thomas-kwa) · 2023-04-19T19:17:37.160Z · LW(p) · GW(p)

I don't know how to engage with the first two comments. As for diffusion being slow, you need to argue that it's so slow as to be uncompetitive with replication times of biological life, and that no other mechanism for placing individual atoms / small molecules could achieve better speed and energy efficiency, e.g. this one.

I don't have the expertise to evaluate the comment by Muireall, so I made a Manifold market.

Replies from: bhauth

↑ comment by bhauth · 2023-04-20T02:50:01.096Z · LW(p) · GW(p)

Such actuator design specifics aren't relevant to my point. If you want to move a large distance, powered by energy from a chemical reaction, you have to diffuse to the target point, then use the chemical energy to ratchet the position. That's how kinesin works. A chemical reaction doesn't smoothly provide force along a range of movement. Thus, larger movements per reaction take longer.

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2024-08-14T12:02:19.100Z · LW(p) · GW(p)

Biological life uses an ATP system. This is an energy currency,  but it's discrete. Like having batteries that can only be empty or full. It doesn't give a good way to apply smaller amounts of energy than 1 atp molecule carries, even if less energy is needed.

Nanobots could have a continuous energy system, or smaller units of energy.

I want to remind everybody how efficient molecular machinery is in terms of thermodynamics:

this molecule [RNA] operates quite near the limit of thermodynamic efficiency [7 kcal/mol] set by the way it is assembled [~10 kcal/mol].

and

 these calculations also establish that the E. coli bacterium produces an amount of heat less than six times (220n_pep/42n_pep) as large as the absolute physical lower bound dictated by its growth rate, internal entropy production, and durability.

From an article Statistical Physics of Self-replication by Jeremy England

deriving a lower bound for the amount of heat that is produced during a process of self-replication in a system coupled to a thermal bath. We find that the minimum value for the physically allowed rate of heat production is determined by the growth rate, internal entropy, and durability of the replicator, and we discuss the implications of this finding for bacterial cell division, as well as for the pre-biotic emergence of self-replicating nucleic acids.

https://aip.scitation.org/doi/10.1063/1.4818538

That said I think that there may be many sweet spots for a combination of macroscopic and microscopic processes. Many industrial chemical processes are such combinations by providing very specialized baths of nutrients and substrates and combining efficient macroscopic flow and transport with microscopic chemical and organic reactions. But there may be more spots that allow for efficiently building up small-scale structures. 

In the context of AI x-risk, I’m mainly interested in

• (1) can an AI use nanotech as a central ingredient of a plan to wipe out humanity, and
• (2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?

[(2) is obviously possible once you have a few billion human-level-intelligent robots, but the question is “can nanotech dramatically reduce the amount of time that the AI is relying on human help, compared to that baseline?”. Presumably “being able to make arbitrarily more chips or chip-equivalents” would be the most difficult ingredient.]

In both cases it seems to me that the answer is “obviously yes”: 

• super-plagues / crop diseases / etc. are an existence proof for (1),
• human brains are an existence proof for (2).

Therefore grey goo as defined in this post doesn’t seem too relevant for my AI-related questions. Like, if the AI doesn’t have a plan to make nanotech things that can exterminate / outcompete microbes living in rocks deep under the seafloor—man, I just don’t care.

None of this is meant to be a criticism of this post, which I’m glad exists, even if I’m not in a position to evaluate it. Indeed, I’m not even sure OP would disagree with my comment here (based on their main AI post).

This might be the most erudite chemistry post [? · GW] ever on Less Wrong. @Eric Drexler [LW · GW] actually comments here on occasion; I wonder what he would have to say. 

I have been trying to sum up my own thoughts without getting too deeply into it. I think I would emphasize first that the capabilities of plain old DNA-based bacteria are already pretty amazing - bacteria already live everywhere from the clouds to our bloodstreams - and if one is worried about what malevolent intent can accomplish on the nanoscale, they already provide reason to be worried. And I think @bhauth [LW · GW] (author of this post) would agree with that. 

Now, regarding the feasibility of an alternative kind of nanobot, with a hard solid exterior, a vacuum interior, and mechanical components... All the physical challenges are real enough, but I'm very wary of supposing that they can't be surmounted. For example, synthesis of diamondoid parts might sound impossibly laborious; then one reads about "direct conversion of CO2 to multi-layer graphene", and thinks, could you have a little nano "sandwich maker" that fills with CO2 (purified by filter), has just the right shape and charge distribution on its inner surfaces to be a substrate for the formation of graphene, and which clamps shut when a particular current flows, opening afterwards to reveal a customized nanosheet that can become a tube or a layer or a surface... 

You're right that cells do all this stuff already, but stochastically, with self-assembled floppy parts in an aqueous environment, and so on. You could view the decades of discussion about mechanical nanotechnology, as a long theoretical study of whether and how rigid mechanical paradigms can also be introduced on the nanoscale. What I would say is that, if one is specifically interested in nanobots made of "diamondoid", one can also investigate the extent to which the biological paradigm can be imported into that world, to create hybrid designs which opportunistically use whichever approach works. 

I really like this post, I hope to see more like it on less wrong, and I strong-upvoted it. That said, let me now go through my thoughts on various points:

2. Rare materials: Yep, this is a real design constraint, but probably not that hard to design around? I'm not expecting nanobots to be made mostly out of iron.

3. Metal surfaces: Why not just build up the metal object in an oxygen-free environment, then add on an external passivation layer at then end? The passivation layer could be engineered to be more stable than the naturally occurring oxidation layer the metal would normally have. There would still be a minimum size for metal objects, of course. (Or more precisely, a minimal curvature.) Corrosion could definitely be a problem, but Cathodic protection might help in some cases.

4. Agree that electrostatic motors are the way to go here. I'm not sure the power supply necessarily has to be an ion gradient, nor that the motor would otherwise need to be made from metal. Metal might be actively bad, because it allows the electrons to slosh around a lot. What about this general scheme for a motor?: Wheel-shaped molecules, with sites that can hold electrons. A high voltage coming from a power supply deposits electrons on one wheel, and produces holes on the other wheel. The corresponding sites are attracted to each other and once they get close enough, the electrons jump into the holes, filling them. Switching is determined by the proximity of various sites on the wheels as they move relative to each other. Considering how electrons are able to jump around between sites in the electron transport chain, this doesn't seem impossible.

5. An intermediate design that I can imagine is a block with a series of tubes and chambers embedded in it. (The bulk of the block can hold electronics.) Most of the tubes are filled with water, so nanobot components can happily bounce around in them. But lots of components are also mounted to the walls of the tubes. You can't clump together if you're bonded to the wall of your tube. A small minority of tubes can be filled with gas, or even under vacuum, for any weird processes that may require those conditions. Pumping energy is volume times pressure, so the energy requirements could be reasonable as long as the volume is small.

6. Yeah, most reactions should probably still be done in water.

7. The reason for doing things at high temperatures is to do reactions with a high activation energy. If we're designing custom catalysts (artificial enzymes) for our nanobots, we can probably finesse it so that the enzyme coaxes the reactants into the high-energy intermediate state, even if the ambient temperature is low (via coupling to a more favorable reaction, for example). Also, covalent single-bonds can rotate, so there's nothing preventing the existence of a covalently bonded structure that can also exhibit conformational changes. Finally, I'm skeptical that existing life has found everything there is to be found here. Plenty of chemicals are useful to humans, but not useful to any living thing, and hence lack an evolved synthesis pathway. Also, I'd guess that the stupidity of evolution has left a lot of low hanging fruit for humans. For example, rather than trying to do a reaction with proteins, we can do it with a group of complicated catalysts synthesized by proteins.

8. I'd bet on diamond synthesis still being possible somehow, but it does seem like a genuinely complicated question, so I'll have to look into it further.

9. Yep, I'm not really even sure how useful diamond would be for lots of nano-stuff. Might mostly just be used for large structures the bots build, rather than being anywhere on the self-replication pathway for the bots. One difference is it's covalent rather than ionic, so there would probably be much less concern about that Ostwald ripening thing for parts made of diamond. So maybe there would be some uses for it because of that.

10. Doesn't have to be rigid, can still be connection based. For example, there could be simple protein-based building blocks that act like legos. An assembly head can assemble these, and move around on the surface of the part it's building by accepting signals that correspond to "move 1 block left", etc. Position is always exactly known, not because there's a rigid beam anywhere in the system, but because we know the exact integer number of steps the assembly head has moved since the start.

12, 13. ATP needs to be everywhere because it supplies energy. Okay, fair enough. Ribosomes need to be almost everywhere, since proteins are needed everywhere and then ribosomes come with their own host of supporting infrastructure like tRNAs and stuff, which have to be everywhere too. I'm being a little unfair to Eukaryotic cells here, which have lots of sophisticated stuff going on, but my general picture is that whenever a cell decides to make proteins somewhere and then transport them somewhere else, that costs genome space, which is very limited, so the cell can't do that very often. Nanobots genuinely have different constraints from life here, in particular they have cheaper genome space, and so they can have custom designed pipes for every type of protein they use, and the pipe leads right to the chamber where that protein is being used. Huge information cost, but if it makes things work way better, it's probably worthwhile for nanobots. I totally believe that life is using those techniques exactly as much as is optimal for it, though.

14. Mostly covered by my answer to 11 where the assembler head walks around on the surface and counts its steps. Larger structures can have many assembler heads working at once. I also don't know why you're scoffing at the potential for building computers here. The supposed "embodied" computation of existing cells is currently computing things that are keeping us alive, which is great, but you can't exactly solve any other important problems on it. It's not a flexible universal computer, in the sense of a Turing machine that can run any program.

This is nice to see, I’ve been generally kind of unimpressed by what have felt like overly generous handwaves re: gray gooey nanobots, and I do think biological cells are probably our best comparison point for how nanobots might work in practice.

That said, I see some of the discussion here veering in the direction of brainstorming novel ways to do harm with biology, which we have a general norm against in the biosecurity community – just wanted to offer a nudge to y’all to consider the cost vs. benefit [LW · GW] of sharing takes in that direction. Feel free to follow up with me over DM!

edit: (link)green goo is plausible [LW · GW]

The AI can kill us and then take over with better optimized biotech very easily.

• Doubling time for
  • Plants (IE:solar powered wet nanotech) > single digit days
  • Algae in ideal conditions 1.5 days
  • E. Coli 20 minutes
• There are piles of yummy carbohydrates lying around (Trees, plants, houses)
  • The AI can go full Tyranid
• The AI can re-use existing cellular machinery. No need to rebuild the photosynthesis or protein building machinery, full digestion and rebuilding at the amino acid level is wasteful.
  • Sub 2 minute doubling times are plausible for a system whose rate limiting step is mechanically infecting plants with a fast acting subversive virus. Spreading flying things are self replicators that steal energy+cellular machinery from plants during infection (IE:mosquito like). Onset time could be a few hours till construction of shoggoth like things. Full biosphere assimilation could be limited by flight speed.

Nature can't do these things since they require substantial non-incremental design changes. Mosquitoes won't simultaneously get plant adapted needles + biological machinery to sort incoming proteins and cellular contents + continuous grow/split reproduction that would allow a small starting population to eat a forest in a day. Nature can't design the virus to do post infection shoggoth construction either.

The only thing that even re-uses existing cellular machinery is viruses and that's because they operate on much faster evolutionary time scales than their victims. Evolution takes so long that winning strategies to eat or subvert existing populations of organisms are self-limiting. The first thing to sort of work wipes out the population and then something else not vulnerable fills the niche.

Design is much more powerful than evolution since individually useless parts can be developed to create a much more effective whole. Evolution can't flip the retina or reroute the recurrent laryngeal nerve even though those would be easy changes a human engineer could make.

Endgame biotech (IE: can design new proteins/DNA/organisms) is very powerful.

But that doesn't mean dry nanotech is useless.

• even if production is expensive it may be worth building some things that way anyways.
• computers
• structural components
  • Biology is largely stuck with ~0.15 Gpa materials (collagen, cellulose, chitin)
  • oriented UHMWPE should be wet synthesizeable (6 Gpa tensile strength)
  • graphene/diamondoid may be worth it in some places to hit 30 Gpa (EG:for things that fly or go to space)
• dry nanotech won't be vulnerable to parasites that can infect a biological system.
• even if the AI has to deal with single day doubling times that's still enough to cover the planet in a month.
• but with the right design parasites really shouldn't be a problem.
  • biological parasite defenses are not-optimal

I'm puzzled that this post is being upvoted. The author does not sound familiar with Drexler's arguments in NanoSystems.

I don't think we should worry much about how nanotech might affect an AI's abilities, but this post does not seem helpful.

Organic Life is Unlikely

(list of reasons why any kind of organic life ought to be impossible, which must to some extent actually be correct because the Fermi Observation shows that it is extremely rare)

I don't really think this approach of listing a bunch of problems is a way to get a high level of certainty about this. In a certain sense, you should treat this like a math problem and insist on a formal proof that nanotech is impossible starting from the Schrodinger Equation. And of course, such a proof would have the very difficult task of ruling out nanotech without ruling out actual bacteria.

self-replicating microscopic machines with some fundamental mechanistic differences from all biological life that make them superior

I think Wet Nanotech might qualify then.

Consider a minor modification to a natural microbe: a different genetic code. I.e., a codon still codes for an amino acid, but which corresponds to which could differ. (This correspondence is universal in natural life, with a few small exceptions.) Such an organism would effectively be immune to all of the viruses that would affect its natural counterpart, and no horizontal gene transfer to natural life would be possible.

One could also imagine further modifications. Greater resistance to mutations, perhaps using a more stable XNA and more repair genes. More types of amino acids. Reversed chirality of various biomolecules as compared to natural life, etc. Such an organism (with the appropriate enzymes) could digest natural life, but not the reverse.

There's nothing here that seems fundamentally incompatible with our understanding of biochemistry, but with enough of these changes, such an organism might then become an invasive species with a massive competitive advantage over natural life, ultimately resulting in an ecophagy scenario.

↑ comment by bhauth · 2023-04-17T04:53:17.207Z · LW(p) · GW(p)

That has already happened naturally and also already been done artificially.

See this paper for reasons why codons are almost universal.

Replies from: JenniferRM

↑ comment by JenniferRM · 2023-04-17T17:20:15.660Z · LW(p) · GW(p)

That third link seems to be full of woo.

Where was the optimization pressure for better designs supposed to have arisen in the "communal" phase?

Thus, we may speculate that the emergence of life should best be viewed in three phases, distinguished by the nature of their evolutionary dynamics. In the first phase, treated in the present article, life was very robust to ambiguity, but there was no fully unified innovation-sharing protocol. The ambiguity in this stage led inexorably to a dynamic from which a universal and optimized innovation-sharing protocol emerged, through a cooperative mechanism. In the second phase, the community rapidly developed complexity through the frictionless exchange of novelty enabled by the genetic code, a dynamic we recognize to be patently Lamarckian (19). With the increasing level of complexity there arose necessarily a lower tolerance of ambiguity, leading finally to a transition to a state wherein communal dynamics had to be suppressed and refinement superseded innovation. This Darwinian transition led to the third phase, which was dominated by vertical descent and characterized by the slow and tempered accumulation of complexity.

They claim that universal horizontal gene transfer (HGT) arose through a "cooperative" mechanism, without saying what that would have looked like at the level of cells, or at the level of some kind of soupy boundary-free chemostat, or something? 

They don't seem to be aware of compensatory mutations or quasi-species or that horizontal transfer is parasitic by default [LW · GW].

Like: where did the new "sloppy but still useful" alleles come from? Why (and how) would any local part of "the communal system" spend energy to generate such things or pass them along starting literally from scratch? This sort of meta-evolutionary cleverness usually requires a long time to arise!

The thing that should be possible (not easy, but possible) only now, with technology, is to invent some new amino acids (leveraging what exists in the biosphere now instead of what was randomly available billions of years ago) AND a new codon system for them, and to boot strap from there, via directed evolution, towards some kind of cohesively viable neolife that (if it turns out to locate a better local optimum than we did) might voraciously consume the current ecology.

The above image is from Figure 2, with the description "Examples of genetically encoded noncanonical amino acids with novel functions" from Schulze's "Expanding the genetic code".

Compensatory mutation is actually a pretty interesting and key concept, because it suggests a method by which one might prevent a gray good scenario on purpose, rather than via "mere finger crossing" regarding limitations that we might pray that we will be lucky enough for it to run into by accident.

We could run evolution in silico at the protein level, with large conceptual jumps, and then print something unlikely to be able to evolve. 

The "unable to easily evolve" thing might be similar to human telomeres, but more robust.

It could make every generation of neolife to almost entirely a degeneration way from "edenic neolife" and towards a mutational meltdown.

Note that this is essentially an "alignment" or "corrigibility" strategy, but at the level of chemistry and molecular biology, where the hardware is much much easier to reason about, in comparison to the "software" of "planning and optimization processes themselves".

If you could cause there to be only a 1 in septillion chance of positive or compensatory mutations on purpose (knowing the mechanisms and math to calculate this risk) and put several fully independent booby traps into the system that will fail after a handful of mutations, then you could have the first X generations "eat and double very very efficiently" and then have the colony switch to "doing the task" for Y generations, and then maybe as meltdown became inevitable within ~Z more generations they could, perhaps, actively prepare for recycling?

I can at least IMAGINE this for genomes, because genomes are mostly not Turing Complete.

I know of nothing similar that could be used to make AI with survive-and-spread powers similarly intrinsically safe.

Replies from: bhauth

↑ comment by bhauth · 2023-04-17T18:02:47.721Z · LW(p) · GW(p)

You're misunderstanding the point of those proposed amino acids. They're proposals for things to be made by (at least partly) non-enzymatic lab-style chemical processes, processed into proteins by ribosomes, and then used for non-cell purposes. Trying to use azides (!) or photocrosslinkers (?) in amino acids isn't going to make cells work better.

There really isn't much improvement to be had by using different amino acids.

Replies from: JenniferRM

↑ comment by JenniferRM · 2023-04-17T18:30:54.054Z · LW(p) · GW(p)

The new aminoacids might be "essential" (not manufacturable internally) and have to come in as "vitamins" potentially. This is another possible way to prevent gray goo on purpose, but hypothetically it might be possible to find ways to move that synthesis into the genome of neolife itself, if that was cheap and safe. These seem like engineering considerations that could change from project to project.

Mostly I have two fundamental points:

1) Existing life is not necessarily bio-chemically optimal because it currently exists within circumscribed bounds that can be transgressed. Those amino acids are weird and cool and might be helpful for something. Only one amino acid (and not even any of those... just anything) has to work to give "neo-life" some kind of durable competitive advantage over normal life.

2) All designs have to come from somewhere, with the optimization pressure supplied by some source, and it is not safe or wise to rely on random "naturally given" limits in the powers of systems that contain an internal open-ended optimization engine. When trying to do safety engineering, and trying to reconcile inherent safety with the design of something involving autonomous (potentially exponential) growth, either (1) just don't do it, or else (2) add multiple well-tested purposeful independent default shutdown mechanisms. If you're "doing it" then look at all your safety mechanisms in a fault tree analysis and if the chance of an error is 1/N then make sure there will definitely not be anything vaguely close to N opportunities for a catastrophe to occur.

Just to be clear, a point which the post seems to take for granted, but which people not familiar with the topic might not think about, is:

Life is already selected for inclusive genetic fitness, so if nanobots do not unlock powerful capacities that life does not already have, then you cannot have a gray goo scenario because ordinary life will outcompete your nanobots for resources.

OK, maybe you want to build some kind of mechanical computers too. Clearly, life doesn't require that for operation, but does that even work? Consider a mechanical computer indicating a position. It has some number, and the high bit corresponds to a large positional difference, which means you need a long lever, and then the force is too weak, so you'd need some mechanical amplifier. So that's a problem.

Drexler absolutely considered thermal noise. Rod logic uses rods at right angles whose positions allow or prevent movement of other rods. That's the amplification since a small force moving one rod can control a later applied larger force on a blocked rod.

http://www.nanoindustries.com/nanojbl/NanoConProc/nanocon2.html#anchor84400

I don't think this is the right mind frame, thinking about how something specific appears too hard or even infeasible. A better frame is "say, you are given $100B/year, can hire the best people in the world, and have 10 years to come up with viable self-replicating nanobots, or else we all die, how would you go about it?" 

This means that the reactions you can do are limited to what organic compounds can do at relatively low temperatures - and existing life can pretty much do anything useful in that category already.

We find that bacteria sometimes do manage to work at higher temperatures as well. Thermus aquaticus that gave us Taq polymerase works for example at higher temperatures than most other bacteria. 

Generally, it's very hard for eukaryotes or prokaryotes to evolve the usage of new amino acids. It's unclear what we could do with artificial designed proteins when we open up the range of amino acids further.

Simply changing the way proteins are coded allows immunizing bacteria against existing phages. Phages are a key reason we have the current diversity of bacteria. 

I understand Eliezer Yudkowsky thinks that someone a little smarter than von Neumann (who didn't invent the "von Neumann architecture" or half the other stuff he took credit for, but that's off topic) would be able to invent "grey goo" type nanobots.

Eliezer model is that an AI with that intelligence will self-improve from "a little smarter than von Neumann" to "a lot smarter than von Neumann"

None of this argues that creating grey goo is an unlikely outcome, just that it's a hard problem. And we have an existence proof of at least one example of a way to make gray goo that covers a planet, which is life-as-we-know-it, which did exactly that.

But solving hard problems is a thing that happens, and unlike the speed of light, this limit isn't fundamental. It's more like the "proofs" that heavier than air flight is impossible which existed in the 1800s, or the current "proofs" that LLMs won't become AGIs - convincing until the counterexample exists, but not at all indicative that no counterexample does or could exist.

Thanks so much for this post, I've been wishing for something like this for a long time. I kept hearing people grumbling about how EY & Drexler were way too bullish about nanotech, but no one had any actual arguments. Now we have arguments & a comment section. :)

Thanks. So helpful to have this comprehensive argument.

If it was advantageous to use structures of those inside cells for reactions somehow, then some organisms would already do that.

Not necessarily. The space of advantageous biologically possible structural configurations seems to me to be intuitively larger than the space of useful configurations currently known to be in use. 
In order for a structure to be evolutionarily feasible, it must not only be advantageous but also there must be a path of individually beneficial (or at minimum not harmful) small steps in between it and currently existing structures. If an adaptation does not lend itself to linearly realized benefit, e.g. one that works really well but only when 90%+ is 'correct', it has no evolutionary way to piece itself together from 0-90%.

low commit here but I've previously used nanotech as an example (rather than a probable outcome) of a class somewhat known unknowns - to portray possible future risks that we can imagine as possible while not being fully conceived. So while grey goo might be unlikely, it seems that precursor to grey goo of a pretty intelligent system trying to mess us up is the thing to be focused on, and this is one of its many possibilities that we can even imagine

Yes, I've actually seen people say that, but cells do use myosin to transport proteins sometimes. That uses a lot of energy, so it's only used for large things.

Cells have compartments with proteins that do related reactions. Some proteins form complexes that do multiple reaction steps. Existing life already does this to the extent that it makes sense to.

Humans or AI designing a transport/ compartmentalization system can go "how many compartments is optimal". Evolution doesn't work like this. It evolves a transport system to transport one specific thing in one specific organism. 

It's like humans invent railways in general. Evolution invents a railway between 2 towns, and if it wants to connect a third town, it needs to invent a railway again from scratch. (Imagine a bunch of very secretive town councils)

>what if a superintelligence finds something I didn't think of?

I'm not a superintelligence, and I know of at least one plausible "green goo" scenario involving rogue microbes.