"Diamondoid bacteria" nanobots: deadly threat or dead-end? A nanotech investigation
post by titotal (lombertini) · 2023-09-29T14:01:15.453Z · LW · GW · 81 commentsThis is a link post for https://titotal.substack.com/p/diamondoid-bacteria-nanobots-deadly
81 comments
Comments sorted by top scores.
comment by Veedrac · 2023-09-30T21:36:53.929Z · LW(p) · GW(p)
I'm not sure how to put this, but while this post is framed as a response to AI risk concerns, those concerns are almost entirely ignored in favor of looking at how plausible it is for near-term human research to achieve it, and only at the end is it connected back to AI risk via a brief aside whose crux is basically that you don't think Yudkowsky-style ASI will exist.
I like a lot of the discussion if I frame it in my head to be about what it is actually arguing for. Taking it as given, it seems instead broadly non-sequiter, as the evidence given basically doesn't relate to resolving the disagreement.
Replies from: lombertini, 1a3orn, Making_Philosophy_Better, faul_sname↑ comment by titotal (lombertini) · 2023-10-01T15:52:23.863Z · LW(p) · GW(p)
At no point did I ever claim that this was a conclusive debunking of AI risk as a whole, only an investigation into one specific method proposed by Yudkowksy as an AI death dealer.
In my post I have explained what DMS is, why it was proposed as a technology, how far along the research went, the technical challenges faced in it's construction, some observations of how nanotech research works, the current state of nanotech research, what near-term speedups can be expected from machine learning, and given my own best guess on whether an AGI could pull off inventing MNT in a short timeframe, based on what was learned.
This is only "broadly non-sequiter" if you think that none of that information is relevant for assessing the feasibility of diamondoid bacteria AI weapons, which strikes me as somewhat ridiculous.
Replies from: Veedrac↑ comment by Veedrac · 2023-10-01T18:24:41.599Z · LW(p) · GW(p)
Rather than focusing on where I disagree with this, I want to emphasize the part where I said that I liked a lot of the discussion if I frame it in my head differently. I think if you opened the Introduction section with the second paragraph of this reply (“In my post I have explained”), rather than first quoting Yudkowsky, you'd set the right expectations going into it. The points you raise are genuinely interesting, and tons of people have worldviews that this would be much more convincing to than Yudkowsky's.
↑ comment by 1a3orn · 2023-10-01T13:04:13.057Z · LW(p) · GW(p)
What would qualify as an evidence against how ASI can do a thing, apart from pointing out the actual physical difficulties in doing the thing?
Replies from: Veedrac↑ comment by Veedrac · 2023-10-01T18:09:08.702Z · LW(p) · GW(p)
apart from pointing out the actual physical difficulties in doing the thing
This excludes most of the potential good arguments! If you can show that large areas of the solution space seem physically unrealizable, that's an argument that potentially generalizes to ASI. For example, I think people can suggest good limits on how ASI could and couldn't traverse the galaxy, and trivially rule out threats like ‘the AI crashes the moon into Earth’, because of physical argument.
To hypothesize an argument of this sort that might be persuasive, at least to people able to verify such claims: ‘Synthesis of these chemicals is not energetically feasible at these scales because these bonds take $X energy to form, but it's only feasible to store $Y energy in available bonds. This limits you to a very narrow set of reactions which seems unable to produce the desired state. Thus larger devices are required, absent construction under an external power source.’ I think a similar argument could plausibly exist around object stickiness, though I don't have the chemistry knowledge to give a good framing for how that might look.
There aren't as many arguments once we exclude physical arguments. If you wanted to argue that it was plausibly physically realizable but that strong ASI wouldn't figure it out, I suppose some in-principle argument that it involves solving a computationally intractable challenge in leu of experiment might work, though that seems hard to argue in reality.
It's generally hard to use weaker claims to limit far ASI, because, being by definition qualitatively and quantitatively smarter than us, it can reason about things in ways that we can't. I'm happy to think there might exist important, practically-solvable-in-principle tasks that an ASI fails to solve, but it seems implausible for me to know ahead of time which tasks those are.
↑ comment by Portia (Making_Philosophy_Better) · 2023-10-01T18:28:08.803Z · LW(p) · GW(p)
I think the text is mostly focussed on the problems humans have run into when building this stuff, because these are known and hence our only solid empirical detailed basis, while the problems AI would run into when building this stuff are entirely hypothetical.
It then makes a reasonable argument that AI probably won't be able to circumvent these problems, because higher intelligence and speed alone would not plausibly fix them, and in fact, a plausible fix might have to be slow, human-mediated, and practical.
One can disagree with that conclusion, but as for the approach, what alternative would you propose when trying to judge AI risk?
Replies from: Veedrac↑ comment by Veedrac · 2023-10-01T18:55:16.127Z · LW(p) · GW(p)
I think I implicitly answered you elsewhere [LW(p) · GW(p)], though I'll add a more literal response to your question here.
On a personal level, none of this is relevant to AI risk. Yudkowsky's interest in it seems like more of a byproduct of his reading choices when he was young and impressionable than anything else, which is not reading I shared. Neither he nor I think this is necessary for xrisk scenarios, with me probably being on the more skeptical side, and me believing more in practical impediments that strongly encourage doing the simple things that work, eg. conventional biotech.
Due to this not being a crux and not having the same personal draw towards discussing it, I basically don't think about this when I think about modelling AI risk scenarios. I think about it when it comes up because it's technically interesting. If someone is reasoning about this because they do think it's a crux for their AI risk scenarios, and they came to me for advice, I'd suggest testing that crux before I suggested being more clever about de novo nanotech arguments.
↑ comment by faul_sname · 2023-10-01T00:45:11.166Z · LW(p) · GW(p)
My impression is that the nanotech is a load bearing part of the "AI might kill all humans with no warning and no signs ofpreparation" story - specifically, "a sufficiently smart ASI could quickly build itself more computing substrate without having to deal with building and maintaining global supply chains, and doing so would be the optimal course of action" seems like the sort of thing that's probably true if it's possible to build self- replicating nanofactories without requiring a bunch of slow, expensive, serial real-world operations to get the first one built and debugged, and unlikely to be true if not.
That's not to say human civilisation is invulnerable, just that "easy" nanotech is a central part of the "everyone dies with no warning and the AI takes over the light cone uncontested" story.
Replies from: Veedrac↑ comment by Veedrac · 2023-10-01T01:41:52.892Z · LW(p) · GW(p)
I was claiming that titotal's post doesn't appear to give arguments that directly address whether or not Yudkowsky-style ASI can invent diamondoid nanotech. I don't understand the relevance to my comment. I agree that if you find titotal's argument persuasive then whether it is load bearing is relevant to AI risk concerns, but that's not what my comment is about.
FWIW Yudkowsky frequently says that this is not load bearing, and that much seems obviously true to me also.
Replies from: adastra22↑ comment by adastra22 · 2024-03-04T08:32:32.479Z · LW(p) · GW(p)
Both this and his earlier article by titotal on computational chemistry make the argument that development of molecular nanotechnology can't be a one-shot process because of intrinsic limitations on simulation capabilities.
Yudkowsky claims that one-shot nanotechnology is not load bearing, yet literally every example he gives when pressed involves one-shot nanotech.
Replies from: Veedrac↑ comment by Veedrac · 2024-03-04T10:20:33.894Z · LW(p) · GW(p)
Could you quote or else clearly reference a specific argument from the post you found convincing on that topic?
Replies from: adastra22↑ comment by adastra22 · 2024-03-04T23:04:22.589Z · LW(p) · GW(p)
OP's post regarding the topic is here (it is also linked to in the body of this article):
https://titotal.substack.com/p/bandgaps-brains-and-bioweapons-the
I'm running a molecular nanotechnology company, so I've had to get quite familiar with the inner workings and limitations of existing computational chemistry packages. This article does a reasonable job of touching on the major issues. I can tell you that experimentally these codes don't even come close to the precision required to find trajectories that reliably drive diamondoid mechanosynthesis. It's really difficult to get experiments in the lab to match the (over simplified, reduced accuracy) simulations which take days or weeks to run.
Having any hope of adequately simulating mechanosynthetic reactions requires codes that scale with O(n^7). No one even writes these codes, because what would be the point? The biggest thing you could simulate is helium. There are approximations with O(n^3) and O(n^4) running time that are sometimes useful guides, but can often have error bars larger than the relevant energy gaps. The acid test here is still experiment: try to do the thing and see what happens.
AI has potential to greatly improve productivity with respect to nano-mechanical engineering designs because you can build parts rigid and large enough, and operating at slow enough speeds that mechanical forces are unlikely to have chemical effects (bond breaking or forming), and therefore faster O(n^2) molecular mechanics codes can be used. But all the intelligence in the world isn't going to make bootstrapping molecular nanotechnology a one-shot process. It's going to be years of hard-won lab work, whether it's done by a human or a thinking machine.
Replies from: Veedrac↑ comment by Veedrac · 2024-03-05T02:23:06.230Z · LW(p) · GW(p)
Well yes, nobody thinks that existing techniques suffice to build de-novo self-replicating nano machines, but that means it's not very informative to comment on the fallibility of this or that package or the time complexity of some currently known best approach without grounding in the necessity of that approach.
One has to argue instead based on the fundamental underlying shape of the problem, and saying accurate simulation is O(n⁷) is not particularly more informative to that than saying accurate protein folding is NP. I think if the claim is that you can't make directionally informative predictions via simulation for things meaningfully larger than helium then one is taking the argument beyond where it can be validly applied. If the claim is not that, it would be good to hear it clearly stated.
Replies from: adastra22↑ comment by adastra22 · 2024-03-05T09:10:02.227Z · LW(p) · GW(p)
I don't understand your objection. People who actually use computational chemistry--like me, like titotal--are familiar with its warts and limitations. These limitations are intrinsic to the problem and not expressions of our ignorance. Diamondoid mechanosynthetic reactions depend on properties which only show up in higher levels of theory, the ones which are too expensive to run for any reasonably sized system. To believe that this limitation won't apply to AI, just because, is magical thinking.
I wasn't saying anything with respect to de-novo self-replicating nano machines, a field which has barely been studied by anyone and which we cannot adequately say much about at all.
Replies from: Veedrac↑ comment by Veedrac · 2024-03-05T17:41:55.801Z · LW(p) · GW(p)
And what reason do you have for thinking it can't be usefully approximated in some sufficiently productive domain, that wouldn't also invalidly apply to protein folding? I think it's not useful to just restate that there exist reasons you know of, I'm aiming to actually elicit those arguments here.
Replies from: adastra22↑ comment by adastra22 · 2024-03-05T21:36:33.606Z · LW(p) · GW(p)
The fact that this has been an extremely active area of research for over 80 years with massive real-world implications, and we're no closer to finding such a simplified heuristic. This isn't like protein folding at all--the math is intrinsically more complicated with shared interaction terms that require high polynomial count running times. Indeed it's provable that you can't have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately.
Of course there could be some heuristic-like approximation which works in the relevant cases for this specific domain, e.g. gemstone crystal lattice structures under ultra-high vacuum conditions and cryogenic temperatures. Just like there are reasonably accurate molecular mechanics models that have been developed for various specific materials science use cases.
But here's the critical point: these heuristics are developed by evolving models based on experiment. In other domains like protein folding you can have a source of truth for generative AI training that uses simulation. That's how AlphaFold was done. But for chemical reactions of the sort we'd be interested here the ab-initio methods that give correct results are simply too slow to use for this. Not "buy a bigger GPU" slow, but "run the biggest supercomputer for a quintillion years" slow. So the solution is to do the process manually, running actual experiments to refine the heuristic models at each step, much machine learning training models would do. These are called semi-empirical methods, and they work. But we don't have a good semi-empirical model for the type of reactions concerned here, and it would take years of painstaking lab work to develop, which no amount of super intelligence can avoid.
It's magical thinking to assume that an AI will just one-shot this into existence.
Replies from: Veedrac↑ comment by Veedrac · 2024-03-10T02:36:30.895Z · LW(p) · GW(p)
Thanks, I appreciate the attempt to clarify. I do though think there's some fundamental disagreement about what we're arguing over here that's making it less productive than it could be. For example,
The fact that this has been an extremely active area of research for over 80 years with massive real-world implications, and we're no closer to finding such a simplified heuristic.
I think both:
- Lack of human progress doesn't necessarily mean the problem is intrinsically unsolvable by advanced AI. Humans often take a bunch of time before proving things.
- It seems not at all the case that algorithmic progress isn't happening, so it's hardly a given that we're no closer to a solution unless you first circularly assume that there's no solution to arrive at.
If you're starting out with an argument that we're not there yet, this makes me think more that there's some fundamental disagreement about how we should reason about ASI, more than your belief being backed by a justification that would be convincing to me had only I succeeded at eliciting it. Claiming that a thing is hard is at most a reason not to rule out that it's impossible. It's not a reason on its own to believe that it is impossible.
With regard to complexity,
- I failed to understand the specific difference with protein folding. Protein folding is NP-hard, which is significantly harder than O(n³).
- I failed to find the source for the claim that O(n³) or O(n⁴) are optimal. Actually I'm pretty confused how this is even a likely concept; surely if the O(n³) algorithm is widely useful then the O(n⁴) proof can't be that strong of a bound on practical usefulness? So why is this not true of the O(n³) proof as well?
It's maybe true that protein folding is easier to computationally verify solutions to, but first, can you prove this, and second, on what basis are you claiming that existing knowledge is necessarily insufficient to develop better heuristics than the ones we already have? The claim doesn't seem to complete to me.
It's magical thinking to assume that an AI will just one-shot this into existence.
Please note that I've not been making the claim that ASI could necessarily solve this problem. I have been making the claim that the arguments in this post don't usefully support the claim that it can't. It is true that largely on priors I expect it should be able to, but my priors also aren't particularly useful ones to this debate and I have tried to avoid making comments that are dependent on them.
Replies from: adastra22↑ comment by adastra22 · 2024-03-10T18:28:25.832Z · LW(p) · GW(p)
Do you have a maths or computer science background? I ask because some of the questions you ask are typical of maths or CS, but don't really make sense in this context.
Take protein folding. Existing techniques for estimating how a protein folds are exponential in complexity (making the problem NP-hard), so running time is bounded by O(2^n). But what's the 'n'? In these algorithms it is the number of amino acids because that's the level at which they are operating. For quantum chemistry the polynomial running times have 'n' being the number of electrons in the system, which can be orders of magnitude higher. That's why O(n^4) can be much, much worse than O(2^n) for relevant problem sizes--there's no hope of doing full protein folding simulations by ab initio methods. The second reason they're not comparable is time steps. Ab initio molecular mechanics methods can't have time steps larger than 1 femtosecond. Proteins fold on the timescale of milliseconds. This by itself is what makes all ab initio molecular mechanics methods infeasible for protein folding, even though it's a constant factor ignored by big-O notation.
Staying on the issue of complexity, from a technical / mathematical foundation perspective every quantum chemistry code should also have exponential running time, and this is trivially proved by the fact that every particle technically does interact with every other particle. But all of these interaction terms fall off with at least 1/r^2, and negative power increases with each corrective term. For 1/r^2 and 1/r^3, there are field density tricks that can be used for accurate sub-exponential modeling. For 1/r^4 or higher it is perfectly acceptable to just have distance cutoffs which cutout the exponential term. How accurate a simulation you will get is a question of how many of these corrective terms you include. The more you include, the more higher-order effects you capture, but you also get hit with higher-order polynomial running times.
To bring a long story to an abrupt end, you don't "prove" correctness of these various codes in a mathematical sense. They're all physically incorrect because they involve approximations. Rather for various use cases you can experimentally show that certain terms dominate, and you need to make sure that the code you use accurately captures the relevant terms for the domain and scale of the problem you are simulating. This is an experimental determination, not a mathematical proof. You literally go measure things in the lab.
The protein data bank contains hundreds of thousands of structures which have had their atomic coordinates determined by various experimental methods, providing a ground truth for protein folding heuristics. There is no such source of truth for diamondoid nano-mechanical machines parts and the mechanosynthesis steps that produce them. If you want to train a better reactive force field heuristic that could make the relevant design simulations tractable, you need a source of truth. The only thing available is various quantum chemistry approximations, as you need to approximate to be computable. These fail as source of truth because you don't know which approximations are reasonable and which are not without experiment data, which we don't have. So you could train a heuristic, yes, but you'd have absolutely no way of assessing its accuracy.
There's a common saying in the AI field: "garbage in, garbage out." The current state of non-empirical ab initio quantum chemistry is garbage. You're saying that AI could take the garbage and by mere application of thought turn it into something useful. That's not in line with the actual history of the development of useful AI outputs.
I'm not saying that AI can't develop useful heuristic approximations for the simulation of gemstone-based nano-mechanical machinery operating in ultra-high vacuum. I'm saying that it can't do so as a one-shot inference without any new experimental work, which is a trope that shows up again and again in certain high-profile people's descriptions of AI x-risk scenarios.
Replies from: Veedrac↑ comment by Veedrac · 2024-03-10T19:10:00.955Z · LW(p) · GW(p)
If you say “Indeed it's provable that you can't have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately” I am quite likely to go on a digression where I try to figure out what proof you're pointing at and why you think it's a fundamental barrier, and it seems now that per a couple of your comments you don't believe it's a fundamental barrier, but at the same time it doesn't feel like any position has been moved, so I'm left rather foggy about where progress has been made.
I think it's very useful that you say
I'm not saying that AI can't develop useful heuristic approximations for the simulation of gemstone-based nano-mechanical machinery operating in ultra-high vacuum. I'm saying that it can't do so as a one-shot inference without any new experimental work
since this seems like a narrower place to scope our conversation. I read this to mean:
- You don't know of any in principle barrier to solving this problem,
- You believe the solution is underconstrained by available evidence.
I find the second point hard to believe, and don't really see anywhere you have evidenced it.
As a maybe-relevant aside to that, wrt.
You're saying that AI could take the garbage and by mere application of thought turn it into something useful. That's not in line with the actual history of the development of useful AI outputs.
I think you're talking of ‘mere application of thought’ like it's not the distinguishing feature humanity has. I don't care what's ‘in line with the actual history’ of AI, I care what a literal superintelligence could do, and this includes a bunch of possibilities like:
- Making inhumanly close observation of all existing data,
- Noticing new, inhumanly-complex regularities in said data,
- Proving new simplifying regularities from theory,
- Inventing new algorithms for heuristic simulation,
- Finding restricted domains where easier regularities hold,
- Bifurcating problem space and operating over each plausible set,
- Sending an interesting email to a research lab to get choice high-ROI data.
We can ignore the last one for this conversation. I still don't understand why the others are deemed unreasonable ways of making progress on this task.
I appreciated the comments on time complexity but am skipping it because I don't expect at this point that it lies at the crux.
Replies from: adastra22↑ comment by adastra22 · 2024-03-10T19:39:56.977Z · LW(p) · GW(p)
If you say “Indeed it's provable that you can't have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately” I am quite likely to go on a digression where I try to figure out what proof you're pointing at and why you think it's a fundamental barrier
By "proof" I meant proof by contradiction. DFT is a great O(n^3) method for energy minimizing structures and exploring electron band structure, and it is routinely used for exactly that purpose. So much so that many people conflate "DFT" with more accurate ab initio methods, which it is not. However DFT utterly ignores exchange correlation terms and so it doesn't model van der Waals interactions at all. Every design for efficient and performant molecular nanotechnology--the ones that get you order-of-magnitude performance increases and therefore any benefit over existing biology or materials science--involve vdW forces almost exclusively in their manufacture and operation. It's the dominant non-covalent bonded interaction at that scale.
That's the most obvious example, but also a lot of the simulations performed by Merkle and Freitas in their minimal toolset paper give incorrect reaction sequences in these lower levels of theory, as they found out when they got money to attempt it in the lab. Without pointing to their specific failure, you can get a hint of this in surface science. Silicon, gold, and other surfaces tend to have rather interesting surface clustering and reorganization effects, which are observable by scanning probe microscopy. These are NOT predicted by the cheaper / computationally tractable codes, and they are an emergent property of higher-order exchange correlations in the crystal structure. These nevertheless have enough of an effect to drastically reshape the surface of these materials, making calculation of those forces absolutely required for any attempt to build off the surface.
Attempting to do cheaper simulations for diamondoid synthesis reactions gave very precise predictions that didn't work as expected in the lab. How would your superintelligent AI know that uncalculated terms dominate in the simulation, and make corrective factors without having access to those incomputable factors?
- Making inhumanly close observation of all existing data
- Noticing new, inhumanly-complex regularities in said data,
- Proving new simplifying regularities from theory
I think you vastly overestimate how much knowledge is left to extracted from the data. AI has made tremendous advances in recent years where it has been able to consume huge amounts of data, far in excess of what any group of humans could analyze. This, on the other hand, is a data-poor regime.
- Inventing new algorithms for heuristic simulation
This is happening right now. There are a variety of machine-learned molecular mechanics force fields that have been published in the last few years. The most interesting one I've found used periodic crystal ab initio simulation methods to create a force field potential that ended up being very good for liquid and gas-phase chemistry, which it was not trained on.
But the relevant question (if you want to talk about AI x-risk by means of bootstrapping nanotech) is how accurate they are outside of the domain where we have heaps of hard evidence, because we don't have a ground truth to compare against in those environments.
- Finding restricted domains where easier regularities hold
- Bifurcating problem space and operating over each plausible set,
Human engineers are very good at this. It's not the limiting factor.
- Sending an interesting email to a research lab to get choice high-ROI data
What lab. There's literally no one doing the relevant research, or equipped to easily do it without years of preparatory chemical synthesis and surface characterization.
Which is really the point and the crux of the matter. It will take an extended, years-long research effort to create molecular nanotechnology. It's not something you can plausibly do in secret, and certainly not something you can shorten by simulation or bayesian inference.
Replies from: ryan_greenblatt, Veedrac↑ comment by ryan_greenblatt · 2024-03-10T21:46:18.015Z · LW(p) · GW(p)
One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
To keep the situation analogous to an AI needing to do this quickly, we'll suppose this team of humans is subject to the same restrictions on compute for simulations (e.g. over the entire billion years they only get 10^30 FLOP) and also can only run whatever experiments the AI would get to run (effectively no interesting experiments).
I feel uncertain about whether this team would succeed, but it does seem pretty plausible that they would be able to succeed. Perhaps I think they're 40% likely to succeed?
Now, suppose the superintelligence is like this, but even more capable.
See also That Alien Message [LW · GW]
Separately, I don't think it's very important to know what an extremely powerful superintelligence could do, because prior to the point where you have an extremely powerful superintelligence, humanity will already be obsoleted by weaker AIs. So, I think Yudkowsky's arguments about nanotech are mostly unimportant for other reasons.
But, if you did think "well, sure the AI might be arbitrarily smart, but if we don't give it access to the nukes what can it even do to us?" then I think that there are many sources of concern and nanotech is certainly one of them.
Replies from: adastra22↑ comment by adastra22 · 2024-03-11T01:45:14.722Z · LW(p) · GW(p)
One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
By merely thinking about it, and not running any experiments? No, absolutely not. I don't you understood my post if you assume I'd think otherwise.
Try this: I'm holding a certain number of fingers behind my back. You and a team of 10,000 of the smartest human engineers and scientists have a billion years to decide, without looking, what your guess will be as to how many fingers I'm holding behind my back. But you only get one chance to guess at the end of that billion years.
That's a more comparable example.
See also That Alien Message
Please don't use That Alien Message as an intuition pump. There's a tremendous amount wrong with the sci-fi story. Not least of which is that it completely violates the same constraint you put into your own post about constraining computation. I suggest doing your own analysis of how many thought-seconds the AI would have in-between frames of video, especially if you assume it to be running as a large inference model.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities. That is where EY went wrong, and a lot of the LW crowd too. If you permit dividing by zero or multiplying by infinity, then you can easily convince yourself of anything. AI isn't magic, and AGI isn't a free pass to believe anything.
PS: We've had AGI since 2017. That'd better be compatible with your world view if you want accurate predictions.
Replies from: ryan_greenblatt↑ comment by ryan_greenblatt · 2024-03-11T04:20:51.575Z · LW(p) · GW(p)
That's a more comparable example.
I don't understand where your confidence is coming from here, but fair enough. It wasn't clear to me if your take was more like "wildly, wildly superintelligent AI will be considerably weaker than a team of humans thinking for a billion years" or more like "literally impossible without either experiments or >>10^30 FLOP".
I generally have an intuition like "it's really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible". It seems you don't share this intuition, fair enough.
(I feel like this applies for nearly all human inventions? Like if you had access to a huge amount of video of the world from 1900 and all written books that existed at this point, and had the affordances I described with a team of 10,000 people, 10^30 FLOP, and a billion years, it seems to me like there is a good chance you'd be able to one-shot reinvent ~all inventions of modern humanity (not doing everything in the same way, in many cases you'd massively over engineer to handle one-shot). Planes seem pretty easy? Rockets seem doable?)
Please don't use That Alien Message as an intuition pump.
I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities
I separately think it's not very important to think about the abilities of wildly, wildly superintelligent AI for most purposes (as I noted in my comment). So I agree that imagining arbitrary capabilities is probablematic. (For some evidence that this isn't post-hoc justification, see this [LW · GW] post on which I'm an author.)
PS: We've had AGI since 2017. That'd better be compatible with your world view if you want accurate predictions.
Uhhhh, I'm not sure I agree with this as it doesn't seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like "able to speak slightly coherant english (GPT-1?) and classify images"?
Replies from: adastra22↑ comment by adastra22 · 2024-03-11T05:55:25.180Z · LW(p) · GW(p)
>>10^30 FLOP
By the way, where's this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
I generally have an intuition like "it's really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible"
Ok let's try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can't actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don't replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don't know that. You're not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn't clear to me. But let's say you do. Which experiments should you discount? Which should you assume to be correct? If you're allowed to start choosing which reported results you believe and which you don't, you've lost the plot. There could be millions of possible heuristics which partially match the literature and there's no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can't. Not any more than Descartes could think his way to absolute truth.
Also for what it's worth you've made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That's hindsight bias, but also selection bias. Nanotechnology doesn't exist. Even if it would work if created, there's no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don't believe this as I'm running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it's essential to point out the logical error.)
(Re: Alien Message) I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
Why? You've gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren't going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I'm not saying it's not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it's a fucking terrible intuition pump.
Uhhhh, I'm not sure I agree with this as it doesn't seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like "able to speak slightly coherant english (GPT-1?) and classify images"?
The transformer architecture introduced in 2017 is:
- Artificial: man-made
- General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
- Intelligent: able to construct efficient solutions to new problems it hasn't seen.
Artificial General Intelligence. A.G.I.
If you're thinking "yeah, but.." then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we're already well within the era of artificial general intelligence.
(Maybe EY's very public meltdown a year ago is making more sense now? But rest easy, EY's predictions about AI x-risk have consistently been wildly off the mark.)
Replies from: ryan_greenblatt, ryan_greenblatt, ryan_greenblatt↑ comment by ryan_greenblatt · 2024-03-11T06:14:40.611Z · LW(p) · GW(p)
then I suggest you taboo the term AGI
FWIW, I do taboo this term and thus didn't use it in this conversation until you introduced it.
Replies from: adastra22↑ comment by adastra22 · 2024-03-11T06:27:49.906Z · LW(p) · GW(p)
You highlighted "disagree" on the part about AGI's definition. I don't know how to respond to that directly, so I'll do so here. Here's the story about how the term "AGI" was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
https://web.archive.org/web/20181228083048/http://goertzel.org/who-coined-the-term-agi/
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym "AGI" that is never specified. My assumption is that when people say AGI here they mean Bostrom's ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
Replies from: ryan_greenblatt, ryan_greenblatt↑ comment by ryan_greenblatt · 2024-03-11T07:03:10.839Z · LW(p) · GW(p)
Definition in the OpenAI Charter:
artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work
A post on the topic by Richard [AF · GW] (AGI = beats most human experts).
↑ comment by ryan_greenblatt · 2024-03-11T07:11:27.300Z · LW(p) · GW(p)
My assumption is that when people say AGI here they mean Bostrom's ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
In case this wasn't clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you'll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying "LW has bad views about X" say "a high fraction of people who comment on LW have bad views about X" or "a high fraction of karma votes seem to be from people with bad views about X". Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don't think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
Replies from: ryan_greenblatt↑ comment by ryan_greenblatt · 2024-03-11T07:11:59.016Z · LW(p) · GW(p)
Also, I'm going to peace out of this discussion FYI.
↑ comment by ryan_greenblatt · 2024-03-11T06:09:31.824Z · LW(p) · GW(p)
By the way, where's this number coming from? (10^30 FLOP) You keep repeating it.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn't cruxy, so I just did a vague guess.
↑ comment by ryan_greenblatt · 2024-03-11T06:07:33.151Z · LW(p) · GW(p)
Why? You've gone into circular logic here.
I wasn't trying to justify anything, just noting my stance.
comment by Max H (Maxc) · 2023-09-29T15:18:18.366Z · LW(p) · GW(p)
Evolution managed to spit out some impressively complex technology at both the cellular level (e.g. mitochondria), chemical level (e.g. venom, hormones), and macro level (e.g. birds) via random iterative mutations of DNA.
Human designers have managed to come up with completely different kinds of complex machinery (internal combustion engines, airplanes, integrated circuits) and chemicals which aren't found anywhere in nature, using intelligent top-down design and industrial processes.
I read "diamondoid bacteria" as synecdoche for the obvious possible synergy between these two design spaces, e.g. modifying natural DNA sequences or writing new ones from scratch using an intelligent design process, resulting in "biological" organisms at points in the design space that evolution could never reach. For example, cells that can make use of chemicals that can (currently) only be synthesized at scale in human-built industrial factories, e.g. diamond or carbon nanotubes.
I think such synergy is pretty likely to allow humans to climb far higher in the tech tree than our current level, with or without the help of AI. And if humans can climb this tech tree at all, then (by definition) human-level AGIs can also climb it, perhaps much more rapidly so.
I'm open to better terminology though, if anyone has suggestions or if there's already something more standard. I think "diamondoid mechanosynthesis" is overly-specific and not really what the term is referring to.
comment by localdeity · 2023-09-29T20:14:01.580Z · LW(p) · GW(p)
Offtopic: I find it hilarious that professor Moriarty is telling us about the technology for world domination.
Replies from: SaidAchmiz↑ comment by Said Achmiz (SaidAchmiz) · 2023-09-29T22:21:20.615Z · LW(p) · GW(p)
Pictured: the explanation received by OP.
comment by MichaelStJules · 2023-09-30T02:16:00.462Z · LW(p) · GW(p)
As a historical note and for further context, the diamondoid scenario is at least ~10 years old, outlined here [LW · GW] by Eliezer, just not with the term "diamondoid bacteria":
The concrete illustration I often use is that a superintelligence asks itself what the fastest possible route is to increasing its real-world power, and then, rather than bothering with the digital counters that humans call money, the superintelligence solves the protein structure prediction problem, emails some DNA sequences to online peptide synthesis labs, and gets back a batch of proteins which it can mix together to create an acoustically controlled equivalent of an artificial ribosome which it can use to make second-stage nanotechnology which manufactures third-stage nanotechnology which manufactures diamondoid molecular nanotechnology and then... well, it doesn't really matter from our perspective what comes after that, because from a human perspective any technology more advanced than molecular nanotech is just overkill. A superintelligence with molecular nanotech does not wait for you to buy things from it in order for it to acquire money. It just moves atoms around into whatever molecular structures or large-scale structures it wants.
The first mention of "diamondoid" on LW (and by Eliezer) is this [LW · GW] from 16 years ago, but not for an AI doom scenario.
comment by Mitchell_Porter · 2023-09-30T02:55:20.101Z · LW(p) · GW(p)
This is just a version of "grey goo", a concept which has been around since 1986 and which was discussed here in April [LW · GW].
DMS research is fairly dead at the moment
I have learned that there are at least two, maybe three private enterprises pursuing it. The "maybe" is the biggest, Atomic Machines.
comment by Metacelsus · 2023-09-29T16:18:11.518Z · LW(p) · GW(p)
>For example, in 2003 the Nanoputian project successfully built a nanoscale model of a person out of organic molecules. They used cleverly chosen reaction pathways to produce the upper body, and cleverly chosen reaction pathways to produce the lower body, and then managed to pick the exact right conditions to mix them together in that would bond the two parts together
As a chemist by training, I don't think this is actually that impressive. They basically did a few Sonogashira couplings, which are rather easy reactions (I did them regularly as an undergrad).
If you want something impressive, look at the synthesis of vitamin B12: https://en.wikipedia.org/wiki/Vitamin_B12_total_synthesis
Replies from: chemslug↑ comment by chemslug · 2023-09-30T00:23:27.158Z · LW(p) · GW(p)
Another way to think about diamandoids is to consider what kind of organic chemistry you need to put them together the "traditional" way. That'll give you some insight into the processes you're going to be competing with as you try to assemble these structures, no matter which technique you use. The syntheses tend to go by rearrangements of other scaffolds that are easier to assemble but somewhat less thermodynamically stable (https://en.wikipedia.org/wiki/Diamantane#Production for example). However, this technique gets arduous beyond 4 or 5 adamantane units:
https://en.wikipedia.org/wiki/Diamondoid
Agreed that the Nanoputians aren't impressive. Lots of drugs are comparably complex, and they're actually designed to elicit a biological effect.
The B12 synthesis is sweet, but I'll put in a vote for the Woodward synthesis of strychnine (done using 1954 technology, no less!):
https://en.wikipedia.org/wiki/Strychnine_total_synthesis#Woodward_synthesis
Replies from: Metacelsus↑ comment by Metacelsus · 2023-09-30T01:44:37.449Z · LW(p) · GW(p)
Yeah, Woodward was a real trailblazer (interestingly, my undergrad PI was one of his last students)
comment by Mikola Lysenko · 2023-09-29T19:45:53.330Z · LW(p) · GW(p)
It's good to hear from an actual expert on this subject. I've also been quite skeptical of the diamondoid nanobot apocalypse on feasibility grounds (though I am still generally worried about AI, this specific foom scenario seems very implausible to me).
Maybe you could also help answer some other questions I have about the scalability of nanomanufacturing. Specifically, wouldn't the energy involved in assembling nanostructures be much much greater than snapping together ready made proteins/nucleic acids to build proteins/cells? I am not convinced that run away nanobots can self assemble or be built in factories at planet scales due to simple thermodynamic limits. For example if you are ripping apart atoms and sticking them together in some new diamondoid configuration, shouldn't the change in gibbs free energy be sufficiently high that energy becomes a limiting factor? If this energy is greater than what could be obtained from nuclear or solar power in some reasonable amount of time, it would rule out most "grey goo" nano-apocalypse scenarios.
My back of the envelope calculation is that there's about 10^20 moles of CO2 in the atmosphere, and it takes about 390 kJ to turn one mole of CO2 into a diamond. The earth receives about 10^17 watts of power from the sun. If we use all of that energy to make diamond bots as fast as we can, then it'll take thousands of years before even 1% of the atmosphere is converted to nano machines.
Granted there's a lot of unknown variables here, and my modeling is probably quite stupid, but I feel like some one must have considered these situations and come up with some way to roughly estimate how much energy would be required to turn the world into a diamond nanobot swarm to check if its even feasible given the energy available on earth (via sunlight or whatever).
My current gut feeling is that its probably more efficient at the end of the day to hijack existing biological materials and processes to build self replicating machines than using covalent bonds to resynthesize everything from scratch, but I don't really know enough to estimate that precisely.
↑ comment by PeterMcCluskey · 2023-09-29T21:23:23.710Z · LW(p) · GW(p)
Freitas' paper on ecophagy has a good analysis of these issues.
Replies from: Mikola Lysenko↑ comment by Mikola Lysenko · 2023-09-29T22:11:47.027Z · LW(p) · GW(p)
That's a great link, thanks!
Though it doesn't really address the point I made, they do briefly mention it:
> Interestingly, diamond has the highest known oxidative chemical storage density because it has the highest atom number (and bond) density per unit volume. Organic materials store less energy per unit volume, from ~3 times less than diamond for cholesterol, to ~5 times less for vegetable protein, to ~10–12 times less for amino acids and wood ...
>
> Since replibots must build energy-rich product structures (e.g. diamondoid) by consuming relatively energy-poor feedstock structures (e.g., biomass), it may not be possible for biosphere conversion to proceed entirely to completion (e.g., all carbon atoms incorporated into nanorobots) using chemical energy alone, even taking into account the possible energy value of the decarbonified sludge byproduct, though such unused carbon may enter the atmosphere as CO2 and will still be lost to the biosphere.
Unfortunately they never bother to follow up on this with the rest of their calculations, and instead base their estimate for replication times on how long it takes the nanobots to eat up all the available atoms. However, in my estimation the bottleneck on nanobot replication is not getting materials, but probably storing up enough joules to overcome the gibbs free energy of assembling another diamondoid nanobot from spare parts. I would love to have a better picture for this estimate since it seems like the determining factor in whether this stuff can actually proceed exponentially or not.
↑ comment by MichaelStJules · 2023-10-01T16:53:11.647Z · LW(p) · GW(p)
Is 1% of the atmosphere way more than necessary to kill everything near the surface by attacking it?
↑ comment by avturchin · 2023-09-30T00:12:24.417Z · LW(p) · GW(p)
Based on you numbers, it would require around 10^26 J to convert all CO2 and this will take 10^9 seconds =30 years.
I like your argument anyway, as it clear that quick solar-powered diamond apocalypses is unfeasible. But if AI kills people first and moves its computations into diamonds, it will have plenty of time.
Replies from: dr_s↑ comment by dr_s · 2023-10-01T10:40:33.847Z · LW(p) · GW(p)
Also unsure why you would go for CO2 in the atmosphere as a source of carbon rather than more low entropy, easily harvested ones (like fossil fuels, plastics, or, well, biomass).
Replies from: MichaelStJules↑ comment by MichaelStJules · 2023-10-01T16:56:24.145Z · LW(p) · GW(p)
Eliezer's scenario uses atmospheric CHON. Also, I guess Eliezer used atmospheric CHON to allow the nanomachines to spread much more freely and aggressively.
comment by Metacelsus · 2023-09-29T15:55:31.858Z · LW(p) · GW(p)
No, but . . . you don't need "diamondoid" technology to make nano-replicators that kill everything. Highly engineered bacteria could do the trick.
Replies from: 1a3orn, MichaelStJules↑ comment by 1a3orn · 2023-09-29T18:49:32.092Z · LW(p) · GW(p)
I think it's good epistemic hygiene to notice when the mechanism underlying a high-level claim switches because the initially-proposed mechanism for the high-level claim turns out to be infeasible, and downgrade the credence you accord the high level claim at least somewhat. Particularly when the former mechanism has been proposed many times.
Alice: This ship is going to sink. I've looked at the boilers, they're going to explode!
Alice: [Repeats claim ten times]
Bob: Yo, I'm an expert in thermodynamics and steel, the boilers are fine for X, Y, Z reason.
Alice: Oh. Well, the ship is still going to sink, it's going to hit a sandbar.
Alice could still be right! But you should try to notice the shift and adjust credence downwards by some amount. Particularly if Alice is the founder of a group talking about why the ship is going to sink.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2023-09-29T19:18:27.445Z · LW(p) · GW(p)
The original theory is sabotage, not specifically boiler explosion. People keep saying "How could you possibly sabotage a ship?", and a boiler explosion is one possible answer, but it's not the reason the ship was predicted to sink. Boiler explosion theory and sabotage theory both predict sinking, but it's a false superficial agreement, these theories are moved by different arguments.
Replies from: 1a3orn↑ comment by 1a3orn · 2023-09-29T21:56:28.563Z · LW(p) · GW(p)
If someone had said "Yo, this one lonely saboteur is going to sink the ship" and consistently responded to requests for how by saying "By exploding the boiler" -- then finding out that it was infeasible for a lone saboteur to sink the ship by exploding the boiler would again be some level of evidence against danger of the lone saboteur, so I don't see how that changes it?
Or maybe I'm misunderstanding you.
Replies from: martin-randall↑ comment by Martin Randall (martin-randall) · 2023-09-30T01:54:24.738Z · LW(p) · GW(p)
To make the analogy more concrete, suppose that Alice posts a 43-point thesis on MacGyver Ruin: A List Of Lethalities, similar to AGI Ruin [LW · GW], that explains that MacGyver is planning to sink our ship and this is likely to lead to the ship sinking. In point 2 of 43, Alice claims that:
MacGyver will not find it difficult to bootstrap to overpowering capabilities independent of our infrastructure. The concrete example I usually use here is exploding the boilers, because there's been pretty detailed analysis of how what definitely look like physically attainable lower bounds on what should be possible with exploding the boilers, and those lower bounds are sufficient to carry the point. My lower-bound model of "how MacGyver would sink the ship, if he didn't want to not do that" is that he gets access to the boilers, reverses the polarity of the induction coils, overloads the thermostat, and then the boilers blow up.
(Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even MacGyver could gain access to the boilers, if he didn't already have a gun?" but one hears less of this after the advent of MacGyver: Lost Treasure of Atlantis, for some odd reason.)
Losing a conflict with MacGyver looks at least as deadly as "there's a big explosion out of nowhere and then the ship sinks".
Then, Bob comes along and posts a 24min reply, concluding with:
I think if there was a saboteur on board, that would increase the chance of the boiler exploding. For example, if they used the time to distract the guard with a clanging sound, they might be able to reach the boiler before being apprehended. So I think this could definitely increase the risk. However, there are still going to be a lot of human-scale bottlenecks to keep a damper on things, such as the other guard. And as always with practical sabotage, a large part of the process will be figuring out what the hell went wrong with your last explosion.
What about MacGyver? Well, now we’re guessing about two different speculative things at once, so take my words (and everyone else’s) with a double grain of salt. Obviously, MacGyver would increase sabotage effectiveness, but I’m not sure the results would be as spectacular as Alice expects.
I suppose this updates my probability of the boilers exploding downwards, just as I would update a little upwards if Bob had been similarly cagey in the opposite direction.
It doesn't measurably update my probability of the ship sinking, because the boiler exploding isn't a load-bearing part of the argument, just a concrete example. This is a common phenomenon in probability when there are agents in play.
Replies from: 1a3orn↑ comment by 1a3orn · 2023-10-01T00:47:30.249Z · LW(p) · GW(p)
It doesn't measurably update my probability of the ship sinking
When you say, doesn't "measurably," do you mean that it doesn't update all or doesn't update much? I'm not saying you should update much. I'm just saying you should update some. Like I'm nodding along at your example, but my conclusion is instead simply the opposite.
Like suppose we've been worried about the imminent unaligned MacGyver threat. Some people say there's no way he can sink the ship; other people say he can. So the people who say he can confer and try to offer 10 different plausible ways he could sink the ship.
If we found out all ten didn't work, then -- considering that these examples were selected for being the clearest ways he can destroy this ship -- it's hard for me to think this shouldn't move you down at all. And so presumably finding out that just one didn't work should move you down by some lesser amount, if finding out 10 didn't work would also do so.
Imagine a a counterfactual world where people had asked, "how can he sink the ship" and people had responded "You don't need to know how, that's would just a concrete example, concrete examples are irrelevant to the principle which is simply that MacGuyver's superior improvisational skills are sufficient to sink the ship." I would have lower credence in MacGyver's sink shipping ability in the world without concrete examples; I think most people would; I think it would be weird not to. So I think moving in the direction of such a world should similarly lower your credence.
Replies from: shankar-sivarajan, martin-randall↑ comment by Shankar Sivarajan (shankar-sivarajan) · 2023-10-03T01:49:46.886Z · LW(p) · GW(p)
I think the chess analogy is better: if I predict that, from some specific position, MacGyver will play some sequence of ten moves that will leave him winning, and then try to demonstrate that by playing from that position and losing, would you update at all?
↑ comment by Martin Randall (martin-randall) · 2023-10-03T13:27:39.110Z · LW(p) · GW(p)
I meant "measurably" in a literal sense: nobody can measure the change in my probability estimate, including myself. If my reported probability of MacGyver Ruin after reading Alice's post was 56.4%, after reading Bob's post it remains 56.4%. The size of a measurable update will vary based on the hypothetical, but it sounds like we don't have a detailed model that we trust, so a measurable update would need to be at least 0.1%, possibly larger.
You're saying I should update "some" and "somewhat". How much do you mean by that?
↑ comment by MichaelStJules · 2023-09-29T17:41:41.552Z · LW(p) · GW(p)
This is the more interesting and important claim to check to me. I think the barriers to engineering bacteria are much lower, but it’s not obvious that this will avoid detection and humans responding to the threat, or that timing and/or triggers in bacteria can be reliable enough.
Replies from: Metacelsus↑ comment by Metacelsus · 2023-09-30T01:45:23.645Z · LW(p) · GW(p)
Unfortunately, explaining exactly what kind of engineered bacteria could be dangerous is a rather serious infohazard.
Replies from: dr_s, shankar-sivarajan, MichaelStJules↑ comment by Shankar Sivarajan (shankar-sivarajan) · 2023-10-03T01:52:46.039Z · LW(p) · GW(p)
Don't worry, I know of a way to stop any engineered bacteria before they can do any harm.
No, I'm not going to tell you what it is. Infohazard.
↑ comment by MichaelStJules · 2023-10-03T16:21:35.904Z · LW(p) · GW(p)
Possibly, but by limiting access to the arguments, you also limit the public case for it and engagement by skeptics. The views within the area will also probably further reflect self-selection for credulousness and deference over skepticism.
There must be less infohazardous arguments we can engage with. Or, maybe zero-knowledge proofs are somehow applicable. Or, we can select a mutually trusted skeptic (or set of skeptics) with relevant expertise to engage privately. Or, legally binding contracts to prevent sharing.
comment by MichaelStJules · 2023-09-30T02:32:27.944Z · LW(p) · GW(p)
High quality quantum chemistry simulations can take days or weeks to run, even on supercomputing clusters.
This doesn't seem very long for an AGI if they're patient and can do this undetected. Even months could be tolerable? And if the AGI keeps up with other AGI by self-improving to avoid being replaced, maybe even years. However, at years, there could be a race between the AGIs to take over, and we could see a bunch of them make attempts that are unlikely to succeed.
Replies from: jacopo, MichaelStJules↑ comment by jacopo · 2023-09-30T19:52:08.343Z · LW(p) · GW(p)
That's one simulation though. If you have to screen hundreds of candidate structures, and simulate every step of the process because you cannot run experiments, it becomes years of supercomputer time.
Replies from: jacopo↑ comment by jacopo · 2023-10-01T13:12:08.655Z · LW(p) · GW(p)
Although they never take the whole supercomputer, so if you have the whole supercomputer for yourself and the calculations do not depend on each other you can run many in parallel
Replies from: adastra22↑ comment by adastra22 · 2024-03-04T08:39:02.163Z · LW(p) · GW(p)
They do in fact routinely take up the entire supercomputer. I don't think you are comprehending the absolute mind-bogglingly huge scale of compute required to get realistic simulations of mechanosynthesis reactions. It's utterly infeasible without radical advances beyond the present state of the art classical compute capabilities.
Replies from: jacopo↑ comment by jacopo · 2024-03-06T08:48:17.305Z · LW(p) · GW(p)
My job was doing quantum chemistry simulations for a few years, so I think I can comprehend the scale actually. I had access to one of the top-50 supercomputers and codes just do not scale to that number of processors for one simulation independently of system size (even if they had let me launch a job that big, which was not possible)
Replies from: adastra22↑ comment by adastra22 · 2024-03-06T09:33:40.308Z · LW(p) · GW(p)
Yeah just for scaling problems are an issue. These are not embarrassingly parallel problems that can be easily sharded onto different CPUs. A lot of interconnect is needed, and as you scale up that becomes the limiting factor.
What codes did you use? The issue here is that DFT gives bad results because it doesn’t model the exchange-correlation energy functional very well, which ends up being load bearing for diamondoid construction reaction pathways performed in UHV using voltage-driven scanning probe microscopy to perform mechanochemistry. You end up having to do some form of perturbation theory, of which there are many variants all giving significantly different results, and yet are all so slow that running ab initio molecular dynamics simulations are utterly out of the question so you end up spending days of supercompute time just to evaluate energy levels of a static system and then use your intuition to guess whether the reaction would proceed. Moving to an even more accurate level of theory that would indisputably give good results for these systems but would have prohibitive scaling factors [O(n^5) to O(n^7)] preventing their use for anything other than toy examples.
I am at a molecular nanotechnology startup trying to bootstrap diamondoid mechanosynthesis build tools, so these sorts of concerns are my job as well.
Replies from: jacopo↑ comment by jacopo · 2024-03-06T17:30:21.931Z · LW(p) · GW(p)
Ahh for MD I mostly used DFT with VASP or CP2K, but then I was not working on the same problems. For thorny issues (biggish and plain DFT fails, but no MD) I had good results using hybrid functionals and tuning the parameters to match some result of higher level methods. Did you try meta-GGAs like SCAN? Sometimes they are suprisingly decent where PBE fails catastrophically...
Replies from: adastra22, jacopo↑ comment by adastra22 · 2024-03-06T23:31:59.745Z · LW(p) · GW(p)
For the most part we're avoiding/designing around compute constraints. Build up our experimental validation and characterization capabilities so that we can "see" what happens in the mechsyn lab ex post facto. Design reaction sequences with large thermodynamic gradients, program the probe to avoid as much as possible side reaction configurations, and then characterize the result to see if it was what you were hoping for. Use the lab as a substitute for simulation.
It honestly feels like we've got a better chance of just trying to build the thing and repeating what works and modifying what doesn't, than even a theoretically optimal machine learning algorithm could do using simulation-first design. Our competition went the AI/ML approach and it killed them. That's part of why the whole AGI-will-design-nanotech thing bugs me so much. An immense amount of money and time has been wasted on that approach, which if better invested could have gotten us working nano-machinery by now. It really is an idea that eats smart people.
↑ comment by jacopo · 2024-03-06T17:34:30.691Z · LW(p) · GW(p)
You could also try to fit an ML potential to some expensive method, but it's very easy to produce very wrong things if you don't know what you're doing (I wouldn't be able for one)
Replies from: adastra22↑ comment by adastra22 · 2024-03-07T05:28:08.000Z · LW(p) · GW(p)
It's coming along quite fast. Here's the latest on ML-trained molecular dynamics force fields that (supposedly) approach ab initio quality:
https://www.nature.com/articles/s41524-024-01205-w
These are potentially tremendously helpful. But in the context of AI x-risk it's still not enough to be concerning. A force field that gives accurate results 90% of the time would tremendously accelerate experimental efforts. But it wouldn't be reliable enough to one-shot nanotech as part of a deceptive turn.
↑ comment by MichaelStJules · 2023-09-30T02:49:43.488Z · LW(p) · GW(p)
Also, maybe we design scalable and efficient quantum computers with AI first, and an AGI uses those to simulate quantum chemistry more efficiently, e.g. Lloyd, 1996 and Zalka, 1996. But large quantum computers may still not be easily accessible. Hard to say.
comment by dr_s · 2023-10-01T05:48:27.403Z · LW(p) · GW(p)
I think you're somewhat downplaying the major impacts even just human level (say, as good as a talented PhD student) AGI could have. The key difference is just by how much the ceiling on specialised intellectual labour would be lifted. Anything theoretical or computational could have almost infinite labour thrown at it, you could try more and risk more. And I'd be really surprised if you couldn't for example achieve decent DFT, or at least a good fast XC functional, using a diffusion model or such, given enough attempts. AGI coupled with robotic chemical synthesis labs (which are already a thing) could try a lot of things in parallel. And if the time horizon for the research was really 60 years, a "simple" speedup of 6X makes that a decade. That's not a lot, and that's just this avenue of research, and assuming no ASI.
comment by transhumanist_atom_understander · 2023-10-02T00:29:17.739Z · LW(p) · GW(p)
Very interesting. A few comments.
I think you mentioned something like this, but Drexler expected a first generation of nanotechnology based on engineered enzymes. For example, in "Engines of Creation", he imagines using enzymes to synthesize airplane parts. Of course the real use of enzymes is much more restricted: cleaning products such as dishwasher detergent, additives in food, pharmaceutical synthesis. It has always seemed to me that someone who really believed Drexler and wanted to bring his imagined future about would actually not be working on anything like the designs in "Nanosystems", but on bringing down the cost of enzyme manufacturing. From that perspective it's interesting that you note that the most promising direction in Drexlery mechanosynthesis is DNA origami. Not quite what Drexler imagined (nucleic acid rather than protein), but still starting with biology.
Also, I think it's very interesting that silicon turned out to be easier than diamond. While I agree with Yudkowsky that biology is nowhere near the limits of what is possible on the nanometer-scale due to constraints imposed by historical accidents, I disagree with Yudkowsky's core example of this, the weak interactions holding proteins in the folded configuration. Stronger bonds make things harder, not easier. Maybe the switch from diamond to silicon is an illustration of that.
Editing to add one more comment... Drexler's definition of "diamondoid" is indeed strange. If we take it literally, it seems that glass is "diamondoid". But then, "diamondoid" microbes already exist, that is, diatoms. Or at least, microbes with "diamondoid" cell walls.
Replies from: pmcarlton-1↑ comment by pmcarlton (pmcarlton-1) · 2023-10-04T01:50:39.053Z · LW(p) · GW(p)
Stronger bonds make things harder, not easier.
Yes, this exactly. I can't envision what kind of informationally-sensitive chemistry is supposed to happen at standard temperature and pressure in an aqueous environment, using "diamondoid".
Proteins are so capable, precisely because they are free to jiggle around, assume different configurations and charge states, etc.
Without a huge amount of further clarification, I think this "nanotech doom" idea has to go. (and I'm not aware of any other instant, undetectable AI takeover scheme suggestions that don't rely on new physics)
comment by Review Bot · 2024-03-10T20:36:09.154Z · LW(p) · GW(p)
The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?
comment by hold_my_fish · 2023-10-05T09:46:47.442Z · LW(p) · GW(p)
I found this very interesting, and I appreciated the way you approached this in a spirit of curiosity, given the way the topic has become polarized. I firmly believe that, if you want any hope of predicting the future, you must at minimum do your best to understand the present and past.
It was particularly interesting to learn that the idea has been attempted experimentally.
One puzzling point I've seen made (though I forget where) about self-replicating nanobots: if it's possible to make nano-sized self-replicating machines, wouldn't it be easier to create larger-sized self-replicating machines first? Is there a reason that making them smaller would make the design problem easier instead of harder?
comment by jacopo · 2023-10-01T13:19:36.141Z · LW(p) · GW(p)
It seems very weird and unlikely to me that the system would go to the higher energy state 100% of the time
I think vibrational energy is neglected in the first paper, it would be implicitly be accounted for in AIMD. Also, the higer energy state could be the lower free energy state - if the difference is big enough it could go there nearly 100% of the time.
comment by avturchin · 2023-09-30T00:03:03.889Z · LW(p) · GW(p)
For significant speedup of computations, super-advanced AI needs new computational medium, and nanotech could be such medium.
But this creates a problem of chicken and an egg: to invent nanotech, the AI has to be able to perform significantly more computations which are available now. But it can't do this without nanotech.
This creates an obstacle to the idea that first AI will be able to rush to create nanotech.
Replies from: MichaelStJules↑ comment by MichaelStJules · 2023-09-30T01:56:49.146Z · LW(p) · GW(p)
Are you thinking quantum computers specifically? IIRC, quantum computers can simulate quantum phenomena much more efficiently at scale than classical computers.
EDIT: For early proofs of efficient quantum simulation with quantum computers, see:
- Lloyd, 1996 https://fab.cba.mit.edu/classes/862.22/notes/computation/Lloyd-1996.pdf
- Zalka, 1996 https://arxiv.org/abs/quant-ph/9603026v2