Posts
Comments
EA has an extraordinary bad image right now, thanks largely to FTX. EA is not a good association to have in any context other than its base.
I suspect the pushback from within NIST has more to do with the fact that their budget has been cut to pay for this and very valuable projects put into indefinite suspension, for a cause that basically no one there supports.
The proof is last paragraph of his post.
It is not fully "local" but who cares?
Non locality is a big deal. That the underlying physics of the universe has a causal speed limit that applies to everything (gravity and QM) yet somehow doesn’t apply to pilot waves is harder to explain away than multiverse. The multiverse makes you uncomfortable, but it is a simpler physical theory than pilot waves.
This seems like a really bad idea. If it happened to me, I’d just leave. I’m not looking for echo chambers, and I often am motivated to come out of lurking and say something when my view is contrarian. That’s just who I am. Such comments often get downvoted by senior people. That’s just how contrary views are handled in online communities.
Is this not what LW wants? Then I guess I’ll just delete my account and go somewhere else.
I think there is a fundamental issue here in the history that is causing confusion. The originators of the AGI term did in fact mean it in the context of narrow vs general AI as described by OP. However they also (falsely!) believed that this general if mediocre capability would be entirely sufficient to kickstart a singularity. So in a sense they simultaneously believed both without contradiction, and you are both right about historical usage. But the events of recent years have shown that the belief AGI=singularity was a false hope/fear.
Thank you for writing this. I have been making the same argument for about two years now, but you have argued the case better here than I could have. As you note in your edit it is possible for goal posts to be purposefully moved, but this irks me for a number of reasons beyond mere obstinacy:
-
The transition from narrow AI to truly general AI is socially transformative, and we are living through that transition right now. We should be having a conversation about this, but are being hindered from doing so because the very concept of Artificial General Intelligence has been co-opted.
-
The confusion originates I think from the belief by many people in the pre-GPT era that achieving general intelligence is all that is required to kick off the singularity. GPT demonstrates quite clearly that this belief is false. This doesn’t mean the foomers/doomers are wrong to be worried about AI, but it is a glaring hole in the standard arguments for their position, and should be talked about more, but confusion over terminology is preventing that from happening.
-
Moving goalposts to define AGI as radically transformative and/or superhuman capabilities is begging the question. To say that we haven’t achieved AGI because modern AI hasn’t literally taken over the world and/or killed all humans is to assume that unaligned AI would necessarily lead to such outcomes. Pre-2017 AI x-risk people did routinely argue that even a middling-level artificial general intelligence would be able to enter a recursive self-improvement cycle and reach superhuman capabilities in short order. Although I have no insider info, I believe this line of thinking is what led to EY’s public meltdown a year or so ago. I disagree with him, but I respect that he took his line of thinking to its logical conclusion and accepted the consequences. Most of the rationalist community has not updated on the evidence of GPT being AGI as EY has, and I think this goalpost moving has a lot to do with that. Be intellectually honest!
The AI x-risk community claimed that the sky was falling, that the development of AGI would end the human race. Well, we’re now 2-7 years out from the birth of AGI (depending on which milestone you choose), and SkyNet scenarios seem no closer to fruition. If the x-risk community wants to be taken seriously, they need to confront this contradiction head-on and not just shift definitions to avoid hard questions.
You highlighted "disagree" on the part about AGI's definition. I don't know how to respond to that directly, so I'll do so here. Here's the story about how the term "AGI" was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
https://web.archive.org/web/20181228083048/http://goertzel.org/who-coined-the-term-agi/
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym "AGI" that is never specified. My assumption is that when people say AGI here they mean Bostrom's ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
>>10^30 FLOP
By the way, where's this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
I generally have an intuition like "it's really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible"
Ok let's try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can't actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don't replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don't know that. You're not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn't clear to me. But let's say you do. Which experiments should you discount? Which should you assume to be correct? If you're allowed to start choosing which reported results you believe and which you don't, you've lost the plot. There could be millions of possible heuristics which partially match the literature and there's no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can't. Not any more than Descartes could think his way to absolute truth.
Also for what it's worth you've made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That's hindsight bias, but also selection bias. Nanotechnology doesn't exist. Even if it would work if created, there's no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don't believe this as I'm running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it's essential to point out the logical error.)
(Re: Alien Message) I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
Why? You've gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren't going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I'm not saying it's not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it's a fucking terrible intuition pump.
Uhhhh, I'm not sure I agree with this as it doesn't seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like "able to speak slightly coherant english (GPT-1?) and classify images"?
The transformer architecture introduced in 2017 is:
- Artificial: man-made
- General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
- Intelligent: able to construct efficient solutions to new problems it hasn't seen.
Artificial General Intelligence. A.G.I.
If you're thinking "yeah, but.." then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we're already well within the era of artificial general intelligence.
(Maybe EY's very public meltdown a year ago is making more sense now? But rest easy, EY's predictions about AI x-risk have consistently been wildly off the mark.)
One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
By merely thinking about it, and not running any experiments? No, absolutely not. I don't you understood my post if you assume I'd think otherwise.
Try this: I'm holding a certain number of fingers behind my back. You and a team of 10,000 of the smartest human engineers and scientists have a billion years to decide, without looking, what your guess will be as to how many fingers I'm holding behind my back. But you only get one chance to guess at the end of that billion years.
That's a more comparable example.
See also That Alien Message
Please don't use That Alien Message as an intuition pump. There's a tremendous amount wrong with the sci-fi story. Not least of which is that it completely violates the same constraint you put into your own post about constraining computation. I suggest doing your own analysis of how many thought-seconds the AI would have in-between frames of video, especially if you assume it to be running as a large inference model.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities. That is where EY went wrong, and a lot of the LW crowd too. If you permit dividing by zero or multiplying by infinity, then you can easily convince yourself of anything. AI isn't magic, and AGI isn't a free pass to believe anything.
PS: We've had AGI since 2017. That'd better be compatible with your world view if you want accurate predictions.
If you say “Indeed it's provable that you can't have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately” I am quite likely to go on a digression where I try to figure out what proof you're pointing at and why you think it's a fundamental barrier
By "proof" I meant proof by contradiction. DFT is a great O(n^3) method for energy minimizing structures and exploring electron band structure, and it is routinely used for exactly that purpose. So much so that many people conflate "DFT" with more accurate ab initio methods, which it is not. However DFT utterly ignores exchange correlation terms and so it doesn't model van der Waals interactions at all. Every design for efficient and performant molecular nanotechnology--the ones that get you order-of-magnitude performance increases and therefore any benefit over existing biology or materials science--involve vdW forces almost exclusively in their manufacture and operation. It's the dominant non-covalent bonded interaction at that scale.
That's the most obvious example, but also a lot of the simulations performed by Merkle and Freitas in their minimal toolset paper give incorrect reaction sequences in these lower levels of theory, as they found out when they got money to attempt it in the lab. Without pointing to their specific failure, you can get a hint of this in surface science. Silicon, gold, and other surfaces tend to have rather interesting surface clustering and reorganization effects, which are observable by scanning probe microscopy. These are NOT predicted by the cheaper / computationally tractable codes, and they are an emergent property of higher-order exchange correlations in the crystal structure. These nevertheless have enough of an effect to drastically reshape the surface of these materials, making calculation of those forces absolutely required for any attempt to build off the surface.
Attempting to do cheaper simulations for diamondoid synthesis reactions gave very precise predictions that didn't work as expected in the lab. How would your superintelligent AI know that uncalculated terms dominate in the simulation, and make corrective factors without having access to those incomputable factors?
- Making inhumanly close observation of all existing data
- Noticing new, inhumanly-complex regularities in said data,
- Proving new simplifying regularities from theory
I think you vastly overestimate how much knowledge is left to extracted from the data. AI has made tremendous advances in recent years where it has been able to consume huge amounts of data, far in excess of what any group of humans could analyze. This, on the other hand, is a data-poor regime.
- Inventing new algorithms for heuristic simulation
This is happening right now. There are a variety of machine-learned molecular mechanics force fields that have been published in the last few years. The most interesting one I've found used periodic crystal ab initio simulation methods to create a force field potential that ended up being very good for liquid and gas-phase chemistry, which it was not trained on.
But the relevant question (if you want to talk about AI x-risk by means of bootstrapping nanotech) is how accurate they are outside of the domain where we have heaps of hard evidence, because we don't have a ground truth to compare against in those environments.
- Finding restricted domains where easier regularities hold
- Bifurcating problem space and operating over each plausible set,
Human engineers are very good at this. It's not the limiting factor.
- Sending an interesting email to a research lab to get choice high-ROI data
What lab. There's literally no one doing the relevant research, or equipped to easily do it without years of preparatory chemical synthesis and surface characterization.
Which is really the point and the crux of the matter. It will take an extended, years-long research effort to create molecular nanotechnology. It's not something you can plausibly do in secret, and certainly not something you can shorten by simulation or bayesian inference.
Do you have a maths or computer science background? I ask because some of the questions you ask are typical of maths or CS, but don't really make sense in this context.
Take protein folding. Existing techniques for estimating how a protein folds are exponential in complexity (making the problem NP-hard), so running time is bounded by O(2^n). But what's the 'n'? In these algorithms it is the number of amino acids because that's the level at which they are operating. For quantum chemistry the polynomial running times have 'n' being the number of electrons in the system, which can be orders of magnitude higher. That's why O(n^4) can be much, much worse than O(2^n) for relevant problem sizes--there's no hope of doing full protein folding simulations by ab initio methods. The second reason they're not comparable is time steps. Ab initio molecular mechanics methods can't have time steps larger than 1 femtosecond. Proteins fold on the timescale of milliseconds. This by itself is what makes all ab initio molecular mechanics methods infeasible for protein folding, even though it's a constant factor ignored by big-O notation.
Staying on the issue of complexity, from a technical / mathematical foundation perspective every quantum chemistry code should also have exponential running time, and this is trivially proved by the fact that every particle technically does interact with every other particle. But all of these interaction terms fall off with at least 1/r^2, and negative power increases with each corrective term. For 1/r^2 and 1/r^3, there are field density tricks that can be used for accurate sub-exponential modeling. For 1/r^4 or higher it is perfectly acceptable to just have distance cutoffs which cutout the exponential term. How accurate a simulation you will get is a question of how many of these corrective terms you include. The more you include, the more higher-order effects you capture, but you also get hit with higher-order polynomial running times.
To bring a long story to an abrupt end, you don't "prove" correctness of these various codes in a mathematical sense. They're all physically incorrect because they involve approximations. Rather for various use cases you can experimentally show that certain terms dominate, and you need to make sure that the code you use accurately captures the relevant terms for the domain and scale of the problem you are simulating. This is an experimental determination, not a mathematical proof. You literally go measure things in the lab.
The protein data bank contains hundreds of thousands of structures which have had their atomic coordinates determined by various experimental methods, providing a ground truth for protein folding heuristics. There is no such source of truth for diamondoid nano-mechanical machines parts and the mechanosynthesis steps that produce them. If you want to train a better reactive force field heuristic that could make the relevant design simulations tractable, you need a source of truth. The only thing available is various quantum chemistry approximations, as you need to approximate to be computable. These fail as source of truth because you don't know which approximations are reasonable and which are not without experiment data, which we don't have. So you could train a heuristic, yes, but you'd have absolutely no way of assessing its accuracy.
There's a common saying in the AI field: "garbage in, garbage out." The current state of non-empirical ab initio quantum chemistry is garbage. You're saying that AI could take the garbage and by mere application of thought turn it into something useful. That's not in line with the actual history of the development of useful AI outputs.
I'm not saying that AI can't develop useful heuristic approximations for the simulation of gemstone-based nano-mechanical machinery operating in ultra-high vacuum. I'm saying that it can't do so as a one-shot inference without any new experimental work, which is a trope that shows up again and again in certain high-profile people's descriptions of AI x-risk scenarios.
It's coming along quite fast. Here's the latest on ML-trained molecular dynamics force fields that (supposedly) approach ab initio quality:
https://www.nature.com/articles/s41524-024-01205-w
These are potentially tremendously helpful. But in the context of AI x-risk it's still not enough to be concerning. A force field that gives accurate results 90% of the time would tremendously accelerate experimental efforts. But it wouldn't be reliable enough to one-shot nanotech as part of a deceptive turn.
For the most part we're avoiding/designing around compute constraints. Build up our experimental validation and characterization capabilities so that we can "see" what happens in the mechsyn lab ex post facto. Design reaction sequences with large thermodynamic gradients, program the probe to avoid as much as possible side reaction configurations, and then characterize the result to see if it was what you were hoping for. Use the lab as a substitute for simulation.
It honestly feels like we've got a better chance of just trying to build the thing and repeating what works and modifying what doesn't, than even a theoretically optimal machine learning algorithm could do using simulation-first design. Our competition went the AI/ML approach and it killed them. That's part of why the whole AGI-will-design-nanotech thing bugs me so much. An immense amount of money and time has been wasted on that approach, which if better invested could have gotten us working nano-machinery by now. It really is an idea that eats smart people.
Yeah just for scaling problems are an issue. These are not embarrassingly parallel problems that can be easily sharded onto different CPUs. A lot of interconnect is needed, and as you scale up that becomes the limiting factor.
What codes did you use? The issue here is that DFT gives bad results because it doesn’t model the exchange-correlation energy functional very well, which ends up being load bearing for diamondoid construction reaction pathways performed in UHV using voltage-driven scanning probe microscopy to perform mechanochemistry. You end up having to do some form of perturbation theory, of which there are many variants all giving significantly different results, and yet are all so slow that running ab initio molecular dynamics simulations are utterly out of the question so you end up spending days of supercompute time just to evaluate energy levels of a static system and then use your intuition to guess whether the reaction would proceed. Moving to an even more accurate level of theory that would indisputably give good results for these systems but would have prohibitive scaling factors [O(n^5) to O(n^7)] preventing their use for anything other than toy examples.
I am at a molecular nanotechnology startup trying to bootstrap diamondoid mechanosynthesis build tools, so these sorts of concerns are my job as well.
The fact that this has been an extremely active area of research for over 80 years with massive real-world implications, and we're no closer to finding such a simplified heuristic. This isn't like protein folding at all--the math is intrinsically more complicated with shared interaction terms that require high polynomial count running times. Indeed it's provable that you can't have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately.
Of course there could be some heuristic-like approximation which works in the relevant cases for this specific domain, e.g. gemstone crystal lattice structures under ultra-high vacuum conditions and cryogenic temperatures. Just like there are reasonably accurate molecular mechanics models that have been developed for various specific materials science use cases.
But here's the critical point: these heuristics are developed by evolving models based on experiment. In other domains like protein folding you can have a source of truth for generative AI training that uses simulation. That's how AlphaFold was done. But for chemical reactions of the sort we'd be interested here the ab-initio methods that give correct results are simply too slow to use for this. Not "buy a bigger GPU" slow, but "run the biggest supercomputer for a quintillion years" slow. So the solution is to do the process manually, running actual experiments to refine the heuristic models at each step, much machine learning training models would do. These are called semi-empirical methods, and they work. But we don't have a good semi-empirical model for the type of reactions concerned here, and it would take years of painstaking lab work to develop, which no amount of super intelligence can avoid.
It's magical thinking to assume that an AI will just one-shot this into existence.
I don't understand your objection. People who actually use computational chemistry--like me, like titotal--are familiar with its warts and limitations. These limitations are intrinsic to the problem and not expressions of our ignorance. Diamondoid mechanosynthetic reactions depend on properties which only show up in higher levels of theory, the ones which are too expensive to run for any reasonably sized system. To believe that this limitation won't apply to AI, just because, is magical thinking.
I wasn't saying anything with respect to de-novo self-replicating nano machines, a field which has barely been studied by anyone and which we cannot adequately say much about at all.
OP's post regarding the topic is here (it is also linked to in the body of this article):
https://titotal.substack.com/p/bandgaps-brains-and-bioweapons-the
I'm running a molecular nanotechnology company, so I've had to get quite familiar with the inner workings and limitations of existing computational chemistry packages. This article does a reasonable job of touching on the major issues. I can tell you that experimentally these codes don't even come close to the precision required to find trajectories that reliably drive diamondoid mechanosynthesis. It's really difficult to get experiments in the lab to match the (over simplified, reduced accuracy) simulations which take days or weeks to run.
Having any hope of adequately simulating mechanosynthetic reactions requires codes that scale with O(n^7). No one even writes these codes, because what would be the point? The biggest thing you could simulate is helium. There are approximations with O(n^3) and O(n^4) running time that are sometimes useful guides, but can often have error bars larger than the relevant energy gaps. The acid test here is still experiment: try to do the thing and see what happens.
AI has potential to greatly improve productivity with respect to nano-mechanical engineering designs because you can build parts rigid and large enough, and operating at slow enough speeds that mechanical forces are unlikely to have chemical effects (bond breaking or forming), and therefore faster O(n^2) molecular mechanics codes can be used. But all the intelligence in the world isn't going to make bootstrapping molecular nanotechnology a one-shot process. It's going to be years of hard-won lab work, whether it's done by a human or a thinking machine.
They do in fact routinely take up the entire supercomputer. I don't think you are comprehending the absolute mind-bogglingly huge scale of compute required to get realistic simulations of mechanosynthesis reactions. It's utterly infeasible without radical advances beyond the present state of the art classical compute capabilities.
Both this and his earlier article by titotal on computational chemistry make the argument that development of molecular nanotechnology can't be a one-shot process because of intrinsic limitations on simulation capabilities.
Yudkowsky claims that one-shot nanotechnology is not load bearing, yet literally every example he gives when pressed involves one-shot nanotech.
It’s a mineral (rock). It’s not ductile at all.
If I make a post or comment starting from the assumption that we are not doomed, and in fact ignore AI x-risk entirely, where would that stand on these moderation guidelines? My reading of the post was that in such a context I would be redirected to read the sequences rather than engaged with.
(Notably the post you link to doesn’t disagree with AI risk, just argues for a long timeline. She explicitly states she agrees with EY on AI x-risk.)
I think you nailed it. The crypto petertodd wrote OpenTimestamps, and his handle is often next to commitment hashes related to that.
Please define what you mean by “AGI” because GPT is AGI. It is:
Artificial — man-made, not natural
General — able to handle any problem domain it is not specifically trained on
Intelligence — solves complex problems using inferred characteristics of the problem domain
What is it that you are imagining AGI to mean, which does not include GPT in its definition?
A key value-prop of LessWrong is that some arguments get to be "reasonably settled", rather than endlessly rehashed.
You are making a huge, and imho unwarranted leap from the article you linked to here. AI risk is very much in the domain of “reasonable people disagree”, unlike the existence of Abrahamic god or the theory of Cartesian dualism.
If moderators are going to start removing or locking posts which disagree on the issue of AI risk, that would be a huge change in the purpose and moderation policy of this site.
A detrimental change, imho.