.

Manifold is pretty weak evidence for anything >=1 year away because there are strong incentives to bet on short term markets.

I'm not sure how to integrate such long-term markets from Manifold. But anyway, that market seems to have a very vague notion of emulation. For example, it doesn't mention anything about the emulation doing any useful cognitive work!

Once we get superintelligence, we might get every other technology that the laws of physics allow, even if we aren't that "close" to these other technologies.

Maybe they believe in a $\approx 38\%$ chance of superintelligence by 2039.

PS: Your comment may have caused it to drop to 38%. :)

An example I love of a helpful brain adaptation with few downsides that I know of, which hasn't spread far throughout mammals is one in seal brains. Seals, unlike whales and dolphins, had an evolutionary niche which caused them to not get as good at holding their breathe as would be optimal for them. They had many years of occasionally diving too deep and dying from brain damage related to oxygen deprivation (ROS in neurons). So, some ancient seal had a lucky mutation that gave them a cool trick. The glial cells which support neurons can easily grow back even if their population gets mostly wiped out. Seals have extra mitochondria in their glial cells and none in their neurons, and export the ATP made in the glial cells to the neurons. This means that the reactive oxygen species from oxygen deprivation of the mitochondria all occur in the glia. So, when a seal stays under too long, their glial cells die instead of their neurons. The result is that they suffer some mental deficiencies while the glia grow back over a few days or a couple weeks (depending on the severity), but then they have no lasting damage. Unlike in other mammals, where we lose neurons that can't grow back.

Given enough time, would humans evolve the same adaptation (if it does turn out to have no downsides)? Maybe, but probably not. There just isn't enough reproductive loss due to stroke/oxygen-deprivation to give a huge advantage to the rare mutant who lucked into it.

But since we have genetic engineering now... we could just give the ability to someone. People die occasionally competing in deep freediving competitions, and definitely get brain damage. I bet they'd love to have this mod if it were offered.

Also, sometimes there are 'valleys of failure' which block off otherwise fruitful directions in evolution. If there's a later state that would be much better, but to get there would require too many negative mutations before the positive stuff showed up, the species may simply never get lucky enough to make it through the valley of failure.

This means that evolution is heavily limited to things which have mostly clear paths to them. That's a pretty significant limitation!

Signaling molecules can potentially take advantage of nature's GRNs. Are you saying that ultrasound might too?

(Just acknowledging that my response is kinda disorganized. Take it or leave it, feel free to ask followups.)

Most easy interventions work on a generational scale. There's pretty easy big wins like eliminating lead poisoning (and, IDK, feeding everyone, basic medicine, internet access, less cannibalistic schooling) which we should absolutely do, regardless of any X-risk concerns. But for X-risk concerns, generational is pretty slow.

This is both in terms of increasing general intelligence, and also in terms of specific capabilities. Even if you bop an adult on the head and make zer +2SDs smarter, ze still would have to spend a bunch of time and effort to train up on some new field that's needed for the next approach to further increasing intelligence. That's not a generational scale exactly, maybe more like 10 years, but still.

We're leaking survival probability mass to an AI intelligence explosion year by year. I think we have something like 0-2 or 0-3 generations before dying to AGI.

To be clear, I'm assuming that when you say "we don't need 7SDs", you mean "we don't need to find an approach that could give 7SDs". (Though to be clear, I agree with that in a literal sense, because you can start with someone who's already +3SDs or whatever.) A problem with this is that approaches don't necessarily stack or scale, just because they can give a +2SD boost to a random person. If you take a starving person and feed zer well, ze'll be able to think better, for sure. Maybe even +2SDs? I really don't know, sounds plausible. But you can't then feed them much more and get them another +2SDs -- maybe you can get like +.5SD with some clever fine-tuning or something. And you can't then also get a big boost from good sleep, because you probably already increased their sleep quality by a lot; you're double counting. Most (though not all!) people in, say, the US, probably can't get very meaningfully less lead poisoned.

Further, these interventions I think would generally tend to bring people up to some fixed "healthy Gaussian distribution", rather than shift the whole healthy distribution's mean upward. In other words, the easy interventions that move the global average are more like "make things look like the developed world". Again, that's obviously good to do morally and practically, but in terms of X-risk specifically, it doesn't help that much. Getting 3x samples from the same distribution (the healthy distribution) barely increases the max intelligence. Much more important (for X-risk) is to shift the distribution you're drawing from. Stronger interventions that aren't generational, such as prosthetic connectivity or adult brain gene editing, would tend to come with much more personal risks, so it's not so scalable--and I don't think in terms of trying to get vast numbers of people to do something, but rather just in terms of making it possible for people to do something if they really want to.

So what this implies is that either

1. your approach can scale up (maybe with more clever technology, but still riding on the same basic principle), or
2. you're so capable that you can keep coming up with different effective, feasible approaches that stack.

So I think it matters to look for approaches that can scale up to large intelligence gains.

To put things in perspective, there's lots of people who say that {nootropics, note taking, meditation, TCMS, blood flow optimization, ...} give them +1SD boost or something on thinking ability. And yet, if they're so smart, why ain't they exploding?

All that said, I take your point. It does make it seem slightly more appealing to work on e.g. prosthetic connectivity, because that's a non-generational intervention and could plausibly be scaled up by putting more effort into it, given an initial boost.

I think brain editing is maybe somewhat less scalable, though I'm not confident (plausibly it's more scalable; it might depend for example on the floor of necessary damage from editing rounds, and might depend on a ceiling of how much you can get given that you've passed developmental windows). Support for thinking (i.e. mental / social / computer tech) seems like it ought to be scalable, but again, why ain't you exploding? (Or in other words, we already see the sort of explosion that gets you; improving on it would take some major uncommon insights, or an alternative approach.) Massive neural transplantation might be scalable, but is very icky. If master regulator signaling molecules worked shockingly well, my wild guess is that they would be a little scalable (by finding more of them), but not much (there probably isn't that much room to overclock neurons? IDK); they'd be somewhat all-or-nothing, I guess?

Meditation has been practiced for many centuries and millions practice it currently.

Please list 3 people who got deeply into meditation, then went on to change the world in some way, not counting people like Alan Watts who changed the world by promoting or teaching meditation.

Got any evidence?

How about TMS/tFUS/tACS => "meditation"/reducing neural noise?

Drastic improvements in mental health/reducing neural noise & rumination are way more feasible than increasing human intelligence (and still have huge potential for very high impact when applied on a population-wide scale [1]), and are possible to do on mass-scale (and there are some experimental TMS protocols like SAINT/accelerated TMS which aim to capture the benefits of TMS on a 1-2 week timeline) [there's also wave neuroscience, which uses mERT and works in conjunction with qEEG, but I'm not sure if it's "ready enough" yet - it seems to involve some sort of guesswork and there are a few negative reviews on reddit]. There are a few accelerated TMS centers and they're not FDA-approved for much more than depression, but if we have fast AGI timelines, the money matters less.

[speeding up feedback loops are also important for mass-adoption - which both accelerated TMS/SAINT and the "intense tACS program" that people like neurofield [Nicholas Dogris/Tiffany Thompson] and James Croall people try to do]. Ideally, the TMS/SAINT or tACS should be done in conjunction with regular monitoring of brainwaves with qEEG or fMRI throughout.

Effect sizes of tFUS are said to be small relative to certain medications/drugs [this is true for neurofeedback/TMS/tACS in general], but part of this may be that people tend to be conservative with tFUS. Leo Zaroff has created an approachable tFUS community in the bay area. Still worth trying b/c the opportunity cost of trying them (with the right people) is very low (and very few people in our communities have heard of them).

There are some like Jeff Tarrant and the Neurofield people (I got to meet many of them at ISNR2024 => many are coming to the Suisun Summit now) who explore these montages.

Making EEG (or EEG+fNIRS) much easier to get can be high impact relative to amount of effort invested [with minimal opportunity cost]). I was pretty impressed with the convenience of Zeto's portable EEG headset at #Sfn24, as well as the convenience of the imedisync at #ISNR2024 [both EEG headsets cost $20,000, which is high but not insurmountable - eg if given some sort of guarantee on quality and useability I might be willing to procure one] but still haven't evaluated the signal quality of each when comparing them to other high-quality EEG montages like the deymed). It also makes it easier to create the true dataset of EEGs (also look into what Jonathan Xu is doing, though his paper is more about visual processing than mental health). We also don't even have proper high-quality EEG+HEG+fMRI+fNIRS datasets of "high intelligence" people relative to others [especially when measuring these potentials in response to cognitive load - I know Thomas Feiner has helped create a freecap support group and has done a lot of thought on ERPs and their response to cognitive load - he helped take my EEG during a brainmaster session at #ISNR2024]

I've found that smart people in general are extremely underexposed to psychometrics/psychonomics (there are not easy ways to enter those fields even if you're a psychology or neuroscience major), and there is a lot of potential for synergy in this area.

[1] esp given the prevalence of anxiety and other mental health issues of people within our communities

These are both addressed in the post.

I think that from like +6.3std the heavytail becomes even a lot stronger because those people can bootstrap themselves extremely good mental software.

I agree something like this happens, I just don't think it's that strong of an effect.

I think one me::Tsvi::+7std person would probably be enough to put humanity on a path to success (given Tsvi timelines), so the "repeatedly" criterion seems a bit off to me.

• A single human still has pretty strong limitations. E.g. fixed skull size (without further intervention); other non-scalable hardware (~one thread of attention, one pair of eyes and hands); self-reprogramming is just hard; benefits of self-reprogramming don't scale (hard to share with other people).
• Coercion is bad; without coercion, a supergenius might just not want to work on whatever is strategically important for humanity.
• It doesn't look to me like we're even close to being able to figure out AGI alignment, or other gnarly problems for that matter (such as a decoding egregores). So we need a lot more brainpower, lots of lottery tickets.
• There's a kind of power that comes from having many geniuses--think Manhattan project.

for the few +6std people on earth it might just give +0.2std or +0.3std,

Not sure what you're referring to here. Different methods have different curves. Adult brain editing would have diminishing returns, but nowhere near that diminishing.

it's sorta vice versa that extremely smart individuals might find ways to significantly leverage their capability

Plausibly, though I don't know of strong evidence for this. For example, my impression is that modern proof assistants still aren't in a state where a genius youngster with a proof assistant can unlock what feels like the possibility of learning a seemingly superhuman amount of math via direct dialogue with the truth--but I could imagine this being created soon. Do you have other evidence in mind?

adult augmentation 2-3std for the average person seems plausible, but for the few +6std people on earth it might just give +0.2std or +0.3std, which tbc I think is incredibly worthwhile.

Such high diminishing returns in g based on genes seems quite implausible to me, but would be happy if you can point to evidence to the contrary. If it works well for people with average Intelligence, I'd expect it to work at most half as well with +6sd.

Yeah, there's a value-drift column in the table of made-up numbers. Values matter and are always at stake, and are relatively more at stake here; and we should think about how to do these things in a way that avoids core value drift.

You have major advantages when creating humans but tweaked somehow, compared to creating de novo AGI.

• The main thing is that you're starting with a human. You start with all the stuff that determines human values--a childhood, basal ganglia giving their opinions about stuff, a stomach, a human body with human sensations, hardware empathy, etc. Then you're tweaking things--but not that much. (Except for in brain emulation, which is why it gets the highest value drift rating.)
• Another thing is that there's a strong built-in limit on the strength of one human: skullsize. (Also other hardware limits: one pair of eyes and hands, one voicebox, probably 1 or 1.5 threads of attention, etc.) One human just can't do that much--at least not without interfacing with many other humans. (This doesn't apply for brain emulation, and potentially applies less for some brain-brain connectivity enhancements.)
• Another key hardware limit is that there's a limit on how much you can reprogram your thinking, just by introspection and thinking. You can definitely reprogram the high-level protocols you follow, e.g. heuristics like "investigate border cases"; you can maybe influence lower-level processes such as concept-formation by, e.g., getting really good at making new words, but you maybe can't, IDK, tell your brain to allocate microcolumns to analyzing commonalities between the top 1000 best current candidate microcolumns for doing some task; and you definitely can't reprogram neuronal behavior (except through the extremely blunt-force method of drugs).
• A third thing is that there's a more plausible way to actually throttle the rate of intelligence increase, compared to AI. With AI, there's a huge compute overhang, and you have no idea what dial you can turn that will make the AI become a genuinely creative thinker, like a human, but not go FOOM. With humans, for the above reasons, you can guess pretty reasonably that creeping up the number of prosthetic connections, or the number of transplanted neurons, or the number of IQ-positive alleles, will have a more continuous effect.
• A fourth major advantage is that you can actually see what happens. In the genomic engineering case, you can see which alleles lead to people being sociopaths or not. You get end-to-end data. And then you can just select against those (but not too strongly). (This is icky, and should be done with extreme care and caution and forethought, but consider status quo bias--are the current selection pressures on new humans's values and behaviors really that great?)

instantly recall any fact and solve any math problem

You mean, recall any fact that's been put into text-searchable form in the past and by you, and solve any calculation problem that's in a reasonably common form.

I'm saying that the effect on philosophical problem-solving is just not very large. Yeah, if you've been spending 80% of your time on manually calculating things and 20% on "leaps of logic", and you could just as well spend 90% on the leaps, then calculators help a lot. But it's not by making you be able to do significantly better leaps. Maybe you can become better by getting more practice or something? But generally skills tend to plateau pretty sharply--there's always new bottlenecks, like a clicker game. If an improvement only addresses some smallish subset of the difficulty involved in some overall challenge, the overall challenge isn't addressed that much.

Like, if you could do calculations with 10x less effort, what calculations would you do to solve alignment, or get AGI banned, or make sure everyone gets food, or fix the housing crisis, or ....?

To put it a different way, I don't think Gödel's lack of a big fast calculator mattered too much?

I think it makes sense to pick the low-hanging fruit first (then attempt incrementally harder stuff with the benefit of being slightly smarter)

No, this doesn't make sense.

I think the stuff you're doing is probably fun / cool / interesting / helpful / something you like. That's great! You don't need to make an excuse for doing it, in terms of something about something else.

But no, that's not the right way to make really smart humans. The right way is to directly create the science and tech. You're saying something like "it stands to reason that if we can get a 5% boost on general intelligence, we should do that first, and then apply that to the tech". But

• It's not a 5% boost to the cognitive capabilities that are the actual bottlenecks to creating the more powerful tech. It's less than that.
• What you're actually doing is doing the 5% boost, and never doing the other stuff. Doing the other stuff is better for the purposes of making a bunch of supergeniuses. (Which, again, doesn't have to be your goal!)

Is the situation so dire with AI intelligence explosion that a human one must exist to counter balance it?

I wouldn't exactly say "counter balance". It's more like we, as humans, want to get ahead of the AI intelligence explosion. Also, I wouldn't advocate for a human intelligence explosion that looks like what an AI explosion would probably look like. An explosion sounds like gaining capability as fast as possible, seizing any new mental technology that's invented and immediately overclocking it to invent the next and the next mental technology. That sort of thing would shred values in the process.

We would want to go about increasing the strength of humanity slowly, taking our time to not fuck it up. (But we wouldn't want to drag our feet either--there are, after all, still people starving and suffering, decaying and dying, by the thousands and tens of thousands every day.)

But yes, the situation with AI is very dire.

augmented humans stand a decent chance of wanting to go full throttle on all things Artificial

I'm not following. Why would augmented humans have worse judgement about what is good for what we care about? Or why would they care about different things?

Ok, I added some links to "Downside risks of genomic selection".

Any of these techniques would surely be available to only a small fraction of the world’s population.

Not true! This consideration is the main reason I included a "unit price" column. Germline engineering should be roughly comparable to IVF, i.e. available to middle class and up; and maybe cheaper given more scale; and certainly ought be subsidized, given the decreased lifetime healthcare costs alone.

greatly increase the distance between the haves and the have-nots

Eh, unless you can explain this more, I think you've been brainwashed by Gattaca or something. Gattaca conflates class with genetic endowment, which is fine because it's a movie about class via a genetics metaphor, but don't be confused that it's about genetics. Did the invention of smart phones increase or decrease the distance? In general, some technologies scale with money, and other technologies scale by bodycount. Each person only gets one brain to receive implants and stuff. Elon Musk, famously extremely rich and baby-obsessed, has what... 12 kids? A peasant could have 12 kids if they wanted to! Germline engineering would therefore be extremely democratic, at least for middle class and up. The solution, of course, is to make the tech even cheaper and more widely available, not to inflict preventable disease and disempowerment on everyone's kids.

Anecdotally, I think there’s somewhat of an inverse correlation between intelligence and empathy in humans.

Stats or GTFO.

Superintelligence will probably affect a person’s personality/values in ways we can’t predict. It could cause depression, psychopathic behavior, who knows.

First, the two specific things you listed are quite genetically heritable. Second, 7 SDs -- which is the most extreme form that I advocate for -- is only a little bit outside the Gaussian human distribution. It's just not that extreme of a change. It seems quite strange to postulate that a highly polygenic trait, if pushed to 5350 out of 10000 trait-positive variants, would suddenly cause major psychological problems, whereas natural-born people with 5250 or 5300 out of 10000 trait-positive variants are fine.

I think the terror reaction is honestly pretty reasonable. ([edit: Not, like, like, necessarily meaning one shouldn't pursue this sort of direction on balance. I think the risks of doing this badly are real and I think the risks of not doing anything are also quite real and probably great for a variety of reasons])

One reason I nonetheless think this is very important to pursue is that we're probably going to end up with superintelligent AI this century, and it's going to be dramatically more alien and scary than the tail-risk outcomes here.

I do think the piece would be improved if it acknowledged and grappled with that more.

What do you mean? Why would they be depressed? Do you mean because they'd be pressured into working on AGI alignment, or something? Yeah, don't do that. Same as with any other kids, you teach them to be curious and good and kind and strong and free and responsible and so on.

I guess that falls under "value drift" in the table. But yeah I think that's extremely unlikely to happen without warning, except in the case of brain emulations. I do think any of these methods would be world-changing, that therefore extremely dangerous and would demand lots of care and caution.

So your suggestion for accelerating strong human intelligence amplification is ...checks notes... "don't do anything"?

Or are you suggesting accelerating AI research in order to use the improved AI faster? I guess technically that would accelerate amplification but seems bad to do.

Maybe AI could help with some parts of the research. But

1. we probably don't need AI to do it, so we should do it now, and
2. if we're not all dead, there will still be a bunch of research that has to be done by humans.

On a psychologizing note, your comment seems like part of a pattern of trying to wriggle out of doing things the way that is hard that will work. Looking for such cheat codes is good but not if you don't aggressively prune the ones that don't actually work -- hard+works is better than easy+not-works.

mentioned in the FAQ