Posts
Comments
I was all set to disagree with this when I reread it more carefully and noticed it said “superhuman reasoning” and not “superintelligence”. Your definition of “reasoning” can make this obviously true or probably false.
The Antarctic Treaty (and subsequent treaties) forbid colonization. They also forbid extraction of useful resources from Antarctica, thereby eliminating one of the main motivations for colonization. They further forbid any profitable capitalist activity on the continent. So you can’t even do activities that would tend toward permanent settlement, like surveying to find mining opportunities, or opening a tourist hotel. Basically, the treaty system is set up so that not only can’t you colonize, but you can’t even get close to colonizing.
Northern Greenland is inhabited, and it’s at a similar latitude.
(Begin semi-joke paragraph) I think the US should pull out of the treaty, and then announce that Antarctica is now part of the US, all countries are welcome to continue their purely scientific activity provided they get a visa, and announce the continent is now open to productive activity. What’s the point of having the world’s most powerful navy if you can’t do a fait accompli once in a while? Trump would love it, since it’s simultaneously unprecedented, arrogant and profitable. Biggest real estate development deal ever! It’s huuuge!
A fascinating recent paper on the topic of human bandwidth is https://arxiv.org/abs/2408.10234. Title and abstract:
The Unbearable Slowness of Being
This article is about the neural conundrum behind the slowness of human behavior. The information throughput of a human being is about 10 bits/s. In comparison, our sensory systems gather data at an enormous rate, no less than 1 gigabits/s. The stark contrast between these numbers remains unexplained. Resolving this paradox should teach us something fundamental about brain function: What neural substrate sets this low speed limit on the pace of our existence? Why does the brain need billions of neurons to deal with 10 bits/s? Why can we only think about one thing at a time? We consider plausible explanations for the conundrum and propose new research directions to address the paradox between fast neurons and slow behavior.
They’re measuring a noisy phenomenon, yes, but that’s only half the problem. The other half of the problem is that society demands answers. New psychology results are a matter of considerable public interest and you can become rich and famous from them. In the gap between the difficulty of supply and the massive demand grows a culture of fakery. The same is true of nutrition— everyone wants to know what the healthy thing to eat is, and the fact that our current methods are incapable of discerning this is no obstacle to people who claim to know.
For a counterexample, look at the field of planetary science. Scanty evidence dribbles in from occasional spacecraft missions and telescopic observations, but the field is intellectually sound because public attention doesn’t rest on the outcome.
Here is a category of book that I really loved at that age: non-embarrasing novels about how adults do stuff. Since, for me, that age was in 1973, the particular books I name might be obsolete. There’s a series of novels by Arthur Hailey, with titles like “Hotel” and “Airport”, that are set inside the titular institutions, and follow people as they deal with problems and interact with each other. And there is no, or at least minimal, sex, so they’re not icky to a kid. They’re not idealized; there is a reasonable degree of fallibility, venality and scheming, but that is also fascinating. And all the motivations, and the way the systems work, is clearly explained, so it can be understood by an unsophisticated reader.
These books were bestsellers back in the day, so you might be able to find a copy in the library. See if he likes it!
Another novel in this vein is “The view from the fortieth floor”, which is about a badly managed magazine going bankrupt. Doesn’t sound amazing, I know, but if you’re a kid, who’s never seen bad managers blunder into ineluctable financial doom, it’s really neat.
My wife is a middle school librarian. I’ll ask her when I see her for more books like this.
Doesn’t matter, because HPMOR is engaging enough on a chapter-by-chapter basis. I read lots of books when I was a kid when I didn’t understand the overarching plot. As long as I had a reasonable expectation that cool stuff would happen in the next chapter, I’d keep reading. I read “Stand On Zanzibar” repeatedly as a child, and didn’t understand the plot until I reread it as an adult last year. Same with the detective novel “A Deadly Shade of Gold”. I read it for the fistfights, snappy dialogue, and insights into adult life. The plot was lost on me.
In general the human body is only capable of healing injuries that are the kind of thing that, if they were smaller, would still leave the victim alive, in the Stone Age. If an injury is of a type that would be immediately fatal in the Stone Age, there’s no evolutionary pressure to make it survivable. For example, we can regrow peripheral nerves, because losing a peripheral nerve means a numb patch and a weak limb, but you could live with this for a few months even if you’re a caveman. On the other hand, we can’t regrow spinal cord, because a transected spinal cord is fatal within a day or two even given the finest Stone Age nursing care (it didn’t become survivable until about 1946.). On the third hand, we can heal brain from strokes, even though brain is more complex than spinal cord, because a small stroke is perfectly survivable as long as you have someone to feed you until you get better. We can survive huge surgical incisions, even though those would be fatal in the Stone Age, because small penetrating wounds were survivable, and the healing mechanisms can just do the same thing all along the incision. This is why we sew wounds up: to convince the healing mechanisms that it’s only a small cut.
Unfortunately this argument suggests regrowing limbs is impossible. An amputation is bad but survivable, and after it heals, you can still get around. But many years of spending a lot of bodily energy on regrowing a limb that is pretty useless for most of that time doesn’t seem worthwhile.
Some particular problems I see:
In humans, there’s no mechanism for a growing limb to connect correctly to an adult injury site. For example, there’s already a bunch of scar tissue there, which has to be cleared away progressively as the limb grows. Evolution has not seen fit to provide us with this complex biochemistry, unlike the case of salamanders.
Children have a high level of circulating growth hormone, which tells the arm cells how fast to grow. If you tried to provide this to an adult, their other bones would also grow, causing deformity (acromegaly).
It’s odd that we can’t grow new teeth when the old ones fall out. More than once, I mean. Drilling for cavities makes sense because the enamel (outer tooth layer) is essentially dead, and doesn’t regrow. But we should be able to grow a whole new tooth from the root when we get a cavity.
To hold the surface out, you need to have a magnetic field tangent to the surface. But you can’t have a continuous magnetic field tangent to every point on the surface of a sphere. That’s a theorem of topology, called the Hairy Ball Theorem. So there has to be some area of the ball that’s unsupported. I guess if the area is small enough, you just let it dimple inwards in tension. The balloon would be covered in dimples, like a golf ball.
Thanks for clearing that up. It sounds like we’re thinking along very similar lines, but that I came to a decision to stop earlier. From a position inside one of major AI labs, you’ll be positioned to more correctly perceive when the risks start outweighing the benefits. I was perceiving events more remotely from over here in Boston, and from inside a company that uses AI as a one of a number of tools, not as the main product.
I’ve been aware of the danger of superintelligence since the turn of the century, and I did my “just now orienting to the question” back in the early 2000s. I decided that it was way too early to stop working on AI back then, and I should just “monitor for new considerations or evidence or events.” Then in 2022, Sydney/Bing came along, and it was of near-human intelligence, and aggressively misaligned, despite the best efforts of its creators. I decided that was close enough to dangerous AI that it was time to stop working on such things. In retrospect I could have kept working safely in AI for another couple of years, i.e. until today. But I decided to pursue the “death with dignity” strategy: if it all goes wrong, at least you can’t blame me. Fortunately my employers were agreeable to have me pivot away from AI; there’s plenty of other work to be done.
I’m not “trying to figure out” whether to work on capabilities, having already decided I’ve figured it out and given up such work. Are you interested in talking about this to someone like me? I can’t tell whether you want to restrict discussion to people who are still in the figuring out stage. Not that there’s anything wrong with that, mind you.
It’s a cute idea, but AI is a terrible fiction writer.
Not only is this true in AI research, it’s true in all science and engineering research. You’re always up against the edge of technology, or it’s not research. And at the edge, you have to use lots of stuff just behind the edge. And one characteristic of stuff just behind the edge is that it doesn’t work without fiddling. And you have to build lots of tools that have little original content, but are needed to manipulate the thing you’re trying to build.
After decades of experience, I would say: any sensible researcher spends a substantial fraction of time trying to get stuff to work, or building prerequisites.
This is for engineering and science research. Maybe you’re doing mathematical or philosophical research; I don’t know what those are like.
High-purity quartz is used as crucibles in which to melt silicon for semiconductors. It’s not a directly consumed raw material. Based on my understanding of the process, the problem is that a little bit of the crucible dissolves into the silicon every time a batch is melted. How long does a crucible last, and can its life be extended by various forms of cleverness? If the lifetime of a crucible is longer than the time it takes to restore the mines to production, then the interruption might not be serious. Assuming that the mines produce extra-fast for a while, to make up for the gap in production, of course.
IIRC, this sort of interruption to chip supply has happened twice, once with a glue factory fire in Nagoya, and once because of floods in Thailand that simultaneously destroyed several assembly plants. Both interruptions lasted about four months before production was restored, and resulted in a brief price increase instead of the Moore’s Law price decreases that semiconductor prices usually enjoy.
Epistemic status: I was trained as an electrical engineer, and worked for many years as a chip designer, but have not actually been in the business in this century, so any detailed knowledge is possibly obsolete.
Yes, things have certainly changed in the four months since I wrote my original comment, with the advent of o1 and Sakana’s Artificial Scientist. Both of those are still incapable of full automation of self-improvement, but they’re close. We’re clearly much closer to a recursive speed up of R&D, leading to FOOM.
I don’t think articles like this belong on Less Wrong, so I downvoted it. Presumably the author agrees, to some extent, or he wouldn’t have felt the need to pre-apologize. If people disagree with me, they are of course free to upvote it.
Also, the article in question was posted in October of 2020. Why bring it up now? It’s not like we can do anything about it.
“Time-Symmetric” and “reversible” mean the same thing to me: if you look at the system with reversed time, it obeys the same law. But apparently they don’t mean the same to OP, and I notice I am confused. In any event, as Mr Drori points out, symmetry/reversibility implies symmetry under time translation. If, further, the system can be described by a Hamiltonian (like all physical systems) then Noether’s Theorem applies, and energy is conserved.
Now I know more! Thanks.
That would suggest that an equal mass of tiny wind turbines would be more efficient. But I see really big turbines all over the midwest. What's the explanation?
Yes. I know him. We met years ago when I was a grad student at the Media Lab. I haven't followed his work on self-reproduction in detail, but from what I've seen he is not aiming at economically self-sufficient devices, while I am. So I'm not too impressed.
As I describe in my first reply to Jackson Wagner above, I can tolerate some inefficiency, as long as I stay above Soviet-style negative productivity. The goal is minimum reproduction time. Once I've scaled up, I can build a rolling mill if needed.
You could mill every single plate in a motor core out of sheet stock on a milling machine...
As you point out, that would be madness. I've got a sheet rolling machine listed, so I assume I can take plate and cold-roll it into sheet. Or heat the plate and hot-roll it if need be. The sheets are only a meter long and a few centimeters wide, so the rolling machine fits inside. They function like shingles for building the outside enclosure, and for various machine guards internally, so they don't have to be big.
where are you quenching the stuff?
I'm quenching in a jar of used lubricant. Or fresh oil, if need be. 6% of the input is oil.
alloys isn't necessarily that you can't substitute X for Y, but that X costs three or four or ten times as much as Y for the specific application that Y is optimized for.
I'm a little reluctant to introduce this kind of evidence, but I've seen lots of machinist videos where they say "I pulled this out of the scrap bin, not sure what it is, but lets use it for this mandrel" (or whatever). And then it works fine. I am happy to believe that different alloys differ by tens of percent in their characteristics, and that getting the right alloy is an important occupation for real engineers. I just don't think that many thousands of them all vary by "three or four or ten times." I think I can get away with six or so.
I was actually thinking of a pair of humanlike arms with many degrees of freedom, and one or more cameras looking at things. You can have dozens of single datum sensors, or one camera. It's much cheaper. Similarly, once you have some robot arms, there's no gain in including many single use motors. For example, when I include an arbor press, I don't mean a motorized press. I mean a big lever that you grab with the robot arm and pull down, to press in a shaft or shape a screw head.
There are two CNC machine tools, to automate some part shaping while the robot does something else.
Yes, absolutely! A fine description of the current state of the art. I upvoted your post by 6 points (didn't know I could do that!).
I'm imagining doing everything the machinist has to do with a mobile pair of robot arms. I can imagine a robot doing everything you listed in your first list of problems. Your "stupider stuff" is all software problems, so will be fixed once, centrally, and for good on the Autofac. The developers can debug their software as it fails, which is not a luxury machinists enjoy.
Call a problem that requires human input a "tough" problem. We can feed the solutions to any tough problems back into the model, using fine-tuning or putting it in the prompt. So ideally, any tough problem will have to be solved once. Or a small number of times, if the VLM is bad at generalizing. The longer we run the Autofacs, the more tough problems we hit, resolve, and never see again. With an exponentially increasing number of Autofacs, we might have to solve an exponentially increasing number of tough problems. This is infeasible and will destroy the scheme. We have to hope that the tough problems per hour per Autofac drops faster than the number of Autofacs increases. It's a hope and only a hope-- I can't prove it's the case.
What's your feeling about the distribution of tough problems?
Wow, I think that comment is as long as my original essay. Lots of good points. Let me take them one by one.
I see a few potential benefits to efficiency-imparing simplifications:
- lt reduces the size/cost/complexity of the initial self-replicating system. (I think this motivation is misplaced, and we should be shooting for a much larger initial size than 1 meter cubed.)
The real motivation for the efficiency-impairing simplifications is none of size, cost or complexity. It is to reduce replication time. We need an Autofac efficient enough that what it produces is higher value than what it consumes. We don't want to reproduce Soviet industry, much of which processed expensive resources into lousy products worth less than the inputs. Having achieved this minimum, however, the goal is to allow the shortest possible time of replication. This allows for the most rapid production of the millions of tons of machinery needed to produce massive effects.
Consider that the Autofac, 50 kg in a 1 m^3, is modeled on a regular machine shop, with the machinist replaced by a robot. The machine shop is 6250 kg in 125 m^3. I just scale it down by a factor of 5, and thereby reduce the duplication time by a factor of 5. So it duplicates in 5 weeks instead of 25 weeks. Suppose we start the Autofac versus the robot machine shop at the same time. After a year, there are 1000 Autofacs versus 4 machine shops; or in terms of mass, 50,000 kg of Autofac and 25,000 kg of machine shop. After two years, 50,000,000 kg of Autofac versus 100,000 kg of machine shop. After 3 years, it's even more extreme. At any time, we can turn the Autofacs from making themselves to making what we need, or to making the tools to make what we need. The Autofac wins by orders of magnitude even if it's teeny and inefficient, because of sheer speed.
That's why I picked a one meter cube. I would have picked a smaller cube, that reproduced faster, but that would scale various production processes beyond reasonable limits. I didn't want to venture beyond ordinary machining into weird techniques only watchmakers use.
I see a few potential benefits to efficiency-imparing simplifications:
- ...
- It reduces the engineering effort needed to design the initial self-replicating system.
This is certainly a consideration. Given the phenomenal reproductive capacity of the Autofac, there's an enormous return to finishing design as quickly as possible and getting something out there.
To me, it seems that the Autofac dream comes from a particular context -- mid-20th-century visions of space exploration -- that have unduly influenced Feynman's current concept.
Let me tell you some personal history. I happened upon the concept of self-reproducing machines as a child or teenager, in an old Scientific American from the fifties. This was in the 1970s. That article suggested building a self-reproducing factory boat, that would extract resources from the sea, and soon fill up the oceans and pile up on beaches. It wasn't a very serious article. Then I went to MIT, in 1979. Self-reproducing machines were in the air-- Eric Drexler was theorizing about mechanical bacteria, and NASA was paying people to think about what eventually became the 1981 lunar factory design study. I thought that sending a self-reproducing factory to the asteroid belt was the obvious right thing, and thought about it, in my baby-engineer fantasy way. But I could tell I was ahead of my time, so I turned my attention to supercomputers and robots and AI and other stuff for a few decades.
A few years ago I picked up the idea of self-reproducing boats again. I imagined a windmill on deck for power, and condensing Seacrete and magnesium from the water for materials. There was a machine shop below decks, building all the parts. But I couldn't make the energy economy work out, even given the endless gales of the Southern Ocean. So I asked myself, what about just the machine shop part? Then I realized the reproduction time was the overriding consideration. How can I figure out the reproduction time? Well, I could estimate the time to do it with a regular human machine shop, and I remembered Eric Drexler's scaling laws. And wow, five weeks?! That's short enough to be a really big deal! So, a certain amount of calculation and spreadsheets later, here we are, the Autofac.
I considered varied environments for situating the Autofac:
- a laboratory in Boston. Good for development, but doesn't allow rapid growth.
- a field near a railroad and power line in the Midwest. Good for the resource inputs, but the neighbors might reasonably complain when the steel mill starts belching flame, or the Autofacs pile up sky-high.
- Baffin Island. Advantages described above.
- Antarctic Icecap. Bigger than Baffin, but useful activities are illegal. Shortage of all elements except carbon, oxygen, nitrogen and hydrogen.
- The Moon. Even bigger. Ironically, shortage of carbon, nitrogen and hydrogen. No wind, so the Autofac has to include solar cell manufacture from the git-go. There will be lots of problems understanding vacuum manufacturing. Obvious first step toward Dyson Sphere.
- Carbonaceous asteroids. Obvious second step toward Dyson Sphere.
So, I decided to propose an intermediate environment. Obviously, it was rooted in the mid-20th-century visions of space exploration. But that didn't set the size, or the use of Baffin Island, or anything else really. We'll build a Dyson Sphere eventually, but I don't feel the need to do it personally.
More to come.
If you look at what I wrote, you will see that I covered both of these.
Yeah, I looked at various forms of printing from powder as a productive system. The problem is that the powder is very expensive, more expensive than most of the parts that can be produced from it. And it can’t produce some parts— like ball bearings or cast iron— so you need tools to make those. And by the time you add those in, it turns out you don’t need powder metallurgy.
#2 is why I’m coming up with this scheme despite my substantial p(doom). I think we can do something like this with subhuman (hence non-dangerous) levels of AI. Material abundance is one of the things we expect from the Singularity. This provides abundance without superhuman AI, reducing the impetus toward it.
You could go some way with 1980s-level integrated circuits for all the onboard electronics. The manufacturing requirements are much more tolerable. But even 1980s semiconductors require a couple of dozen chemically exotic and ultra pure feedstocks. The Autofacs would have to build a complex chemical industry before they could start building chips.
I’ve been avoiding Factorio. I watched a couple of videos of people playing it, and it was obviously the most interesting game in the world, and if I tried it my entire life would get sucked in. So I did the stoic thing, and simply didn’t allow myself to be tempted.
Yes. The alternate approach to achieving a self-reproducing machine is to build a humanoid robot that can be dropped into existing factories, then gradually replace the workers that build it with robots. That path may well be the one that succeeds. Either path delivers an enormous expansion of industrial capabilities.
Well, I seem to be talking to someone who knows more about alloys than I do. How many alloys do you think I need? I figure there's a need for Neodymium Iron Boron, for motor cores, Cast Iron in the form of near-net-shape castings for machine frames, and some kind of hardenable tool steel for everything else. But I'm uncertain about the "everything else".
I don't think the "staggering number of standardized alloys" needs to alarm us. There are also a staggering number of standardized fasteners out there, but I think 4 sizes of machine screws will suffice for the Autofac. We don't need the ultimate in specialized efficiency that all those alloys give us.
They depend on lapping, which can be done "manually" by the robot arm. I forgot to list "abrasive powder" in my list of vitamin ingredients. Fixed now.
The fancier optical techniques provide precision on the order of a wavelength, which is far in excess of our needs. All we need is eyeball-class optical techniques like looking along an edge to make sure it's straghtish, or pressing a part against a surface plate and seeing if light passes under it.
I've operated a lathe and a mill (both entirely manual), various handheld power tools, a robot arm, robot eyes, autonomous mobile robots, and a data center. For the rest, I've read books and watched videos.
I've built and/or maintained various kinds of robots.
I have no experience with cutting-edge VLMs.
Yes, that's totally part of the starter pack. All the electronics are imported-- CPUs, radios, cameras, lights, voltage converters, wire harnesses, motor controllers...
I don't know how to plan the split between the part of the thinking that is done inside the Autofac and the part that is done in the data center.
Nowadays we use carbide bits, but we used to use steel bits to cut steel. It's called high speed steel. It differs from regular steel by being a different crystal structure, that is harder (and more brittle). It used to be perfectly common to cut a steel-cutting tool out of steel, then apply heat treatment to induce the harder crystal structure, and use the hardened tool to cut the softer original steel. It's one of the reasons I specified steel instead of aluminum or brass.
The machine shop can use a tool until it wears down too much, then un-harden it (a different heat treatment), cut it back to have a sharp edge again, and then re-harden. Steel really is amazing stuff.
I've looked into machine tool techniques pretty closely, and I believe I can make them with only 2% by weight that's not steel or lubricant. In a lot of ways, it's going back to the designs they used a hundred years ago, before they had good plastics or alloys. For example, the only place you HAVE to use plastic is as a flexible wire insulation.
I welcome your suggestions as to inputs I may have overlooked.
I think you're substantially underestimating the difficulty here,
I thought I was pretty careful not to say how hard I estimated the difficulty to be, but just to be clear: I think it will be a large project and many years of effort. Can you point to a place where you got the opposite impression? Or was it my breezy style?
and the proportion of effort which goes into the "starter pack" (aka vitamins) relative to steelworking.
I too was surprised by how large the steel input was compared to the vitamins, and in turn how much of that was lubricant relative to everything else. I got these proportions by scaling off the US economy as a whole. Compared to how much steel we consume, the vitamins are really that small a fraction. It also seems like a reasonable fraction of non-steel parts in a bunch of machine tools in a metal box. Of course those vitamins are far more valuable per kilogram compared to the steel.
initially focusing on just one of producing parts, assembling parts, or controlling machinery. If your system works when teleoperated, or can assemble but not produce a copy of itself, etc., that's already a substantial breakthrough.
Thank you for your suggestions of how to demonstrate only part of the system; I've been trying to come up with a minimum viable product that is less difficult to get to. I'm an old guy, not in the full flower of my health, so I'm gonna let someone else build the startup company to do the whole job.
Thanks for the reference to Chirikjian 2004; I wasn't aware of that one.
There are standard ways to make more precise tools from less precise tools. The methods were invented 1750-1840 to allow upgrading handmade metal tools to the precision of thousandths of inches we enjoy today. We just have to apply such methods a little bit at every generation to keep the level of precision constant.
“Pigs may have been chosen due to the fact that they are especially resistant to prion disease.”
I assume you’re talking about the original selection of pig brains as the source of Cerebrolysin in 1951. This is entirely impossible, as prions were unknown in 1951. The only prion disease known at the time was scrapie in sheep, but of course they didn’t know what caused it.
Well, it seems like you have very high standards for “epistemic footing”; indeed standards so high that nothing can meet them. I’m willing to settle for mere empirical verification, mathematical elegance, and logical coherence. All of which are satisfied by our present understanding of quantum field theory.
The controversy over “underlying reality” continues because all theories of underlying reality reproduce identical experimental predictions, so arguments in this area are philosophy rather than physics, and so rather inconclusive.
Of course we don’t know how to reconcile our empirically valid theory of quantum mechanics with our empirically valid theory of gravity, so at least one of the theories is wrong.
Because present LLMs can’t pass the Turing test. They think at or above human level, for a lot of activities. But over the past couple of years, we’ve identified lots of tasks that they absolutely suck at, in a way no human would. So an assiduous judge would have no trouble distinguishing them from a human.
But, I hear you protest, that’s obeying the letter of the Turing test, not the spirit. To which I reply: well, now we’re arguing about the spirit of an ill-defined test from 74 years ago. Turing expected that the first intelligent machine would have near-human abilities across the board, instead of the weird spectrum of abilities that they actually have. AI didn’t turn out like Turing expected, rendering the question of whether a machine could pass his test a question without scientific interest.
Two comments:
We understand quantum physics. I mean, I don’t, personally. But I know people who do. And I own books that could explain the whole thing to me if I had the stamina to get all the way through them.
The kinetic theory of gases was developed decades before quantum mechanics. It gives the same results whether its constituent particles are quantum or classical, at least for ordinary phenomena like explosions.
I’m 62, so I was adult in the ‘80s and ‘90s. My sense is that the world was different. The consequences of expressing divergent opinion seem much more serious now.
We deceive or manipulate portions of ourselves all the time. There are portions of our mind that are sufficiently simple that our consciousness can understand and outthink them.
At first this seems like an absurd claim, so here’s a few examples. I can enjoy sexual arousal by directing my gaze toward pictures of smiling bare-breasted women. I deceive my lust management system into thinking it is about to have sex with a willing woman, because it is too stupid to understand the existence of pictures. When my live-in girlfriend dumps me, I move out, because seeing her every day causes me heartache. I deceive my love management system into not thinking about her (as much) because for it, out of sight is (mostly) out of mind. When I have to give a urine sample at the doctor’s office, and I can’t pee right then, I turn on a water tap to produce a rushing sound. I deceive my bladder management system by making it think I am in a good place to pee.
By analogy, a sufficiently superhuman system would be deceptive or manipulative toward the human “portion” of itself. In all the above examples, I’m not operating in order to hurt the other portions of my mind; I am merely manipulating them into accomplishing my greater goals. I don’t care how a portion of myself feels, as long as the whole is better off. But an AI with this attitude would be no more trustworthy than any other.
So it seems like this is another alignment scheme that breaks down in the limit of sufficiently superhuman intelligence. And perhaps in the limit of sufficiently coherent near-human intelligence. Darn.
Here’s a stronger example of deceiving part of yourself. Suppose you watch a disgusting movie, and are so nauseated that you throw up. The movie has deceived some segment of your nervous system into thinking that something disgusting has happened, and as a precaution, it carries out the vomiting sequence. The disgust perceiver is kind of stupid, so it doesn’t realize it’s only a movie. It’s part of the self, but sometimes operates at cross-purposes to the self as a whole.
I assumed that being struck by lightning was fairly common, but today I learned I was wrong. Apparently it only kills about 30 Americans per year, and I assumed it was more like 3000, or even 30,000.
As a child, I was in an indoor swimming pool in a house that was struck by lightning, and as a young man, I was in a car that was about 40 feet from a telephone pole that was struck by lightning. In both cases I was fine because of the Faraday cage effect, but the repeated near-misses and spectacular electrical violence made me think that lightning was a non-negligible hazard. I suppose that’s a rationalist lesson: don’t generalize from your experience of rare events if you can actually look up the probabilities.
I guess I wasted all that time training my kids in lightning safety techniques.
Why should we accept as evidence something that you perceived while you were dreaming? Last night I dreamed that I was walking barefoot through the snow, but it wasn’t cold because it was summer snow. I assume you don’t take that as evidence that warm snow is an actual summer phenomenon, so why should we take as evidence your memory of having two consciousnesses?
It seems to me that a correctly organized consciousness would occur once per body. Consciousness is (at least in part) a system for controlling our actions in the medium and long term. If we had two consciousnesses, and they disagree as to what to do next, it would result in paralysis. And if they agree, then one of them is superfluous, and we’d expend less brain energy if we only had one.
It’s curious that you ask for personal experience or personal research. Did you mean to discount the decades of published research in using xenon for anesthesia and MRI contrast? Anyway, if you’ll accept the opinion of someone who has merely read some books on anesthesia and gas physiology: my opinion is that this guy is full of it. The physiology of small-molecule anaesthetics and serotoninergic hallucinogens is completely different. And he doesn’t seem like a serious person.
Electrical engineer here. I read the publicity statement, and from my point of view it is both (a) a major advance, if true, and (b) entirely plausible. When you switch from a programmable device (e.g. GPU) to a similarly sized special purpose ASIC, it is not unreasonable to pick up a factor of 10 to 50 in performance. The tradeoff is that the GPU can do many more things than the ASIC, and the ASIC takes years to design. They claim they started design in 2022, on a transformer-only device, on the theory that transformers were going to be popular. And boy, did they luck out. I don‘t know if other people can tell, but to me, that statement oozes with engineering glee. They’re so happy!
I would love to see a technical paper on how they did it.
Of course they may be lying.
In a great deal of detail, apparently, since it has a recommended reading time of 131 minutes.
I read along in your explanation, and I’m nodding, and saying “yup, okay”, and then get to a sentence that makes me say “wait, what?” And the whole argument depends on this. I’ve tried to understand this before, and this has happened before, with “the universal prior is malign”. Fortunately in this case, I have the person who wrote the sentence here to help me understand.
So, if you don’t mind, please explain “make them maximally probable”. How does something in another timeline or in the future change the probability of an answer by writing the wrong answer 10^100 times?
Side point, which I’m checking in case I didn’t understand the setup: we’re using the prior where the probability of a bit string (before all observations) is proportional to 2^-(length of the shortest program emitting that bit string). Right?
I will speak to the question of “what are some situations in real life, other than "AI takeoff", where the early/mid/late game metaphor seems useful?”. It seems to me that such a metaphor is useful in any situation with
—two or more competitors,
—who start small and expand,
—in a fixed-size field of contention,
—and such that bigger competitors tend to beat small ones.
The phases in such a competition can be described as
—Early: competitors are building up power and resources more or less independently, because they’re not big enough to run into each other significantly. Important to strategize correctly. You can plan longer term because other players can’t upset your plans yet.
—Mid: what other players do matters very much to what you do. Maximum coupling leads to maximum confusion.
—End: time for the leading player to grind down the smaller players. Becomes easier to understand as hope disappears.
Chess is an example, where there are two competitors, and the resource is “pieces that have been moved and not yet taken”. This also applies to multiplayer tabletop games (which is where I thought of it). It also applies to companies moving into a new product area, like desktop operating systems in the ‘80s. It applied to European colonial powers moving into new continents.