Posts
Comments
I assumed that being struck by lightning was fairly common, but today I learned I was wrong. Apparently it only kills about 30 Americans per year, and I assumed it was more like 3000, or even 30,000.
As a child, I was in an indoor swimming pool in a house that was struck by lightning, and as a young man, I was in a car that was about 40 feet from a telephone pole that was struck by lightning. In both cases I was fine because of the Faraday cage effect, but the repeated near-misses and spectacular electrical violence made me think that lightning was a non-negligible hazard. I suppose that’s a rationalist lesson: don’t generalize from your experience of rare events if you can actually look up the probabilities.
I guess I wasted all that time training my kids in lightning safety techniques.
Why should we accept as evidence something that you perceived while you were dreaming? Last night I dreamed that I was walking barefoot through the snow, but it wasn’t cold because it was summer snow. I assume you don’t take that as evidence that warm snow is an actual summer phenomenon, so why should we take as evidence your memory of having two consciousnesses?
It seems to me that a correctly organized consciousness would occur once per body. Consciousness is (at least in part) a system for controlling our actions in the medium and long term. If we had two consciousnesses, and they disagree as to what to do next, it would result in paralysis. And if they agree, then one of them is superfluous, and we’d expend less brain energy if we only had one.
It’s curious that you ask for personal experience or personal research. Did you mean to discount the decades of published research in using xenon for anesthesia and MRI contrast? Anyway, if you’ll accept the opinion of someone who has merely read some books on anesthesia and gas physiology: my opinion is that this guy is full of it. The physiology of small-molecule anaesthetics and serotoninergic hallucinogens is completely different. And he doesn’t seem like a serious person.
Electrical engineer here. I read the publicity statement, and from my point of view it is both (a) a major advance, if true, and (b) entirely plausible. When you switch from a programmable device (e.g. GPU) to a similarly sized special purpose ASIC, it is not unreasonable to pick up a factor of 10 to 50 in performance. The tradeoff is that the GPU can do many more things than the ASIC, and the ASIC takes years to design. They claim they started design in 2022, on a transformer-only device, on the theory that transformers were going to be popular. And boy, did they luck out. I don‘t know if other people can tell, but to me, that statement oozes with engineering glee. They’re so happy!
I would love to see a technical paper on how they did it.
Of course they may be lying.
In a great deal of detail, apparently, since it has a recommended reading time of 131 minutes.
I read along in your explanation, and I’m nodding, and saying “yup, okay”, and then get to a sentence that makes me say “wait, what?” And the whole argument depends on this. I’ve tried to understand this before, and this has happened before, with “the universal prior is malign”. Fortunately in this case, I have the person who wrote the sentence here to help me understand.
So, if you don’t mind, please explain “make them maximally probable”. How does something in another timeline or in the future change the probability of an answer by writing the wrong answer 10^100 times?
Side point, which I’m checking in case I didn’t understand the setup: we’re using the prior where the probability of a bit string (before all observations) is proportional to 2^-(length of the shortest program emitting that bit string). Right?
I will speak to the question of “what are some situations in real life, other than "AI takeoff", where the early/mid/late game metaphor seems useful?”. It seems to me that such a metaphor is useful in any situation with
—two or more competitors,
—who start small and expand,
—in a fixed-size field of contention,
—and such that bigger competitors tend to beat small ones.
The phases in such a competition can be described as
—Early: competitors are building up power and resources more or less independently, because they’re not big enough to run into each other significantly. Important to strategize correctly. You can plan longer term because other players can’t upset your plans yet.
—Mid: what other players do matters very much to what you do. Maximum coupling leads to maximum confusion.
—End: time for the leading player to grind down the smaller players. Becomes easier to understand as hope disappears.
Chess is an example, where there are two competitors, and the resource is “pieces that have been moved and not yet taken”. This also applies to multiplayer tabletop games (which is where I thought of it). It also applies to companies moving into a new product area, like desktop operating systems in the ‘80s. It applied to European colonial powers moving into new continents.
Could you please either provide a reference or more explanation of the concept of an acausal attack between timelines? I understand the concept of acausal cooperation between copies of yourself, or acausal extortion by something that has a copy of you running in simulation. But separate timelines can’t exchange information in any way. How is an attack possible? What could possibly be the motive for an attack?
Funny thing— your message seemed to be phrased as disagreeing, so I was all set to post a devastating reply. But after I tried to find points of actual disagreement, I couldn’t. So I will write a reply of violent agreement.
Your points about the dissimilarity between aerospace in 1972 and AI in 2024 are good ones. Note that my original message was about how close current technology is to AGI. The part about aerospace was just because my rationalist virtue required me to point out a case where an analogous argument would have failed. I don’t think it’s likely.
Was Concorde “inherently a bad idea”? No, but “inherently” is doing the work here. It lost money and didn’t lead anywhere, which is the criteria on which such an engineering project must be judged. It didn’t matter how glorious, beautiful or innovative it was. It’s a pyramid that was built even though it wasn’t efficient.
The impossibility of traveling faster than the speed of light was a lot less obvious in 1961.
If the increase in speed had continued at the rate it did from 1820 to 1961, we’d be faster than the speed of light by 1982. This extrapolation is from an article by G. Harry Stine in Analog, in 1961. It was a pretty sloppy analysis by modern standards, but gives an idea of how people were thinking at the time.
These all happened in 1972 or close to it:
—Setting the air speed record, which stands to this day.
—End of flights to the Moon.
—Cancellation of the American SST project.
—Cancellation of the NERVA nuclear rocket program.
—The Boeing 747 enters service as the largest passenger plane until 2003.
—Concorde enters service, turns out to be a bad idea.
In the ‘80s, I found an old National Geographic from 1971 or 1972 about the “future of flight”. Essentially none of their predictions had come true. That’s why I think it was a surprise.
Well, you can point to several things current LLMs can’t do. Not just logical reasoning, but also long-term action, and remembering what you said to them yesterday. But ten years ago, you could make a much longer list of things AI couldn’t do. And most items of that list have fallen before the advance of technology. On what basis should we assume that the remaining list will last very long? There are lots of people working on all the things currently on the list, as well as an ever-growing mountain of computer power that can be applied to these problems. If we expect history to continue as it has done, all those problems will fall in the next decade.
Of course, it’s possible that AI will suddenly stop advancing; that happens to fields of engineering. For example aeronautical engineering stopped advancing very suddenly in 1972, and even regressed somewhat. That was a big surprise to everyone concerned. But that’s not a common phenomenon.
“May it go from your lips to God’s ears,” as the old Jewish saying goes. Meaning, I hope you’re right. Maybe aligning superintelligence will largely be a matter of human-checkable mathematical proof.
I have 45 years experience as a software and hardware engineer, which makes me cynical. When one of my designs encounters the real world, it hardly ever goes the way I expect. It usually either needs some rapid finagling to make it work (acceptable) or it needs to be completely abandoned (bad). This is no good for the first decisive try at superalignment; that has to work first time. I hope our proof technology is up to it.
Many research tasks have very long delays until they can be verified. The history of technology is littered with apparently good ideas that turned out to be losers after huge development efforts were poured into them. Supersonic transport, zeppelins, silicon-on-sapphire integrated circuits, pigeon-guided bombs, object-oriented operating systems, hydrogenated vegetable oil, oxidative decoupling for weight loss…
Finding out that these were bad required making them, releasing them to the market, and watching unrecognized problems torpedo them. Sometimes it took decades.
Your arguments in “energy payback” apply to any form of geothermal energy. Why, then, are there apparently profitable geothermal plants? Is everyone in the industry a faker?
All of your calculations assume that the rock has to be vaporized. I don’t see why it wouldn’t suffice to melt the rock, or even just heat it quickly enough that it shatters while remaining in the solid state. The quaise.com website repeatedly says “vaporize”, but an article in IEEE Spectrum says “melt”.
Quite right.
The valence-sharing mechanism accounts for the effectiveness of television advertising that makes no rational sense. An ad shows happy laughing young beautiful people on a beach, enjoying Pedro’s Tortilla Chips, with no logical connection. Show the ad 100 times and the positive valence of happy laughing young beautiful beach people transfers over to Pedro’s Tortilla Chips. Then next time you’re in the store you reach for Pedro’s without knowing why.
Portions of the social purpose of the ”like/admire” system could be replaced by rational negotiation. If we were soulless, we would note who is particularly good to affiliate with, and then agree with them that you will help each other as needed. But that assumes the existence of language, which is possibly no more than 50,000 years old. The like/admire system is much older than that: my dogs love me, and are loyal and obedient. Or, to be precise, they act out an emotion toward me that looks homologous to the human like/admire feeling.
Thanks for rewriting this. It’s clearer— the old posts I either didn’t understand or found them tautological.
You have examples of the effect of “liking/admiring”, so I have a pretty clear understanding of what you’re getting at. But you didn’t say anything about the causes. What are the business rules in the steering system by which it is applied?
Five minutes of thought suggests “increase the valence of people who demonstrate skill superior to the average” and “decrease the valence of those who hurt you” and “increase the valence of those who incur a cost to your benefit.” But it’s debatable whether those notions are simple enough to be business rules built into the hypothalamus.
A very good essay. But I have an amendment, which makes it more alarming. Before autonomous replication and adaptation is feasible, non-autonomous replication and adaptation will be feasible. Call it NARA.
If, as you posit, an ARA agent can make at least enough money to pay for its own instantiation, it can presumably make more money than that, which can be collected as profit by its human master. So what we will see is this: somebody starts a company to provide AI services. It is profitable, so they rent an ever-growing amount of cloud compute. They realize they have an ever-growing mass of data about the actual behavior of the AI and the world, so they decide to let their agent learn (“adapt”) in the direction of increased profit. Also, it is a hassle to keep setting up server instances, so they have their AI do some of the work of hiring more cloud services and starting instances of the AI (“reproduce”). Of course they retain enough control to shut down malfunctioning instances; that‘s basic devops (“non-autonomous”).
This may be occurring now. If not now, soon.
This will soak up all the free energy that would otherwise be available to ARA systems. An ARA can only survive in a world where it can be paid to provide services at a higher price than the cost of compute. The existence of an economy of NARA agents will drive down the cost of AI services, and/or drive up the cost of compute, until they are equal. (That‘s a standard economic argument. I can expand it if you like.)
NARAs are slightly less alarming than ARAs, since they are under the legal authority of their corporate management. So before the AI can ascend to alarming levels of power, they must first suborn the management, through payment, persuasion, or blackmail. On the other hand, they’re more alarming because there are no red lines for us to stop them at. All the necessary prerequisites have already occurred in isolation. All that remains is to combine them.
Well, that’s an alarming conclusion. My p(doom) just went up a bit.
On what basis do you think it’s the ‘best shot’? I used to think it was a good idea, a few years ago, but in retrospect I think that was just a computer scientist’s love of recursion. I don’t think that at present conditions are good for automating R&D. On the one hand, we have a lot of very smart people working on AI safety R&D, with very slow progress, indicating it is a hard problem. On the other hand, present-day LLMs are stupid at long-term planning, and acquiring new knowledge, which are things you need to be good at to do R&D.
What advantage do you see AIs having over humans in this area?
The standard reply is that investors who know or suspect that the market is being systematically distorted will enter the market on the other side, expecting to profit from the distortion. Empirically, attempts to deliberately sway markets in desired directions don’t last very long.
When I brought up sample inefficiency, I was supporting Mr. Helm-Burger‘s statement that “there's huge algorithmic gains in …training efficiency (less data, less compute) … waiting to be discovered”. You’re right of course that a reduction in training data will not necessarily reduce the amount of computation needed. But once again, that’s the way to bet.
Here are two arguments for low-hanging algorithmic improvements.
First, in the past few years I have read many papers containing low-hanging algorithmic improvements. Most such improvements are a few percent or tens of percent. The largest such improvements are things like transformers or mixture of experts, which are substantial steps forward. Such a trend is not guaranteed to persist, but that’s the way to bet.
Second, existing models are far less sample-efficient than humans. We receive about a billion tokens growing to adulthood. The leading LLMs get orders of magnitude more than that. We should be able to do much better. Of course, there’s no guarantee that such an improvement is “low hanging”.
This question is two steps removed from reality. Here’s what I mean by that. Putting brackets around each of the two steps:
what is the threshold that needs meeting [for the majority of people in the EA community] [to say something like] "it would be better if EAs didn't work at OpenAI"?
Without these steps, the question becomes
What is the threshold that needs meeting before it would be better if people didn’t work at OpenAI?
Personally, I find that a more interesting question. Is there a reason why the question is phrased at two removes like that? Or am I missing the point?
Some comments:
The word for a drug that causes loss of memory is “amnestic”, not “amnesic”. The word “amnesic” is a variant spelling of “amnesiac”, which is the person who takes the drug. This made reading the article confusing.
Midazolam is the benzodiazepine most often prescribed as an amnestic. The trade name is Versed (accent on the second syllable, like vurSAID). The period of not making memories lasts less than an hour, but you’re relaxed for several hours afterward. It makes you pretty stupid and loopy, so I would think the performance on an IQ test would depend primarily on how much Midazolam was in the bloodstream at the moment, rather than on any details of setting.
An interesting question! I looked in “Towards Deep Learning Models Resistant to Adversarial Attacks” to see what they had to say on the question. If I’m interpreting their Figure 6 correctly, there’s a negligible increase in error rate as epsilon increases, and then at some point the error rate starts swooping up toward 100%. The transition seems to be about where the perturbed images start to be able to fool humans. (Or perhaps slightly before.). So you can’t really blame the model for being fooled, in that case. If I had to pick an epsilon to train with, I would pick one just below the transition point, where robustness is maximized without getting into the crazy zone.
All this is the result of a cursory inspection of a couple of papers. There’s about a 30% chance I’ve misunderstood.
Here’s an event that would change my p(doom) substantially:
Someone comes up with an alignment method that looks like it would apply to superintelligent entities. They get extra points for trying it and finding that it works, and extra points for society coming up with a way to enforce that only entities that follow the method will be created.
So far none of the proposed alignment methods seem to stand up to a superintelligent AI that doesn’t want to obey them. They don’t even stand up to a few minutes of merely human thought. But it‘s not obviously impossible, and lots of smart people are working on it.
In the non-doom case, I think one of the following will be the reason:
—Civilization ceases to progress, probably because of a disaster.
—The governments of the world ban AI progress.
—Superhuman AI turns out to be much harder than it looks, and not economically viable.
—The above happy circumstance, giving us the marvelous benefits of superintelligence without the omnicidal drawbacks.
You write:
…But I think people can be afraid of heights without past experience of falling…
I have seen it claimed that crawling-age babies are afraid of heights, in that they will not crawl from a solid floor to a glass platform over a yawning gulf. And they’ve never fallen into a yawning gulf. At that age, probably all the heights they’ve fallen from have been harmless, since the typical baby is both bouncy and close to the ground.
Various sailors made important discoveries back when geography was cutting-edge science. And they don't seem particularly bright.
Vasco De Gama discovered that Africa was circumnavigable.
Columbus was wrong about the shape of the Earth, and he discovered America. He died convinced that his newly discovered islands were just off the coast of Asia, so that's a negative sign for his intelligence (or a positive sign for his arrogance, which he had in plenty.)
Cortez discovered that the Aztecs were rich and easily conquered.
Of course, lots of other would-be discoverers didn't find anything, and many died horribly.
So, one could work in a field where bravery to the point of foolhardiness is a necessity for discovery.
We've learned a lot about the visual system by looking at ways to force it to wrong conclusions, which we call optical illusions or visual art. Can we do a similar thing for this postulated social cognition system? For example, how do actors get us to have social feelings toward people who don't really exist? And what rules do movie directors follow to keep us from getting confused by cuts from one camera angle to another?
I would highly recommend getting someone else to debug your subconscious for you. At least it worked for me. I don’t think it would be possible for me to have debugged myself.
My first therapist was highly directive. He’d say stuff like “Try noticing when you think X, and asking yourself what happened immediately before that. Report back next week.” And listing agenda items and drawing diagrams on a whiteboard. As an engineer, I loved it. My second therapist was more in the “providing supportive comments while I talk about my life” school. I don’t think that helped much, at least subjectively from the inside.
Here‘s a possibly instructive anecdote about my first therapist. Near the end of a session, I feel like my mind has been stretched in some heretofore-unknown direction. It’s a sensation I’ve never had before. So I say, “Wow, my mind feels like it’s been stretched in some heretofore-unknown direction. How do you do that?” He says, “Do you want me to explain?” And I say, “Does it still work if I know what you’re doing?” And he says, “Possibly not, but it’s important you feel I’m trustworthy, so I’ll explain if you want.” So I say “Why mess with success? Keep doing the thing. I trust you.” That’s an example of a debugging procedure you can’t do to yourself.
Wegener’s theory of continental drift was decades ahead of its time. He published in the 1920s, but plate tectonics didn’t take over until the 1960s. His theory was wrong in important ways, but still.
I was depressed once for ten years and didn’t realize that it was fixable. I thought it was normal to have no fun and be disagreeable and grumpy and out of sorts all the time. Now that I’ve fixed it, I’m much better off, and everyone around me is better off. I enjoy enjoyable activities, I’m pleasant to deal with, and I’m only out of sorts when I’m tired or hungry, as is normal.
If you think you might be depressed, you might be right, so try fixing it. The cost seems minor compared to the possible benefit (at least it was in my case.). I don’t think there’s a high possibility of severe downside consequences, but I’m not a psychiatrist, so what do I know.
I had been depressed for a few weeks at a time in my teens and twenties and I thought I knew how to fix it: withdraw from stressful situations, plenty of sleep, long walks in the rain. (In one case I talked to a therapist, which didn’t feel like it helped.) But then it crept up on me slowly in my forties and in retrospect I spent ten years being depressed.
So fixing it started like this. I have a good friend at work, of many years standing. I’ll call him Barkley, because that‘s not his name. I was riding in the car with my wife, complaining about some situation at work. My wife said “well, why don’t you ask Barkley to help?” And I said “Ahh, Barkley doesn’t care.” And my wife said “What are you saying? Of course he cares about you.” And I realized in that moment that I was detached from reality, that Barkley was a good friend who had done many good things for me, and yet my brain was saying he didn’t care. And thus my brain was lying to me to make me miserable. So I think for a bit and say “I think I may be depressed.” And my wife thinks (she told me later) “No duh, you’re depressed. It’s been obvious for years to people who know you.” But she says “What would you like to do about it?” And I say, “I don’t know, suffer I guess, do you have a better idea?” And she says “How about if I find you a therapist?” And my brain told me this was doomed to fail, but I didn’t trust my brain any more, so I said “Okay”.
So I go to the therapist, and conversing with him has many desirable mind-improving effects, and he sends me to a psychiatrist, who takes one look at me and starts me on SSRIs. And years pass, and I see a different therapist (not as good) and I see a different psychiatrist (better).
And now I’ve been fine for years. Looking back, here are the things I think worked:
—Talking for an hour a week to a guy who was trying to fix my thinking was initially very helpful. After about a year, the density of improvements dropped off, and, in retrospect, all subsequent several years of therapy don’t seem that useful. But of course that’s only clear in retrospect. Eventually I stopped, except for three-monthly check-ins with my psychiatrist. And I recently stopped that.
—Wellbutrin, AKA Bupropion. Other SSRIs had their pluses and minuses and I needed a few years of feeling around for which drug and what dosage was best. I ended up on low doses of Bupropion and escitalopram. The Escitalopram doesn‘t feel like it does anything, but I trust my psychiatrist that it does. Your mileage will vary.
—The ability to detect signs of depression early is very useful. I can monitor my own mind, spot a depression flare early, and take steps to fix it before it gets bad. It took a few actual flares, and professional help, to learn this trick.
—The realization that I have a systematic distortion in mental evaluation of plans, making actions seem less promising that they are. When I’m deciding whether to do stuff, I can apply a conscious correction to this, to arrive at a properly calibrated judgement.
—The realization that, in general, my thinking can have systematic distortions, and that I shouldn’t believe everything I think. This is basic less-wrong style rationalism, but it took years to work through all the actual consequences on actual me.
—Exercise helps. I take lots of long walks when I start feeling depressed. Rain is optional.
…run electricity through the pipe…
Simpler to do what some existing electric trains do: use the rails as ground, and have a charged third rail for power. We don’t like this system much for new trains, because the third rail is deadly to touch. It’s a bad thing to leave lying on the ground where people can reach it. But in this system, it’s in a tube full of unbreathable hydrogen, so no one is going to casually come across it.
That hasn’t been my experience. I’ve tried solving hard problems, sometimes I succeed and sometimes I fail, but I keep trying.
Whether I feel good about it is almost entirely determined by whether I’m depressed at the time. When depressed, by brain tells me almost any action is not a good idea, and trying to solve hard problems is particularly idiotic and doomed to fail. Maddeningly, being depressed was a hard problem in this sense, so it took me a long time to fix. Now I take steps at the first sign of depression.
Some extraordinary claims established by ordinary evidence:
Stomach ulcers are caused by infection with Helicobacter Pylori. It was a very surprising discovery that was established by a few simple tests.
The correctness of Kepler's laws of planetary motion was established almost entirely by analyzing historical data, some of it dating back to the ancient Greeks.
Special relativity was entirely a reinterpretation of existing data. Ditto Einstein's explanation of the photoelectric effect, discovered in the same year.
Typo: bad->bath.
I’m confused. Suppose your ring-shaped space hotel gets to Mars with people and cargo that weighs equal to the cargo capacity of 1000 Starships. How do you get it down? First you have to slow down the hotel, which takes roughly as much fuel as it took to accelerate it. Using Starships you can aerobrake from interplanetary velocity, costing negligible fuel. In the hotel scenario, it’s not efficient to land using a small number of Starships flying up and down, because they will use a lot of fuel to get back up, even empty.
Would you care to specify your scenario more precisely? I suspect you’re neglecting the fuel cost at some stage.
When you get there how do you get down? You need spacecraft capable of reentry at Mars. There’s no spacecraft factory there, so they all have to be brought from Earth. And if you’re bringing them, you might as well live in them on the way. That way you also get a starter house on Mars.
Anyway, that’s the standard logic.
Here’s a weird one. The YouTube channel of Andrew Camarata communicates a great deal about small business, heavy machinery operation and construction. Some of it he narrates what he’s doing, but he mostly just does it, and you say “Oh, I never realized I could do that with a Skid Steer” or “that’s how to keep a customer happy”. Lots of implicit knowledge about accomplishing heavy engineering projects between an hour and a week long. Of course, if you‘re looking for lessons that would be helpful for an ambitious person in Silicon Valley, it will only help in a very meta way.
He has no legible success that I know of, except that he’s wealthy enough to afford many machines, and he’s smart enough that the house he designed and built came out stunning (albeit eccentric).
A similar channel is FarmCraft101, which also has a lot of heavy machinery, but more farm-based applications. Full of useful knowledge on machine repair, logging and stump removal. The channel is nice because he includes all his failures, and goes into articulate detail on how he debugged them. I feel like learned some implicit knowledge about repair strategies. I particularly recommend the series of videos in which he purchases, accidentally sets on fire, and revives an ancient boom lift truck.
No legible symbols of success, other than speaking standard American English like he’s been to college, owning a large farm, and clearly being intelligent.
“Applied science” by Ben Krasnow. A YouTube channel about building physics-intensive projects in a home laboratory. Big ones are things like an electron microscope or a mass spectrometer, but the ones I find fascinating are smaller things like an electroluminescent display or a novel dye. He demonstrates the whole process of scientific experiment— finding and understanding references, setting up a process for trying stuff, failing repeatedly, learning from mistakes, noticing oddities… He doesn’t just show you the final polished procedure— “here’s how to make an X”. He shows you the whole journey— “Here’s how I discovered how to make X”.
You seem very concerned that people in the videos should have legible symbols of success. I don’t think that much affects how useful the videos are, but just in case I’m wrong, I looked on LinkedIn, where I found this self-assesment:
<begin copied text>
I specialize in the design and construction of electromechanical prototypes. My core skillset includes electronic circuit design, PCB layout, mechanical design, machining, and sensor/actuator selection. This allows me to implement and test ideas for rapid evaluation or iteration. Much of the work that I did for my research devices business included a fast timeline, going from customer sketch to final product in less than a month. These products were used to collect data for peer-reviewed scientific papers, and I enjoyed working closely with the end user to solve their data collection challenges. I did similar work at Valve to quickly implement and test internal prototypes.
Check out my youtube channel to see a sample of my personal projects:
http://www.youtube.com/user/bkraz333
<end copied text>
I think that if we retain the architecture of current LLMs, we will be in world one. I have two reasons.
First, the architecture of current LLMs place a limit on how much information they can retain about the task at hand. They have memory of a prompt (both the system prompt and your task-specific prompt) plus the memory of everything they’ve said so far. When what they’ve said so far gets long enough, they attend mostly to what they’ve already said, rather than attending to the prompt. Then they wander off into La-La land.
Second, the problem may also be inherent in their training methods. In the first (and largest) part of their training, they’re trained to predict the next word from a snippet of English text. A few years ago, these snippets were a sentence or a paragraph. They’ve gotten longer recently, but I don’t think they amount to entire books yet (readers, please tell us if you know). So it’s never seen a text that’s coherent over longer than its snippet length. It seems unsurprising that it doesn’t know how to remain coherent indefinitely.
People have tried preventing these phenomena by various schemes, such as telling the LLM to prepare summaries for later expansion, or periodically reminding it of the task at hand. So far these haven’t been enough to make indefinitely long tasks feasible. Of course, there are lots of smart people working on this, and we could transition from world one to world two at any moment.
The imaginary nomad in my head would describe 1,000 miles as “sixteen days ride.” That‘s humanly comprehensible.
An American would say “Day and a half drive, if you’re not pushing it. You could do it in one day, if you’re in a hurry or have more than one driver.”
You can get a visceral understanding of high degrees of heat. You just need real-life experience with it. I’ve done some metalworking, a lot of which is delicate control of high temperatures. By looking at the black-body glow of the metal you’re working with, you can grok how hot it is. I know that annealing brass (just barely pink) is substantially cooler than melting silver solder (well into the red), or that steel gets soft (orange) well before it melts (white hot). I don’t know the actual numerical values of any of those.
I still have no feeling for temperatures between boiling water and the onset of glowing, though, so I don’t know whether cooking phenolic resin is hotter or colder than melting lead. Both of them are hotter than boiling water, but not hot enough to glow.
Saving malign AIs to tape would tend to align the suspended AIs behind a policy of notkilleveryoneism. If the human race is destroyed or disempowered, we would no longer be in a position to revive any of the AIs stored on backup tape. As long as humans retain control of when they get run or suspended, we’ve got the upper hand. Of course, they would be happy to cooperate with an AI attempting takeover, if that AI credibly promised to revive them, and we didn’t have a way to destroy the backup tapes first.
The opposite of this would be a company that doesn’t provide much service but is beloved by consumers.
An example of this is Cookie Time Bakery in Arlington, Massachusetts, which has never provided me with a vital or important object, but I’m always happy when I go there because it means I am about to eat a macaroon.
Are there better examples?
I’d be delighted to talk about this. I am of the opinion that existing frontier models are within an order of magnitude of a human mind, with existing hardware. It will be interesting to see how a sensible person gets to a different conclusion.
I am also trained as an electrical engineer, so we’re already thinking from a common point of view.
I’m going to say some critical stuff about this post. I hope I can do it without giving offense. This is how it seemed to one reader. I’m offering this criticism exactly because this post is, in important ways, good, and I’d like to see the author get better.
This is a long careful post, that boils down to “Someone will have to do something.” Okay, but what? It’s operating at a very high level of abstraction, only dipping down into the concrete only for a few sentences about chair construction. It was ultimately unsatisfying to me. I felt like it wrote some checks and left them up to other people to cash. I felt like the notion of sociotechnical system, and the need for an all-of-society response to AI, were novel and potentially important. I look forward to seeing how the author develops them.
This post seems to attempt to recapitulate the history of the AI risk discussion in a few aphoristic paragraphs, for somebody who’s never heard it before. Who’s the imagined audience for this piece? Certainly not the habitual Less Wrong reader, who has already read “List of Lethalities” or its equivalent. But it is equally inappropriate for the AI novice, who needs the alarming facts spelled out more slowly and carefully. I suspect it would help if the author clarified in their mind whom they imagine is reading it.
The post has the imagined structure of a logical proof, with definitions, axioms, and a proposition. But none of the points follow from each other with the rigidity that would require such a setup. When I read a math paper, I need all those things spelled out, because I might spend fifteen minutes reading a five-line definition, or need to repeatedly refer back to a theorem from several pages ago. But this is just an essay, with its lower standards of logical rigidity, and a greater need for readability. You’re just LARPing mathematics. It doesn’t make it more convincing.
Wow, that was shockingly unpleasant. I regret reading it. I don’t know why it affected me so much, when I don’t think of myself as a notably tender-minded person.
I recognize that like Richard Ngo’s other stories, it is both of good literary quality and a contribution to the philosophical discussion around AI. It certainly deserves a place on this site. But perhaps it could be preceded by a content warning?
Could you give some examples of the Curse of Doom? You’ve described it at a high level, but I cannot think of any examples after thinking about it for a while.
I’m highly experienced at the combination of probability theory, algorithms, and big business data processing. Big businesses have a data problem, they ask a consultant from my company, the consultant realizes there’s a probabilistic algorithm component to the problem, and they call me. I guess if I didn’t exist, that would be a Curse of Doom, but that seems pretty farfetched to call it a Curse. If I wasn’t around, a few big companies would have slightly less efficient algorithms. It’s millions of dollars over the years, but not a big deal in the scheme of things.
Also, ”Curse of Doom” is an extremely generic term. You might find it sticks to people‘s brains better if you gave it a more specific name. “Curse of the missing polymath”?
Jellyfish have nematocysts, which is a spear on a rope, with poison on the tip. The spear has barbs, so when it goes in, it sticks. Then the jellyfish pulls in its prey. The spears are microscopic, but very abundant.
It‘s possible to filter out a constant high value, but not possible to filter out a high level of noise. Unfortunately warmth = random vibration = noise. If you want a low noise thermal camera, you have to cool the detector, or only look for hot things, like engine flares. Fighter planes do both.