Posts
Comments
Although the argument you outline might be an argument against ever fully trusting tests (usually called "evals" on this site) that this or that AI is aligned, alignment researchers have other tools in their toolbox besides running tests or evals.
It would take a long time to explain these tools, particularly to someone unfamiliar with software development or a related field like digital-electronics design. People make careers in studying tools to make reliable software systems (and reliable digital designs).
The space shuttle is steered by changing the direction in which the rocket nozzles point relative to the entire shuttle. If at any point in flight, one of the rocket nozzles had pointed in a direction a few degrees different from the one it should point in, the shuttle would have been lost and all aboard would have died. The pointing or aiming of these rocket nozzles is under software control. How did the programmers at NASA make this software reliable? Not merely through testing!
These programmers at NASA relied on their usual tools (basically engineering-change-order culture) and did not need a more advanced tool called "formal verification", which Intel turned to to make sure their microprocessors did not have any flaw necessitating another expensive recall after they spent 475 million dollars in 1994 in a famous product recall to fix the so-called Pentium FDIV bug.
Note that FDIV refers to division of (floating-point) numbers and that it is not possible in one human lifetime to test all possible dividends and divisors to ensure that a divider circuit is operating correctly. I.e., the "impossible even theoretically" argument you outline would have predicted that it is impossible to ensure the correct operation of even something as simple as a divider implemented in silicon, and yet Intel has during the 30 years since the 1994 recall avoided another major recall of any of their microprocessors for reasons of a mistake in their digital design.
"Memory allocation" errors (e.g., use-after-free errors) are an important source of bugs and security vulnerabilities in software, and testing has for decades been an important way to find and eliminate these errors (Valgrind probably being the most well-known framework for doing the testing) but the celebrated new programming language Rust completely eliminates the need for testing for this class of programming errors. Rust replaces testing with a more reliable methodology making use of a tool called a "borrow checker". I am not asserting that a borrow checker will help people create an aligned super-intelligence: I am merely pointing at Rust and its borrow checker as an example of a methodology that is superior to testing for ensuring some desirable property (e.g., the absence of use-after-free errors that an attacker might be able to exploit) of a complex software artifact.
In summary, aligning a superhuman AI is humanly possible given sufficiently careful and talented people. The argument for stopping frontier AI research (or pausing it for 100 years) depends on considerations other than the "impossible even theoretically" argument you outline.
Methodologies that are superior to testing take time to develop. For example, the need for a better methodology to prevent "memory allocation" errors was recognized in the 1970s. Rust and its borrow checker are the result of a line of investigation inspired by a seminal paper published in 1987. But Rust has become a realistic option for most programming projects only within the last 10 years or less. And an alignment methodology that continue to be reliable even when the AI becomes super-humanly capable is a much taller order than a methodology to prevent use-after-free errors and related memory-allocation errors.
The strongest sign an attack is coming that I know of is firm evidence that Russia or China is evacuating her cities.
Another sign that would get me to flee immediately (to a rural area of the US: I would not try to leave the country) is a threat by Moscow that Moscow will launch an attack unless Washington takes action A (or stops engaging in activity B) before specific time T.
Western Montana is separated from the missile fields by mountain ranges and the prevailing wind direction and is in fact considered the best place in the continental US to ride out a nuclear attack by Joel Skousen. Being too far away from population centers to be walkable by refugees is the main consideration for Skousen.
Skousen also likes the Cumberland Plateau because refugees are unlikely to opt to walk up the escarpment that separates the Plateau from the population centers to its south.
The overhead is mainly the "fixed cost" of engineering something that works well, which suggest re-using some of the engineering costs already incurred in making it possible for a person to make a hands-free phone call on a smartphone.
Off-topic: most things (e.g., dust particles) that land in an eye end up in the nasal cavity, so I would naively expect that protecting the eyes would be necessary to protect oneself fully from respiratory viruses:
https://www.ranelle.com/wp-content/uploads/2016/08/Tear-Dust-Obstruction-1024x485.jpg
Does anyone care to try to estimate how much the odds ratio of getting covid (O(covid)) decreases when we intervene by switching a "half-mask" respirator such as the ones pictured here for a "full-face" respirator (which protects the eyes)?
The way it is now, when one lab has an insight, the insight will probably spread quickly to all the other labs. If we could somehow "drive capability development into secrecy," that would drastically slow down capability development.
Malice is a real emotion, and it is a bad sign (but not a particularly strong sign) if a person has never felt it.
Yes, letting malice have a large influence on your behavior is a severe character flaw, that is true, but that does not mean that never having felt malice or being incapable of acting out of malice is healthy.
Actually, it is probably rare for a person never to act out of malice: it is probably much more common for a person to just be unaware of his or her malicious motivations.
The healthy organization is to be tempted to act maliciously now and then, but to be good at perceiving when that is happening and to ignore or to choose to resist the temptation most of the time (out of a desire to be a good person).
I expect people to disagree with this answer.
their >90% doom disagrees with almost everyone else who thinks seriously about AGI risk.
The fact that your next sentence refers to Rohin Shah and Paul Christiano, but no one else, makes me worry that for you, only alignment researchers are serious thinkers about AGI risk. Please consider that anyone whose P(doom) is over 90% is extremely unlikely to become an alignment researcher (or to remain one if their P(doom) became high when they were an alignment researcher) because their model will tend predict that alignment research is futile or that it actually increases P(doom).
There is a comment here (which I probably cannot find again) by someone who was in AI research in the 1990s, then he realized that the AI project is actually quite dangerous, so he changed careers to something else. I worry that you are not counting people like him as people who have thought seriously about AGI risk.
I disagree. I think the fact that our reality branches a la Everett has no bearing on our probability of biogensis.
Consider a second biogenesis that happened recently enough and far away enough that light (i.e., information, causal influence) has not had enough time to travel from it to us. We know such regions of spacetime "recent enough and far away enough" exist and in principle could host life, but since we cannot observe a sign of life or a sign of lack of life from them, they are not relevant to our probability of biogenesis whereas by your logic, they are relevant.
new cities like Los Alamos or Hanover
You mean Hanford.
When you write,
a draft research paper or proposal that frames your ideas into a structured and usable format,
who is "you"?
I know you just said that you don't completely trust Huberman, but just today, Huberman published a 30-minute video titled "Master your sleep and be more alert when awake". I listened to it (twice) to refresh my memory and to see if his advice changed.
He mentions yellow-blue (YB) contrasts once (at https://www.youtube.com/watch?v=lIo9FcrljDk&t=502s) and at least thrice he mentions the desirability of exposure to outdoor light when the sun is at a low angle (close to the horizon). As anyone can see by looking around at dawn and again at mid-day, at dawn some parts of the sky will be yellowish (particularly, the parts of the sky near the sun) or even orange whereas other parts will range from pale blue to something like turquoise to deep blue whereas at mid-day the sun is white, the part of the sky near the sun is blue and the blue parts of the sky are essentially all the same shade or hue of blue.
He also says that outdoor light (directly from the sun or indirectly via atmospheric scattering) is the best kind of light for maintaining a healthy circadian rhythm, but that if getting outdoors early enough that the sun is still low in the sky is impractical, artificial light can be effective, particularly blue-heavy artificial light.
I've been help greatly over the last 2 years by a protocol in which I get outdoor light almost every morning when the YB contrasts are at its most extreme, namely between about 20 min before sunrise and about 10 min after sunrise on clear days and a little later on cloudy days. (The other element of my protocol that I know to be essential is strictly limit my exposure to light between 23:00 and 05:00.) I was motivated to comment on your post because it did not contain enough information to help someone sufficiently similar to me (the me of 2 years ago) to achieve the very welcome results I achieved: I'm pretty sure that even very bright artificial light from ordinary LED lights that most of us have in our home (even very many of them shining all at once) would not have helped me nearly as much.
Huberman is not so insistent on getting outside during this 30-minute interval of maximum YB contrast as my protocol is. In fact in today's video he says that he himself often gets outside only after the sun has been out for an hour or 2 and is consequently no longer particularly near the horizon.
Health-conscious people apply a (software-based) filter to their screens in the evening to reduce blue light emitted from the screen. On iOS this is called Night Shift. If your rendition of the effects of light on the circadian rhythm (CR) is complete, then they're doing everything they can do, but if YB contrasts have important effects on the CR, it might be useful in addition to eliminate YB contrasts on our digital devices (which Night Shift and its analogs on the other platforms do not eliminate). This can be done by turning everything shades of gray. (On iOS for example, this can be achieved in Settings > Accessibility > Display & Text Size > Color Filters > Grayscale and can be combined with or "overlaid on" Night Shift.) I and others do this (turn of a filter that makes everything "grayscale") routinely to make it more likely that we will get sleepy sufficiently early in the evening. Additional people report that they like to keep their screens grayscale, but do not cite the CR as the reason for their doing so.
Is a computer screen bright enough such that YB contrasts on the screen can activate the machinery in the retina that is activated by a sunrise? I'm not sure, but I choose to eliminate YB contrasts on my screens just in case it is.
Finally let me quote what I consider the main takeaway from the video Huberman posted today, which I expect we both agree with:
Get up each morning, try to get outside. I know that can be challenging for people, but anywhere from 2 to 10 min of sun exposure will work well for most people. If you can't do it every day or if you sleep through this period of early-day low-solar angle, don't worry about it. The systems in the body -- these hormone systems and neurotransmitter systems -- that make you awake at certain periods of the day and sleepy at other times are operating by averaging when you view the brightest light.
This post paints a partially inaccurate picture. IMHO the following is more accurate.
Unless otherwise indicated, the following information comes from Andrew Huberman. Most comes from Huberman Lab Podcast #68. Huberman opines on a great many health topics. I want to stress that I don't consider Huberman a reliable authority in general, but I do consider him reliable on the circadian rhythm and on motivation and drive. (His research specialization for many years was the former and he for many years has successfully used various interventions to improve his own motivation and drive, which is very high.)
Bright light (especially bluish light) makes a person more alert. (Sufficiently acute exposure to cold, e.g., a plunge into 45-degree water, is an even stronger cause of alertness. Caffeine is another popular intervention for causing alertness.) After many hours of being alert and pursuing goals, a person will get tired, and this tiredness tends to help the person go to sleep. However, the SCN operates independently of exposure to bright light (and cold and caffeine) and independently of how many hours in a row the person has already been alert. A good illustration of that is what happens when a person pulls an all-nighter: at about 4:30 it becomes easier for most people pulling an all-nighter to stay awake even if the person is not being exposed to bright light and even if the person has already been awake for a very long time. Without any light as a stimulus, at around 04:30 the SCN decides to get the brain and the body ready for wakefulness and activity. So, let us inquire how the SCN would stay in sync with the sun in the ancestral environment before the availability of artificial lighting. How does the SCN know that dawn is coming soon?
The answer is that it is complicated, like most things in biology, but I think most neuroscientists agree that the stimulus that most potently entrains the SCN (i.e., that is most effective at ensuring the the SCN is in sync with the sun) is yellow-blue (YB) contrasts. Specifically, the SCN knows it is 4:30 and consequently time to start making the body alert because of the person's exposure to these "YB contrasts" on previous days. Exposure in the evening has an effect, but the strongest effect is exposure circa dawn.
When the sun is at a high angular elevation, it is white and the surrounding sky is dark blue (assuming a cloudless sky). When the sun is slightly above or below the horizon, the part of the sky near the sun is yellow or even orange or pink and with further (angular) distance from the sun, the sky gets steadily bluer. (Note that even 30 minutes before sunrise, the sky is already much brighter than your cell phone's screen or most indoor environments: there is an optical illusion whereby people underestimate the brightness of a light source when the source is spread over a large (angular) area and overestimate the brightness of "point sources" like light bulbs.)
The sensing of these YB contrasts is done by a system distinct from the usual visual system (i.e., the system that gives visual information that is immediately salient to the conscious mind) and in particular there are fewer "sensing pixels" and they are spread further apart than the "pixels" in the usual visual system. The final page of this next 5-page paper has a nice image of the author's estimate of a bunch of "sensing pixels" depicted as dotted circles laid over a photo of a typical sunrise:
https://pmc.ncbi.nlm.nih.gov/articles/PMC8407369/pdf/nihms-1719642.pdf
A light bulb named Tuo is recommended at least half-heartedly by Huberman for controlling the circadian rhythm. Huberman says IIRC it works by alternating between yellow light and blue light many times a second. Huberman explained IIRC that both "spatial" YB contrasts and "temporal" YB contrasts serve a signal that "dawn or dusk is happening". I definitely recall Huberman saying that outdoor light is preferred to this Tuo light bulb, and I understood him to mean that it is more likely to work because no one understands SCN entrainment well enough right now to design an artificial light source guaranteed to work.
The web site for this light bulb says,
Most light therapy products on the market today are based on blue light. This is 15-year-old science that has since been proven incorrect. New science based on laboratory research conducted by the University of Washington, one of the world's top vision and neuroscience centers, shows that blue light has little to no effect in shifting your circadian rhythm. High brightness levels of blue light can have some effect, but this falls short when compared to the power of TUO.
High lux products can work, but they require 10,000 lux of light at a distance of under 2 feet for the duration of treatment. This light level at this distance is uncomfortable for most people. Treatment also needs to happen first thing in the morning to be most effective. Who has time to sit within 2 feet of a bulb for up to a half hour when first waking up? High lux products offer dim settings to make use more comfortable and typically downplay the distance requirement. Unfortunately, at less than 10,000 lux and 2 feet of distance, high lux products have little to no impact.
Huberman recommends getting as much bright light early in the day as possible -- "preferably during the first hour after waking, but definitely during the first 3 hours". This encourages the body to produce cortisol and dopamine, which at this time of day is very good for you (and helps you be productive). But this bright light won't have much effect on keeping your circadian clock entrained with the schedule you want it be entrained with unless the light contains YB contrasts; i.e., getting bright light right after waking is good and getting sufficiently-bright light containing YB contrasts right after waking is also good, but they are good for different reasons (though the first kind of good contributes to a small extent to the second kind of good through the mechanism I described in my second paragraph).
Huberman is insistent that it is not enough to expose yourself to light after your normal wake-up time: you also have to avoid light when you are normally asleep. Suppose your normal wake-up time is 06:00. To maintain a strong circadian rhythm and to get to sleep at a regular time each night (which is good for you and which most people should strive to do) it is essential to avoid exposure to light during the 6 hours between 23:00 and 05:00. Whereas dim light has little positive effect after wake-up time, even quite dim light or light of brief duration between 23:00 and 05:00 tends to have pronounced negative effects.
Light during these 6 hours not only confuses the circadian clock (which is bad and makes it hard to get to sleep at a healthy hour) but it also decreases the amount of motivation and drive available the next morning (by sending signals to a brain region called the habenula). I personally have noticed a strong increase in my level of motivation and drive on most mornings after I instituted the habits described in this comment. (And I more reliably start my sleep at what I consider a healthy hour, but that was less critical in my case because insomnia was never a huge problem of mine.)
Huberman says that getting outside at dawn works to keep the SCN in sync with the sun even on completely overcast days, but it requires longer duration of exposure: 20 minutes instead of 5 minutes IIRC. He says that there are YB contrasts in the overcast sky when the sun is near the horizon that are absent when the angle of the sun is high.
To this point in this comment I merely repeated information I learned from Huberman (and maybe a bit from Wikipedia or such -- it is hard to remember) although I hasten to add that this information certainly jibes with my own experience of going outside at dawn almost every day starting about 2 years ago. Allow me to add one thing of my own invention, namely, what to call this 6-hour interval every night when it is a bad idea to let your eyes be exposed to light: I humbly suggest "curfew". The original meaning of "curfew" was a time every night during which it was illegal in medieval London to have a fire going even in your own fireplace in your own home. (I.e., it was a measure to prevent urban fires.)
“Game theoretic strengthen-the-tribe perspective” is a completely unpersuasive argument to me. The psychological unity of humankind OTOH is persuasive when combined with the observation that this unitary psychology changes slowly enough that the human mind’s robust capability to predict the behavior of conspecifics (and manage the risks posed by them) can keep up.
Although I agree with another comment that Wolfram has not "done the reading" on AI extinction risk, my being able to watch his face while he confronts some of the considerations and arguments for the first time made it easier, not harder, for me to predict where his stance on the AI project will end up 18 months from now. It is hard for me to learn anything about anyone by watching them express a series of cached thoughts.
Near the end of the interview, Wolfram say that he cannot do much processing of what was discussed "in real time", which strongly suggests to me that he expects to process it slowly over the next days and weeks. I.e., he is now trying to reassure himself that the AI project won't kill his four children or any grandchildren he has or will have. Because Wolfram is much better AFAICT at this kind of slow "strategic rational" deliberation than most people at his level of life accomplishment, there is a good chance he will fail to find his slow deliberations reassuring, in which case he probably then declares himself an AI doomer. Specifically, my probability is .2 that 18 months from now, Wolfram will have come out publicly against allowing ambitious frontier AI research to continue. P = .2 is much much higher than my P for the average 65-year-old of his intellectual stature who is not specialized in AI. My P is much higher mostly because I watched this interview; i.e., I was impressed by Wolfram's performance in this interview despite his spending the majority of his time on rabbit holes than I could quickly tell had no possible relevance to AI extinction risk.
My probability that he will become more optimistic about the AI project over the next 18 months is .06: mostly likely, he goes silent on the issue or continues to take an inquisitive non-committal stance in his public discussions of it.
If Wolfram had a history of taking responsibility for his community, e.g., campaigning against drunk driving or running for any elected office, my P of his declaring himself an AI doomer (i.e., becoming someone trying to stop AI) would go up to .5. (He might in fact have done something to voluntarily take responsibility for his community, but if so, I haven't been able to learn about it.) If Wolfram were somehow forced to take sides, and had plenty of time to deliberate calmly on the choice after the application of the pressure to choose sides, he would with p = .88 take the side of the AI doomers.
Some people are more concerned about S-risk than extinction risk, and I certainly don't want to dismiss them or imply that their concerns are mistaken or invalid, but I just find it a lot less likely that the AI project will lead to massive human suffering than its leading to human extinction.
the public seems pretty bought-in on AI risk being a real issue and is interested in regulation.
There's a huge gulf between people's expressing concern about AI to pollsters and the kind of regulations and shutdowns that would actually avert extinction. The people (including the "safety" people) whose careers would be set back by many years if they had to find employment outside the AI field and the people who've invested a few hundred billion into AI are a powerful lobbying group in opposition to the members of the general public who tell pollsters they are concerned.
I don't actually know enough about the authoritarian countries (e.g., Russia, China, Iran) to predict with any confidence how likely they are to prevent their populations from contributing to human extinction through AI. I can't help but notice though that so far the US and the UK have done the most to advance the AI project. Also, the government's deciding to shut down movements and technological trends is much more normalized and accepted in Russia, China and Iran than it is in the West, particularly in the US.
I don't have any prescriptions really. I just think that the OP (titled "why the 2024 election matters, the AI risk case for Harris, & what you can do to help", currently standing at 23 points) is badly thought out and badly reasoned, and I wish I had called for readers to downvote it because it encourages people to see everything through the Dem-v-Rep lens (even AI extinction risk, whose causal dependence on the election we don't actually know) without contributing anything significant.
The CNS contains dozens of "feedback loops". Any intervention that drastically alters the equilibrium point of several of those loops is generally a bad idea unless you are doing it to get out of some dire situation, e.g., seizures. That's my recollection of Huberman's main objection put into my words (because I dont recall his words).
Supplementing melatonin is fairly unlikely to have (much of) a permanent effect on the CNS, but you can waste a lot of time by temporarily messing up CNS function for the duration of the melatonin supplementation (because a person cannot make much progress in life with even a minor amount of messed-up CNS function).
A secondary consideration is that melatonin is expensive to measure quantitatively, so the amount tends to vary a lot from what is on the label. In particular, there are reputational consequences and possible legal consequences to a brand's having been found to have less than the label says, so brands tend to err on the side of putting too much melatonin in per pill, which ends up often being manyfold more than the label says.
There are many better ways to regularize the sleep rhythm. My favorite is ensuring I get almost no light at night (e.g., having foil on the windows of the room I sleep in) but then get the right kind of light in the morning, which entails understanding how light affects the intrinsically photosensitive retinal ganglion cells and how those cells influence the circadian rhythm. In fact, I'm running my screens (computer screen and iPad screen) in grayscale all day long to prevent yellow-blue contrasts on the screen from possibly affecting my circadian rhythm. I also use magnesium and theanine according to a complex protocol of my own devising.
Are Eliezer and Nate right that continuing the AI program will almost certainly lead to extinction or something approximately as disastrous as extinction?
A lot of people e.g. Andrew Huberman (who recommends many supplements for cognitive enhancement and other ends) recommend against supplementing melatonin except to treat insomnia that has failed to respond to many other interventions.
It's also important to consider the selection pressure keeping intelligence low, namely, the fact that most animals chronically have trouble getting enough calories, combined with the high caloric needs of neural tissue.
It is no coincidence that human intelligence didn't start rising much till humans were reliably getting meat in their diet and were routinely cooking their food, which makes whatever calories are in food easier to digest, allowing the human gut to get smaller, which in turn reduced the caloric demands of the gut (which needs to be kept alive 24 hours a day even on a day when the person finds no food to eat).
Surely you cannot change the climate of Antarctica without changing the climate of Earth as a whole.
The reason most dental practices were owned by dentists and their families rather than by investors before the passage of the affordable care act is probably that there is no practical way for the investors to tell whether the dentists running the practice are cheating them (e.g., by hiring his cousin to work at twice the rate at which the cousin's labor is really worth).
In contrast, VCs (and the investors into VC funds) can trust the startups they invest in because they pick only companies that have a plan to grow very rapidly (more rapidly than is possible if revenue scales linearly with such a constrained resource as dentist labor or oral-surgeon labor). When the proceeds of an IPO or sale of the company are several orders of magnitude larger than the amount invested, any "self-dealing" on the part of the execs of the company being IPOed or sold become irrelevant -- or at least easier for the investors to detect and to deal with.
Although private-equity funds gets paid mostly by IPO or sale-to-a-larger-company just like VC does, because they don't limit themselves to investments that have the chance of growing by orders of magnitude, they have a more difficult time raising money. They must rely more on loans (as opposed to sale of stock) to raise money, and what sale of stock they do do is to a larger extent to high-net-worth individuals with an expertise in the business being invested in or at least an expertise in the purchase, sale and management of mature companies where "mature" means "no longer has a realistic chance of growing very much larger".
During the dotcom boom, there was a startup out of LA whose plan involved search-engine optimization, but the young men running the startup decided to spend some of the money they raised to make a movie. When the investors found out about the movie, they hired private security contractors who teamed up with LA police to raid the startup and claw back as much of their investment as they could (causing the startup to dissolve). I offer this as an example that self-dealing is possible in a startup, but it is easier to deal with: when a private-equity fund gets together 100 million dollars to buy Kleenex Corp or whatever, then finds out that the execs at Kleenex are spending a significant amount of the corp's money to make a movie (which they didn't tell the investors about) they can't just walk away from their investment: they have to fire the execs and hire new ones or step in and run the corp themselves, both of which will result in an interval of time during which the company is being run by exec still learning about the company, which is a costly undertaking.
VCs are already doing this. They have offered to buy both the oral surgery practice and the dental practice I use in town.
Investors have offered to buy both, but why do you believe those investors were VCs? It seem very unlikely to me that they were.
I worry about that, and I worry about the first AI-based consumer products that are really fun (like video games are) because pleasure is the motivation in most motivated cognition (at least in Western liberal societies).
I've been a heavy consumer of health services and health information since 1985, where "health" here definitely includes mental health (and specifically the effects of childhood trauma).
YT started being my main source of insights about my health and how I might make it better about 3 years ago. During that time I've managed to improve my health faster than the rate I managed in the decades before that.
How do you assess the expertise of those YT creators?
A person could write a book about that (which would tend to overlap a lot with a book about general rationality). I've probably changed some habit of mine (diet, exercise, etc) about ten times over the last 3 years in response to learnings from YT. I watch carefully for effects of those changes. This watching process involves taking voluminous notes. E.g., I write down everything I ingest every day. So, that is one way I assess experts: basically I experiment on myself.
Even though I prefer the written word, I get most of my mental-health (and health) info from Youtube these years. These videos for example taught me important things about trauma, and I think I already knew a lot about it before watching them: https://youtu.be/QHUoSrCOBGE https://youtu.be/LAEB5DIPPX8.
Many hundreds of deep experts on health and mental health have made and uploaded to YT tens of thousands of hours of video, and it is significantly easier for me to find the deep experts on health and mental health on YT than it is on the textual web, but if you do not already know a lot about health and mental health, it might not possible for you to tell which YT creators are the deep experts.
The textual web is still superior for many queries / quests, e.g., what are the names of the active forms of vitamin B6? Do any of the experts who treat or research CFS consider disorders of oxalate metabolism and important component, cause or consequence of CFS (chronic fatigue syndrome)? But if the goal is to learn more about CFS or oxalates (rather than the connection or intersection of CFS and oxalates) or trauma disorder or a particular mental-health condition, I would search YT first.
Personally I’m interested in perspectives from developing world countries
I have nothing relevant here.
Why do you multiply time-to-understand by time-to-implement rather than add them?
plug that utility function (the one the first AI wrote) into it
Could some team make an good AGI or ASI that someone could plug a utility function into? It would be very different from all the models being developed by the leading labs. I'm not confident that humanity could do it in the time we have left.
A brief warning for those making their own purifier: five years ago, Hacker News ran a story, "Build a do-it-yourself air purifier for about $25," to which someone replied,
One data point: my father made a similar filter and was running it constantly. One night the fan inexplicably caught on fire, burned down the living room and almost burned down the house.
Meditation has been practiced for many centuries and millions practice it currently.
Please list 3 people who got deeply into meditation, then went on to change the world in some way, not counting people like Alan Watts who changed the world by promoting or teaching meditation.
I'm not saying that MIRI has some effective plan which more money would help with. I'm only saying that unlike most of the actors accepting money to work in AI Safety, at least they won't use a donation in a way that makes the situation worse. Specifically, MIRI does not publish insights that help the AI project and is very careful in choosing whom they will teach technical AI skills and knowledge.
TsviBT didn't recommend MIRI probably because he receives a paycheck from MIRI and does not want to appear self-serving. I on the other hand have never worked for MIRI and am unlikely ever to (being of the age when people usually retire) so I feel free to recommend MIRI without hesitation or reservation.
MIRI has abandoned hope of anyone's solving alignment before humanity runs out of time: they continue to employ people with deep expertise in AI alignment, but those employees spend their time explaining why the alignment plans of others will not work.
Most technical alignment researchers are increasing P(doom) because they openly publish results that help both the capability research program and the alignment program, but the alignment program is very unlikely to reach a successful conclusion before the capability program "succeeds", so publishing the results only shortens the amount of time we have to luck into an effective response or resolution to the AI danger (which again if one appears might not even involve figuring out how to align an AI so that it stays aligned as it becomes an ASI).
There are 2 other (not-for-profit) organizations in the sector that as far as I can tell are probably doing more good than harm, but I don't know enough about them for it to be a good idea for me to name them here.
I'm going to be a little stubborn and decline to reply till you ask me a question without "simulate" or "simulation" in it. I have an unpleasant memory of getting motte-and-baileyed by it.
Essentially the same question was asked in May 2022 although you did a better job in wording your question. Back then the question received 3 answers / replies and some back-and-forth discussion:
https://www.lesswrong.com/posts/vaX6inJgoARYohPJn/
I'm the author of one of the 3 answers and am happy to continue the discussion. I suggest we continue it here rather than in the 2-year-old web page.
Clarification: I acknowledge that it would be sufficiently easy for an ASI to spare our lives that it would do so if it thought that killing us all carried even a one in 100,000 chance of something really bad happening to it (assuming as is likely that the state of reality many 1000s of years from now matters to the ASI). I just estimate the probability of the ASI's thinking the latter to be about .03 or so -- and most of that .03 comes from considerations other than the consideration (i.e., that the ASI is being fed fake sensory data as a test) we are discussing here. (I suggest tabooiing the terms "simulate" and "simulation".)
I appreciate it when people repost here things Eliezer has written on Twitter or Facebook because it makes it easier for me to stay away from Twitter and Facebook.
(OTOH, I'm grateful to Eliezer for participating on Twitter because posting on Twitter has much higher impact than posting here does.)
You spend a whole section on the health of US democracy. Do you think if US democracy gets worse, then risks from AI get bigger?
It would seem to me that if the US system gets more autocratic, then it becomes slightly easier to slow down the AI juggernaut because fewer people would need to be convinced that the juggernaut is too dangerous to be allowed to continue.
Compare with climate change: the main reason high taxes on gasoline haven't been imposed in the US like they have in Europe is that US lawmakers have been more afraid of getting voted out of office by voters angry that it cost them more to drive their big pickup trucks and SUVs than their counterparts in Europe have been. "Less democracy" in the US would've resulted in a more robust response to climate change! I don't see anything about the AI situation that makes me expect a different outcome there: i.e., I expect "robust democracy" to interfere with a robust response to the AI juggernaut, especially in a few years when it becomes clear to most people just how useful AI-based products and services can be.
Another related argument is that elites don't want themselves and their children to be killed by the AI juggernaut any more than the masses do: its not like castles (1000 years ago) or taxes on luxury goods where the interests of the elites are fundamentally opposed to the interests of the masses. Elites (being smarter) are easier to explain the danger to than the masses are, so the more control we can give the elites relative to the masses, the better our chances of surviving the AI juggernaut, it seems to me. But IMHO we've strayed too far from the original topic of Harris vs Trump, and one sign that we've strayed too far is that IMHO Harris's winning would strengthen US elites a little more than a Trump win would.
Along with direct harms, a single war relevant to US interests could absorb much of the nation’s political attention and vast material resources for months or years. This is particularly dangerous during times as technologically critical as ours
I agree that a war would absorb much of the nation's political attention, but I believe that that effect would be more than cancelled out by how much easier it would become to pass new laws or new regulations. To give an example, the railroads were in 1914 as central to the US economy as the Internet is today -- or close to it -- and Washington nationalized the railroads during WWI, something that simply would not have been politically possible during peacetime.
You write that a war would consume "vast material resources for months or years". Please explain what good material resources do, e.g., in the hands of governments, to help stop or render safe the AI juggernaut? It seems to me that if we could somehow reduce worldwide material-resource availability by a factor of 5 or even 10, our chances of surviving AI get much better: resources would need to be focused on maintaining the basic infrastructure keeping people safe and alive (e.g., mechanized farming, police forces) with the result that there wouldn't be any resources left over to do huge AI training runs or to keep on fabbing ever more efficient GPUs.
I hope I am not being perceived as an intrinsically authoritarian person who has seized on the danger from AI as an opportunity to advocate for his favored policy of authoritarianism. As soon as the danger from AI is passed, I will go right back to being what in the US is called a moderate libertarian. But I can't help but notice that we would be a lot more likely to survive the AI juggernaut if all of the world's governments were as authoritarian as for example the Kremlin is. That's just the logic of the situation. AI is a truly revolutionary technology. Americans (and Western Europeans to a slightly lesser extent) are comfortable with revolutionary changes; Russia and China much less so. In fact, there is a good chance that as soon as Moscow and Beijing are assured that they will have "enough access to AI" to create a truly effective system of surveilling their own populations, they'll lose interest in AI as long as they don't think they need to continue to invest in it in order to stay competitive militarily and economically with the West. AI is capable of transforming society in rapid, powerful, revolutionary ways -- which means that all other things being equal, Beijing (who main goal is to avoid anything that might be described as a revolution) and Moscow will tend to want to supress it as much as practical.
The kind of control and power Moscow and Beijing (and Tehran) have over their respective populations is highly useful for stopping those populations from contributing to the AI juggernaut. In contrast, American democracy and the American commitment to liberty makes the US relatively bad at using political power stopping some project or activity being done by its population. (And the US government was specifically designed by the Founding Fathers to make it a lot of hard work to impose any curb or control on the freedom of the American people). America's freedom, particularly economic and intellectual freedom, in contrast, is highly helpful to the Enemy, namely, those working to make AI more powerful. If only more of the world's countries were like Russia, China and Iran!
I used to be a huge admirer of the US Founding Fathers. Now that I know how dangerous the AI juggernaut is, I wish that Thomas Jefferson had choked on a chicken bone and died before he had the chance to exert any influence on the form of any government! (In the unlikely event that the danger from AI is successfully circumnavigated in my lifetime, I will probably go right back to being an admirer of Thomas Jefferson.) It seemed like a great idea at the time, but now that we know how dangerous AI is, we can see in retrospect that it was a bad idea and that the architects of the governmental systems of Russia, China and Iran were in a very real sense "more correct": those choices of governmental architectures make it easier for humanity to survive the AI gotcha (which was completely hidden from any possibility of human perception at the time those governmental architectural decisions were made, but still, right is right).
I feel that the people who recognize the AI juggernaut for the potent danger that it is are compartmentalizing their awareness of the danger in a regrettable way. Maybe a little exercise would be helpful. Do you admire the inventors of the transistor? Still? I used to, but no longer do. If William Shockley had slipped on a banana peel and hit his head on something sharp and died before he had a chance to assist in the invention of the transistor, that would have been a good thing, I now believe, because the invention of the transistor would have been delayed -- by an expected 5 years in my estimation -- giving humanity more time to become collectively wiser before it must confront the great danger of AI. Of course, we cannot hold Shockley morally responsible because there is no way he could have known about the AI danger. But still, if your awareness of the potency of the danger from AI doesn't cause you to radically re-evaluate the goodness or badness of the invention of the transistor, then you're showing a regrettable lapse in rationality IMHO. Ditto most advances in computing. The Soviets distrusted information technology. The Soviets were right -- probably for the wrong reason, but right is still right, and no one who recognizes AI for the potent danger it is should continue to use the Soviet distrust of info tech as a point against them.
(This comment is addressed to those readers who consider AI to be so dangerous as to make AI risk the primary consideration in this conversation. I say that to account for the possbility that the OP cares mainly about more mundane political concerns and brought up AI safety because he (wrongly IMHO) believes it will help him make his argument.)
Our situation is analogous to someone who has been diagnosed with cancer and told he has a low probability of survival, but in consolation, he has identified a nifty investment opportunity.
Applying decision theory to scenarios involving mutually untrusting agents.
Even people who know English pretty well might prefer to consume information in their native language, particularly when they aren't in a task-oriented frame of mind and do not consider themselves to be engaged in work, which I'm guessing is when people are most receptive to learning more about AI safety.
buy up most of the GPUs or the resources used to produce them
That would backfire IMHO. Specifically, GPUs would become more expensive, but that would last only as long as it takes for the GPU producers to ramp up production (which is very unlikely to take more than 5 years) after which GPU prices would go lower than they would've gone if we hadn't started buying them up (because of better economies of scale).
GPUs and the products and services needs to produce GPUs are not like the commodity silver where if you buy up most of the silver, the economy probably cannot respond promptly by producing a lot more silver. If you could make leading-edge fabs blow up in contrast that would make GPUs more expensive permanently (by reducing investment in fabs) or at least it would if you could convince investors that leading-edge fabs are likely to continue to blow up.
We innately know a lot about "the physical environment", but not a lot about math (because the math abilities of your ancestors have not been under selection pressure for any evolutionary-significant length of time). Although it is true that neuroscience has found brain circuitry specialized for understanding non-negative integers, it remains the fact that much more of an educated person's knowledge about math must be acquired through deliberate practice, which is slow, than his or her knowledge about stacking boxes or replacing an alternator in a car.
In summary, there is no analog in math of your experience of having your knowledge about the physical environment unlock just because you chose to pay more attention to the details of your the physical environment.
We all have an innate drive to understand (i.e., curiosity) and also an innate drive to try to win arguments. Unlocking those 2 motivations is the closest analog I can think of to of your experience in the factory of getting a skill or innate ability to unlock, but before you can wield math towards profitable ends, you must spend many hundreds of hours reading about math and doing math. The 2 motivations I just described merely make it possible for you to put in those many hundreds of hours with less use of willpower or discipline.
Because of the difficulty of predicting a "safe upper bound" on size of rock below which the risk of human extinction is within acceptable limits, I prefer the idea of destroying all the leading-edge fabs in the world or reducing the supply of electricity worldwide to low enough levels that the AI labs cannot compete for electricity with municipalities who need some electricity just to maintain social order and cannot compete with the basic infrastructure required just to keep most people alive. If either of those 2 outcomes weren't hard enough, we would have to maintain such an improved state of affairs (i.e., no leading-edge fab capability or severely degraded electricity generation capability) long enough (i.e., probably at least a century in my estimation) for there to come into being some other, less drastic way of protecting against reckless AI development.
Neither OP's metal-eating bacteria, the large rock from space nor either of the 2 interventions I just described is feasible enough to be worth thinking about much, IMO (and again the large rock from space carries much higher extinction risk than the other 3).
My answer to your first question was, I don't know, but in the interest of comprehensiveness I mentioned a minor exception, which you asked about, so here is more info on the minor exception.
what does encouraging/discouraging sources of intrinsic motivation look like for you?
It looks like making a habit of patiently watching for the desired change. My software environment is organized enough that I can usually arrange for my future self to re-read a written reminder. So, (years ago) I wrote a reminder to watch for any instance where I think my behavior and my decisions are being motivated by interpersonal altruism or I experience pleasure or satisfaction from having achieve an altruistic interpersonal outcome. Note that this did not result in incontrovertible evidence of a significant increase in frequency of altruistic behavior. But I certainly stopped my addiction to the flow motivator (over the course of many years, except I relapse when I'm under stress, but these years it takes a lot of stress) and am pretty sure that the patient watching strategy helped a lot there. (And "just watching" helped me make other kinds of internal mental changes.)
My mind seems to works such that if the only conscious effort I make to effect some internal change is to get into the habit of watching or checking to see if the change has already occurred, my subconscious sometimes seems to figure out a way to effect the change if I watch long enough (months or years).
There are much faster and more potent ways to increase motivation and drive for most people: avoiding all exposure to light between 23:00 and 05:00 every night; getting as much very bright light as possible during the first 3 hours of wakefulness; making sure to get enough tyrosine (a dopamine precursor); deliberate cold exposure; avoiding spending too much of the day in perfectly-safe pleasurable activities; doing enough things you find aversive or outright painful; doing enough risky things. But you didn't ask about that, I don't think: "passion" almost always refers to intrinsic motivation (i.e., basically something that feels good or that a person wants to do for it own sake rather than as part of a plan to achieve some other outcome), whereas the motivation for most of what I (and probably you) do is extrinsic. E.g., I take the time to cook a healthy meal not because I enjoy cooking but rather because I anticipate that eating well will pay dividends mostly in future years in helping me achieve outcomes that I haven't even decided to pursue yet. I took stuff like that to be outside the scope of your question.
value(SHUT IT ALL DOWN) > 0.2 > value(MAGIC) > 0 = value(US first, no race)=value(US first, race)=value(PRC first, no race)=value(PRC first, race)=value(PRC first, race)=value(extinction)
Yes, that is essentially my preference ordering / assignments, which remains the case even if the 0.2 is replaced with 0.05 -- in case anyone is wondering whether there are real human beings outside MIRI who are that pessimistic about the AI project.
No, I haven't made a successful effort to get myself to be passionate about doing X where I was not passionate about X before -- with one minor exception.
I try to factor my motivations into things that seem like complex function adaptations. For example, my motivation for coding projects comes from a strong desire for my being in the flow state and for being dissociated (which might or might not be the same complex function adaptation), but also the desire to "get ahead" or advance my status or position in society and occasionally from curiosity. Curiosity I consider a mildly good intrinsic motivation for me to have (but it does cause me to waste a lot of time doing web searches or listening to long Youtube videos). I wish to strengthen my desire to get ahead; my desire to be in the flow state or to be dissociated I strong try to discourage (because I need to learn ways to keep my nervous system calm other than the flow state and because I was addicted to flow for many many years). I try to encourage what small amount of intrinsic motivation I have to help other people.
So the "one minor exception" I refer to above is that over many years, my efforts to encourage some intrinsic motivators while discouraging others has had effects, the strongest of which has been that these years I mostly succeed in my policy of staying out of deep flow states unless I'm under severe stress. (To reach deep flow, I need to be fairly active, e.g., coding or doing math or playing chess online. My guess is that I'm also entering a flow state when I'm watching an absorbing movie, but if so, passive activities like movie watching never activate deep flow. I should probably mention that these years I never enter a flow state when I am coding AFAICT: my knowing that I'm a more effective at coding when I stay out of the flow state makes it a lot easier for me to stay out of the flow state while coding.) The way I understand motivation to work, by making that one source of intrinsic motivation less influential on my behavior, I am automatically making the other intrinsic motivators more influential. (And one of the major disadvantages of being addicted to something is that it is almost impossible to be intrinsically motivated by anything other than the addictive activity.)
There is a trend toward simplifying model architectures. For example, AlphaGo Zero is simpler than AlphaGo in that it was created without using data from human games. AlphaZero in turn was simpler than AlphaGo Zero (in some way that I cannot recall right now).
Have you tried to find out whether any of the next-generation LLMs (or "transformer-based models") being trained now even bothers to split text into tokens?
OK.
Decision theorists holds that for every sequence of observations and every utility function (set of goals), there is exactly one best move or optimal action (namely, the one that maximizes expected utility conditional on the observations). Does trying to use decision theory as much as possible in one's life tend to push one into having a totalizing worldview?