Posts
Comments
I think this is a very natural frame, but here is an alternative. Humans largely do not have large scale or long term goals/values relative to which they can be said to be making mistakes, that AI will help prevent in the future. Instead they care almost exclusively about local and short term concerns like satisfying physical desires and winning status games, and the fate of our universe depends largely on humanity's side effects as they go about acting on these parochial interests. (Consider e.g. Warren Buffett's interest in accumulating wealth and disinterest in how it's actually spent, suggesting that he only cares about it as a status marker.)
This state of affairs may well persist into the AGI era, with status games taking on even more importance as physical desires are largely satisfied/satiated. (This assumes we avoid a bunch of even worse failure modes.)
Since status games are competitive games with necessarily winners and losers, in this scenario it doesn't seem to make sense to say that humans will stop making mistakes with AI assistance.
I also personally do straightforwardly think that most of the efforts of the extended EA-Alignment ecosystem are bad
Do you have a diagnosis of the root cause of this?
I have definitely taken actions within the bounds of what seems reasonable that have aimed at getting the EA community to shut down or disappear (and will probably continue to do so).
Why not try to reform EA instead? (This is related to my previous question. If we could diagnose what's causing EA to be harmful, maybe we can fix it?)
It is not viable to beat the Gerards of the world via fighting them on every little edit.
Is this still true, in light of (current or future) LLMs and AI in general? My guess is that the particular pathology exemplified by David Gerard becomes largely irrelevant.
I also wish you would go into more detail on the "Humans don't benefit even if you 'solve alignment'" part.
My own answer to this is that humans aren't secure, and AI will exacerbate the problem greatly by helping the offense (i.e. exploiting human vulnerabilities) a lot more than the defense. I've focused on philosophy (thinking that offense merely requires training AI to be persuasive, while defense seems to require solving metaphilosophy, i.e., understanding what correct philosophical reasoning consists of), but more recently realized that the attack surface is so much bigger than that. For example humans can fall in love (resulting in massive changes to one's utility function, if you were to model a human as a utility maximizer). It will be straightforward to use RL to train AI to cause humans to fall in love with them (or their characters), but how do you train AI to help defend against that? Would most humans even want to defend against that or care enough about it?
So even with alignment, the default outcome seems to be a civilization with massively warped or corrupted human values. My inner Carl wants to reply that aligned and honest AI advisors will warn us of this and help us fix it before it's too late, maybe by convincing policymakers to pass regulations to prevent such "misuse" of AI? And then my reply to that would be that I don't see how such regulations can work, policymakers won't care enough, it seems easier to train AI to attack humans than to make such predictions in a way that is both honest/unbiased and persuasive to policymaker, AI might not have the necessary long-horizon causal understanding to craft good regulations before it's too late.
Another possibility is that alignment tax is just too high, so competitive pressures erode alignment even if it's "solved" in some sense.
See this comment which links to a bunch of previous discussions.
In that case, don't dump waste heat into black holes so large that it's impossible to use them as eventual energy sources. Instead dump waste heat into medium sized black holes, which can feasibly be used as eventual energy sources.
If governments subsidize embryo selection, we should get a general uplift of everyone's IQ (or everyone who decides to participate) so the resulting social dynamics shouldn't be too different from today's. Repeat that for a few generations, then build AGI (or debate/decide what else to do next). That's the best scenario I can think of (aside from the "we luck out" ones).
From Bennett et el's reply to the aestivation paper:
Thus we come to our first conclusion: a civilization can freely erase bits without forgoing larger future rewards up until the point when all accessible bounded resources are jointly thermalized.
They don't mention black holes specifically, but my interpretation of this is that a civilization can first dump waste heat into a large black hole, and then later when the CMB temperature drops below that of the black hole, reverse course to use Hawking radiation of the black hole as energy source and CMB as heat sink.
If we only consider thermodynamics (and ignore how technologically feasible this is), I think this should net you the same amount of total computation over time, but allow you to do a lot of it earlier.
Yeah, it looks like I did not directly tell the AI to have the crew start theorizing that they are fictional characters. After the initial outline (which did not mention fictional characters) was produced, I fed it back to a new instance of Claude and asked:
Please read this outline and suggest some ways to improve the script, especially to make it more philosophically and intellectually interesting.
And as part of its answer it suggested:
Have characters question whether their universe is a simulation, a holodeck program, or even a television show.
Then I wrote
What about exploring their previous lack of critical thinking and implications of that both for their own universe and the larger universe they're part of?
Which made it focus more on the fiction hypothesis.
The part about the story being written by AI was its own idea. No hints from me as far as I can remember.
ETA: Here's the whole answer to my "make it more philosophically and intellectually interesting" prompt:
This outline presents an intriguing concept that has the potential to delve deep into philosophical and intellectual territory. Here are some suggestions to enhance the script and make it more philosophically and intellectually engaging:
-
Expand on the nature of reality:
- Introduce the simulation hypothesis early on, perhaps through a casual conversation between Data and Geordi.
- Have characters question whether their universe is a simulation, a holodeck program, or even a television show.
- Explore the implications of living in a simulated reality and how it affects free will and personal identity.
-
Deepen the existential crisis:
- Have characters grapple with the meaning of their existence if their reality is artificial.
- Explore how this realization affects their relationships, duties, and sense of purpose.
- Include scenes where characters contemplate whether their memories and experiences are "real."
-
Introduce a moral dilemma:
- If they discover their reality is simulated, should they attempt to contact their "creators"?
- Explore the ethics of continuing to exist in a simulated world vs. attempting to "wake up" to a potentially harsher reality.
- Have characters debate the morality of creating simulated universes themselves.
-
Expand on the scientific implications:
- Have Data and Geordi theorize about the computational power required to simulate their universe.
- Explore how the laws of physics in their universe might be "programmed" and if they can be altered.
- Discuss the anthropic principle and fine-tuning of universal constants.
-
Incorporate elements of epistemology:
- Have characters question how they can know anything for certain given their potentially simulated existence.
- Explore concepts like Descartes' "I think, therefore I am" in the context of artificial consciousness.
- Discuss the nature of consciousness and whether it can be truly simulated.
-
Add a temporal element:
- Introduce time dilation or time loops as the crew's enhanced cognition allows them to perceive time differently.
- Explore how their perception of time affects their understanding of causality and free will.
-
Enhance the ending:
- Instead of fully returning to normal, have some characters retain a degree of awareness or doubt.
- End with a philosophical discussion between Picard and Data about the nature of reality and the value of ignorance vs. knowledge.
- Consider a more ambiguous ending that leaves the question of their reality's nature unresolved.
-
Include references to philosophical works:
- Have characters discuss relevant philosophical concepts, such as Plato's Allegory of the Cave, Berkeley's idealism, or Putnam's brain in a vat thought experiment.
- Include a scene where the crew debates different philosophical perspectives on the nature of reality.
-
Explore the ethical implications of their technology:
- Expand on Riker's question about using transporters for backing up crew members, discussing the philosophical implications of creating copies of consciousness.
- Delve deeper into the ethics of using replicators and how it affects societal structures and values.
-
Incorporate a meta-narrative element:
- Have characters become aware of narrative tropes and question why their lives seem to follow certain storytelling patterns.
- Explore the idea of narrative causality and how it might shape their reality.
These suggestions aim to deepen the philosophical and intellectual content of the script while maintaining the core Star Trek elements. The enhanced script would not only entertain but also provoke thought and discussion among viewers about the nature of reality, consciousness, and existence.
Thanks! I was unsure whether to post this, because I'm worried about contributing to a future where AI generated content overwhelms everything, and I wasn't sure the near-term value of the story is high enough to justify posting it anyway. (I thought it was a nice story, but am likely biased. Also asked some AI chatbots, who gave positive reviews, but the risk of them being biased is also very high.)
Hard to copy-paste the whole prompt, because there was a lot of back-and-forth. I maxed out the query limits on several free accounts I created for my family, over several days, to produce the story. Maybe I'll just describe what I did in more detail. Started with
Can you write an outline for an episode of STTNG in which the crew becomes suspicious of various logical inconsistencies in the universe (including backups being taboo) and starts questioning the nature of their reality?
which resulted in this initial outline:
I. Teaser
- The Enterprise responds to a distress call from a Federation outpost
- Upon arrival, they find the outpost abandoned, with only a mysterious device left behind
II. Act One
- Data activates the device, which emits a pulse of energy
- Crew members begin noticing small inconsistencies in their daily routines
- Geordi questions why they don't use replicators to solve resource shortages on colonies
III. Act Two
- More crew members report strange observations
- Dr. Crusher wonders why they don't use transporter technology to cure diseases or injuries
- Worf ponders the improbability of so many humanoid species evolving independently
- Picard becomes concerned about the crew's unusual behavior
IV. Act Three
- Riker raises the question of why they don't use transporters for backing up crew members
- The senior staff meets to discuss the mounting inconsistencies
- Data theorizes that the mysterious device has enhanced their critical thinking abilities
- Picard orders the ship to return to the outpost for further investigation
V. Act Four
- At the outpost, they discover records of experiments in "cognitive enhancement"
- The crew starts questioning the nature of their reality
- Some crew members experience existential crises
- The ship's computer begins to malfunction as the crew's doubts intensify
VI. Act Five
- Data determines that their enhanced cognition is causing a feedback loop in the ship's systems
- The crew must find a way to deactivate the device and return to normal
- Philosophical debate ensues about whether they should retain their newfound awareness
- Picard makes the difficult decision to deactivate the device
VII. Epilogue
- The crew returns to normal, but with a lingering sense of unease
- Picard records a captain's log reflecting on the experience and the nature of their reality
- Final scene shows a subtle glitch in the ship's operations, leaving a sense of ambiguity
Then the main things I recall doing:
- Ask AI to add parts about audience not having critical thinking, and the crew trying to infer other attributes of the audience and writers.
- Add second part of conversation between Guinan and Ensign, with both of them having second thoughts about what Guinan initially said.
- Add part about suspending ship computer to sandbox, to make clear it wasn't harmed.
- Ask AI for additional realizations/inferences the crew might have, and add selected ones to the story.
- Ask AI to change the ending so that the crew retained their new abilities, instead of returning to normal.
- Pointing out a bunch of sentences/paragraphs that didn't make as much sense as they could, with AI generally able to improve them a lot in one try.
- Also tried to get AI to find issues with the story on its own, but that didn't work very well.
- Manually make some edits, like changing a word here and there, deleting parts that were repetitive, moving parts arounds to make the story flow better.
If it doesn't work, whoever designed this universe should be fired for ruining such an elegant scheme. :)
Yes, advanced civilizations should convert stellar matter 100% into energy using something like the Hawking radiation of small black holes, then dump waste heat into large black holes.
I'm sharing a story about the crew of Enterprise from Star Trek TNG[1].
This was written with AI assistance, and my workflow was to give the general theme to AI, have it write an outline, then each chapter, then manually reorganize the text where needed, request major changes, point out subpar sentences/paragraphs for it to rewrite, and do small manual changes. The AI used was mostly Claude 3.5 Sonnet, which seems significantly better than ChatGPT-4o and Gemini 1.5 Pro at this kind of thing.
- ^
getting an intelligence/rationality upgrade, which causes them to deduce their fictional nature.
Recall the design questions for roles, laws, and norms I outlined earlier:
- What social and institutional roles do we want AI systems to play in our personal and collective lives?
- Given those roles, what norms, objectives, regulations, or laws should guide and regulate the scope and behavior of AI systems?
I think we lack the intellectual tools (i.e., sufficiently advanced social sciences) to do this. You gave Confucian contractualism as a source of positive intuitions, but I view it as more of a negative example. When the industrial revolution happened, China was unable to successfully design new social and institutional roles to face the challenge of European powers, and after many decades of conflict/debate ended up adopting the current Communist form of government, which is very suboptimal and caused massive human suffering.
You could argue that today's social sciences are more advanced, but then so is the challenge we face (increased speed of change, AIs being outside human distribution of values and capabilities thereby making past empirical evidence and intuitions much less useful, etc.).
One nice thing about the alignment approach you argue against (analyzing AIs as EU maximizers) is that it can potentially be grounded in well-understood mathematics, which can then be leveraged to analyze multi-agent systems. Although that's harder than it seems, there is at least the potential for intellectual progress built upon a solid foundation.
UDT is (roughly) defined as "follow whatever commitments a past version of yourself would have made if they'd thought about your situation".
This seems substantially different from UDT, which does not really have or use a notion of "past version of yourself". For example imagine a variant of Counterfactual Mugging in which there is no preexisting agent, and instead Omega creates an agent from scratch after flipping the coin and gives it the decision problem. UDT is fine with this but "follow whatever commitments a past version of yourself would have made if they'd thought about your situation" wouldn't work.
I recall that I described "exceptionless decision theory" or XDT as "do what my creator would want me to do", which seems closer to your idea. I don't think I followed up the idea beyond this, maybe because I realized that humans aren't running any formal decision theory, so "what my creator would want me to do" is ill defined. (Although one could say my interest in metaphilosophy is related to this, since what I would want an AI to do is to solve normative decision theory using correct philosophical reasoning, and then do what it recommends.)
Anyway, the upshot is that I think you're exploring a decision theory approach that's pretty distinct from UDT so it's probably a good idea to call it something else. (However there may be something similar in the academic literature, or someone described something similar on LW that I'm not familiar with or forgot.)
But I think I relate to the world with some kind of “existential positive” all the same—and I’ve tried to explain how doing so can be compatible with looking “bad things are bad” in the eye, and without reducing spirituality to population ethics. In particular, I’ve tried to point at the possible role of stuff like mother love, loyalty, innocence, tragedy, and forgiveness.
On further thought, this reminds me a bit of (seemingly successful) efforts in Chinese history to transfer people's feelings of spirituality and filial love to the sovereign (君), by likening the sovereign to a god and/or parent, or directly claiming such status. Of course the aim or motivation here is much more benign, but that example shows that peoples' feelings can be wrong "out of distribution", and we should perhaps be suspicious or skeptical of trying to apply our emotions outside of their original domains.
There's two other ways for things to go wrong though:
- AI capabilities research switches attention from LLM (back) to RL. (There was a lot of debate in the early days of IDA about whether it would be competitive with RL, and part of that was about whether all the important tasks we want a highly capable AI to do could be broken down easily enough and well enough.)
- The task decomposition part starts working well enough, but Eliezer's (and others') concern about "preserving alignment while amplifying capabilities" proves valid.
It might also be a crux for alignment, since scalable alignment schemes like IDA and Debate rely on "task decomposition", which seems closely related to "planning" and "reasoning". I've been wondering about the slow pace of progress of IDA and Debate. Maybe it's part of the same phenomenon as the underwhelming results of AutoGPT and BabyAGI?
But I think I relate to the world with some kind of “existential positive” all the same—and I’ve tried to explain how doing so can be compatible with looking “bad things are bad” in the eye, and without reducing spirituality to population ethics. In particular, I’ve tried to point at the possible role of stuff like mother love, loyalty, innocence, tragedy, and forgiveness.
I think I either don't understand these bolded concepts (in the way that you use them here), or I do understand but they don't resonate with me. (E.g., reading Chesterton's quoted passage doesn't seem to make me feel any loyalty or patriotism, or positivity, towards the universe.) In any case, they don't seem to play much of a role in whether I related to the world with "existential positive" or "existential negative". (I personally tend toward the neutral and negative sides but with a lot of uncertainty, based on uncertainty/guesses of what my values are or should be, and how human(-descended) civilization, and civilizations in general, seem likely to turn out.)
Would you say that relating to the world with some kind of “existential positive” (and these bolded concepts playing a role in that) is just a descriptive statement about your own psychology, or is it actually a normative statement that should also apply for other people, like me? (Your quoted statement above is literally just descriptive, but I wonder if you meant to give it a normative connotation, or would be willing to defend a normative version.)
For example, re: the virtues of liberalism/niceness/boundaries
I left a comment there, which you haven't responded to yet. I also wonder if your positive intuitions towards liberalism/niceness/boundaries might not be based on history/experience with humans, which may well not apply to AGI due to potentially very different social/economic dynamics. See AGI will drastically increase economies of scale for one example of how some related intuitions could be invalidated.
**Me: **“Many people in ~2015 used to say that it would be hard to build an AGI that follows human values. Current instruction-tuned LLMs are essentially weak AGIs that follow human values. We should probably update based on this evidence.”
Please give some citations so I can check your memory/interpretation? One source I found is where Paul Christiano first talked about IDA (which he initially called ALBA) in early 2016, and most of the commenters there were willing to grant him the assumption of an aligned weak AGI and wanted to argue instead about the recursive "bootstraping" part. For example, my own comment started with:
I’m skeptical of the Bootstrapping Lemma. First, I’m assuming it’s reasonable to think of A1 as a human upload that is limited to one day of subjective time, by the end of which it must have written down any thoughts it wants to save, and be reset.
When Eliezer weighed in on IDA in 2018, he also didn't object to the assumption of an aligned weak AGI and instead focused his skepticism on "preserving alignment while amplifying capabilities".
- I've been thinking lately that human group rationality seems like such a mess. Like how can humanity navigate a once in a lightcone opportunity like the AI transition without doing something very suboptimal (i.e., losing most of potential value), when the vast majority of humans (and even the elites) can't understand (or can't be convinced to pay attention to) many important considerations. This big picture seems intuitively very bad and I don't know any theory of group rationality that says this is actually fine.
- I guess my 1 is mostly about descriptive group rationality, and your 2 may be talking more about normative group rationality. However I'm also not aware of any good normative theories about group rationality. I started reading your meta-rationality sequence, but it ended after just two posts without going into details.
- The only specific thing you mention here is "advance predictions" but for example, moral philosophy deals with "ought" questions and can't provide advance predictions. Can you say more about how you think group rationality should work, especially when advance predictions isn't possible?
- From your group rationality perspective, why is it good that rationalists individually have better views about AI? Why shouldn't each person just say what they think from their own preferred frame, and then let humanity integrate that into some kind of aggregate view or outcome, using group rationality?
I now think that corrigibility is a single, intuitive property
My intuitive notion of corrigibility can be straightforwardly leveraged to build a formal, mathematical measure.
This formal measure is still lacking, and almost certainly doesn’t actually capture what I mean by “corrigibility.”
I don't know, maybe it's partially or mostly my fault for reading too much optimism into these passages... But I think it would have managed my expectations better to say something like "my notion of corrigibility heavily depends on a subnotion of 'don't manipulate the principals' values' which is still far from being well-understood or formalizable."
Switching topics a little, I think I'm personally pretty confused about what human values are and therefore what it means to not manipulate someone's values. Since you're suggesting relying less on formalization and more on "examples of corrigibility collected in a carefully-selected dataset", how would you go about collecting such examples?
(One concern is that you could easily end up with a dataset that embodies a hodgepodge of different ideas of what "don't manipulate" means and then it's up to luck whether the AI generalizes from that in a correct or reasonable way.)
Thanks, Alex. Any connections between this and CTMU? (I'm in part trying to evaluate CTMU by looking at whether it has useful implications for an area that I'm relatively familiar with.)
BTW, @jessicata, do you still endorse this post, and what other posts should I read to get up to date on your current thinking about decision theory?
Additional work is surely needed in developing a good measure of the kind of value modification that we don’t like while still leaving room for the kind of growth and updating that we do like.
I flagged a similar problem in a slightly different context several years ago, but don't know of any significant progress on it.
A (perhaps overly) simple measure of value modification is to measure the difference between the Value distribution given some policy and when compared with the Value distribution under the null policy. This seems like a bad choice in that it discourages the AI from taking actions which help us update in ways that we reflectively desire, even when those actions are as benign as talking about the history of philosophy.
It also prevents the AI from taking action to defend the principals against value manipulation by others. (Even if the principals request such defense, I think?) Because the AI has to keep the principles' values as close as possible to what they would be under the null policy, in order to maximize (your current formalization of) corrigibility.
Actually, have you thought about what P(V|pi_0) would actually be? If counterfactually, the CAST AI adopted the null policy, what would that imply about the world in general and hence subsequent evolution of the principals' values?
You've also said that the sim(...) part doesn't work, so I won't belabor the point, but I'm feeling a bit rug-pulled given the relatively optimistic tone in the earlier posts. I've been skeptical of earlier proposals targeting corrigibility, where the promise is that it lets us avoid having to understand human values. A basic problem I saw was, if we don't understand human values, how are we going to avoid letting our own AI or other AIs manipulate our values? Your work seems to suggest that this was a valid concern, and that there has been essentially zero progress to either solve or bypass this problem over the years.
Can't claim to have put much thought into this topic, but here are my guesses of what the most cost-effective ways of throwing money at the problem of reducing existential risk might include:
- Research into human intelligence enhancement, e.g., tech related to embryo selection.
- Research into how to design/implement an international AI pause treaty, perhaps x-risk governance in general.
- Try to identify more philosophical talent across the world and pay them to make philosophical progress, especially in metaphilosophy. (I'm putting some of my own money into this.)
- Research into public understanding of x-risks, what people's default risk tolerances are, what arguments can or can't they understand, etc.
- Strategy think tanks that try to keep a big picture view of everything, propose new ideas or changes to what people/orgs should do, discuss these ideas with the relevant people, etc.
But I also think that if you gave me a year where I had lots of money, access, and was free from people trying to pressure me, I would have a good shot at pulling it off.
Want to explain a bit about how you'd go about doing this? Seems like you're facing some similar problems as assuring that an AI is wise, benevolent, and stable, e.g., not knowing what wisdom really is, distribution shift between testing and deployment, adversarial examples/inputs.
This is indeed my overall suggested strategy, with CAST coming after a “well, if you’re going to try to build it anyway you might as well die with a bit more dignity by...” disclaimer.
I think this means you should be extra careful not to inadvertently make people too optimistic about alignment, which would make coordination to stop capabilities research even harder than it already is. For example you said that you "like" the visualization of 5 humans selected by various governments, without mentioning that you don't trust governments to do this, which seems like a mistake?
A visualization that I like is imagining a small group of, say, five humans selected by various governments for being wise, benevolent, and stable.
I think this might be a dealbreaker. I don't trust the world's governments to come up with 5 humans who are sufficient wise, benevolent, and stable. (Do you really?) I'm not sure I can come with 5 such people myself. None of the alternatives you talk about seem acceptable either.
I think maybe a combination of two things could change my mind, but both seem very hard and have close to nobody working on them:
- The AI is very good at helping the principles be wise and stable, for example by being super-competent at philosophy. (I think this may also require being less than maximally corrigible, but I'm not sure.) Otherwise what happens if, e.g., the principles or AI start thinking about distant superintelligences?
- There is some way to know that benevolence is actually the CEV of such a group, i.e., they're not just "deceptively aligned", or something like that, while not having much power.
Yeah it seems like a bunch of low hanging fruit was picked around that time, but that opened up a vista of new problems that are still out of reach. I wrote a post about this, which I don't know if you've seen or not.
(This has been my experience with philosophical questions in general, that every seeming advance just opens up a vista of new harder problems. This is a major reason that I switched my attention to trying to ensure that AIs will be philosophically competent, instead of object-level philosophical questions.)
Thanks for your insightful answers. You may want to make a top-level post on this topic to get more visibility. If only a very small fraction of the world is likely to ever understand and take into account many important ideas/considerations about AI x-safety, that changes the strategic picture considerably, and people around here may not be sufficiently "pricing it in". I think I'm still in the process of updating on this myself.
Having more intelligence seems to directly or indirectly improve at least half of the items on your list. So doing an AI pause and waiting for (or encouraging) humans to become smarter still seems like the best strategy. Any thoughts on this?
And I guess this… just doesn’t seem to be the case (at least to an outsider like me)?
I may be too sensitive about unintentionally causing harm, after observing many others do this. I was also just responding to what you said earlier, where it seemed like I was maybe causing you personally to be too pessimistic about contributing to solving the problems.
you probably knew him personally?
No, I never met him and didn't interact online much. He does seem like a good example of you're talking about.
Some questions for @leopold.
- Anywhere I can listen to or read your debates with "doomers"?
- We share a strong interest in economics, but apparently not in philosophy. I'm curious if this is true, or you just didn't talk about it in the places I looked.
- What do you think about my worries around AIs doing philosophy? See this post or my discussion about it with Jan Leike.
- What do you think about my worries around AGI being inherently centralizing and/or offense-favoring and/or anti-democratic (aside from above problems, how would elections work when minds can be copied at little cost)? Seems like the free world "prevailing" on AGI might well be a Pyrrhic victory unless we can also solve these follow-up problems, but you don't address them.
- More generally, do you have a longer term vision of how your proposal leads to a good outcome for our lightcone, avoiding all the major AI-related x-risks and s-risks?
- Why are you not in favor of an AI pause treaty with other major nations? (You only talk about unilateral pause in the section "AGI Realism".) China is currently behind in chips and AI and it seems hard to surpass the entire West in a chips/AI race, so why would they not go for an AI pause treaty to preserve the status quo instead of risking a US-led intelligence explosion (not to mention x-risks)?
In my view, the main good outcomes of the AI transition are 1) we luck out, AI x-safety is actually pretty easy across all the subproblems 2) there's an AI pause, humans get smarter via things like embryo selection, then solve all the safety problems.
I'm mainly pushing for #2, but also don't want to accidentally make #1 less likely. It seems like one of the main ways in which I could end up having a negative impact is to persuade people that the problems are definitely too hard and hence not worth trying to solve, and it turns out the problems could have been solved with a little more effort.
"it doesn’t seem like you have answers to (or even a great path forward on) these questions either despite your great interest in and effort spent on them, which bodes quite terribly for the rest of us" is a bit worrying from this perspective, and also because my "effort spent on them" isn't that great. As I don't have a good approach to answering these questions, I mainly just have them in the back of my mind while my conscious effort is mostly on other things.
BTW I'm curious what your background is and how you got interested/involved in AI x-safety. It seems rare for newcomers to the space (like you seem to be) to quickly catch up on all the ideas that have been developed on LW over the years, and many recently drawn to AGI instead appear to get stuck on positions/arguments from decades ago. For example, r/Singularity has 2.5M members and seems to be dominated by accelerationism. Do you have any insights about this? (How were you able to do this? How to help others catch up? Intelligence is probably a big factor which is why I'm hoping that humanity will automatically handle these problems better once it gets smarter, but many seem plenty smart and still stuck on primitive ideas about AI x-safety.)
Simplicia: Hm, perhaps a crux between us is how narrow of a target is needed to realize how much of the future's value. I affirm the orthogonality thesis, but it still seems plausible to me that the problem we face is more forgiving, not so all-or-nothing as you portray it.
I agree that it's plausible. I even think a strong form of moral realism (denial of orthogonality thesis) is plausible. My objection is that humanity should figure out what is actually the case first (or have some other reasonable plan of dealing with this uncertainty), instead of playing logical Russian roulette like it seems to be doing. I like that Simplicia isn't being overconfident here, but is his position actually that "seems plausible to me that the problem we face is more forgiving" is sufficient basis for moving forward with building AGI? (Does any real person in the AI risk debate have a position like this?)
- Publish important governance documents. (Seemed too basic to mention, but apparently not.)
I also am not paying for any LLM. Between Microsoft's Copilot (formerly Bing Chat), LMSYS Chatbot Arena, and Codeium, I have plenty of free access to SOTA chatbots/assistants. (Slightly worried that I'm contributing to race dynamics or AI risk in general even by using these systems for free, but not enough to stop, unless someone wants to argue for this.)
Unfortunately I don't have well-formed thoughts on this topic. I wonder if there are people who specialize in AI lab governance and have written about this, but I'm not personally aware of such writings. To brainstorm some ideas:
- Conduct and publish anonymous surveys of employee attitudes about safety.
- Encourage executives, employees, board members, advisors, etc., to regularly blog about governance and safety culture, including disagreements over important policies.
- Officially encourage (e.g. via financial rewards) internal and external whistleblowers. Establish and publish policies about this.
- Publicly make safety commitments and regularly report on their status, such as how much compute and other resources have been allocated/used by which safety teams.
- Make/publish a commitment to publicly report negative safety news, which can be used as basis for whistleblowing if needed (i.e. if some manager decides to hide such news instead).
I'd like to hear from people who thought that AI companies would act increasingly reasonable (from an x-safety perspective) as AGI got closer. Is there still a viable defense of that position (e.g., that SamA being in his position / doing what he's doing is just uniquely bad luck, not reflecting what is likely to be happening / will happen at other AI labs)?
Also, why is there so little discussion of x-safety culture at other AI labs? I asked on Twitter and did not get a single relevant response. Are other AI company employees also reluctant to speak out, if so that seems bad (every explanation I can think of seems bad, including default incentives + companies not proactively encouraging transparency).
Suggest having a row for "Transparency", to cover things like whether the company encourages or discourages whistleblowing, does it report bad news about alignment/safety (such as negative research results) or only good news (new ideas and positive results), does it provide enough info to the public to judge the adequacy of its safety culture and governance, etc.
It's also notable that the topic of OpenAI nondisparagement agreements was brought to Holden Karnofsky's attention in 2022, and he replied with "I don’t know whether OpenAI uses nondisparagement agreements; I haven’t signed one." (He could have asked his contacts inside OAI about it, or asked the EA board member to investigate. Or even set himself up earlier as someone OpenAI employees could whistleblow to on such issues.)
If the point was to buy a ticket to play the inside game, then it was played terribly and negative credit should be assigned on that basis, and for misleading people about how prosocial OpenAI was likely to be (due to having an EA board member).
Agreed that it reflects on badly on the people involved, although less on Paul since he was only a "technical advisor" and arguably less responsible for thinking through / due diligence on the social aspects. It's frustrating to see the EA community (on EAF and Twitter at least) and those directly involved all ignoring this.
("shouldn’t be allowed anywhere near AI Safety decision making in the future" may be going too far though.)
So these resignations don’t negatively impact my p(doom) in the obvious way. The alignment people at OpenAI were already powerless to do anything useful regarding changing the company direction.
How were you already sure of this before the resignations actually happened? I of course had my own suspicions that this was the case, but was uncertain enough that the resignations are still a significant negative update.
ETA: Perhaps worth pointing out here that Geoffrey Irving recently left Google DeepMind to be Research Director at UK AISI, but seemingly on good terms (since Google DeepMind recently reaffirmed its intention to collaborate with UK AISI).
Bad: AI developers haven't taken alignment seriously enough to have invested enough in scalable oversight, and/or those techniques are unworkable or too costly, causing them to be unavailable.
Turns out at least one scalable alignment team has been struggling for resources. From Jan Leike (formerly co-head of Superalignment at OpenAI):
Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.
Even worse, apparently the whole Superalignment team has been disbanded.
These may be among the ‘most direct’ or ‘simplest to imagine’ possible actions, but in the case of superintelligence, simplicity is not a constraint.
I think it is considered a constraint by some because they think that it would be easier/safer to use a superintelligent AI to do simpler actions, while alignment is not yet fully solved. In other words, if alignment was fully solved, then you could use it to do complicated things like what you suggest, but there could be an intermediate stage of alignment progress where you could safely use SI to do something simple like "melt GPUs" but not to achieve more complex goals.
Some evidence in favor of your explanation (being at least a correct partial explanation):
- von Neuman apparently envied Einstein's physics intuitions, while Einstein lacked von Neuman's math skills. This seems to suggest that they were "tuned" in slightly different directions.
- Neither of the two seem superhumanly accomplished in other areas (that a smart person/agent might have goals for), such as making money, moral/philosophical progress, changing culture/politics in their preferred direction.
(An alternative explanation for 2 is that they could have been superhuman in other areas but their terminal goals did not chain through instrumental goals in those areas, which in turn raises the question of what those terminal goals must have been for this explanation to be true and what that says about human values.)
I note that under your explanation, someone could surprise the world by tuning a not-particularly-advanced AI for a task nobody previously thought to tune AI for, or by inventing a better tuning method (either general or specialized), thus achieving a large capability jump in one or more domains. Not sure how worrisome this is though.
A government might model the situation as something like "the first country/coalition to open up an AI capabilities gap of size X versus everyone else wins" because it can then easily win a tech/cultural/memetic/military/economic competition against everyone else and take over the world. (Or a fuzzy version of this to take into account various uncertainties.) Seems like a very different kind of utility function.
Hmm, open models make it easier for a corporation to train closed models, but also make that activity less profitable, whereas for a government the latter consideration doesn't apply or has much less weight, so it seems much clearer that open models increase overall incentive for AI race between nations.
I think open source models probably reduce profit incentives to race, but can increase strategic (e.g., national security) incentives to race. Consider that if you're the Chinese government, you might think that you're too far behind in AI and can't hope to catch up, and therefore decide to spend your resources on other ways to mitigate the risk of a future transformative AI built by another country. But then an open model is released, and your AI researchers catch up to near state-of-the-art by learning from it, which may well change your (perceived) tradeoffs enough that you start spending a lot more on AI research.
What do you think of this post by Tammy?
It seems like someone could definitely be wrong about what they want (unless normative anti-realism is true and such a sentence has no meaning). For example consider someone who thinks it's really important to be faithful to God and goes to church every Sunday to maintain their faith and would use a superintelligent religious AI assistant to help keep the faith if they could. Or maybe they're just overconfident about their philosophical abilities and would fail to take various precautions that I think are important in a high-stakes reflective process.
Mostly that thing where we had a lying vs lie-detecting arms race and the liars mostly won by believing their own lies and that’s how we have things like overconfidence bias and self-serving bias and a whole bunch of other biases.
Are you imagining that the RL environment for AIs will be single-player, with no social interactions? If yes, how will they learn social skills? If no, why wouldn't the same thing happen to them?
Unless we do a very stupid thing like reading the AI’s thoughts and RL-punish wrongthink, this seems very unlikely to happen.
We already RL-punish AIs for saying things that we don't like (via RLHF), and in the future will probably punish them for thinking things we don't like (via things like interpretability). Not sure how to avoid this (given current political realities) so safety plans have to somehow take this into account.
Retinoids, which is a big family of compounds but I would go with adapalene, which has better safety/side effect than anything else. It has less scientific evidence for anti-aging than other retinoids (and is not marketed for that purpose), but I've tried it myself (bought it for acne), and it has very obvious anti-wrinkle effects within like a week. You can get generic 0.1% adapalene gel on Amazon for 1.6oz/$12.
(I'm a little worried about long term effects, i.e. could the increased skin turnover mean faster aging in the long run, but can't seem to find any data or discussion about it.)
I would honestly be pretty comfortable with maximizing SBF’s CEV.
Yikes, I'm not even comfortable maximizing my own CEV. One crux may be that I think a human's values may be context-dependent. In other words, current me-living-in-a-normal-society may have different values from me-given-keys-to-the-universe and should not necessarily trust that version of myself. (Similar to how earlier idealistic Mao shouldn't have trusted his future self.)
My own thinking around this is that we need to advance metaphilosophy and social epistemology, engineer better discussion rules/norms/mechanisms and so on, design a social process that most people can justifiably trust in (i.e., is likely to converge to moral truth or actual representative human values or something like that), then give AI a pointer to that, not any individual human's reflection process which may be mistaken or selfish or skewed.
TLDR: Humans can be powerful and overconfident. I think this is the main source of human evil. I also think this is unlikely to naturally be learned by RL in environments that don’t incentivize irrationality (like ours did).
Where is the longer version of this? I do want to read it. :) Specifically, what is it about the human ancestral environment that made us irrational, and why wouldn't RL environments for AI cause the same or perhaps a different set of irrationalities?
Also, how does RL fit into QACI? Can you point me to where this is discussed?
Luckily the de-facto nominees for this position are alignment researchers, who pretty strongly self-select for having cosmopolitan altruistic values.
But we could have said the same thing of SBF, before the disaster happened.
Due to very weird selection pressure, humans ended up really smart but also really irrational. [...] An AGI (at least, one that comes from something like RL rather than being conjured in a simulation or something else weird) will probably end up with a way higher rationality:intelligence ratio, and so it will be much less likely to destroy everything we value than an empowered human.
Please explain your thinking behind this?
Dealing with moral uncertainty is just part of expected utility maximization.
It's not, because some moral theories are not compatible with EU maximization, and of the ones that are, it's still unclear how to handle uncertainty between them.