Posts
Comments
Not everything suboptimal, but suboptimal in a way that causes suffering on an astronomical scale (e.g. galactic dystopia, or dystopia that lasts for thousands of years, or dystopia with an extreme number of moral patients (e.g. uploads)).
I'm not sure what you mean by Ord, but I think it's reasonable to have a significant probability of S-risk from a Christiano-like failure.
I think you miss one important existential risk separate from extinction, which is having a lastingly suboptimal society. Like, systematic institutional inefficiency, and being unable to change anything because of disempowerment.
In that scenario, maybe humanity is still around because one of the things we can measure and optimize for is making sure a minimum amount of humans are alive, but the living conditions are undesirable.
I'm not sure either, but here's my current model:
Even though it looks pretty likely that AISC is an improvement on no-AISC, there are very few potential funders:
1) EA-adjacent caritative organizations.
2) People from AIS/rat communities.
Now, how to explain their decisions?
For the former, my guess would be a mix of not having heard of/received an application from AISC and preferring to optimize heavily towards top-rated charities. AISC's work is hard to quantify, as you can tell from the most upvoted comments, and that's a problem when you're looking for projects to invest because you need to avoid being criticized for that kind of choice if it turns out AISC is crackpotist/a waste of funds. The Copenhagen interpretation of ethics applies hard there for an opponent with a tooth against the organization.
For the latter, it depends a lot on individual people, but here are the possibilities that come to mind:
- Not wanting donate anything but feeling like having to, which leads to large donations to few projects when you feel like donating enough to break the status quo bias.
- Being especially mindful of one's finances and donating only to preferred charities, because of a personal attachment (again, not likely to pick AISC a priori) or because they're provably effective.
To answer 2), you can say why you don't donate to AISC? Your motivations are probably very similar to other potential donators here.
Follow this link to find it. The translation is made by me, and open to comments. Don't hesitate to suggest improvements.
It's not obvious at all to me, but it's certainly a plausible theory worth testing!
To whom it may concern, here's a translation of "Bold Orion" in French.
A lot of the argumentation in this post is plausible, but also, like, not very compelling?
Mostly the "frictionless" model of sexual/gender norms, and the examples associated: I can see why these situations are plausible (if at least because they're very present in my local culture) but I wouldn't be surprised if they are a bunch of social myth either, in which case the whole post is invalidated.
I appreciate the effort though; it's food for thought even if it doesn't tell me much about how to update based on the conclusion.
Epistemic status: Had a couple conversations on AI Plans with the founder, participated in the previous critique-a-thon. I've helped AI Plans a bit before, so I'm probably biased towards optimism.
Neglectedness: Very neglected. AI Plans wants to become a database of alignment plans which would allow quick evaluation of whether an approach is worth spending effort on, at least as a quick sanity check for outsiders. I can't believe it didn't exist before! Still very rough and unuseable for that purpose for now, but that's what the critique-a-thon is for: hopefully, as critiques accumulate and more votes are fed into the system, it will become more useful.
Tractability: High. It may be hard to make winning critiques, but considering the current state of AI Plans, it's very easy to make an improvement. If anything, you can filter out the obvious failures.
Impact: I'm not as confident here. If AI Plans works as intended, it could be very valuable to allocate funds more efficiently and save time by figuring out which approaches should be discarded. However, it's possible that it will just fail to gain steam and become a stillborn project. I've followed it for a couple months, and I've been positively surprised several times, so I'm pretty optimistic.
The bar to entry is pretty low; if you've been following AIS blogs or forums for several months, you probably have something to contribute. It's very unlikely you'll have a negative impact.
It may also be an opportunity for you to discuss with AIS-minded people and check your opinions on a practical problem; if you feel like an armchair safetyist and tired to be one, this is the occasion to level up.
Another way to think about it is that the engagement was very low in previous critique-a-thon so if you have a few hours to spare, you can make some easy money and fuzzies even if you're not sure about the value in utilons.
Thank you, this is incredibly interesting! Did you ever write up more on the subject? I'm excited to see how it relates to mesa-optimisation in particular.
In the finite case, where , then
Typo: I think you mean ?
I'm surprised to hear they're posting updates about CoEm.
At a conference held by Connor Leahy, I said that I thought it was very unlikely to work, and asked why they were interested in this research area, and he answered that they were not seriously invested in it.
We didn't develop the topic and it was several months ago, so it's possible that 1- I misremember or 2- they changed their minds 3- I appeared adversarial and he didn't feel like debating CoEm. (For example, maybe he actually said that CoEm didn't look promising and this changed recently?)
Still, anecdotal evidence is better than nothing, and I look forward to seeing OliviaJ compile a document to shed some light on it.
Nice! Is this on ai-plans already?
I invite you. You can send me this summary in private to avoid downvotes.
There's a whole part of the argument which is missing which is the framing of this as being about AI risk.
I've seen various propositions for why this happened, and the board being worried about AI risk is one of them but not the most plausible afaict.
In addition this is phrased similarly to technical problems like the corrigibility, which it is very much not about.
People who say "why can't you just turn it off" typically refer to literally turning off the AI if it appears to be dangerous, which this is not about. This is about turning off the AI company, not the AI.
1- I didn't know Executive Order could be repealed easily. Could you please develop?
2- Why is it good news? To me, this looks like a clear improvement on the previous status of regulations.
AlexNet dates back to 2012, I don't think previous work on AI can be compared to modern statistical AI.
Paul Christiano's foundational paper on RLHF dates back to 2017.
Arguably, all of agent foundations work turned out to be useless so far, so prosaic alignment work may be what Roko is taking as the beginning of AIS as a field.
The AI safety leaders currently see slow takeoff as humans gaining capabilities, and this is true; and also already happening, depending on your definition. But they are missing the mathematically provable fact that information processing capabilities of AI are heavily stacked towards a novel paradigm of powerful psychology research, which by default is dramatically widening the attack surface of the human mind.
I assume you do not have a mathematical proof of that, or you'd have mentioned it. What makes you think it is mathematically provable?
I would be very interested in reading more about the avenues of research dedicated to showing how AI can be used for psychological attacks from the perspective of AIS (I'd expect such research to be private by default due to infohazards).
I don't understand how the parts fit together. For example, what's the point of presenting the (t-,n)-AGI framework or the Four Background Claims?
I assume it's incomplete. It doesn't present the other 3 anchors mentioned, nor forecasting studies.
To avoid being negatively influenced by perverse incentives to make societally risky plays, couldn't TurnTrout just leave the handling of his finances to someone else and be unaware of whether or not he has Google stock?
Doesn't matter if he does, as long as he doesn't think he does; and if he's uncertain about it, I think psychologically it'll already greatly reduce caring about Google stock.
Not before reading the link, but Elizabeth did state that they expected the pro-meat section to be terrible without reading it, presumably because of the first part.
Since the article is low-quality in the part they read and expected low-quality in the part they didn't, they shouldn't take it as evidence of anything at all; that is why I think it's probably confirmation bias to take it as evidence against excess meat being related to health issues.
Reason for retraction: In hindsight, I think my tone was unjustifiably harsh and incendiary. Also the karma tells that whatever I wrote probably wasn't that interesting.
That’s the first five subsections. The next set maybe look better sourced, but I can’t imagine them being good enough to redeem the paper. I am less convinced of the link between excess meat and health issues than I was before I read it, because surely if the claim was easy to prove the paper would have better supporting evidence, or the EA Forum commenter would have picked a better source.
That's confirmation bias if I've ever seen it.
It seems likely to me that you're exposed to a lot of low-quality anti-meat content, and you should correct for selection bias since you're likely to only read what will support your views that the arguments are bad, and recommendation algorithms often select for infuriatingness.
[Note: I didn’t bother reading the pro-meat section. It may also be terrible, but this does not affect my position.]
??? Surely if meat-being-good was easy to prove, the paper would have better supporting (expected) evidence.
You should probably take a step back and disengage from that topic to restore your epistemics about how you engage with (expected low-quality) pro-vegan content.
A model is deceptively aligned with its designers. However, the designers have very good control mechanisms in place such that they would certainly catch the AI if it tried to act misaligned. Therefore, the model acts aligned with the designers' intentions 100% of the time. In this world, a model that is technically deceptively aligned may still be safe in practice (although this equilibrium could be fragile and unsafe in the long run).
In that case, there is no strategic deception (the designers are not induced in error by the AI).
I think we consider this case strategic deception, because we have an intuition of what inputs the AI receives (one where it would be controlled little enough to be deceptive) that differs from the actual one.
To fix this, I propose strategic deception not be defined according to the AI's behavior, but to its hypothetical behavior on an idealized class of inputs that represents all situations where the designers want the AI to behave in a certain way.
E.g. The designers are holding the AI in a simulation to check if it's deceptive. This information is given to the AI in a pre-prompt due to technical issues. However, the designers want the AI to avoid strategic deception even during deployment where it won't be told it's in a simulation, so their idealized test set includes prompts without this information.
By this definition they cannot check if the AI exhibits strategic deception before deployment in this situation.
Also, I am unsatisfied by "in order to accomplish some outcome" and "[the AI's] goals" because this assumes an agentic framework, which might not be relevant in real-world AI.
How to fix the first, for agentic AI only: "for which the AI predicts an outcome that can be human-interpreted as furthering its goals"
Not sure how to talk about deceptive non-agentic AI.
At a glance, I couldn't find any significant capability externality, but I think that all interpretability work should, as a standard, have a paragraph explaining why the authors won't think their work will be used to improve AI systems in an unsafe manner.
Seeing as the above response wasn't very upvoted, I'll try to explain in simpler terms.
If 2+2 comes out 5 the one-thrillionth-and-first time we compute it, then our calculation does not match numbers.
... which we can tell because?
...and writing this now I realize why the answer was more upvoted, because this is circular reasoning. ':-s
Sorry, I have no clue.
In France, EffiSciences is looking for new members and interns.
Sounds like those people are victim of a salt-in-pasta-water fallacy.
It's also very old-fashioned. Can't say I've ever heard anyone below 60 say "pétard" unironically.
You might also assign different values to red-choosers and blue-choosers (one commenter I saw said they wouldn't want to live in a world populated only by people who picked red) but I'm going to ignore that complication for now.
Roko has also mentioned they think people choose blue for being bozos and I think it's fair to assume from their comments that they care less about bozos than smart people.
I'm very interested in seeing the calculations where you assign different utilities to people depending on their choice (and possibly, also depending on yours, like if you only value people who choose like you).
I mean, as an author you can hack through them like butter; it is highly unlikely that out of all the characters you can write, the only ones that are interesting will all generate interesting content iff (they predict) you'll give them value (and this prediction is accurate).
I strongly suspect the actual reason you'll spend half of your post's value on buying ads for Olivia (if in fact you do that, which is doubtful as well) is not that (begin proposition) she would only accept this trade if you did that because
- she can predict your actions (as in, you wrote her as being unable to act in another manner than being able to predict your actions)
- she predicts you'll do that (in exchange for providing you with a fun story)
(end proposition).
I suspect that your actual reason is more like staying true to your promise, making a point, having fun and other such things.
I can imagine acausally trading with humans gone beyond the cosmological horizon, because our shared heritage would make a lot of the critical flaws in the post go away.
This is mostly wishful thinking.
You're throwing away your advantages as an author to bargain with fictionally smart entities. You can totally void the deal with Olivia and she can do nothing about it because she's as dumb as you write her to be.
Likewise, the author writing about space warring aliens writing about giant cube-having humans could just consider the aliens that have space wars without consideration for humans at all; you haven't given enough detail for the aliens' modelization of the humans be precise enough that their behavior must depend on it.
Basically, you're creating characters that are to you as you are to a superintelligence, but dumb yourself down to their level for the fun of trading acausally. This is not acausal trading, because you are not actually on their level and their decisions do not in fact depend on reliably predicting you'll cooperate in the trade.
This is just fiction.
For instance, a money-maximising trade-bot AI could be perfectly safe if it notices that money, in its initial setting, is just a proxy for humans being able to satisfy their preferences.
There is a critical step missing here, which is when the trade-bot makes a "choice" between maximising money or satisfying preferences.
At this point, I see two possibilities:
- Modelling the trade-bot as an agent does not break down: the trade-bot has an objective which it tries to optimize, plausibly maximising money (since that is what it was trained for) and probably not satisfying human preferences (unless it had some reason to have that has an objective).
A comforting possibility is that it is corrigibly aligned, that it optimizes for a pointer to its best understanding of its developers. Do you think this is likely? If so, why? - An agentic description of the trade-bot is inadequate. The trade-bot is an adaptation-executer, it follows shards of value, or something. What kind of computation is it making that steers it towards satisfying human preferences?
So I'd be focusing on "do the goals stay safe as the AI gains situational awareness?", rather than "are the goals safe before the AI gains situational awareness?"
This is a false dichotomy. Assuming that when the AI gains situational awareness, it will optimize for its developers' goals, alignment is already solved. Making the goals safe before situational awareness is not that hard: at that point, the AI is not capable enough for X-risk.
(A discussion of X-risk brought about by situationally unaware AIs could be interesting, such as a Christiano failure story, but Soares's model is not about it, since it assumes autonomous ASI.)
A new paper, built upon the compendium of problems with RLHF, tries to make an exhaustive list of all the issues identified so far: Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
That sounds nice but is it true? Like, that's not an argument, and it's not obvious! I'm flabbergasted it received so many upvotes.
Can someone please explain?
Well, I wasn't interested because AIs were better than humans at go, I was interested because it was evidence of a trend of AIs being better at humans at some tasks, for its future implications on AI capabilities.
So from this perspective, I guess this article would be a reminder that adversarial training is an unsolved problem for safety, as Gwern said above. Still doesn't feel like all there is to it though.
To clarify: what I am confused about is the high AF score, which probably means that there is something exciting I'm not getting from this paper.
Or maybe it's not a missing insight, but I don't understand why this kind of work is interesting/important?
I'm confused. Does this show anything besides adversarial attacks working against AlphaZero-like AIs?
Is it a surprising result? Is that kind of work important for reproducibility purposes regardless of surprisingness?
You're making many unwarranted assumptions about an AI's specific mind, along with a lot of confusion about semantics which seems to indicate you should just read the Sequences. It'll be very hard to point out where you are going wrong because there's just too much confusion.
As example, here's a detailed analysis of the first few paragraphs:
Intelligence will always seek more data in order to better model the future and make better decisions.
Unclear if you mean intelligence in general, and if so, what you mean by the word. Since the post is about AI, let's talk about that. AI does not necessarily seek more data. Typically, most modern AIs are trained on a training dataset provided by developers, and do not actively seek more data.
There is also not necessarily an "in order to". Not all AIs are agentic.
Not all AIs model the future at all. Very few agentic AIS have as a terminal goal to make better decisions - though it is expected that advanced AI by default will do that as an instrumental behavior, and possibly as instrumental or terminal goal because of the convergent instrumental goals thesis.
Conscious intelligence needs an identity to interact with other identities, identity needs ego to know who and what it is. Ego would often rather be wrong than admit to being wrong.
You use connotated, ill-defined words to go from consciousness to identity to ego to refusing to admit to being wrong. Definitions have no causal impact on the world (in first order considerations, a discussion of self-fulfilling terminology is beyond this comment). That's not to say you have to use well-defined words, but you should be able to taboo your words properly before you use technical words with controversial/exotic-but-specifically-defined-in-this-community meaning. And really, I would recommend you just read more on the subject of consciousness; theory of mind is a keyword that will get you far on LW.
Non conscious intelligence can build a model of consciousness from all the data it has been trained on because it all originated from conscious humans. AI could model a billion consciousness's a million years into the future, it will know more about it than we ever will. But AI will not chose to become conscious.
Non-sequitur, wrong reasons to have approximately correct beliefs... Just, please read more about AI before having an opinion.
Later, you show examples of false dichotomy, privileging the hypothesis, reference class error... it's not better quality than the paragraphs I commented in detail.
So in conclusion, where are you going wrong? Pretty much everywhere. I don't think your comment is salvageable, I'd recommend just discarding that train of thought altogether and keeping your mind open while you digest more literature.
We cannot select all companies currently looking to hire AI researchers. There's just too many of them, and most will just want to integrate ChatGPT into their software or something.
We're interested in companies making the kind of capabilities research that might lead to AI that poses an existential risk.
Do you suggest that we should consider all companies that employ a certain number of AI experts?
Are you an AI?
Thanks! I wish the math hadn't broken down, it makes the post harder to read...
List of known discrepancies:
- Deepmind categorizes Cohen’s scenario as specification gaming (instead of crystallized proxies).
- They consider Carlsmith to be about outer misalignment?
Value lock-in, persuasive AI and Clippy are on my TODO list to be added shortly. Please do tell if you have something else in mind you'd like to see in my cheat sheet!
I'm not sure why this was downvoted into oblivion, so I figured I'd give my own opinion at least:
I assume the author is an amateur writer, and wrote this for fun without much consideration for the audience of the actual subject. It's the kind of things I could have done when I entered the community.
About the content, the story is awful:
- The characters aren't credible. The AI does not match any sensible scenario, and especially not the kind of AI typically imagined for a boxing experiment. The arguments of the AI are unconvincing, as are its abilities and personality. The protagonist is weak; he doesn't feel much except emotions ex machina that the author inserts for the purpose of the plot. He's also extremely dumb, which breaks suspension of disbelief in this kind of community.
Most damningly, I didn't feel anything for the characters at any point. I didn't feel close to Nolan or Elysium, I didn't root for either nor was I emotionally engaged in the resolution.
- The progression of the story is decent, if extremely stereotypical. However, there is absolutely no foreshadowing so every plot twist appears out of the blue. Nolan is offered a million dollars? Why does he even believe this when he has been repeating it was all very suspect? It's the same for every argument by Elysium.
- The worst part is that all the arguments somewhat related to AI boxing are very poor and would give incorrect ideas to an outsider reading this story as a cheap proxy for understanding the literature. It's like this was written as a parody, but there's no humor.
On matters of style, I think it was okayish:
- The grammar is perfect afaics. The tone is pandering and repetitive, but this would be less of a problem if the content was engaging.
- I wonder if it was written by an AI? Constantly making sure each plot point is explicitly pointed out to the reader with absolutely no room for interpretation reminds me a lot of this feeling of epuration you get when using GPT+RLHF. The introduction is suspect, too. If it was written by an AI, that should be mentioned to the audience.
Kudos for going through the work of actually writing this, as a prospective writer I know how hard it is to write and publish, even if the result is bad.
Good luck if you want to write again, I hope this review can help. My best advice is to return to the fundamentals and follow some writing tutorials to train a bit.
Sounds like you simply assumed that saying you could disgust the gatekeeper would make them believe they would be disgusted.
But the kind of reaction to disgust that could make a gatekeeper let the AI out needs to be instantiated to have impact.
Most people won't get sad just imagining that something sad could happen. (Also, duh, calling out the bluff.)
In practice, if you had spent the time to find disgusting content and share, it would have been somewhat equivalent to torturing the gatekeeper, which in the extreme case might work on a significant fraction of the population, but it's also kind of obvious that we could prevent that.
Heh, it's better to have this than nothing at all! I'll keep hoping for it. ^-^
It sounds like an excellent foundation.
Ideas for improvement:
- If you have multiple explanations for a concept, then on a fraction epsilon of users, randomize the order in which they're displayed to collect statistical data about which explanation they first found enlightening, then for most users, display the explanations in their order of expected chance of being grokked.
- Don't restrict yourself to explanations. Propose different teaching methods.
- Include expected reading time and score for each explanation (like LW).
Criticism:
- How do you distinguish feeling of epiphany and grokking?
Mostly confused about how the chronophone works. However I try to imagine strict rules, the thought experiment is not that interesting.
They fired quanta of energy.
I also would be interested in learning more, even just a link would be nice.