Posts
Comments
Twitter is almost always blocked on my devices, as well.
What is the formal definition of 'power seeking'?
The freshly updated paper answers this question in great detail; see section 6 and also appendix B.
Why do people have such low credences for "The effect of First contact is mostly harmful (e.g., selfish ETI, hazards)"? Most alien minds probably don't care about us? But perhaps caring about variety is evolutionarily convergent? If not, why wouldn't our "first contact" be extremely negative (given their tech advantage)?
Over the last 2.5 years, I've read a lot of math textbooks. Not using Anki / spaced repetition systems over that time has been an enormous mistake. My factual recall seems worse-than-average among my peers, but when supplemented with Anki, it's far better than average (hence, I was able to learn 2000+ Japanese characters in 90 days, in college).
I considered using Anki for math in early 2018, but I dismissed it quickly because I hadn't had good experience using that application for things which weren't languages. I should have at least tried to see if I could repurpose my previous success! I'm now happily using Anki to learn measure theory and ring theory, and I can already tell that it's sticking far better.
This mistake has had real consequences. I've gotten far better at proofs and I'm quite good at real analysis (I passed a self-administered graduate qualifying exam in the spring), but I have to look things some up for probability theory. Not a good look in interviews. I might have to spend weeks of extra time reviewing things I could have already stashed away in an Anki deck.
Oops!
What a beautiful, bold, and chaotic story. Thanks for writing, Stuart.
I also did give arguments above, but people mostly made jokes about my punctuation! #grumpy
This is a timeless part of the LessWrong experience, my friend.
Argument screens off authority, and I'm interested in hearing arguments. While that information should be incorporated into your prior, I don't see why it's worth mentioning as a counterargument. (To be sure, I'm not claiming that jacobjacob isn't a good predictor in general.)
I remarked to my brother, Josh, that when most people find themselves hopefully saying "here's how X can still happen!", it's a lost cause and they should stop grasping for straws and move on with their lives. Josh grinned, pulled out his cryonics necklace, and said "here's how I can still not die!"
Do I intend to do something with people's predictions? Not presently, but I think people giving predictions is good both for the reader (to ingrain the concepts by thinking things through enough to provide a credence / agreement score) and for the community (to see where people stand wrt these ideas).
The catastrophic convergence conjecture was originally formulated in terms of "outer alignment catastrophes tending to come from power-seeking behavior." I think that this was a mistake: I meant to talk about impact alignment catastrophes tending to be caused by power-seeking. I've updated the post accordingly.
jacobjacob once again seems too pessimistic, posterior is very heavily that when habryka makes a 60% yes prediction for a decision he has (partial) control over about a functionality which the community has glommed onto thus far, the community is also justified in expressing ~60% belief that the feature ships. :)
Also, we aren't selecting from "most possible features"!
I actually waited for this feature to post!
For me, it's because there's disjunctively many ways that AGI could not happen (global totalitarian regime, AI winter, 55% CFR avian flu escapes a BSL4 lab, unexpected difficulty building AGI & the planning fallacy on timelines which we totally won't fall victim to this time...), or that alignment could be solved, or that I could be mistaken about AGI risk being a big deal, or...
Granted, I assign small probabilities to several of these events. But my credence for P(AGI extinction | no more AI alignment work from community) is 70% - much higher than my 40% unconditional credence. I guess that means yes, I think AGI risk is huge (remember that I'm saying "40% chance we just die to AGI, unconditionally"), and that's after incorporating the significant contributions which I expect the current community to make. The current community is far from sufficient, but it's also probably picking a good amount of low-hanging fruit, and so I expect that its presence makes a significant difference.
EDIT: I'm decreasing the 70% to 60% to better match my 40% unconditional, because only the current alignment community stops working on alignment.
”Catastrophic” is normally used in the term ”global catastrophic risk” and means something like “kills 100,000s of people”, so I do think “doesn’t necessarily kill but could’ve killed a couple of people” is a fairly different meaning.
Agreed. In retrospect, I might have opted for "pre-AGI nearly-deadly accident caused by deceptive alignment."
In retrospect I realize that I put my answer to the second question far too high — if it just means “a deceptive aligned system nearly gives a few people in hospital a fatal dosage but it’s stopped and we don’t know why the system messed up” then it’s quite plausible nothing this substantial will happen as a result of that.
I intended the situation to be more like "we catch the AI pretending to be aligned, but actually lying, and it almost or does kill at least a few people as a result of that."
With #1, I'm trying to have people predict the "deception is robustly instrumental behavior, but AIs will be bad at it at first and so we'll catch them." #2 is trying to operationalize whether this would be viewed as a fire alarm.
Some ways you might think scenario #1 won't happen:
- You don't think deception will be incentivized
- Fast takeoff means the AI is never smart enough to deceive but dumb enough to get caught
- Our transparency tools won't be good enough for many people to believe it was actually deceptively aligned
In the following, an event is "catastrophic" if it endangers several human lives; it need not be an existential catastrophe.
Edit: I meant to say "deceptive alignment", but the meaning should be clear either way.
Smiles knowingly from behind steepled hands
You've just handed me so much power to prove people right or wrong on the internet...
This looks so good! Great work by both Ought and the LW mods.
Perhaps this can also gauge approval (from 0% (I hate it) to 100% (I love it)), or even function as a poll (first decile: research agenda 1 seems most promising; second decile: research area 2...)... or Ought we not do things like that?
Another thing which seems to "gain something" every time it hops up a level of meta: Corrigibility as Outside View. Not sure what the fixed points are like, if there are any, and I don't view what I wrote as attempting to meet these desiderata. But I think there's something interesting that's gained each time you go meta.
Sure, but why would those changes tend to favor Trump as you get outside of a small neighborhood? Like, why would Biden / (Biden or Trump win) < .5? I agree it would at least approach .5 as the neighborhood grows. I think.
Why do you think that? How do you know that?
I read someone saying that ~half of the universes in a neighborhood of ours went to Trump. But... this doesn't seem right. Assuming Biden wins in the world we live in, consider the possible perturbations to the mental states of each voter. (Big assumption! We aren't thinking about all possible modifications to the world state. Whatever that means.)
Assume all 2020 voters would be equally affected by a perturbation (which you can just think of as a decision-flip for simplicity, perhaps). Since we're talking about a neighborhood ("worlds pretty close to ours"), each world-modification is limited to N decision flips (where N isn't too big).
- There are combinatorially more ways for a race to be close (in popular vote) than for it to not be close. But we're talking perturbations, and so since we're assuming Biden wins in this timeline, he's still winning in most other timelines close to ours
- I don't know whether the electoral college really changes this logic. If we only consider a single state (PA), then it probably doesn't?
- I'm also going to imagine that most decision-flips didn't have too many downstream effects, but this depends on when the intervention takes place: if it's a week beforehand, maybe people announce changes-of-heart to their families? A lot to think about there. I'll just pretend like they're isolated because I don't feel like thinking about it that long, and it's insanely hard to play out all those effects.
- Since these decision-flips are independent, you don't get any logical correlations: the fact that I randomly changed my vote, doesn't change how I expect people like me to vote. This is big.
Under my extremely simplified model, the last bullet is what makes me feel like most universes in our neighborhood were probably also Biden victories.
Forgive me if I missed it in the post, but might it be cheaper to use some kind of visor if you're only illuminating for personal benefit? Much easier and cheaper to fill your FOV with light that way.
I did this with my family! We had a lot of fun.
(Misread prompt as "you have 10 years of food & water")
1. Kick down the door
2. Unlock the door
3. Use high-pressure water source to explode the room
4. Contact someone and get them to open the door
5. Learn how to pick locks and pick the locks
6. Eat through the door
7. Bust through the thin drywall
8. End up in a low-probability universe where you teleport outside the room
9. Wait for someone to come open the door
10. Upload yourself and have yourself reconstructed on the outside
11. Build an AGI to solve the problem
12. Hire someone to come blow up the door
13. Go on an extreme diet and slide under the door
14. Train hard at karate so you can break off the hinges with one blow
15. Use a rocket launcher
16. Climb out through ventilation shaft
17. Synthesize corrosive compound that eats through the wall
18. Tunnel underground with your hands
19. Take screw from glasses, burrow out hole in door so you can reach hand through and unlock it
20. Take your shoe and hit the bolts out
21. Use hemp in food to grow a rope and then slip under the door and wait for someone to trip and then let them know you're there
22. Scream at resonant frequency of the wall so it explodes
23. Trigger sprinkler system to soften drywall
24. Burn down whole room and hope your remains are enterred outside of the room
25. Wear your way through the wall
26. Eat so much food that you get bigger than the room and then it explodes
27. Download blueprints for the room and then exploit weak points
28. Use avocado pits to cut out cinderblock
29. Find out how to make explosives from food (methane gas?)
30. Use acidity of something to wear away metal in room
31. Train super hard to run fast enough opposite to Earth's spin to stop its spin and blow the room away via inertia (this wouldn't work)
32. Build a 3D printer that takes food as input and produces keys to the door
33. Jump hard enough to bust head through ceiling
34. Use light + unlimited energy to wood burn out of the room
35. Wave your hands so fast the air turns to plasma
36. Call the police and tell them you have weed in the room
37. Let food rot; bug eggs hatch; bugs chew through the door
38. Hire a hitman to kidnap you
39. Climb out through the window
40. Rip off two of your toes and rub them together to make a fire
41. Wait for tornado
42. Wait for earthquake
43. Wait until sun expands and then "your" molecules likely end up outside of the former room's molecules
44. Spin arms at right angle to get lift to turn yourself into a human helicopter and fly out of the skylight
45. Use tine on belt to pick lock
46. Get the building condemned (via your phone) and torn down
47. Sand away log walls with sandblock
48. Drill through the wall by spinning your hand quickly
49. Scratch out with really long fingernails
50. Yell for help
If you're tempted to write "clearly" in a mathematical proof, the word quite likely glosses over a key detail you're confused about. Use that temptation as a clue for where to dig in deeper.
At least, that's how it is for me.
The answer to this seems obvious in isolation: shaping helps with credit assignment, rescaling doesn't (and might complicate certain methods in the advantage vs Q-value way). But I feel like maybe there's an important interaction here that could inform a mathematical theory of how a reward signal guides learners through model space?
I really like this post. Thanks for writing it!
(Why didn't you mention the Slaughterbot Slip-up of '24, though?)
Ideally within the next month!
(I have a big google doc analyzing corrigibility & manipulation from the attainable utility landscape frame; I’ll link it here when the post goes up on LW)
Epistemic status: not an expert
Understanding Newton's third law, .
Consider the vector-valued velocity as a function of time, . Scale this by the object's mass and you get the momentum function over time. Imagine this momentum function wiggling around over time, the vector from the origin rotating and growing and shrinking.
The third law says that force is the derivative of this rescaled vector function - if an object is more massive, then the same displacement of this rescaled arrow is a proportionally smaller velocity modification, because of the rescaling!
And also, forces have opposite reactions (by conservation of momentum) and equal reactions (by conservation of energy).
I'm going to assume the evil forces knew I had it, but can't read my mind or know everything I did to hide it. They'll spend a reasonable amount of resources looking, but not an infinite amount, and they won't torture me to find out where it is (they know how unreliable torture is as information gain!).
- Put the pen in a small steel box, go on a multi-day hike, and randomly bury the box somewhere along the hike. Remember where you put it.
- Travel to a new country and keep it in a bank deposit box under a fake name. (Did they have those in 1855?)
- Acquire a bunch of pens that look exactly the same, put a different color marking on the interior of each, and give each to a friend. (Figure out some way to make sure the Evil People don't just steal all of the pens?)
- Put the pen inside the wall of a friend's house.
- Destroy the pen and then remake it in 50 years.
- Embed it in a tree core. (Would the tree grow to crush it?)
- Commission a copy of the pen and give it to them, claiming it's the real thing.
- Have 10 friends over and give them each a pen, only one of which is the Destined Pen. Then, each of them meets 10 people and gives one of them the pen, and so on once again. The final person who receives the actual pen is given the appropriate instructions and payment plan for delivering the pen to Einstein in 50 years (DDOS their resources - they probably can't investigate 1000 people).
- Find a box and put the pen in it.
- Disassemble the pen and entrust the pieces to a pensmith (is that a thing?)
- Preserve the pen in some kind of resin
- Form a concrete block around the pen, or otherwise embed it in some kind of wall, and then take it out when needed
- Go spelunking and leave the pen wedged in one of the caves in a memorable but rarely frequented location (I notice I'm focusing on the "hide from dedicated group of people" constraint, which is slowing me down. I'm going to focus on just "places you could put a pen" without worrying at all about hiding for now.)
- On the ground.
- In the attic.
- Bird's nest
- Buried in dirt
- Library
- Carriage
- Warehouse (Ok, back to solutions that have at least a small chance of working)
- Pay someone to hide the pen, split up the location info into a cryptographically secure 3-out-of-5 secret-sharing scheme, and then dole out the info to me and four close friends who live far away and move frequently. In 1855, it would be logistically hard to coordinate to track 3 of us down at the same time. They pass on the secret to their children and spouses, in case any of us die before 50 years pass.
- Unethical: kill the evil forces?
- Unethical: take over the local government and have them protect the pen.
- Talk the bad guys out of wanting the pen.
- Set up an elaborate dungeon below my house (Legend of Zelda-style), at the end of which is a fake copy of the pen. Keep the real pen in my desk drawer.
- Similarly, go to a lot of trouble to look like I'm building up a vault for the pen, all the while keeping "the rest" of my pens on a pedestal in the middle of my house. Of course, the real pen is there.
- Persuade the bad guys that pens are really annoying and you should use a mechanical pencil (that totally existed in 1855, right?), if you aren't going to use a digital writing device (which they aren't going to use).
- Give a fake pen to a major world power, and then tell the bad guys it's already out of my hands, and eat popcorn while the two groups fight.
- Find Einstein's ancestors and convince them to pass down the pen.
- Using extreme computational resources and technological prowess, build a capsule that's shot into LEO which is timed to reenter Earth's atmosphere in front of Einstein right when he's looking for a new pen.
- Deduce exactly what kind of simulation we're on, then execute a bizarre policy that mind-hacks whoever's observing us into spawning 1x Pen of Destiny for Einstein at exactly the right time.
- Compute a Butterfly Effect policy for a molecularly identical pen spontaneously assembling itself in front of Einstein at the appropriate time (this is totally a thing, right?)
- Wedge the sanitized pen between my radius and ulna. That totally wouldn't go wrong, and it'd be too gross for them to want to retrieve.
- Buy a farm and hide it in a hay bale.
- Go to a stream and instruct a friend to wait some unknown # of miles downstream. Put the pen in a bottle, float it downstream, and the friend retrieves it and waits downstream. Maybe a festival could be going on near the stream at the same time. In any case, even if I'm being trailed at that moment, they wouldn't know how far downstream it went, which would give my friend an important head start for hiding it in a different town.
- Pay someone to take it to a tribe which will be convinced that it should be worshipped as a sacred object, and then steal it back in 50 years.
- Memorize a binary sequence using a memory palace, which I use as an XOR cipher on a series of coin flips which indicate: "heads: go north 100 feet; tails: go east 100 feet". Flip 100 coins and write down the result, and then bury the coin in the place indicated by the flips XOR the sequence. (This is basically a one-time pad for north-eastern lattice paths)
- Do the above, but just through the pseudo-randomly generated memorized sequence. Also, have a habit of taking these north-eastern lattice path walks randomly for a few days before and after actually burying the pen, so they don't know the walk on which I buried the pen.
- Enter into a 2-out-of-2 secret sharing scheme with someone difficult to intimidate, like a major world leader.
- Do 38, but using a fall tree branch as my (not technically) pseudorandom source: given a branch, choose the one that has the reddest leaf on it, using an appropriate embedding from branch choices into directional bearings for the next step.
- Put the pen inside a football that won't be used. Who would do that?
- Preserve the pen in permafrost somewhere stable (is that a thing? would it be crushed by changing pressures?)
- Encase the pen in diamond or some even harder material, which Einstein will need to invent molecular nanotech to undo. He wants his miracle papers, let him do the impossible!
- If I'm making myself as capable as in #43, why not just build an AGI to ensure Einstein gets it? Might as well.
- Toss the pen into a furnace in front of the bad guys; unbeknownst to them, the furnace has a secret compartment in which the pen will be safe from the heat.
- Keep the pen inside a wooden compartment, which I nail to the underside of the nearest bridge (so that if the pen falls, it falls over land).
- Fake my own death and inter the pen with the fake body.
- 47, but hide the pen in the gravestone.
- 48, but a friend surreptitiously removes the pen from the gravestone compartment during the funeral.
- Just give the bad guys the pen and take it back before Einstein needs it.
OK, but there's a difference between "here's a definition of manipulation that's so waterproof you couldn't break it if you optimized against it with arbitrarily large optimization power" and "here's my current best way of thinking about manipulation." I was presenting the latter, because it helps me be less confused than if I just stuck to my previous gut-level, intuitive understanding of manipulation.
Edit: Put otherwise, I was replying more to your point (1) than your point (2) in the original comment. Sorry for the ambiguity!
In your original comment, you wrote:
one possible implication being he will also not accept the results
The first article in that search says:
Before leaving the state, Biden told reporters his comments were ‘taken a little out of context’ and added that ‘I’m going to accept the outcome of this election, period.’
Not Stuart, but I agree there's overlap here. Personally, I think about manipulation as when an agent's policy robustly steers the human into taking a certain kind of action, in a way that's robust to the human's counterfactual preferences. Like if I'm choosing which pair of shoes to buy, and I ask the AI for help, and no matter what preferences I had for shoes to begin with, I end up buying blue shoes, then I'm probably being manipulated. A non-manipulative AI would act in a way that increases my knowledge and lets me condition my actions on my preferences.
Sources?
I'm not very satisfied by answers like "X won't support him, because that's illegal" and "it's unconstitutional for the federal government to do X, so they won't." I think these usually are correct, but over the last four years we have seen rapid deterioration of our ability to: agree on an objective reality, have an executive branch which abides by the law absent immediate and tangible enforcement mechanisms (remember when congressional subpoenas were at least often answered? now they're ~always ignored AFAICT), have common knowledge that the law is the Law and if you break it you will be punished (obviously, rich&powerful would get more leeway in this calculation), etc.
I think many of these things have degraded and am no longer sure that anything would really stop red states from ignoring the popular results and sending their own set of electors. It's already being discussed, and red officials have admitted they are discussing it without immediately walking it back / distancing themselves from the prospect. Maybe the Supreme Court would be enough to stop that, if they so chose. (Would they? Aren't states technically allowed to choose electors however they please?)
I went to the doctor's yesterday. This was embarrassing for them on several fronts.
First, I had to come in to do an appointment which could be done over telemedicine, but apparently there are regulations against this.
Second, while they did temp checks and required masks (yay!), none of the nurses or doctors actually wore anything stronger than a surgical mask. I'm coming in here with a KN95 + goggles + face shield because why not take cheap precautions to reduce the risk, and my own doctor is just wearing a surgical? I bought 20 KN95s for, like, 15 bucks on Amazon.
Third, and worst of all, my own doctor spouted absolute nonsense. The mildest insinuation was that surgical facemasks only prevent transmission, but I seem to recall that many kinds of surgical masks halve your chances of infection as well.
Then, as I understood it, he first claimed that coronavirus and the flu have comparable case fatality rates. I wasn't sure if I'd heard him correctly - this was an expert talking about his area of expertise, so I felt like I had surely misunderstood him. I was taken aback. But, looking back, that's what he meant.
He went on to suggest that we can't expect COVID immunity to last (wrong) but also that we need to hit 70% herd immunity (wrong). How could you even believe both of these things at the same time? Under those beliefs, are we all just going to get sick forever? Maybe he didn't notice the contradiction because he made the claims a few minutes apart.
Next, he implied that it's not a huge deal that people have died because a lot of them had comorbidities. Except that's not how comorbidities and counterfactual impact works. "No one's making it out of here alive", he says. An amusing rationalization.
He also claimed that nursing homes have an average stay length of 5 months. Wrong. AARP says it's 1.5 years for men, 2.5 years for women, but I've seen other estimate elsewhere, all much higher than 5 months. Not sure what the point of this was - old people are 10 minutes from dying anyways? What?
Now, perhaps I misunderstood or misheard one or two points. But I'm pretty sure I didn't mishear all of them. Isn't it great that I can correct my doctor's epidemiological claims after reading Zvi's posts and half of an epidemiology textbook? I'm glad I can trust my doctor and his epistemology.
I read "feasible" as something like "rationalizable." I think it would have been much clearer if you had said "if no strategy profiles are Pareto over any others."
Wikipedia says if , it's a "non-expansive map". But yes, contraction maps have some Lipschitz constant that enforces the behavior you describe. However, notice we still have the math "intuitive contraction" here, so it has the reverse-direction correspondence. Intriguingly, we're missing part of "intuitive contraction = bring things closer together" for the case, as you point out, so we don't have the forward direction fulfilled.
You're thinking of a Kaldor-Hicks optimality frontier for {outcomes with maximal total payoff}, while the Pareto frontier is {maximal elements in the unanimous-agreement preference ordering over outcomes}.
Right, I understand how this correctly labels certain cases, but that doesn't seem to address my question?
So, we could consider a game completely adversarial if it has a structure like this: no strategy profiles are a Pareto improvement over any others. In other words, the feasible outcomes of the game equal the game's Pareto frontier. All possible outcomes involve trade-offs between players.
I must have missed some key word - by this definition, wouldn't common-payoff games be "completely adversarial", because the "feasible" outcomes equal the Pareto frontier under the usual assumptions?
(Vague spoilers about non-obvious puzzles in Fez)
I don't remember there being an abrupt change; it felt like there were secrets under the surface. I didn't have to read around to figure out that the game brims with coded messages, it felt like a natural part of exploring the world and understanding what happened to it. But perhaps it's in the eye of the beholder.
I think you missed out on most of the game. (Spoilers as to the nature of the game's deeper puzzles)
I remember enjoying Fez when I was in college. Past-me would probably put it at Tier 2.
What is "real"? I think about myself as a computation embedded in some other computation (i.e. a universe-history). I think "real" describes hypotheses about the environment where my computation lives. What should I think is real? That which an "ideal embedded reasoner" would assign high credence. However that works.
This sensibly suggests that Gimli-in-actual-Ea (LOTR) should believe he lives in Ea, and that Ea is real, even though it isn't our universe's Earth. Also, the notion accounts for indexical uncertainty by punting it to how embedded reasoning should work (a la radical probabilism), without being tautological. Also, it supports both the subjective nature of what one should call "real", and the notion of an actual out-there-somewhere shared reality (multiple computations can be embedded within the same universe-history).
Reasoning about learned policies via formal theorems on the power-seeking incentives of optimal policies
One way instrumental subgoals might arise in actual learned policies: we train a proto-AGI reinforcement learning agent with a curriculum including a variety of small subtasks. The current theorems show sufficient conditions for power-seeking tending to be optimal in fully-observable environments; many environments meet these sufficient conditions; optimal policies aren't hard to compute for the subtasks. One highly transferable heuristic would therefore be to gain power in new environments, and then figure out what to do for the specific goal at hand. This may or may not take the form of an explicit mesa-objective embedded in e.g. the policy network.
Later, the heuristic has the agent seek power for the "real world" environment.
(Optimal Farsighted Agents Tend to Seek Power is rather dated and will be updated soon.)
Once formed, a contract acts as custom, private law between the parties.
This is a cool way of understanding contracts.
I'm putting this on the shelf of facepalm-obvious-but-beautiful realizations like
- Medicine is the science of healing, not just a collection of random facts about what pills to take
- Math is about the inescapable and immutable consequences of basic rules, not just about playing with integrals and numbers
- Physics is, in large part, about discovering the transition rules of the universe
- Machine learning is about the beautiful ideal learn- yeah, no, machine learning is still just a mess