Posts
Comments
I would say:
A theory always takes the following form: "given [premises], I expect to observe [outcomes]". The only way to say that an experiment has falsified a theory is to correctly observe/set up [premises] but then not observe [outcomes].
If an experiment does not correctly set up [premises], then that experiment is invalid for falsifying or supporting the theory. The experiment gives no (or nearly no) Bayesian evidence either way.
In this case, [premises] are the assumptions we made in determining the theoretical pendulum period; things like "the string length doesn't change", "the pivot point doesn't move", "gravity is constant", "the pendulum does not undergo any collisions", etc. The fact that (e.g.) the pivot point moved during the experiment invalidates the premises, and therefore the experiment does not give any Bayesian evidence one way or another against our theory.
Then the students could say:
"But you didn't tell us that the pivot point couldn't move when we were doing the derivation! You could just be making up new "necessary premises" for your theory every time it gets falsified!"
In which case I'm not 100% sure what I'd say. Obviously we could have listed out more assumptions that we did, but where do you stop? "the universe will not explode during the experiment"...?
By "reliable" I mean it in the same way as we think of it for self-driving cars. A self-driving car that is great 99% of the time and fatally crashes 1% of the time isn't really "high skill and unreliable" - part of having "skill" in driving is being reliable.
In the same way, I'm not sure I would want to employ an AI software engineer that 99% of the time was great, but 1% of the time had totally weird inexplicable failure modes that you'd never see with a human. It would just be stressful to supervise, to limit its potential harmful impact to the company, etc. So it seems to me that AI's won't be given control of lots of things, and therefore won't be transformative, until that reliability threshold is met.
Two possibilities have most of the "no agi in 10 years" probability mass for me:
- The next gen of AI really starts to scare people, regulation takes off, and AI goes the way of nuclear reactors
- Transformer style AI goes the way of self driving cars and turns out to be really hard to get from 99% reliable to the necessary 99.9999% that you need for actual productive work
Well sure, but the interesting question is the minimum value of P at which you'd still push
I also agree with the statement. I'm guessing most people who haven't been sold on longtermism would too.
When people say things like "even a 1% chance of existential risk is unacceptable", they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).
Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim
Not sure I understand; if model runs generate value for the creator company, surely they'd also create value that lots of customers would be willing to pay for. If every model run generates value, and there's ability to scale, then why not maximize revenue by maximizing the number of people using the model? The creator company can just charge the customers, no? Sure, competitors can use it too, but does that really override losing an enormous market of customers?
I won't argue with the basic premise that at least on some metrics that could be labeled as evolution's "values", humans are currently doing very well.
But, the following are also true:
- Evolution has completely lost control. Whatever happens to human genes from this point forward is entirely dependent on the whims of individual humans.
- We are almost powerful enough to accidentally cause our total extinction in various ways, which would destroy all value from evolution's perspective
- There are actions that humans could take, and might take once we get powerful enough, that would seem fine to us but would destroy all value from evolution's perspective.
Examples of such actions in (3) could be:
- We learn to edit the genes of living humans to gain whatever traits we want. This is terrible from evolution's perspective, if evolution is concerned with maximizing the prevalence of existing human genes
- We learn to upload our consciousness onto some substrate that does not use genes. This is also terrible from a gene-maximizing perspective
None of those actions is guaranteed to happen. But if I were creating an AI, and I found that it was enough smarter than me that I no longer had any way to control it, and if I noticed that it was considering total-value-destroying actions as reasonable things to maybe do someday, then I would be extremely concerned.
If the claim is that evolution has "solved alignment", then I'd say you need to argue that the alignment solution is stable against arbitrary gains in capability. And I don't think that's the case here.
That's great. "The king can't fetch the coffee if he's dead"
Wow. When I use GPT-4, Ive had a distinct sense of "I bet this is what it would have felt like to use one of the earliest computers". Until this post I didnt realize how literal that sense might be.
This is a really cool and apt analogy - computers and LLM scaffolding really do seem like the same abstraction. Thinking this way seems illuminating as to where we might be heading.
I always assumed people were using "jailbreak" in the computer sense (e.g. jailbreak your phone/ps4/whatever), not in the "escape from prison" sense.
Jailbreak (computer science), a jargon expression for (the act of) overcoming limitations in a computer system or device that were deliberately placed there for security, administrative, or marketing reasons
I think the definition above is a perfect fit for what people are doing with ChatGPT
I am going to go ahead and say that if males die five times as often from suicide, that seems more important than the number of attempts. It is kind of stunning, or at least it should be, to have five boys die for every girl that dies, and for newspapers and experts to make it sound like girls have it worse here.
I think the strength of your objection here depends on which of two possible underlying models is at play:
- The boys who attempt suicide and the girls who attempt suicide are in pretty much the same mental state when they attempt suicide (that is: definitely wanting to end their life). But, for whatever reason, the boys use more effective methods, and so they end up actually dying at a higher rate.
- There are two categories of mental states for suicide-attempters: one in which they genuinely and definitely want to die, and will therefore seek out and use effective methods; and one in which they are ok with dying but also would be ok with living and receiving attention, in which case they may use a less effective method
If (1) is the case, then I think it is at least arguable that girls have it worse here, since they end up in the mental state of "definitely wanting to die" more often than boys, and that sucks. That said, it's still true that they're not actually dying as much, and so I think it's still kinda disingenuous to frame it the way the newspaper and experts have here.
If (2) is the case, then that means that boys are ending up in the "definitely wanting to die" state much more often than girls, in which case I'd agree that it's very wrong to say that girls have it worse.
If you're getting comments like that from friends and family, it's possible that you havent been epistemically transparent with them? E.g. do you think your friends who made those comments would be able to say why you believe what you do? Do you tell them about your reaearch process and what kinds of evidence you look for, or do you just make contrarian factual assertions?
There's a big difference between telling someone "the WHO is wrong about salt, their recommendations are potentially deadly" versus "Ive read a bunch of studies on salt, and from what Ive found, the WHOs recommendations don't seem to agree with the latest research. Their recs are based on [studies x,y] and say to do [a], but [other newer/better studies] indicate [b]."
Cut to a few decades later, and most people think that the way it's been done for about two or three generations is the way it's always been done (it isn't)
As possibly one of those people myself, can you give a few examples of what specifically is being done differently now? Are you talking about things like using lots of adderall?
I'm also morbidly curious what the model would do in <|bad|> mode.
I'm guessing that poison-pilling the <|bad|> sentences would have a negative effect on the <|good|> capabilities as well? I.e. It seems like the post is saying that the whole reason you need to include the <|bad|>s at all in the training dataset is that the model needs them in order to correctly generalize, even when predicting <|good|> sentences.
It seems plausible to me that within the next few years we will have:
- The next gen of language models, perhaps with patches to increase memory of past conversations
- The next gen of image/video models, able to create real-time video of a stable character conversing with the user
- The next gen of AI voice synthesis, though current capabilities might be enough
- The next gen of AI voice recognition, though current capabilities are probably enough
And with these things, you'd have access to a personalized virtual partner who you can video chat, phone call, or text with.
It does seem like AI dating will start to become a big thing in the near future. And I'm also not sure how to feel about that.
I think the point of this post is more "how do we get the AI to do what we want it do to", and less "what should we want the AI to do"
That is, there's value in trying to figure out how to align an LLM to any goal, regardless of whether a "better" goal exists. And the technique in the post doesnt depend on what target you have for the LLM: maybe someone wants to design an LLM to only answer questions about explosives, in which case they could still use the techniques described in the post to do that.
Well, really every second that you remain alive is a little bit of bayesian evidence for quantum immortality: the likelihood of death during that second according to quantum immortality is ~0, whereas the likelihood of death if quantum immortality is false is >0. So there is a skewed likelihood ratio in favor of quantum immortality each time you survive one extra second (though of course the bayesian update is very small until you get pretty old, because both hypotheses assign very low probaility to death when young)
I just want to say that I appreciate this post, and especially the "What it might look like if this gap matters" sections. They were super useful for contextualizing the more abstract arguments, and I often found myself scrolling down to read them before actually reading the corresponding section.
I'll definitely agree that most people seem to prefer having their own kids to adopting kids. But is this really demonstrating an intrinsic desire to preserve our actual physical genes, or is it more just a generic desire to "feel like your kids are really yours"?
I think we can distinguish between these cases with a thought experiment: Imagine that genetic engineering techniques become available that give high IQs, strength, height, etc., and that prevent most genetic diseases. But, in order to implement these techniques, lots and lots of genes must be modified. Would parents want to use these techniques?
I myself certainly would, even though I am one of the people who would prefer to have my own kids vs adoption. For me, it seems that the genes themselves are not actually the reason I want my own kids. As long as I feel like the kids are "something I created", or "really mine", that's enough to satisfy my natural tendencies. I suspect that most parents would feel similarly.
More specifically, I think what parents care about is that their kids kind of look like them, share some of their personality traits, "have their mother's eyes", etc. But I don't think that anyone really cares how those things are implemented.
So basically you admit that humans are currently an enormous success according to inclusive fitness, but at some point this will change - because in the future everyone will upload and humanity will go extinct
Not quite - I take issue with the certainty of the word "will" and with the "because" clause in your quote. I would reword your statement the following way:
"Humans are currently an enormous success according to inclusive fitness, but at some point this may change, due to any number of possible reasons which all stem from the fact that humans do not explicitly care about / optimize for our genes"
Uploading is one example of how humans could become misaligned with genetic fitness, but there are plenty of other ways too. We could get really good at genetic engineering and massively reshape the human genome, leaving only very little of Evolution's original design. Or we could accidentally introduce a technology that causes all humans to go extinct (nuclear war, AI, engineered pandemic).
(Side note: The whole point of being worried about misalignment is that it's hard to tell in advance exactly how the misalignment is going to manifest. If you knew in advance how it was going to manifest, you could just add a quick fix onto your agent's utility function, e.g. "and by the way also assign very low utility to uploading". But I don't think a quick fix like this is actually very helpful, because as long as the system is not explicitly optimizing for what you want it to, it's always possible to find other ways the system's behavior might not be what you want)
My point is that I'm not confident that humans will always be aligned with genetic fitness. So far, giving humans intelligence has seemed like Evolution's greatest idea yet. If we were explicitly using our intelligence to maximize our genes' prevalence, then that would probably always remain true. But instead we do things like create weapons arsenals that actually pose a significant risk to the continued existence of our genes. This is not what a well-aligned intelligence that is robust to future capability gains looks like.
I see your point, and I think it's true right at this moment, but what if humans just haven't yet taken the treacherous turn?
Say that humans figure out brain uploading, and it turns out that brain uploading does not require explicitly encoding genes/DNA, and humans collectively decide that uploading is better than remaining in our physical bodies, and so we all upload ourselves and begin reproducing digitally instead of thru genes. There is a sense in which we have just destroyed all value in the world, from the anthropomorphized Evolution's perspective.
If we say that "evolutions goal" is to maximize the number of human genes that exist, then it has NOT done a good job at aligning humans in the limit as human capabilities go to infinity. We just havent reached the point yet where "humans following our own desires" starts to diverge with evolution's goals. But given that humans do not care about our genes implicitly, there's a good chance that such a point will come eventually.
Thanks for coming today, everyone! For anyone who is interested in starting a regular Princeton meetup group / email list / discord, shoot me an email at dskumpf@gmail.com, and I'll set something up!
I agree. I find myself in an epistemic state somewhat like: "I see some good arguments for X. I can't think of any particular counter-argument that makes me confident that X is false. If X is true, it implies there are high-value ways of spending my time that I am not currently doing. Plenty of smart people I know/read believe X; but plenty do not"
It sounds like that should maybe be enough to coax me into taking action about X. But the problem is that I don't think it's that hard to put me in this kind of epistemic state. Eg, if I were to read the right blogs, I think I could be brought into that state for a bunch of different values of X. A few of the top of my head that seem plausible:
- Climate change
- Monetary policy/hyperinflation
- Animal suffering
So I don't feel super trusting of my epistemic state. I guess I feel a sort of epistemic learned helplessness, where I am suspicious of smart bloggers' ability to get me to think an issue is important and worth dedicating my life to.
Not totally sure how to resolve this, though I suppose it would involve some sort of "going off on my own and actually thinking deeply about what it should take to convince me"
A few more instances of cheap screening of large numbers:
- I've seen people complain about google-style technical interviews, because implementing quicksort in real-time is probably not indicative of what you'll be doing as a software engineer on the job. But google has enough applicants that it doesn't matter if the test is noisy; some genuinely good candidates may fail the test, but there are enough candidates that it's more efficient to just test someone else than to spend more time evaluating any one candidate
- Attractive women on dating apps. A man's dating profile is a noisy signal of his value as a partner, but it's extremely cheap for an attractive woman to just "swipe left" and try the next one. This strategy will certainly pass up people who would have made good partners, but the cost of evaluating a new profile is so low that it makes sense to just ignore any profiles that aren't obviously great
I'll offer up my own fasting advice as well:
I (and the couple of people I know who have also experimented with fasting) have found it to be a highly trainable skill. Doing a raw 36-hour fast after never having fasted before may be miserable; but doing the same fast after two weeks of 16-8 intermittent fasting will probably be no big deal.
Before I started intermittent fasting, I'd done a few 30-hour fasts, and all of them got very difficult towards the end. I would get headaches, feel very fatigued, and not really be able to function from hours 22-30. When I started IF, the first week was quite tough. I'd have similar symptoms as the fasting window was ending: headaches, trouble focusing. But then right around the two week mark, things changed. The symptoms went away, and the hunger became a much more "passive" feeling. Rather than hunger directly causing discomfort, the hunger now feels more like a "notification". Just my body saying "hey, just so you know, we haven't eaten for a while", rather than it saying "you're going to die if you don't eat right this moment". This change has been persistent, even during periods where I've stopped IF.
Both of the two others I've seen try IF have reported something similar, that the first few weeks are tough, but then the character of hunger itself starts to change. Today, I can go 24 hours without eating fairly trivially, ie without much distraction or performance decreases from hunger.
Going 36 will still be a challenge, but some pre-training may make it easier! Of course you may be specifically trying to test your willpower, in which case making it easier may be counter productive. Either way, this seems like a cool idea for a secular holiday. Best of luck!
I'm in a similar place, and had the exact same thought when I looked at the 80k guide.
Yes that was my reasoning too. The situation presumably goes:
- Omicron chooses a random number X, either prime or composite
- Omega simulates you, makes its prediction, and decides whether X's primality is consistent with its prediction
- If it is, then:
- Omega puts X into the box
- Omega teleports you into the room with the boxes and has you make your choice
- If it's not, then...? I think the correct solution depends on what Omega does in this case.
- Maybe it just quietly waits until tomorrow and tries again? In which case no one is ever shown a case where the box does not contain Omicron's number. If this is how Omega is acting, then I think you can act as though your choice affects Omircon's number, even though that number is technically random on this particular day.
- Maybe it just picks its own number, and shows you the problem anyway. I believe this was the assumption in the post.
I remember hearing from what I thought was multiple sources that your run-of-the-mill PCR test had something like a 50-80% sensitivity, and therefore a pretty bad bayes factor for negative tests. But that doesnt seem to square with these results - any idea what Im thinking of?
I agree. It makes me really uncomfortable to think that while Hell doesn't exist today, we might one day have the technology to create it.
I’m disappointed that a cooperative solution was not reached
I think you would have had to make the total cooperation payoff greater than the total one-side-defects payoff in order to get cooperation as the final result. From a "maximize money to charity" standpoint, defection seems like the best outcome here (I also really like the "pre-commit to flip a coin and nuke" solution). You'd have to believe that the expected utility/$ of the "enemy" charity is less than 1/2 of the expected utility/$ of yours; otherwise, you'd be happier with the enemy side defecting than with cooperation. I personally wouldn't be that confident about the difference between AMF and MIRI.
For those of us who don't have time to listen to the podcasts, can you give a quick summary of which particular pieces of evidence are strong? I've mostly been ignoring the UFO situation due to low priors. Relatedly, when you say the evidence is strong, do you mean that the posterior probability is high? Or just that the evidence causes you to update towards there being aliens? Ie, is the evidence sufficient to outweigh the low priors/complexity penalties that the alien hypothesis seems to have?
FWIW, my current view is something like:
- I've seen plenty of videos of UFOs that seemed weird at first that turned out to have a totally normal explanation. So I treat "video looks weird" as somewhat weak Bayesian evidence.
- As for complexity penalties: If there were aliens, it would have to be explained why they mostly-but-not-always hide themselves. I don't think it would be incompetence, if they're the type of civilization that can travel stellar distances.
- It would also have to be explained why we haven't seen evidence of their (presumably pretty advanced) civilization
- And it would have to be explained why there hasn't been any real knock-down evidence, eg HD close-up footage of an obviously alien ship (unless this is the type of evidence you're referring to?). A bunch of inconclusive, non-repeatable, low-quality data seems to be much more likely in the world where UFOs are not aliens. Essentially there's a selection effect where any sufficiently weird video will be taken as an example of a UFO. It's easier for a low-quality video to be weird, because the natural explanations are masked by the low quality. So the set of weird videos will include more low-quality data sources than the overall ratio of existing high/low quality sources would indicate. Whereas, if the weird stuff really did exist, you'd think the incidence of weird videos would match the distribution of high/low quality sources (which I don't think it does? as video tech has improved, have we seen corresponding improvements in average quality of UFO videos?).
I really like this post for two reasons:
- I've noticed that when I ask someone "why do you believe X", they often think that I'm asking them to cite sources or studies or some such. This can put people on the defensive, since we usually don't have ready-made citations in our heads for every belief. But that's not what I'm trying to ask; I'm really just trying to understand what process actually caused them to believe X, as a matter of historical fact. That process could be "all the podcasters I listen to take X as a given", or "my general life experience/intuition has shown X to be true". You've put this concept into words here and solidified the idea for me: that it's helpful to communicate why you actually believe something, and let others do with that what they will.
- The point about uncertainty is really interesting. I'd never realized before that if you present your conclusion first, and then the evidence for it, then it sure looks like you already had that hypothesis for some reason before getting a bunch of confirming evidence. Which implies that you have some sort of evidence/intuition that led you to the hypothesis in addition to the evidence you're currently presenting.
I've wondered why I enjoy reading Scott Alexander so much, and I think that the points you bring up here are a big reason why. He explains his processes really well, and I usually end up feeling that I understand what actually caused him to believe his conclusions.
In a similar vein, there's a bunch of symphony of science videos. These are basically remixes of random quotes by various scientists, roughly grouped by topic into a bunch of songs.
If, on the other hand, heritability is high, then throwing more effort/money at how we do education currently should not be expected to improve SAT scores
I agree with spkoc that this conclusion doesn't necessarily follow from high heritability. I think it would follow from high and stable heritability across multiple attempted interventions.
An exaggerated story for the point I'm about to make: imagine you've never tried to improve SAT scores, and you measure the heritability. You find that, in this particular environment, genetic variance explains 100% of SAT scores. You can predict someone's SAT scores perfectly just by looking at their genome. You decide to take the half of the population with the highest predicted scores, and keep the SAT a secret from them until the day they take the test. And for the lower half, you give them dedicated tutors to help them prepare. Given the 100% heritability, you expect scores to stay exactly the same. But wait! What no one told you was that before your intervention, the learning environment had been magically uniform for every student. There had been no environmental variance at all, and so the only thing left to explain test scores was genetics. What you didn't realize is that your heritability estimate gave you no information at all about how environmental changes would affect scores - because there was no environmental variance at all!
A single heritability measurement only tells you, roughly, the ratio of "[existing environmental variance] times [sensitivity to environmental variance]" to "[existing genetic variance] times [sensitivity to genetic variance]". But it doesn't do anything to disentangle the sensitivities-to-variances from the actual variances. What if there's practically zero variance in the environment, but a high sensitivity of the trait you're looking at to environmental variance? You'd find heritability is very high, but changes to the environment will cause large decreases to heritability. Same thing with genes: what if your trait is 100% determined by genes, but it just so happens that everyone has the exact same genes? You'd find that genetic variance explains zero percent of your trait, but if you then tried some genetic engineering, you'd find heritability shoot upward.
In order to disentangle the "sensitivity of X to environmental variance" from "the level of environmental variance", you'd have to run multiple interventions over time, and measure the heritability of X after each one (or be confident that your existing environment has lots of variance).
People get fat eating fruits
Are you implying that there are examples of people like BDay mentioned, who are obese despite only eating fruits/nuts/meat/veggies? Or just that people can get fat while including fruit in the diet? I'd be surprised and intrigued if it were the former.
I've tried the whole foods diet, and I've personally found it surprisingly hard to overeat, even when I let myself eat as many fruits and nuts as I want. You can only eat so many cashews before they start to feel significantly less appetizing. And after I've eaten 500 cal of cashews in one sitting, the next time I'm feeling snacky, those cashews still sound kinda meh. Fruit is certainly easier to eat, but still after the fourth or fifth clementine I feel like "ok that's enough" (and that's probably only ~300 calories). Whereas I could easily eat 500 cal of candy without breaking a sweat.
I think one major roadblock to overeating with fruit is that it takes effort to eat. You have to peel an orange, or cut up a kiwi or melon, or bite off the green part of a strawberry. There's a lot more work involved in eating 500 cal of fruit than there is in unwrapping a candy bar or opening a party size bag of chips.
So all of this rambling is just to say that I'm somewhat skeptical of the claims that "fruit (nuts) are mostly sugar (fat) and are calorie dense, and you can overeat them just like you can with junk food". I think it's surprisingly hard in practice to do so (and it's much less enjoyable than overeating junk food).
Couple more:
"he wasn't be treated"
"Club cast cast Lumos"
It seems to me that the hungry->full Dutch book can be resolved by just considering the utility function one level deeper: we don't value hungriness or fullness (or the transition from hungry to full) as terminal goals themselves. We value moving from hungry to full, but only because doing so makes us feel good (and gives nutrients, etc). In this case, the "feeling good" is the part of the equation that really shows up in the utility function, and a coherent strategy would be one for which this amount of "feeling good" can not be purchased for a lower cost.
In the event anyone reading this has objective, reliable external metrics of extremely-high ability yet despite this feels unworthy of exploring the possibility that they can contribute directly to research
Huh, that really resonates with me. Thanks for this advice.
For the record, here's what the 2nd place CooperateBot [Insub] did:
- On the first turn, play 2.
- On other turns:
- If we added up to 5 on the last round, play the opponent's last move
- Otherwise, 50% of the time play max(my last move, opponents last move), and 50% of the time play 5 minus that
My goal for the bot was to find a simple strategy that gets into streaks of 2.5's as quickly as possible with other cooperation-minded bots. Seems like it mostly worked.
Is something strange going on in the Round 21-40 plot vs the round 41-1208 plot? It looks like the line labeled MeasureBot in the Round 21-40 plot switches to be labeled CooperateBot [Insub] in the Round 41-1208 plot. I hope my simple little bot actually did get second place!