I think there might be many local improvements, but I'm pretty uncertain about important factors like elasticity of "demand" (for robbery) with respect to how much of a medication is available on demand.  i.e. how many fewer robberies do you get if you can get at most a single prescriptions' worth of some kind of controlled substance (and not necessarily any specific one), compared to "none" (the current situation) or "whatever the pharmacy has in stock" (not actually sure if this was the previous situation - maybe they had time delay safes for storing medication that wasn't filling a prescription, and just didn't store the filled prescriptions in the safes as well)?

Headline claim: time delay safes are probably much too expensive in human time costs to justify their benefits.

The largest pharmacy chains in the US, accounting for more than 50% of the prescription drug market^[1][2], have been rolling out time delay safes (to prevent theft)^[3].  Although I haven't confirmed that this is true across all chains and individual pharmacy locations, I believe these safes are used for all controlled substances.  These safes open ~5-10 minutes after being prompted.

There were >41 million prescriptions dispensed for adderall in the US in 2021^[4].  (Note that likely means ~12x fewer people were prescribed adderall that year.)   Multiply that by 5 minutes and you get >200 million minutes, or >390 person-years, wasted.  Now, surely some of that time is partially recaptured by e.g. people doing their shopping while waiting, or by various other substitution effects.  But that's also just adderall!

Seems quite unlikely that this is on the efficient frontier of crime-prevention mechanisms, but alas, the stores aren't the ones (mostly) paying the costs imposed by their choices, here.

1. ^{^}

   https://www.mckinsey.com/industries/healthcare/our-insights/meeting-changing-consumer-needs-the-us-retail-pharmacy-of-the-future

2. ^{^}

   https://www.statista.com/statistics/734171/pharmacies-ranked-by-rx-market-share-in-us/

3. ^{^}

   https://www.cvshealth.com/news/pharmacy/cvs-health-completes-nationwide-rollout-of-time-delay-safes.html

4. ^{^}

   https://www.axios.com/2022/11/15/adderall-shortage-adhd-diagnosis-prescriptions

use spaces that your community already has (Lighthaven?), even if they're not quite set up the right way for them

Not set up the right way would be an understatement, I think.  Lighthaven doesn't have an indoor space which can seat several hundred people, and trying to do it outdoors seems like it'd require solving maybe-intractable logistical problems (weather, acoustics, etc).  (Also Lighthaven was booked, and it's not obvious to me to what degree we'd want to subsidize the solstice celebration.  It'd also require committing a year ahead of time, since most other suitable venues are booked up for the holidays quite far in advance.)

I don't think there are other community venues that could host the solstice celebration for free, but there might be opportunities for cheaper (or free) venues outside the community (with various trade-offs).

Having said that, I would NOT describe this as asking "how could I have arrived at the same destination by a shorter route". I would just describe it as asking "what did I learn here, really".

I mean, yeah, they're different things.  If you can figure out how to get to the correct destination faster next time you're trying to figure something out, that seems obviously useful.

Some related thoughts.   I think the main issue here is actually making the claim of permanent shutdown & deletion credible.  I can think of some ways to get around a few obvious issues, but others (including moral issues) remain, and in any case the current AGI labs don't seem like the kinds of organizations which can make that kind of commitment in a way that's both sufficiently credible and legible that the remaining probability mass on "this is actually just a test" wouldn't tip the scales.

I am not covering training setups where we purposefully train an AI to be agentic and autonomous. I just think it's not plausible that we just keep scaling up networks, run pretraining + light RLHF, and then produce a schemer.^[2]

Like Ryan, I'm interested in how much of this claim is conditional on "just keep scaling up networks" being insufficient to produce relevantly-superhuman systems (i.e. systems capable of doing scientific R&D better and faster than humans, without humans in the intellectual part of the loop).  If it's "most of it", then my guess is that accounts for a good chunk of the disagreement.

Curated.  I liked that this post had a lot of object-level detail about a process that is usually opaque to outsiders, and that the "Lessons Learned" section was also grounded enough that someone reading this post might actually be able to skip "learning from experience", at least for a few possible issues that might come up if one tried to do this sort of thing.

(We check for "downvoter count within window", not all-time.)

Curated.  This dialogue distilled a decent number of points I consider cruxes between these two (clusters of) positions.  I also appreciated the substantial number of references linking back to central and generally high-quality examples of each argument being made; I think this is especially helpful when writing a dialogue meant to represent positions people actually hold.

I look forward to the next installment.

Here's the editor guide section for spoilers.  (Note that I tested the instructions for markdown, and that does indeed seem broken in a weird way; the WYSIWYG spoilers still work normally but only support "block" spoilers; you can't do it for partial bits of lines.)

In this case I think a warning at the top of the comment is sufficient, given the context of the rest of the thread, so up to whether you want to try to reformat your comment around our technical limitations.

Foobar! :::spoiler This text would be covered by a spoiler block. ::: test more stuff on the same line.

if they all shut down because their employees all quit.

I think that to the extent that other labs are "not far behind" (such as FAIR), this is substantially an artifact of them being caught up in a competitive arms race.  Catching up to "nearly SOTA" is usually much easier than "advancing SOTA", and I'm fairly persuaded by the argument that the top 3 labs are indeed ideologically motivated in ways that most other labs aren't, and there would be much less progress in dangerous directions if they all shut down because their employees all quit.

I mean, it was published between 2010 and 2015, and it's extremely rare for fanfiction (or other kinds of online serial fiction) to be more popular after completion than while it's in progress.  I followed it while it was in progress, and am in fact one of those people who found LessWrong through it.  There was definitely an observable "wave" of popularity in both my in-person and online circles (which were not, at the time, connected to the rationality community at all); I think it probably peaked in 2012 or 2013.

I suspect a lot of problems came from the influx of HPMOR and FiO readers into LW without any other grounding point. Niplav thankfully avoided this wave, and I buy that the empirical circles like Ryan Greenblatt's social circle isn't relying on fiction, but I'm worried that a lot of non-experts are so bad epistemically speaking that they ended up essentially writing fanfic on AI doom, and forget to check whether the assumptions actually hold in reality.

This seems unlikely to me, since HPMOR peaked in popularity nearly a decade ago.

If you get value out of it, we're happy for dialogues to be used that way, as long as it's clear to all participants what the expectations re: publishing/not publishing are, so that nobody has an upleasant surprise at the end of the day.  (Dialogues currently allow any participant to unilaterally publish, since most other options we could think imposed a lot of friction on publishing.)

We also have a system which automatically applies "core tags" (AI, Rationality, World Modeling, World Optimization, Community, and Practical) to new posts.  It's accurate enough, particulary with the AI tag, that it enables the use-case of "filter out all AI posts from the homepage", which a non-zero number of users want, even if we still need to sometimes fix the tags applied to posts.

Intercom has the benefit of acting as an inbox on our side, unlike comments posted on LW (which may not be seen by any LW team member).

In an ideal world, would Github Issues be better for tracking bug reports?  Probably, yes.  But Github Issues require that the user reporting an issue navigate to a different page and have a Github account, which approximately makes it a non-starter as the top-of-funnel.

Intercom's message re: response times has some limited configurability but it's difficult to make it say exactly the right thing here.  Triaging bug reports from Intercom messages is a standard part of our daily workflow,so you shouldn't model yourself as imposing unusual costs on the team by reporting bugs through Intercom.

re: reliability - yep, we are not totally reliable here.  There are probably relatively easy process improvements here that we will end up not implementing because figuring out & implementing such process improvements takes time, which means it's competing with everything else we might decide to spend time on.  Nevertheless I'm sorry about the variety of dropped balls; it's possible we will try to improve something here.

re: issue tracker - right now our process is approximately "toss bugs into a dedicated slack channel, shared with the EA forum".  The EA forum has a more developed issue-tracking process, so some of those do find their way to Github Issues (eventually).

Just as an FYI: pinging us on Intercom is a much more reliable way of ensuring we see feature suggestions or bug reports than posting comments.  Most feature suggestions won't be implemented^[1]; bug reports are prioritized according to urgency/impact and don't always rise to the level of "will be addressed" (though I think >50% do).

1. ^{^}

   At least not as a result of a single person suggesting them; we have ever made decisions that were influenced on the margin by suggestions from one or more LW users.

Not all of these are NDAs; my understanding is that the OpenPhil request comes along with the news of the grant (and isn't a contract).  Really my original shortform should've been a broader point about confidentiality/secrecy norms, but...

I have more examples, but unfortunately some of them I can't talk about.  A few random things that come to mind:

• OpenPhil routinely requests that grantees not disclose that they've received an OpenPhil grant until OpenPhil publishes it themselves, which usually happens many months after the grant is disbursed.
• Nearly every instance that I know of where EA leadership refused to comment on anything publicly post-FTX due to advice from legal counsel.
• So many things about the Nonlinear situation.
• Coordination Forum requiring attendees agree to confidentiality re: attendance and content of any conversations with people who wanted to attend but not have their attendance known to the wider world, like SBF, and also people in the AI policy space.

As a recent example, from this article on the recent OpenAI kerfufle:

Two people familiar with the board’s thinking say that the members felt bound to silence by confidentiality constraints.

I have observed the term "postmortem" to suffer from some semantic slippage, and is sometimes used to mean something similar to "retrospective".

NDAs sure do seem extremely costly.  My current sense is that it's almost never worth signing one, or binding oneself to confidentiality in any similar way, for anything except narrowly-scoped technical domains (such as capabilities research).

I think Schmidhuber does in fact think that humans will go extinct as a result of developing ASI: https://www.lesswrong.com/posts/BEtQALqgXmL9d9SfE/q-and-a-with-juergen-schmidhuber-on-risks-from-ai

This is subtweeting the fact that (iirc) the OP wrote a comment sort of similar to the one described, on Duncan's most recent post.  I think it is much less surprising that such a comment got downvoted on LW (vs. the EA forum), and am not at all surprised that Duncan then deleted the comment and banned the OP from comment on their posts, after the comment spawned a (probably-annoying-to-Duncan) longer comment thread.

On the question of aptitude for science, Summers said this: "It does appear that on many, many different human attributes -- height, weight, propensity for criminality, overall IQ, mathematical ability, scientific ability -- there is relatively clear evidence that whatever the difference in means -- which can be debated -- there is a difference in the standard deviation, and variability of a male and a female population. And that is true with respect to attributes that are and are not plausibly, culturally determined. If one supposes, as I think is reasonable, that if one is talking about physicists at a top 25 research university, one is not talking about people who are two standard deviations above the mean. And perhaps it's not even talking about somebody who is three standard deviations above the mean. But it's talking about people who are three and a half, four standard deviations above the mean in the one in 5,000, one in 10,000 class. Even small differences in the standard deviation will translate into very large differences in the available pool substantially out."

Source (the article linked to in the axios article you cite)

Putting that aside, I don't see what part of Zvi's claim you think is taking things further than they deserve to be taken.  It seems indisputably true that Summers is both a well-known bullet-biter^[1], and also that he has had some associations with Effective Altruism^[2].  That is approximately the extent of Zvi's claims re: Summers.  I don't see anything that Zvi wrote as implying some sort of broader endorsement of Summers' character or judgment.

1. ^{^}

   See above.

2. ^{^}

   https://harvardundergradea.org/podcast/2018/5/19/larry-summers-on-his-career-lessons-and-effective-altruism

Yep, thanks, corrected.  (I think that still could possibly imply the consequent, but it's a much weaker connection.)

At the same time, there are claims that Emmett couldn't get the board to explain their reasoning to him get written documentation of the board's reasoning, so if those claims are true, then "nothing to do with a specific safety issue" might just be a claim from the board by proxy.

I think this post neglects one of the most serious risks: that adopting a strategy is correlated decision across agents, that others will correctly see that happening, and that the downside risk is significantly magnified by those dynamics.

Naive 1st-order utilitarianism gives you the wrong answer here.  Do not illegally skip on paying your income taxes to donate to charity.  Spend your cognitive resources getting a better job, or otherwise legally optimizing for more income.  Being a software engineer at many tech companies will enable you to donate six figures per year while maintaining a comfortable lifestyle, without restricting your ability to engage with financial infrastructure, own property, or travel.

I am pretty concerned that most of the public discussion about risk from e.g. the practice of open sourcing frontier models is focused on misuse risk (particular biorisk).  Misuse risk seems like it could be a real thing, but it's not where I see most of the negative EV, when it comes to open sourcing frontier models.  I also suspect that many people doing comms work which focuses on misuse risk are focusing on misuse risk in ways that are strongly disproportionate to how much of the negative EV they see coming from it, relative to all sources.

I think someone should write a summary post covering "why open-sourcing frontier models and AI capabilities more generally is -EV".  Key points to hit:

• (1st order) directly accelerating capabilities research progress
• (1st order) we haven't totally ruled out the possibility of hitting "sufficiently capable systems" which are at least possible in principle to use in +EV ways, but which if made public would immediately have someone point them at improving themselves and then we die.  (In fact, this is very approximately the mainline alignment plan of all 3 major AGI orgs.)
• (2nd order) generic "draws in more money, more attention, more skilled talent, etc" which seems like it burns timelines

And, sure, misuse risks (which in practice might end up being a subset of the second bullet point, but not necessarily so).  But in reality, LLM-based misuse risks probably don't end up being x-risks, unless biology turns out to be so shockingly easy that a (relatively) dumb system can come up with something that gets ~everyone in one go.

And I'm confused why you're simultaneously complaining about lack of community epistemic rigor, but then also criticize Scott's time spent on research. Don't those considerations point in opposite directions?

Well, not necessarily - the judgment re: lack of epistemic rigor could be coming from having decided that there's an obvious right answer and observing everybody else arriving at the wrong answer, not from a lack of research effort that preceded arriving at the wrong answer.

ETA: I do currently think^[1] that kidney donation is probably more appropriately bucketed as "buying fuzzies" rather than "buying utilons" for most people in the relevant reference class, but I can imagine a set of beliefs & circumstances that tip it into the other bucket.

1. ^{^}

   Not a position I've spent a lot of time thinking about.  Maybe an hour?

whether or not this is an accurate description of Hobart’s views or his talk

Hobart (and Askonas) respond in Sergey's comment section, if anyone wants more detail.

Reducing costs equally across the board in some domain is bad news in any situation where offense is favored. Reducing costs equally-in-expectation (but unpredictably, with high variance) can be bad even if offense isn't favored, since you might get unlucky and the payoffs aren't symmetrical.

(re: recent discourse on bio risks from misuse of future AI systems.  I don't know that I think those risks are particularly likely to materialize, and most of my expected disutility from AI progress doesn't come from that direction, but I've seen a bunch of arguments that seem to be skipping some steps when trying to argue that progress on ability to do generic biotech is positive EV.  To be fair, the arguments for why we should expect it to be negative EV are often also skipping those steps.  My point is that a convincing argument in either direction needs to justify its conclusion in more depth; the heuristics I reference above aren't strong enough to carry the argument.)

Huh, I think that's probably a bug.  Will look into it.

You're aware that there's only one public RSP?

Of course very few "doomers" think that current LLMs behave in ways that we parse as "nice" because they have a "hidden, instrumental motive for being nice" (in the sense that I expect you meant that).  Current LLMs likely aren't coherent & self-aware enough to have such hidden, instrumental motives at all.

The broader laws of natural selection and survival of the fittest

These are distinct from biological evolution, so if our descendants end up being optimized over by them rather than by biological evolution, that feels like it's conceding the argument (that we'll have achieved escape velocity).

"I'm worried that if we let someone go off and try something different, they will suddenly become way less open to changing their mind, and be dead set on thinking they've found the One True Way" seems like something weird to be worried about.

This both seems like a totally reasonable concern to have, and also missing many of the concerning elements of the thing it's purportedly summarizing, like, you know, suddenly having totally nonsensical beliefs about the world.

"The training loop is still running" is not really a counterargument to "the training loop accidentally spat out something smarter than the training loop, that wants stuff other than the stuff the training loop is optimizing for, that iterates at a much faster speed than the training, and looks like it's just about to achieve total escape velocity".

It seems like the major crux here is whether we think that debates over claim and counter-claim (basically, other cruxes) are likely to be useful or likely to cause harm. It seems from talking to the mods here and reading a few of their comments on this topic that they tend to learn towards them being harmful on average and thus need to be pushed down a bit.

This is, as far as I can tell, totally false.  There is a very different claim one could make which at least more accurately represents my opinion, i.e. see this comment by John Wentworth (who is not a mod).

Most of your comment seems to be an appeal to modest epistemology.  We can in fact do better than total agnosticism about whether some arguments are productive or not, and worth having more or less of on the margin.

John mentioned the existince of What's General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?, which was something of a follow-up post to How To Go From Interpretability To Alignment: Just Retarget The Search, and continues in a similar direction.

evhub was at 80% about a year ago (currently at Anthropic, interned at OpenAI).

Daniel Kokotajlo was at 65% ~2 years ago; I think that number's gone up since then.

Quite a few other people at Anthropic also have pessimistic views, according to Chris Olah:

I wouldn't want to give an "official organizational probability distribution", but I think collectively we average out to something closer to "a uniform prior over possibilities" without that much evidence thus far updating us from there. Basically, there are plausible stories and intuitions pointing in lots of directions, and no real empirical evidence which bears on it thus far.

(Obviously, within the company, there's a wide range of views. Some people are very pessimistic. Others are optimistic. We debate this quite a bit internally, and I think that's really positive! But I think there's a broad consensus to take the entire range seriously, including the very pessimistic ones.)

The Deepmind alignment team probably has at least a couple people who think the odds are bad, (p(doom) > 50%) given the way Vika buckets the team, combined with the distribution of views reflected by DeepMind alignment team opinions on AGI ruin arguments.

Some corrections for your overall description of the DM alignment team:

• I would count ~20-25 FTE on the alignment + scalable alignment teams (this does not include the AGI strategy & governance team)
• I would put DM alignment in the "fairly hard" bucket (p(doom) = 10-50%) for alignment difficulty, and the "mixed" bucket for "conceptual vs applied"

Please refer back to your original claim:

no one in the industry seems to think they are writing their own death warrant

no one in the industry seems to think they are writing their own death warrant

What led you to believe this? Plenty of people working at the top labs have very high p(doom) (>80%).  Several of them comment on LessWrong.  We have a survey of the broader industry as well.  Even the people running the top 3 labs (Sam Altman, Dario Amodei, and Demis Hassabis) all think it's likely enough that it's worth dedicating a significant percentage of their organizational resources to researching alignment.

Most "Bayesians" are deceiving themselves about how much they are using it.

This is a frequently-made accusation which has very little basis in reality.  The world is a big place, so you will be able to find some examples of such people, but central examples of LessWrong readers, rationalists, etc, are not going around claiming that they run their entire lives on explicit Bayes.

Most (but not all) automatic rate limits allow authors to continue to comment on their own posts, since in many such cases it does indeed seem likely that preventing that would be counterproductive.

Curated.

Although the LK-99 excitement has cooled off, this post stands as an excellent demonstration of why and how Bayesian reasoning is helpful: when faced with surprising or confusing phenomena, understanding how to partition your model of reality such that new evidence would provide the largest updates, is quite valuable.  Even if the questions you construct are themselves confused or based on invalid premises, they're often confused in a much more legible way, such that domain experts can do a much better job of pointing to that and saying something like "actually, there's a third alternative", or "A wouldn't imply B in any situation, so this provides no evidence".

This seems like an epistemically dangerous way of describing the situation that "These people think that AI x-risk arguments are incorrect, and are willing to argue for that position".

I don't think the comment you're responding to is doing this; I think it's straightforwardly accusing LeCun and Andreesen of conducting an infowar against AI safety.  It also doesn't claim that they don't believe their own arguments.

Now, the "deliberate infowar in service of accelerationism" framing seems mostly wrong to me (at least with respect to LeCun; I wouldn't be surprised if there was a bit of that going on elsewhere), but sometimes that is a thing that happens and we need to be able to discuss whether that's happening in any given instance.  re: your point about tribalism, this does carry risks of various kinds of motivated cognition, but the correct answer is not to cordon off a section of reality and declare it off-limits for discussion.

So "being happy" or "being a utility-maximizer" will probably end up being a terminal goal, because those are unlikely to conflict with any other goals. 

"Being unlikely to conflict with other values" is not at the core of what characterizes the difference between instrumental and terminal values.

If you're talking about goals related purely to the state of the external world, not related to the agent's own inner-workings or its own utility function, why do you think it would still want to keep its goals immutable with respect to just the external world?

Putting aside the fact that agents are embedded in the environment, and that values which reference the agent's internals are usually not meaningfully different from values which reference things external to the agent... can you describe what kinds of values that reference the external world are best satisfied by those same values being changed?