Utilitarianism is just an approximate theory. I don’t think it’s truly possible to compare happiness and pain, and certainly one can not balance the other. The Repugnant Conclusion should be that Utilitarianism is being stretched outside of its bounds. It’s not unlike Laplace’s demon in physics: it’s impossible to know enough about the system to make those sorts of choices.

You would have to look at each individual. I order to get a sufficiently detailed picture of their life, it takes a lot of time. Happiness isn’t a number. It’s more like a vector in high-dimensional space, where it can depend on any number of factors, including the mental state of one’s neighbors. Comparing requires combinatorics, so again, these hypothetical computations would blow up to impracticality.

Utilitarianism is instead an approximate theory. We are accepting the approximation that happiness and pain are a one-dimensional. It’s not real, but it makes the math easier to deal with. It’s useful, because that approximation works for most cases, without knowing the details, similar to statistical mechanics, but once you start getting into edge cases, the wheels fall off. That shouldn’t be surprising, as we are collapsing a high-dimensional vector into a single point. We’re losing fidelity, to gain computability.

I think it’s fair to say that humans are incapable of truly understanding each other. Relationships of that approximate level of knowledge take years to develop, and in most cases never do. Without that you don’t know their preferences, and without that you can’t know the vectors of their mental state, and therefore you can’t really compare for the level of detail needed to truly know if the world would be better in one state or another.

So, we approximate. Which is fine, as long as you remember that it is an approximation. I don’t think that it is possible to have a perfect ethical system with no contradictions. The best we can do is hold several ethical models, and see how they compare as a guide for our actions in an uncertain world.

Ever since NIST put a backdoor in Dual Elliptic Curve Deterministic Random Bit Generator, they have the problem that many people no longer trust them.

I guess it might be possible to backdoor AI Safety Evaluations (e.g. suppose there is some  know very dangerous thing that National Security Agency is doing with AI, and NIST deliberately rigs their criteria to not stop this very dangerous thing).

But apart from the total loss of public trust in them as an institution, NIST has done ground-breaking work in the computer security field in the past, so it wouldnt be so unusal for them to develop AI criteria.

The whole dual elliptic curve fiasco is possibly a lesson that criteria should be developed by international groups, because a single country;s standards body, like NIST, can be subverted by their spy agencies.

This is a very interesting paper, thanks.

Sir Gowers actually has a number of playlists around thinking about problems in real time; haven't looked at them myself, but may be worthwhile to mention that this series is one instance of multiple playlists of the same type, each focused on its own domain.

Wow, this seems like an interesting topic to explore.

The people threatening to resign (are there any? without specific information, this could possibly be entirely made up), could be useful to ask them if they have any objections against Paul Christiano, or just EA in general, and if it is the latter, what sources they got their information from, and perhaps what could possibly change their minds.

I think FHI was an extremely special place and I was privileged to get to spend time there. 

I applaud attempts to continue its legacy. However, I'd feel gut-level more optimistic about plans that feel more grounded in thinking about how circumstances are different now, and then attempting to create the thing that is live and good given that, relative to attempting to copy FHI as closely as possible. 

Differences in circumstance

You mention not getting to lean on Bostrom's research taste as one driver of differences, and I think this is correct but may be worth tracing out the implications of even at the stage of early planning. Other things that seem salient and important to me:

• For years, FHI was one of the only places in the world that you could seriously discuss many of these topics
  • There are now much bigger communities and other institutions where these topics are at least culturally permissible (and some of them, e.g. AI safety, are the subject of very active work)
  • This means that:
    • One of FHI's purposes was serving a crucial niche which is now less undersupplied
    • FHI benefited from being the obvious Schelling location to go to think about these topics
      • Whereas even in Berkeley you want to think a bit about how you sit in the ecosystem relative to Constellation (which I think has some important FHI-like virtues, although makes different tradeoffs and misses on others)
• FHI benefited from the respectability of being part of the University
  • In terms of getting outsiders to take it seriously, getting meetings with interesting people, etc.
  • I'm not saying this was crucial for its success, and in any case the world looks different now; but I think it had some real impact and is worth bearing in mind
• As you mention -- you have a campus!
  • I think it would be strange if this didn't have some impact on the shape of plans that would be optimal for you

Pathways to greatness

If I had to guess about the shape of plans that I think you might engage in that would lead to something deserving of the name "FHI of the West", they're less like "poll LessWrong for interest to discover if there's critical mass" (because I think that whether there's critical mass depends a lot on people's perceptions of what's there already, and because many of the people you might most want probably don't regularly read LessWrong), and more like thinking about pathways to scale gracefully while building momentum and support.

When I think about this, two ideas that seem to me like they'd make the plan more promising (that you could adopt separately or in conjunction) are (1) starting by finding research leads, and/or (2) starting small as-a-proportion-of-time. I'll elaborate on these:

Finding research leads

I think that Bostrom's taste was extremely important for FHI. There are a couple of levels this was true on:

• Cutting through unimportant stuff in seminars
  • I think it's very easy for people, in research, to get fixated on things that don't really matter. Sometimes this is just about not asking enough which the really important questions are (or not being good enough at answering that); sometimes it's kind of performative, about people trying to show off how cool their work is
  • Nick had low tolerance for this, as well as excellent taste. He wasn't afraid to be a bit disagreeable in trying to get to the heart of things
  • This had a number of benefits:
    • Helping discussions in seminars to be well-focused
    • Teaching people (by example) how to do the cut-through-the-crap move
    • Shaping incentives for researchers in the institute, towards tackling the important questions head on
• Gatekeeping access to the space
  • Bostrom was good at selecting people who would really contribute in this environment
    • This wasn't always the people who were keenest to be there; and saying "no" to people who would add a little bit but not enough (and dilute things) was probably quite important
    • In some cases this meant finding outsiders (e.g. professors elsewhere) to visit, and keeping things intellectually vibrant by having discussions with people with a wide range of current interests and expertise, rather than have FHI just become an echo chamber
• Being a beacon
  • Nick had a lot of good ideas, which meant that people were interested to come and talk to him, or give seminars, etc.

If you want something to really thrive, at some point you're going to have to wrestle with who is providing these functions. I think that one thing you could do is to start with this piece. Rather than think about "who are all the people who might be part of this? does that sound like critical mass?", start by asking "who are the people who could be providing these core functions?". I'd guess if you brainstorm names you'll end up with like 10-30 that might be viable (if they were interested). Then I'd think about trying to approach them to see if you can persuade one or more to play this role. (For one thing, I think this could easily end up with people saying "yes" who wouldn't express interest on the current post, and that could help you in forming a strong nucleus.)

I say "leads" rather than "lead" because it seems to me decently likely that you're best aiming to have these responsibilities be shared over a small fellowship. (I'm not confident in this.)

Your answer might also be "I, Oliver, will play this role". My gut take would be excited for you to be like one of three people in this role (with strong co-leads, who are maybe complementary in the sense that they're strong at some styles of thinking you don't know exactly how to replicate), and kind of weakly pessimistic about you doing it alone. (It certainly might be that that pessimism is misplaced.)

Starting small as-a-proportion-of-time

Generally, things start a bit small, and then scale up. People can be reluctant to make a large change in life circumstance (like moving job or even city) for something where it's unknown what the thing they're joining even is. By starting small you get to iron out kinks and then move on from there.

Given that you have the campus, I'd seriously consider starting small not as-a-number-of-people but as-a-proportion-of-time. You might not have the convening power to get a bunch of great people to make this their full time job right now (especially if they don't have a good sense who their colleagues will be etc.). But you probably do have the convening power to get a bunch of great people to show up for a week or two, talk through big issues, and spark collaborations. 

I think that you could run some events like this. Maybe to start they're just kind of like conferences / workshops, with a certain focus. (I'd still start by trying to find something like "research leads" for the specific events, as I think it would help convening power as well as helping the things to go well.) In some sense that might be enough for carrying forward the spirit of FHI -- it's important that there are spaces for it, not that these spaces are open 365. But if it goes well and they seem productive, it could be expanded. Rather than just "research weeks", offer "visiting fellowships" where people take a (well-paid) 1-3 month sabbatical from their regular job to come and be in person all at the same time. And then if that's going well consider expanding to a permanent research group. (Or not! Perhaps the ephemeral nature of short-term things, and constantly having new people, would prove even more productive.)

I thought that the point was that either managed-interface-only access, or API access with rate limits, monitoring, and an appropriate terms of service, can prevent use of some forms of scaffolding. If it's staged release, this makes sense to do, at least for a brief period while confirming that there are not security or safety issues.

These days it's rare for a release to advance the frontier substantially.

This seems to be one crux. Sure, there's no need for staged release if the model doesn't actually do much more than previous models, and doesn't have unpatched vulnerabilities of types that would be identified by somewhat broader testing.

The other crux, I think, is around public release of model weights. (Often referred to, incorrectly, as "open sourcing.") Staged release implies not releasing weights immediately - and I think this is one of the critical issues with what companies like X have done that make it important to demand staged release for any models claiming to be as powerful or more powerful than current frontier models. (In addition to testing and red-teaming, which they also don't do.)

If you think there’s something mysterious or unknown about what happens when you make two copies of yourself

Eliezer talked about some puzzles related to copying and anticipation in The Anthropic Trilemma [LW · GW] that still seem quite mysterious to me. See also my comment [LW(p) · GW(p)] on that post.

I would imagine I would also react to it with smile in the context of an informal call. When used as brand / "fill interest form here" I just think it's not a good name, even if I am sympathetic to proposals to create more places to do big picture thinking about future.