LessWrong 2.0 Reader

I would strongly suggest considering hires who would be based in DC or who would hop between DC and Berkeley. In my experience, being in DC (or being familiar with DC & having a network in DC) is extremely valuable for being able to shape policy discussions, know what kinds of research questions matter, know what kinds of things policymakers are paying attention to, etc.

I would go as far as to say something like "in 6 months, if MIRI's technical governance team has not achieved very much, one of my top 3 reasons for why is that they did not engage enough with DC people//US policy people. As a result, they focused too much on questions that Bay Area people are interested in and too little on questions that Congressional offices and executive branch agencies are interested in. And relatedly, they didn't get enough feedback from DC people. And relatedly, even the good ideas they had didn't get communicated frequently enough or fast enough to relevant policymakers. And relatedly... etc etc."

I do understand this trades off against everyone being in the same place, which is a significant factor, but I think the cost is worth it. 

I strongly agree with Owen's suggestions about figuring out a plan grounded in current circumstances, rather than reproducing what was.

Here's some thoughts about what this might look like:

• Rationality as more of a focus area: Given that Lightcone runs Less Wrong, an obvious path to explore is whether rationality could be further developed by providing people either a fellowship or a permanent position to work on developing the art:
  • Being able to offer such paid positions might allow you to draw contributions from people with rare backgrounds. For example, you might decide it would be useful to better understand anthropology as a way of better understanding other cultures and practices and so you could directly hire an anthropologist to help with that.
  • It would also help with projects that would be valuable, but which would be a slog and require specific expertise. For example, it would be great to have someone update the sequences in light of more recent psychological research.
• Greater focus on entrepreneurship:
  • You've already indicated your potential interest in taking it this direction by adding it as one of the options on your form.
  • This likely makes sense given that Lightcone is located in the Bay Area, the city with the most entrepreneurs and venture capitalists in the world.
• Response to the rise of AI:
  • One of the biggest shifts in the world since FHI was started has been the dramatic improvements in AI
  • One response to this would be to focus more on the risks and impacts from AI. However, there are already a number of institutions focusing on this, so this might simply end up being a worse version of them:
    • You may also think that you might be able to find a unique angle, for example, given how Eliezer was motivated to create rationality in order to help people understand his arguments on AI safety, it might be valuable for there to be a research program which intertwines those two elements.
    • Or you might identify areas, such as AI consciousness, that feel ignored/underappreciated by the mainstream.
  • Another response would be to try to figure out how to leverage AI:
    • As an example, how could AI be used to develop wisdom.
  • Another response would be to decide that better orgs are positioned to pursue these projects.
• Is there anything in the legacy of MIRI, CFAR of FHI that is particularly ripe for further development?:
  • For example, would it make sense for someone to try to publish an explanation of some of the ideas produced by MIRI on decision theory in a mainstream philosophical journal?
  • Perhaps some techniques invented by CFAR could be tested with a rigorous academic study?
• Potential new sources of ideas:
  • There seems to have been a two-way flow of ideas between LW/EA and FHI.
  • Are there any new communities that could provide a novel source of ideas?:
    • A few immediately come to mind: post-rationality, progress studies, sensemaking, meditation, longevity.
• Opportunities for alpha from philosophical rigour:
  • This was one of the strengths of FHI - bringing philosophical rigour to new areas - how can this be preserved/carried forward?
  • One of the strengths of academic philosophy - compared to the more casual writing that is popular on Less Wrong - is its focus on rigour and drawing out distinctions. If this institute were able to recruit people with strong philosophical backgrounds, are there any areas that would be particularly ripe for applying this style of thinking?

I do think evaporative cooling is a concern, especially if everyone (or a very significant amount) of people left. But I think on the margin more people should be leaving to work in govt. 

I also suspect that a lot of systemic incentives will keep a greater-than-optimal proportion of safety-conscious people at labs as opposed to governments (labs pay more, labs are faster and have less bureaucracy, lab people are much more informed about AI, labs are more "cool/fun/fast-paced", lots of govt jobs force you to move locations, etc.)

I also think it depends on the specific lab– EG in light of the recent OpenAI departures, I suspect there's a stronger case for staying at OpenAI right now than for DeepMind or Anthropic. 

This works too, yeah.

https://en.wikipedia.org/wiki/Jeep_problem https://en.wikipedia.org/wiki/Tsiolkovsky_rocket_equation

Twitter, probably.

Sounds like a lot of political alliances! (And "these two political actors are aligned" is arguably an even weaker condition than "these two political actors are allies".)

At the end of the day, of course, all of these analogies are going to be flawed. AI is genuinely a different beast.

Be faithless to your cause and betray it to a stronger enemy.

This is such an incredible quote. I feel like I could write a whole essay on this.

Google fires 28 employees working on cloud and AI services for doing a ten hour sit in where they occupied their boss’s office until the police were eventually involved. And yes, if what you do at work is spend your time blockading your boss’s office until your policy demands are met, it seems like you are going to get fired?

In a company other than Google, I would say: yes, obviously.

But remember, when James Damore wrote his document, and as a reaction other people stopped doing their work in protest, it was he who was fired, not them. How were they supposed to know that this time it will be different?

you should substitute more specific language in place of “rational”: “The self-benefiting thing to do is to run away, but I hope I would at least try to drag the child off the railroad tracks,”

Wouldn't it be correct to say that it would be 'instrumentally rational' to run away in this case? It sounds rational to me, as far as you 'winning' means you 'surviving'.