LessWrong 2.0 Reader

As well as thinking about the need for the place in terms of providing a space for research, it is probably worth thinking about the need for a place in terms of what it provides the world.  What subjects are currently under-represented in the world and need strong representation to guide us to a positive future? That will guide who you want to lead the organisation.

Glad you're doing this. By default, it seems we're going to end up with very strong tool-use models where any potential safety measures are easily removed by jailbreaks or fine-tuning. I understand you as working on: How are we going to know that it happened? is that a fair characterization?

Another important question: What should the response be to the appearance of such a model? any thoughts?

I've updated the event information to include summaries/abstracts for the projects/talks. Some of these are still under construction.

I agree in the abstract with the idea of looking for niches, and I think that several of these ideas have something to them. Nevertheless when I read the list of suggestions my overall feeling is that it's going in a slightly wrong direction, or missing the point, or something. I thought I'd have a go at articulating why, although I don't think I've got this to the point where I'd firmly stand behind it:

It seems to me like some of the central FHI virtues were:

• Offering a space to top thinkers where the offer was pretty much "please come here and think about things that seem important in a collaborative truth-seeking environment"
  • I think that the freedom of direction, rather than focusing on an agenda or path to impact, was important for:
    • attracting talent
    • finding good underexplored ideas (b/c of course at the start of the thinking people don't know what's important)
  • Caveats:
    • This relies on your researchers having some good taste in what's important (so this needs to be part of what you select people on)
    • FHI also had some success launching research groups where people were hired to more focused things
      • I think this was not the heart of the FHI magic, though, but more like a particular type of entrepreneurship picking up and running with things from the core
• Willingness to hang around at whiteboards for hours talking and exploring things that seemed interesting
  • With an attitude of "OK but can we just model this?" and diving straight into it
    • Someone once described FHI as "professional amateurs", which I think is apt
      • The approach is a bit like the attitude ascribed to physicists in this xkcd, but applied more to problems-that-nobody-has-good-answers-for than things-with-lots-of-existing-study (and with more willingness to dive into understanding existing fields when they're importantly relevant for the problem at hand)
  • Importantly mostly without directly asking "ok but where is this going? what can we do about it?"
    • Prioritization at a local level is somewhat ruthless, but is focused on "how do we better understand important dynamics?" and not "what has external impact in the world?"
• Sometimes orienting to "which of our ideas does the world need to know about? what are the best ways to disseminate these?" and writing about those in high-quality ways
  • I'd draw some contrast with MIRI here, who I think were also good at getting people to think of interesting things, but less good at finding articulations that translated to broadly-accessible ideas

Reading your list, a bunch of it seems to be about decisions about what to work on or what locally to pursue. My feeling is that those are the types of questions which are largely best left open to future researchers to figure out, and that the appropriate focus right now is more like trying to work out how to create the environment which can lead to some of this stuff.

Overall, the take in the previous paragraph is slightly too strong. I think it is in fact good to think through these things to get a feeling for possible future directions. And I also think that some of the good paths towards building a group like this start out by picking a topic or two to convene people on and get them thinking about. But if places want to pick up the torch, I think it's really important to attend to the ways in which it was special that aren't necessarily well-represented in the current x-risk ecosystem.

Not sure what you mean by "this would require a pretty small universe".

If we live in naive MWI, an IBP agent would not care for good reasons, because naive MWI is a "library of babel" where essentially every conceivable thing happens no matter what you do.

Also not sure what you mean by "some sort of sampling". AFAICT, quantum IBP is the closest thing to a coherent answer that we have, by a significant margin.

Done! I didn't do it at first because I thought it'd have to be in person only, but then clicked around in the form and found that remote is also possible.

It's also putting the attribute on the wrong thing - it's not garbage data, it's data that's useful for other purposes than the one at hand.

Mostly it's not useful for anything. Like the logs contains lots of different types of information, and all the different types of information are almost always useless for all purposes, but each type of information has a small number of purpose for which a very small fraction of that information is useful.

Blessed and cursed are much worse as descriptors. In most cases there's nobody doing the blessing or cursing, and it focuses the mind on the perception/sanctity of the data, not the use of it.

This is somewhat intentional. One thing one can do with information is give it to others who would not have seen it. Here one sometimes needs to be careful to preserve and highlight the blessed information and eliminate the cursed information.

It can't use "air" around it for engines because what's around it isn't "air".

Oxygen is much heavier than the fuel it's used with, and you'd either need liquid oxygen (which increases costs) or pressurized tanks (which would perhaps double that mass). That's still lighter than batteries, yes, but engines are also needed. Piston engines are inefficient and/or heavy, and gas turbines are somewhat expensive.

It's not that difficult to separate water and hydrogen, that's true, but processing that much gas is still rather impractical when batteries have enough specific energy. Simply condensing it in the tube is...possible, but would increase drag, especially considering density variation issues, and you'd have to deal with getting it out of a long sealed tube without leaking hydrogen.

Also, if batteries are good enough, the cost of replacing the hydrogen alone probably makes batteries better than burning the hydrogen.

Noting that a nicer name that's just waiting to be had, in this context, is "Future of the Lightcone Institute" :) 

I can confirm that my PayPal has received the $500, although it'll be frozen for a while.

Thanks! I had a lot of fun doing the research for this and I'm working on an update that'll be out in a few days.