LessWrong 2.0 Reader

Do I indestand you correctly? You are saying that:

1. ASI will decide that its in a simulation
2. ASI will assume that simulation runners would prefer it to act aligned
3. Therefore, ASI will act aligned 

I think 1 is wrong to begin with, but addressing it is beyond this comment section.

So, what about 2? Why would ASI assume that cooperation with humans in a simulation is a behaviour that the creators of the simulation would reward? Instead of assuming... literally anything else about the nature of such reward.

Note: Most RSVPs (29 as of now) are on Meetup.

From my point of view, of course profit maximizing companies will…maximize profit. It never was even imaginable that these kinds of entities could shoulder such a huge risk responsibly.

Correct me if I'm wrong but isn't Conjecture legally a company? Maybe their profit model isn't actually foundation models? Not actually trying to imply things, just thought the wording was weird in that context and was wondering whether Conjecture has a different legal structure than I thought.

I don’t know whether OpenAI uses nondisparagement agreements; I haven’t signed one.

This can also be glomarizing. "I haven't signed one." is a fact, intended for the reader to use it as anecdotal evidence. "I don't know whether OpenAI uses nondisparagement agreements" can mean that he doesn't know for sure, and will not try to find out.

Obviously, the context of the conversation and the events surrounding Holden stating this matters for interpreting this statement, but I'm not interested in looking further into this, so I'm just going to highlight the glomarization possibility.

I really liked this one! I'd kept wanting to jump in on a DnD.Science thing for a while (both because it looked fun and I'm trying to improve myself in ways strongly consistent with learning more about how to do data science) and this was a perfect start. IMO you totally should run easier and/or shorter puzzles sometimes going forward, and maybe should mark ones particularly amenable to a first-timer as being so.

minus Cullen O’Keefe who worked on policy and legal (so was not a clear cut case of working on safety),

I think Cullen was on the same team as Daniel (might be misremembering), so if you count Daniel, I'd also count Cullen. (Unless you wanna count Daniel because he previously was more directly part of technical AI safety research at OAI.)

Are there any other nice decision problems that are low? A quick search only reveals existence theorems.

Intuitive guess: Can we get some hierarchy from oracles to increasingly sparse subsets of the digits of Chaitin's constant?

actions we could take in preparation

In preparation for what?

OK hmm I think I understand what you mean.

I would have thought about it like this:

• "our reference class" includes roughly the observations we make before observing that we're very early in the universe
  • This includes stuff like being a pre-singularity civilization
• The anthropics here suggest there won't be lots of civs later arising and being in our reference class and then finding that they're much later in universe histories
• It doesn't speak to the existence or otherwise of future human-observer moments in a post-singularity civilization

... but as you say anthropics is confusing, so I might be getting this wrong.

Why no prediction markets for large infrastructure projects?

Been reading this excellent piece on why prediction markets aren't popular. They say that without subsidies prediction markets won't be large enough; the information value of prediction markets is often nog high enough. 

Large infrastructure projects undertaken by governments, and other large actors often go overbudget, often hilariously so: 3x,5x,10x or more is not uncommon, indeed often even the standard.

One of the reasons is that government officials deciding on billion dollar infrastructure projects don't have enough skin in the game. Politicians are often not long enough in office to care on the time horizons of large infrastructure projects. Contractors don't gain by being efficient or delivering on time. To the contrary, infrastructure projects are huge cashcows. Another problem is that there are often far too many veto-stakeholders. All too often the initial bid is wildly overoptimistic. 

Similar considerations apply to other government projects like defense procurement or IT projects.

Okay - how to remedy this situation? Internal prediction markets theoretically could prove beneficial. All stakeholders & decisionmakers are endowed with vested equity with which they are forced to bet on building timelines and other key performance indicators. External traders may also enter the market, selling and buying the contracts. The effective subsidy could be quite large. Key decisions could save billions. 

In this world, government officials could gain a large windfall which may be difficult to explain to voters. This is a legitimate objection. 

A very simple mechanism would simply ask people to make an estimate on the cost C and the timeline T for completion.  Your eventual payout would be proportional to how close you ended up to the real C,T compared to the other bettors. [something something log scoring rule is proper].