LessWrong 2.0 Reader

What kind of madman puts the "Very Left" side of the plot on the right?

Assuming all went well, I just donated £5,000 through the Anglo-American charity, which should become about (£5000 * 1.25 * 96% = £6000 ≈ $7300) to lightcone.

I had further questions to their how to give page, so:

• You can return the forms by email, no need to post them. (I filled them in with Firefox's native "draw/write on this pdf" feature, handwriting my signature with a mouse.)
• If donating by bank transfer, you send the money to "anglo-american charity limited", not "anglo-american charitable foundation".
• For lightcone's contact details I asked on LW intercom. Feels rude to put someone's phone number here, so if you're doing the same as me, I'm not gonna save you that step.

You understand you can just block her on Reddit and Facebook, and move on with your life?

What do you mean by "geometry of program synthesis"?

When you say there's "no such thing as a state," or "we live in a density matrix," these are statements about ontology: what exists, what's real, etc.

Density matrices use the extra representational power they have over states to encode a probability distribution over states. If we regard the probabilistic nature of measurements as something to be explained, putting the probability distribution directly into the thing we live in is what I mean by "explain with ontology."

Epistemology is about how we know stuff. If we start with a world that does not inherently have a probability distribution attached to it, but obtain a probability distribution from arguments about how we know stuff, that's "explain with epistemology."

In quantum mechanics, this would look like talking about anthropics, or what properties we want a measure to satisfy, or solomonoff induction and coding theory.

What good is it to say things are real or not? One useful application is predicting the character of physical law. If something is real, then we might expect it to interact with other things. I do not expect the probability distribution of a mixed state to interact with other things.

Whew! That's pretty intense, and pretty smart. I didn't read it all because I don't have time and I'm not in the same emotional position you're in.

I do want to say that I've thought an awful lot and researched a lot about the situation we're in with AI and as a world and a society. I feel a lot more uncertainty than you seem to about outcomes. AI and AGI are going to create very, very major changes. There's a ton of risk of different sorts, but there's also a very real chance (relative to what we reliably know) that things get way, way better after AGI, if we can align it and manage the power distribution issues. And I think there are very plausible routes to both of those.

This is primarily based on uncertainty. I've looked in depth at the arguments for pessimism about both alignment and societal power structures. They are just as incomplete and vibes-based as the arguments for optimism. There's a lot of real substance on both sides, but not enough to draw firm conclusions. Some very well informed people think we're doomed, other equally well-informed people think odds are in favor of a vastly better future. We simply don't know how this is going to turn out.

There is still time to hope, and to help.

See my If we solve alignment, do we die anyway? [LW · GW] and the other posts and sources I link there for more on all of these claims.

That's fair enough and a good point. 

I think that the key difference is that in the case of profitable-but-bad technologies, someone, somewhere, will probably invent them because there's great incentive to do so.

In the case of gain-of-function, if there stops being grants and the academics who do it become pariahs, then the incentive to do the gain-of-function research is gone. 

Well that puts my concern to rest. Thanks!

Partially agreed. I've tested this a little personally; Claude successfully predicted their own success probability on some programming tasks, but was unable to report their own underlying token probabilities. The former tests weren't that good, the latter ones somewhat were okay, I asked Claude to say the same thing across 10 branches and then asked a separate thread of Claude, also downstream of the same context, to verbally predict the distribution.

Would also love to take the tests. If possible you could grab human test subjects from certain areas: a less wrong group, a reddit group, etc.