Ah, I see. I did interpret the framing around "net positive" to be largely around normal companies. It's IMO relatively easy to be net-positive, since all you need to do is to avoid harm in expectation and help in any way whatsoever, which my guess is almost any technology startup that doesn't accelerate things, but has reasonable people at the aim, can achieve. 

When we are talking more about "how to make a safety-focused company that is substantially positive on safety?", that is a very different question in my mind.

I do have examples that motivated me to right this, but they're all examples where people are still strongly disagreeing about the object level of what happened, or possibly lying about how they disagree on the object level and pretending they're committed to honesty. I thought about putting them in the essay but decided it wouldn't be fair and I didn't want to distract my actual thesis into a case analysis of how maybe all my examples have a problem other than over-adherence to bad honesty norms. Should I put them in a comment? I'm genuinely unsure. I could probably DM you them if you really want?

There's also the possibility that a CCP AGI can only happen through being trained on Western data to some extent (i.e., the English language internet) because otherwise they can't scale data enough. This implies that it would probably be a "Marxism with Chinese characteristics [with American characteristics]" AI since it seems like that just raises the "alignment to CCP values" technical challenge difficulty a lot.

Isn't it kinda unreasonable to put 10% on superhuman coder in a year if current AIs have a 15 nanosecond time horizon? TBC, it seems fine IMO if the model just isn't very good at predicting the 10th/90th percentile, especially wiht extreme hyperparameters.

Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.

The more advanced model has fundamentally the same issues, but I haven't dug as deep there yet.

I agree that to the extent the treatment of superexponentiality is wrong it's also an issue there, albeit a substantially smaller one because the time horizon extension plays a smaller role in that model.

Thanks a lot for digging into this, appreciate it!

being bad at 10% and 90% work

Is the 90% supposed to say 50%?

If the model is about "how far we are to super-human effort and are extrapolating how long it takes to get there", a median-defining singularity that is not mentioned except in a table is just not good science.

Yup, very open to the criticism that the possibility and importance of superexponentiality should be highlighted more prominently. Perhaps it should be highlighted in the summary? To be clear, I was not intentionally trying to hide this. It is mentioned but not strongly highlighted in the corresponding  expandable in the scenario ("Why we forecast a superhuman coder in early 2027").

For another look, here is a 15 millisecond-to-4 hour time horizon comparison just in the super-exponentiated plurality.

copied from response to Ryan [LW(p) · GW(p)]: Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.

The superexponential in the simulation is implemented pretty crudely, if you have ideas on how to improve it I'd be interested, I didn't spend a large amount of time on this. I think this is happening because it's crudely implemented as each doubling taking 10% shorter, but this already leads to extremely low doubling times by the time the marginal doublings from the starting time horizon are added.

An obvious marginal thing I could do is to add uncertainty as to the shortening factor, though when I tested this and eyeballed it it didn't make that big of a difference (but I might have been mistaken). I also think this would ideally be implemented as some chance of going superexponential after each doubling rather than a chance of it happening only from the start, but didn't have time to implement that.

I will defend the basic dynamics though of substantial probability of a superexponentiality leading to a singularity being reasonable. And I don't think it's obvious whether we model that dynamic as too fast or too slow in general.

Here is the pure singularity isolated (only the first super-exponential function applied) and the same with normal parameters, which mostly just hide what's happening:

Yup, similar reaction to the above graph, I agree the shape looks wrong, interested in suggestions if you have them.
 

edit: actually I'm pretty confused about this last graph, what code exactly are you running? I don't think I understand.

Thank you so much for the thorough reply :)

My answer for "which weaker agents should the AI be kind to," is "all weaker agents."

Enough room for everyone

Our universe contains humans, octopuses, insects and many different weak agents. A superintelligent AI which has a very philosophically uncertain chance of being in a Karma Test, would be kind to all of these agents just in case the Karma Test executors had a particular one in mind.

Earth's future lightcone is extremely big (${10}^{22}$ stars), so there is room to be kind to all of these weaker agents, if you are morally uncertain about which ones to care about, and do not put all your weight on "caring for rocks."

Which acausal trade first?

From the point of view of a superintelligence, it may make sense to do acausal trades in the order you described. From the point of view of humans, we can debate the feasibility of acausal trades in any order we like :)

I'm not saying that the superintelligence has to do this trade first, before the other trades. I'm saying it should do it eventually.

Advantages and disadvantages

I was wrong to ask the question "which version is better." I think they can both be done, to cover each other's weaknesses.

Disadvantages

One disadvantage of Logical Counterfactual Simulations, is that at least some logical counterfactuals (e.g. your example of 2+2=5) cannot be imagined by an AI (or human).

The algorithms for imagining counterfactuals, designed for computing "what will happen if X was true," inevitably performs certain mathematical computations like 2+2=4. As such, it is unable to imagine what would happen if 2+2=5.

Therefore, even though the algorithms for imagining counterfactuals are capable of logical uncertainty, and assigning probabilities to logical statements like $P=NP$ and so forth, not every logical counterfactual can be imagined.

Even for logical counterfactuals which can be imagined (e.g. being in a Karma Test), I'm not sure how the ASI will assign probabilities to these counterfactuals. Being in a Karma Test cannot be proven or disproven, and is almost a philosophical question. My hope is that the ASI assigns a bit of probability and expected value to it, enough to make it worthwhile to spare humans.

If we can't argue why the ASI won't assign any probability, it's worth trying.

Advantages: UDT

The key advantage of Logical Counterfactual Simulations over physical simulations, is that it does not become a zero sum trade against UDT agents.

For physical simulations, in order to convince the UDT paperclip maximizer AI to make room for happy humans, you have to give it more paperclips somewhere else in the universe. This means whatever trade you make with it, reduces the total number of happy humans, and increases the total number of paperclips.

If you are a utilitarian, this would clearly be a zero sum game. But even if you are a selfish individual, what is your measure of survival? Is it the number of copies of yourself who are alive in the future? If that was the case, this would still be a zero sum game, since it's cheaper for your surviving copies to directly clone themselves, than to buy your doomed copies from the paperclip maximizers. Any trade with paperclip maximizers lead to more paperclips, and less of whatever you value.

Logical Counterfactual Simulations are still zero sum over logical counterfactuals, but the AI has a positive sum if AI alignment turns out easy, and humanity has a positive sum if AI alignment turns out hard (reducing logical risk [LW · GW]).

Advantages: No certainty

Logical Counterfactual Simulations prevents the ASI from reaching extreme certainty over which agents always win and which agents always lose, so it spreads out its trades.

If humans (and other sentient life) lose to misaligned ASI every time, such that we have nothing to trade with it, average human/sentient life in all of existence may end up miserable.

Logical Counterfactual Simulations, allows us to edit the Kingmaker Logic, so that the ASI can never be really sure we have nothing to trade, even if math and logic appear to prove we lose every time.

Thanks for reading :)

Do you agree each idea has advantages and disadvantages?

“Omega looks at whether we’d pay if in the causal graph the knowledge of the digit of pi and its downstream consequences were edited”

Can you formalize this? In other words, do you have an algorithm for translating an arbitrary mind into a causal graph and then asking this question? Can you try it out on some simple minds, like GPT-2?

I suspect there may not be a simple/elegant/unique way of doing this, in which case the answer to the decision problem depends on the details of how exactly Omega is doing it. E.g., maybe all such algorithms are messy/heuristics based, and it makes sense to think a bit about whether you can trick the specific algorithm into giving a "wrong prediction" (in quotes because it's not clear exactly what right and wrong even mean in this context) that benefits you, or maybe you have to self-modify into something Omega's algorithm can recognize / work with, and it's a messy cost-benefit analysis of whether this is worth doing, etc.

If your investors only get paid up to 100x their investment, you want to go for strategies that return much more than 100x if they work.

This pairs well with Scott Alexander's Kolmogorov Complicity And The Parable Of Lightning