LessWrong 2.0 Reader

The possibility of sudden spikes in prices during a crisis also incentivize "speculators" to stockpile the good in advance. If they buy during normalcy in anticipation of a crisis, they smooth out the price curve by raising the price before the crisis has hit. As they hold the inventory in the mean time, these speculators are effectively creating a buffer in a potentially otherwise just-in-time supply chain.

I think this is a story about how AI was severely regulated by the courts, leading to incentives away from AI progress combined with global catastrophic events setting humanity's technology back by a lot, so AIs didn't kill us.

Fun unimportant thought: As tunnel density increases, there's a phase shift in how warfare works (in the area with dense tunnels).

Consider e.g. a city that has a network of tunnels/catacombs/etc. underneath. Attackers can advance on the surface, and/or they can advance under the ground.

For tunnel networks of historically typical density, it's better to advance on the surface. Why? Because (a) the networks are sparse enough that the defenders can easily man every chokepoint underground, and (b) not being able to use airplanes or long-ranged weaponry underground seems to advantage the defender more than the attacker (e.g. can't use artillery to soften up defenders, can't use tanks, can't scout or bomb from the air). OTOH your attacking forces can get real close before they themselves can be shot at -- but this doesn't seem to be sufficient compensation.

Well, as the density of the network increases, eventually factor (a) reverses. Imagine a network so dense that in a typical 1km stretch of frontline, there are 100 separate tunnels passing beneath, such that you'd need at least 100 defensive chokepoints or else your line would have an exploitable hole. Not enough? Imagine that it's 1000... The point is, at some point it becomes more difficult to defend underground than to defend on the surface. Beyond that point fighting would mostly happen underground and it would be... infantry combat, but in three dimensions? Lots of isolated squads of men exploring dark tunnel networks, occasionally clashing with each other, forming impromptu chokepoints and seeking to outflank?

Factor (b) might reverse as well. Perhaps it's entangled with factor (a). In a superdense tunnel network, I could see it being the case that the advantage of the attacker (being able to advance unseen, being able to advance without being caught in fields of fire by machineguns and artillery, until they are literally in the same hallway as the enemy) outweigh the advantage of the defender (good cover everywhere, defensive positions can't be spotted in advance and pounded by artillery and bombs) Whereas in a light tunnel network, there are only a few ways to approach so the defender just has to fortify them and wait.



 

Yes it does require limiting the spread of AGI. I only referred to it briefly in the phrase "...even if AGI does not proliferate widely". I discuss it more in "do we all die anyway". I think it's not quite as clear as needing to shut down all other AGI projects or we're doomed; a small number of AGIs under control of different humans might be stable with good communication and agreements, at least until someone malevolent or foolish enough gets involved. That's why I'm using the term proliferation; I think the dynamics are somewhat similar to the nuclear standoff, where we've actually seen stability with a handful of actors.

I'm hoping the need to reduce proliferation will become apparent to anyone who sees the potential of real AGI and who thinks about international politics, including terrorism. I'm hoping the potential of AGI will be much more intuitively apparent to anyone having a conversation with something that's smarter than them and just as agentic. We shall see.

Note that I did address your core point from the other comments you linked: human values aren't well-defined, so you can't align anything to them. I think aligning a superintelligence to anything in the neighborhood of common human preferences would be close enough to be pretty happy with, even if you can't do something more clever and provide lots of freedom to future beings without letting them abuse it at others' expense. Hopefully we can have a long reflection, or good answers seem obvious if we have a superhuman intelligence helping us think through it. I have some ideas, but that's a whole different project than surviving the first AGIs and so getting to have that discussion and that choice.

I think without any progress on understanding human values we could still have a world thousands of times better than anything we've had, in the opinions of the vast majority of human beings who have or will live. That's good enough for me, at least for now.

The French rich idiot who bought 75 million dollar of Trump is an EA hero win or lose.

LW loves prediction markets. EA loves them. I love them. You love them.

See: https://worksinprogress.co/issue/why-prediction-markets-arent-popular/

Problem is, unlike financial markets prediction markets are zero sum. That limits how much informed traders -"sharps"- are incentivized.

In theory this could be remedied by subsidizing the market by a party that is willing to pay for the information. The information is public so this suffers from standard tragedy of the commons.

Mister Theo bought 75 million dollars worth of Yes Trump. He seems to have no inside information. In expectation then he is subsidizing prediction markets, providing the much needed financial incentive to attract the time, effort and skills of sophisticated sharps.

Perhaps we should think of uninformed "noise traders" on prediction markets as engaging in prosocial behaviour. Their meta-epistemic delusion and gambling addiction provide the cold hard cash to finance accurate pricefinding in the long-run.

In many respects, I expect this to be closer to what actually happens than "everyone falls over dead in the same second" or "we definitively solve value alignment". Multipolar worlds, AI that generally follows the law (when operators want it to) but cannot fully be trusted, and generally muddling through are the default future. I'm hoping we don't get instrumental survival drives though.

Summary threads of two recent papers which seem like significant evidence in favor of the Simulators [LW · GW] view of LLMs (especially after just pretraining): https://x.com/aryaman2020/status/1852027909709382065 https://x.com/DimitrisPapail/status/1844463075442950229 

These numbers are dictated by the regulator. What mechanism is there to make them have any relation to the real world?

Does this plan necessarily factor through the using the intent-aligned AGI to quickly commit some sort of pivotal act [LW · GW] that flips the gameboard and prevents other intent-aligned AGIs from being used malevolently [? · GW] by self-interested or destructive (human) actors to gain a decisive strategic advantage? After all, it sure seems less than ideal to find yourself in a position where you can solve the theoretical parts of value alignment,^[1] but you cannot implement that in practice because control over the entire future light cone has already been permanently taken over by an AGI intent-aligned to someone who does not care about any of your broadly prosocial goals...

1. ^{^}

   In so far as something like this even makes sense, which I have already expressed [LW(p) · GW(p)] my skepticism of many times [LW(p) · GW(p)], but I don't think I particularly want to rehash this discussion with you right now...

This is IMO the type of experiment that I like, even though it is evidence against my thesis on AI alignment generalization, and if done more could be a much more productive method of settling the current disputes that currently exists in the AI safety field.

I will also say that at the moment, it's looking like a lot of the proposed alignment methods are unlikely to be adversarially robust, which while mostly fine for an alignment proposal in a vacuum, is going to require rather extreme info-security to make the plan work out in practice without unleashing extinction or worse.

I don't go as far as Zach Stein Perlman here, but I agree with the broad take that most plans for controlling/aligning AIs rely fairly critically on the model not being stolen by other parties:

https://www.lesswrong.com/posts/eq2aJt8ZqMaGhBu3r/zach-stein-perlman-s-shortform#ckNQKZf8RxeuZRrGH [LW(p) · GW(p)]