LessWrong 2.0 Reader

I haven't, no. I really wish I could somehow investigate all 3 pillars of a good watermark (Decisiveness, Invisibility, Robustness), but I couldn't think of any way to quantify a general text watermark's invisibility. For any given watermark you can technically rate "how invisible it is" by using an LLM's loss function to see how different the watermarked text is from the original text, but I can't come up with a way to generalize this.
So unfortunately my analysis was only about the interplay between decisiveness and robustness.

Here is another more narrow way to put this argument:

• Let's say Nate is 35 (arbitrary guess).
• Let's say that branches which deviated 35 years ago would pay for our branch (and other branches in our reference class). The case for this is that many people are over 50 (thus existing in both branches), and care about deviated versions of themselves and their children etc. Probably the discount relative to zero deviation is less than 10x.
• Let's say that Nate thinks that if he didn't ever exist P(takeover) would go up by 1 / 10 billion (roughly 2^-32). If it was wildly lower than this, that would be somewhat surprising and might suggest different actions.
• Nate existing is sensitive to a bit of quantum randomness 35 years ago, so other people as good as Nate existing could be created with a bit of quantum randomness. So, 1 bit of randomness can reduce risk by at least 1 / 10 billion.
• Thus, 75 bits of randomness presumably reduces risk by > 1 / 10 billion which is >> 2^-75.

What about these case where we only go back 10 years? We can apply the same argument, but instead just use some number of bits (e.g. 10) to make Nate work a bit more, say 1 week of additional work via changing whether Nate ends up getting sick (by adjusting the weather or which children are born, or whatever). This should also reduce doom by 1 week / (52 weeks/year) / (20 years/duration of work) * 1 / 10 billion = 1 / 10 trillion.

And surely there are more efficient schemes.

I mean that if turing machine is computing universe according to the laws of quantum mechanics,

I assume you mean the laws of QM except the collapse postulate.

observers in such universe would be distributed uniformly,

Not at all. The problem is that their observations would mostly not be in a classical basis.

not by Born probability.

Born probability relates to observations, not observers.

So you either need some modification to current physics, such as mangled worlds,

Or collapse. Mangled worlds is kind of a nothing burger--its a variation on the idea than interference between superposed states leads to both a classical basi and the Born probabilities, which is an old idea, but wihtout making it any more quantiative.

or you can postulate that Born probabilities are truly random.

??

Catholicism never would have collected the intelligence necessary to invent a nuke. Their worldview was not compatible with science. It was an inferior organizing principle. ("inferior" meaning less capable of coordinating a collective intelligence needed to build nukes.)

You believe intelligence is such a high good, a high virtue, that it would be hard for you to see how intelligence is deeply and intricately causal with the destruction of life on this planet, and therefore the less intelligent, less destructive religions actually have more ethical ground to stand on, even though they were still fairly corrupt. 

But it's a straightforward comparison. 

Medieval "dark" ages = almost no technological progress, very little risk of blowing up the planet in any way; relatively, not inspiring, but still - kudos for keeping us from hurtling toward extinction, and at this point, we're fine with rewarding this even though it's such a "low bar"

Today = massive, exponential technological progress, nuclear war could already take us all out, but we have a number of other x-risks to worry about. And we're so identified with science and tech that we aren't willing to stop, even as we admit OUT LOUD that it could cause extinction-level catastrophe. This is worse than the Crusades by a long shot. We're not talking about sending children to war. We're talking about the end of children. Just no more children. This is worse than suicide cults that claim we go to heaven as long as we commit suicide. We don't even think what we're doing will necessarily result in heaven, and we do it anyway. We have no evidence we can upload consciousnesses at all. Or end aging and death. Or build a friendly AI. At least the Catholics were convinced a very good thing would happen by sending kids to war. We're not even convinced, and we are willing to risk the lives of all children. Do you see how this is worse than the Catholics? 

My expectation is that it's the same reason there was so much outcry about the completely toothless Executive Order. To quote myself: [LW(p) · GW(p)]

The EO might be a bit more important than seems at a glance. My sense is that the main thing it does isn't its object-level demands, but the fact that it introduces concepts like "AI models" and "model weights" and "compute thresholds" and "datacenters suitable for training runs" and so on into the framework of the legislation.

That it doesn't do much with these variables is a secondary matter. What's important is that it defines them at all, which should considerably lower the bar for defining further functions on these variables, i. e. new laws and regulations.

I think our circles may be greatly underappreciating this factor, accustomed as we are to thinking in such terms. But to me, at least, it seems a bit unreal to see actual government documents talking about "foundation models" in terms of how many "floating-point operations" are used to "train" them.

Coming up with a new fire safety standard for restaurants and passing it is relatively easy if you already have a lot of legislation talking about "restaurants" — if "a restaurant" is a familiar concept to the politicians, if there's extant infrastructure for tracking the existence of "restaurants" nationwide, etc. It is much harder if your new standard needs to front-load the explanation of what the hell a "restaurant" even is.

By analogy, it's not unlike academic publishing, where any conclusion (even the extremely obvious ones) that isn't yet part of some paper can't be referred to.

Similarly, SB 1047 would've introduced a foundational framework on which other regulations could've later been built. Keeping the-government-as-a-system unable to even comprehend the potential harms AGI Labs could unleash is a goal in itself.

Thesis: while consciousness isn't literally epiphenomenal, it is approximately epiphenomenal. One way to think of this is that your output bandwidth is much lower than your input bandwidth. Another way to think of this is the prevalence of akrasia, where your conscious mind actually doesn't have full control over your behavior. On a practical level, the ecological reason for this is that it's easier to build a general mind and then use whatever parts of the mind that are useful than to narrow down the mind to only work with a small slice of possibilities. This is quite analogous to how we probably use LLMs for a much narrower set of tasks than what they were trained for.

• If I understand correctly, it's that grants by NSF to institutions have to have a section explaining their DEI initiatives. 

Grants to individual researchers now also require the DEI statement. I think the AI is not a very good analogy because:

• it's apolitical
• I assume that in your example historians and cancer researchers would not lose their jobs for saying heretical things in their AI statements.

A more appropriate analogy would be abortion. Suppose the progressive admins in the NSF are replaced with Christian conservatives who replace the DEI statement with the statement on the “sanctity of life”. To get grants, all researchers must demonstrate how their work is relevant to the fight against the murder of unborn children. Those who refuse or argue in favor of pro-choice policies do not get funding and, after a few years, are forced to quit academic institutions. Would you consider this policy a restriction on the freedom of speech and conscience or just a shift in government priorities?

It would probably be more constructive if we focus on one issue at a time, so I suggest we finish this topic before discussing others.

possible worlds that split off when the photon was created

I don't think this is a very good way of thinking about what happens. I think worlds appear as fairly robust features of the wavefunction when quantum superpositions get entangled with large systems that differ in lots of degrees of freedom based on the state of the superposition.

So, when the intergalactic photon interacts non-trivially with a large system (e.g. Earth), a world becomes distinct in the wavefunction, because there's a lump of amplitude that is separated from other lumps of amplitude by distance in many, many dimensions. This means it basically doesn't interact with the rest of the wavefunction, and so looks like a distinct world.

https://mason.gmu.edu/~rhanson/mangledworlds.html

I mean that if turing machine is computing universe according to the laws of quantum mechanics, observers in such universe would be distributed uniformly, not by Born probability. So you either need some modification to current physics, such as mangled worlds, or you can postulate that Born probabilities are truly random.

This paper seems (to me) to do a very good job of laying out its assumptions and reasoning based on them. For that, I applaud it, and I'm very glad it exists. I specifically disagree with several of the key assumptions, like the idea that "human level" is a "wide 'middle' of the range of AI capabilities." I very much appreciate that it shows that even granted no decisive strategic advantage there is strong reason to believe the usual mechanisms that reduce and limit war among humans mostly won't apply.

At some point I'd like to see similar reasoning applied to the full decision tree of assumptions. I think most of the assumption are unnecessary in the sense that all the plausible options for that node lead to some form of "Humans most likely lose badly." I don't think most people would read such a thing, or properly understand it, but you can show them the big graph at the end that shows something like, "Out of 8192 paths AGI development could take, all but 3 lead to extinction or near-extinction, and the only one of those paths where we have control of which outcome we get is to not build AGI."