LessWrong 2.0 Reader

I keep hearing about dual-use risk concerns when I mention automated AI safety research. Here’s a simple solution that could even work in a startup setting:

Keep all of the infrastructure internally and only share with vetted partners/researchers.

You can hit two birds with one stone:

• Does not turn into a mass-market product that leads to dual-use risks.
• Builds a moat where you have complex internal infrastructure which is not shared, only the product of that system is shared. Investors love moats, you just got to convince them that this is the way to go for a product like this these days.

You don’t market the product to mass-market, you just find partners and use the system to spin out products and businesses that have nothing to do with frontier models. So, you can repurpose the system for specific application areas without releasing the platform and process, which would be copied in a day in the age of AI anyways.

Isn't this what the "coherent" part is about? (I forget.)

I think there's a strong motivation to believe in hell for other people. The wicked flourish like the green bay tree, and where is justice?

Alternatively, belief in hell for other people is mere spitefulness.

Also, I believe inventing the tortures of hell is very like the same drive that causes people to write horror fiction, though I have no idea of why they do it, or why I like horror fiction.

It's evidence that God loves complexity even more than He loves beetles.

Is there really some particular human whose volition you'd like to coherently extrapolate over eternity but where you refrain because you're worried it will generate infighting? Or is it more like, you can't think of anybody you'd pick, so you want a decision procedure to pick for you?

If there is some particular human, who is it?

Thanks for raising this important point. When modeling these situations carefully, we need to give terms like "today" a precise semantics that's well-defined for the agent. With proper semantics established, we can examine what credences make sense under different ways of handling indexicals. Matthias Hild's paper "Auto-epistemology and updating" demonstrates how to carefully construct time-indexed probability updates. We could then add centered worlds or other approaches for self-locating probabilities.

Some cases might lead to puzzles, particularly where epistemic fixed points don't exist. This might push us toward modeling credences differently or finding other solutions. But once we properly formalize "today" as an event, we can work on satisfying richness conditions. Whether this leads to inconsistent attitudes depends on what constraints we place on those attitudes - something that reasonable people might disagree about, as debates over sleeping beauty suggest.
 

Pasek did couchsurf at my place in the days after a LessWrong Community Weekend in Berlin. That was before he went to the Bay Area, so probably 8 or 9 years ago and before he seemed to make contact with Ziz which was after Pazek left the Bay Area and moved to live with other rationalists in a group house in Gran Canaria. Pazek's contact with Ziz seemed to be mostly online while living in Gran Canaria.

If you read Pasek's post where he thinks about committing suicide, there's plenty of TDT-thinking in it. I matched my idea of how Pasek thinks even before engaging with Ziz.

Pasek was TDT-ish vegan. 

Pasek had some QS tracking for how he spent every waking hour of the day that he did on paper and seemed to not suffer from akrasia while guiding his actions. 

If I remember right, that he said that stealing is okay in cases where the TDT calculation would be in favor of stealing where traditional morality would say stealing is bad. I don't think that resulted in Pasek actually stealing things but I think we talked about some case where he thought it was justified to steal which surprised me at the time. My memory is here very fussy.

I dislike raw oysters quite a bit, but they're okay cooked.

Speaking of logical fallacies, the fact that one person loves a thing means that other people will even tolerate it is not strongly likely. I don't know that people even have an obligation to try things other people love.

And yet, the temptation to think that other people do or should love what one loves it very strong. "I think this is great!" just doesn't feel as true as "This is great!".

I'm not sure if I fully understand why this is supposed to pose a problem, but maybe it helps to say that by "meaningfully consider" we mean something like, is actually part of the agent's theory of the world. In your situation, since the agent is considering which envelope to take, I would guess that to satisfy richness she should have a credence in the proposition. 

Okay, then I believe you definetely have a problem with this example and would be glad to show you where exactly.

I think (maybe?) what makes this case tricky or counterintuitive is that the agent seems to lack any basis for forming beliefs about which envelope contains the money - their memory is erased each time and the location depends on their previous (now forgotten) choice.

However, this doesn't mean they can't or don't have credences about the envelope contents. From the agent's subjective perspective upon waking, they might assign 0.5 credence to each envelope containing the money, reasoning that they have no information to favor either envelope.

Let's suppose that the agent does exactly that. Suppose they believe that on every awakening there is 50% chance that money is in envelope 1. Then picking envelope 1 every time will in expectation lead to winning 350$ per experiment.

But this is clearly false. The experiment is specifically designed in such a manner that the agent can win money only on the first awakening. On every other day (6 times out of 7) the money would be in the envelope 2. 

So should the agent believe that there is only 1/7 chance that money are in envelope 1 then? Also no. I suppose you can see why. As soon as he tries to act on such belief it will turn out that 6 times out of 7 the money are in envelope 1.

In fact, we can notice, that there is no coherent value of credence for statement "Today the money are in envelope 1" that would not lead the agent to irrational behavior. This is because the term "Today" is not well-defined in the setting of such experiment. 

By which I mean that in the same iteration of the experiment propositions including "Today" may not have a unique value. On the first day of the experiment statement "Today money are in envelope 1" may be true, while on the second day it may be false, so in the single iteration of the experiment that lasts 7 days the statement is simultaneously true and false!

Which means that "Today money are in envelope 1" isn't actually an event from the event space of the experiment and therefore doesn't have a probability value, as probability function's domain is event space.

But this is a nuance of formal probability theory that most people do not notice, or even try to ignore outright. Our intuitions are accustoimed to situations where statements about "Today" can be represented as well-defined events from the event space and therefore we assume that they can always be "meaningfully considered". 

And so if you try to base you decision theory framework on what feels as meaningfull to an agent instead of what is formalizable mathematically, you will end up with a bunch of paradoxical situations, like the one I've just described.

Ah, so not like, A is strongly preferred to B and B is strongly preferred to A, but more of a violation of transitivity. Then I still think that the Broome paper is a place I'd look at, since you get that exact kind of structure in preference aggregation. 

The Bradley paper assumes everything is transitive throughout, so I don't think you get the kind of structure you want there. I'm not immediately aware of any work of that kind of inconsistency in JB that isn't in the social choice context, but there might be some. I'll take a look. 

There are ways to think about degrees and measures of incoherence, and how that connects up to decision making. I'm thinking mainly of this paper by Schervish, Seidenfeld, and Kadane, Measures of Incoherence: How Not to Gamble if You Must. There might a JB-style version of that kind of work, and if there isn't, I think it would be good to have one. 

But to your core goal or weakening the preference axioms to more realistic standards, you can definitely do that in JB by weakening the preference axioms, but still keeping the background objects of preference be propositions in a single algebra. I think this would still preserve many of what I consider the naturalistic advantages of the JB system. For modifying the preference axioms, I would guess descriptively you might want something like prospect theory, or something else along those broad lines. Also depends on what kinds of agents we want to describe.