LessWrong 2.0 Reader

Ok, so this sounds like it talks about cardinality in the sense of 1 or 3, rather than in the sense of 2. I guess I default to 2 because it's more intuitive due to the transfer property, but maybe 1 or 3 are more desirable due to being mathematically richer.

From Goldblatt, since {0..H} is internal.

From my personal perspective, I think Roam brought to attention a different type of software and vibe which were dominated SV type startups.  Through Roam, I've learned about all those fancy-old ideas and concepts. I guess I was coming from a more mainstream corner of the tech scene.

My excitement declined as the hype too, and as you describe, I couldn't understand its real benefits as I didn't have a strong enough reason to use it. 

Coming to Roam a few years later, after starting a publication, has made the difference as I'm less focused on [[double-bracketing]] but just taking notes, gradually and only when I'm feeling like it.

"Drink only water" is a good one. Depends if you think forgoing deliciousness, or other effects (alcohol, caffeine) counts as a tradeoff. 

Given you don't think forgoing the convenience/enjoyment of using your phone in bed counts as a tradeoff, I'm guessing this likely is the sort of thing you're looking for.

I expect (~ 75%) that the decision to "funnel" EAs into jobs at AI labs will become a contentious community issue in the next year. I think that over time more people will think it is a bad idea. This may have PR and funding consequences too.

I love this idea!

Some other books this could work for:

• The Ancestor's Tale
• The Art of Game Design
• The Anthropocene Reviewed
• The LessWrong review books 😉

Many textbooks have a few initial "core" chapters, and then otherwise a bunch of independent chapters on applications or assorted advanced topics.

(x-posted from the EA Forum [EA(p) · GW(p)])

We should expect that the incentives and culture for AI-focused companies to make them uniquely terrible for producing safe AGI. 

From a “safety from catastrophic risk” perspective, I suspect an “AI-focused company” (e.g. Anthropic, OpenAI, Mistral) is abstractly pretty close to the worst possible organizational structure for getting us towards AGI. I have two distinct but related reasons:

1. Incentives
2. Culture

From an incentives perspective, consider realistic alternative organizational structures to “AI-focused company” that nonetheless has enough firepower to host multibillion-dollar scientific/engineering projects:

1. As part of an intergovernmental effort (e.g. CERN’s Large Hadron Collider, the ISS)
2. As part of a governmental effort of a single country (e.g. Apollo Program, Manhattan Project, China’s Tiangong)
3. As part of a larger company (e.g. Google DeepMind, Meta AI)

In each of those cases, I claim that there are stronger (though still not ideal) organizational incentives to slow down, pause/stop, or roll back deployment if there is sufficient evidence or reason to believe that further development can result in major catastrophe. In contrast, an AI-focused company has every incentive to go ahead on AI when the cause for pausing is uncertain, and minimal incentive to stop or even take things slowly. 

From a culture perspective, I claim that without knowing any details of the specific companies, you should expect AI-focused companies to be more likely than plausible contenders to have the following cultural elements:

1. Ideological AGI Vision AI-focused companies may have a large contingent of “true believers” who are ideologically motivated to make AGI at all costs and
2. No Pre-existing Safety Culture AI-focused companies may have minimal or no strong “safety” culture where people deeply understand, have experience in, and are motivated by a desire to avoid catastrophic outcomes. 

The first one should be self-explanatory. The second one is a bit more complicated, but basically I think it’s hard to have a safety-focused culture just by “wanting it” hard enough in the abstract, or by talking a big game. Instead, institutions (relatively) have more of a safe & robust culture if they have previously suffered the (large) costs of not focusing enough on safety.

For example, engineers who aren’t software engineers understand fairly deep down that their mistakes can kill people, and that their predecessors’ fuck-up have indeed killed people (think bridges collapsing, airplanes falling, medicines not working, etc). Software engineers rarely have such experience.

Similarly, governmental institutions have institutional memories with the problems of major historical fuckups, in a way that new startups very much don’t.

Not just any set.

Almost any set: only the empty set is excluded. The identities of the elements themselves are irrelevant to the mathematical structure. Any further restrictions are not a part of mathematical definition of probability space, but some particular application you may have in mind.

If elementary event {A} has P(A) = 0, then we can simply not include outcome A into the sample space for simplicity sake.

In some cases this is reasonable, but in others it is impossible. For example, when defining continuous probability distributions you can't eliminate sample set elements having measure zero or you will be left with the empty set.

There is a potential source of confusion in the "credence" category. Either you mean it as a synonym for probability, and then it follows all the properties of probability, including the fact that it can only measure formally defined events from the event space, which have stable truth value during an iteration of probability experiment.

It is a synonym for probability in the sense that it is a mathematical probability: that is, a measure over a sigma-algebra for which the axioms of a probability space are satisfied. I use a different term here to denote this application of the mathematical concept to a particular real-world purpose. Beside which, the Sleeping Beauty problem explicit uses the word.

I also don't quite know what you mean by the phrase "stable truth value". As defined, a universe state either satisfies or does not satisfy a proposition. If you're referring to propositions that may vary over space or time, then when modelling a given situation you have two choices: either restrict the universe states in your set to locations or time regions over which all selected propositions have a definite truth value, or restrict the propositions to those that have a definite truth value over the selected universe states. Either way works.

Semantic statement "Today is Monday" is not a well-defined event in the Sleeping Beauty problem.

Of course it is. I described the structure under which it is, and you can verify that it does in fact satisfy the axioms of a probability space. As you're looking for a crux, this is probably it.

Universe states can be distinguished by time information, and in problems like this where time is part of the world-model, they should be. The mathematical structure of a probability space has nothing to do with it, as the mathematical formalism doesn't care what the elements of a sample space are.

Otherwise you can't model even a non-coin flip form of the Sleeping Beauty problem in which Beauty is always awoken twice. If the problem asks "what should be Beauty's credence that it is Monday" then you can't even model the question without distinguishing universe states by time.

I lean heavily to #5.  On very narrow topics, measurement and rationality tends to win, and there's not actually much debate about it.  There really aren't many ideological rocket launches, for instance.  But once you start talking about human behavior or any other complex or game-theoretic situations, it stops being legible and tractable for logic to really do much good without a whole lot of modeling choices being involved.

Oh, I was making a joke about timelines.