Yes, the new reading is "Politics isn't a good place to practice rationality unless all the discussants are already rational". Not that you shouldn't engage in discussion of politics, just that you shouldn't train rationality there (when not already well practiced in other areas).

fwiw, I'd kind of like to see this book epistemically-spot-checked before building too much off it (I was chatting about it recently and some of the claims seemed iffy to me. It seemed like it should probably be able to identify easier-to-check claims and check to make sure he's at least getting obvious things right)

My understanding is that the last time people got really into Left Brain / Right Brain as a dichotomy it ended getting kinda pop-sci-simplified (which eventually resulted in it falling out of favor), and I'd like to "do it right this time" if it's going to be a thing people are building theories and models around.

My thoughts after the first cycle: I found listing my bugs, Yoda Timers, and TAPs very helpful. The later days were interesting to read, but they did not help me solve many of my bugs (time calibration is a very useful skill, but I was already pretty good at it, at least for the short projects I can get quick feedback on). I think my largest benefits from reading this came from setting aside time to fix various things I had been procrastinating on, and motivating me to actually start doing things I had been planning to do for a while.

The counter-examples are of that type because the examples are often of that type - presented formally, so vulnerable to a formal solution.

It does not seem to me the cluster of concepts in corrigibility, Clarifying AI Alignment, and my comment on it are presented formally. They feel very, very informal (to the point that I think we should try to make them more formal, though I'm not optimistic about getting them to the level of formality you typically use).

(I still need to get a handle on ascription universality, which might be making these concepts more formal, but from what I understand of it so far it's still much less formal than you usually work with.)

we need to define what a "reasonable" utility function

My argument is that we don't need to define this formally; we can reason about it informally and still get justified confidence that we will get good outcomes, though not justified confidence in < 1-in-a-billion chance of failure.

Thanks. I've added a newline for the first one, and just deleted the entire paragraph about the branch naming because we never really stuck to it.

Ian McGilchrist came out a book on brain hemispheres and their specialized roles called The Master and His Emissary. This summary was useful: https://www.reddit.com/r/streamentry/comments/b39n4x/the_divided_brain_and_awakening_theorycommunity/

The Left Hemisphere handles narrow focus (like a bird trying to pick out a seed among a bunch of pebbles and dirt), while the Right Hemisphere handles broad, open focus (the same bird keeping some attention on the background for predators). The LH is associated with tool use and manipulation of objects. The RH is associated with exploration and experiential data gathering.

I don't immediately know how the hemispheres may be involved in the types of Curiosity. But a plausible hypothesis might be that Active Curiosity would be more left-brained and Open Curiosity would be more right-brained.

3 years ago I created a map of different ideas about possible AI failures (LW-post, pdf). Recently I converted it into an article "Classification of global catastrophic risks connected with artificial intelligence". I think there is around 100 failure modes which we could imagine now, and obviously some unimaginable.

However, my classification looks different from the one above: it is a classification of external behaviours, not of internal failure modes. It start from the risks of AI which is below human level, like "narrow-AI viruses" or narrow AI used to create advance weapons, like biological weapons.

Then I look into different risks during AI takeoff and after it. The interesting ones are:

• AI kills human to make world simpler.
• Two AIs go into war
• AI blackmails humanity by a doomsday weapon to get what it needs.

Next is the difference between non-friendly AIs and failures of friendliness. For example, if AI wireheads everybody, it is failure of friendliness, as well as dangerous value learners.

Another source of failures is technical: that is bugs, accumulation of errors, conflicting subgoals and general problems related to complexity. AI's self-wireheading also belongs here. All this could result into unpredictable halting of AI-Singleton with catastrophic consequences for all humanity about which it now cares.

The last source of possible AI-halting is unresolvable philosophical problems, which effectively halt it. We could imagine several, but not all. Such problems are something like: unsolvable "meaning of life" (or "is-ought" problem) and the problem that the result of computation doesn't depend on AI's existence, so it can't prove to itself that it actually exist.

AI also could encounter more advance alien AI (or its signals) and fail its victim.

Correct. I suspect it's too broad a category, and imprecisely defined (or at least different from things commonly measured) to have much basis OTHER than analogies and anecdotes. Fortunately, it's also irrelevant to most of us - no action is proposed or expected as part of this discussion/debate.

[ edit: I'm embarassed to say that until now I hadn't even looked up the legality of it. In the US, blackmail is specifically about demanding payment for not informing of a violation of US law, not for any other topics of gossip. It's also a relatively minor offence. Many jurisdictions just treat it as a special case of extortion, which is severe, but unclear whether that's because it's often violent or because it's bad on it's own.]

This resonates with me quite a lot. I have been focused on trying to find out what I'm really good at and enjoy doing (like writing for Scott, or playing the piano for his brother) these past few years. I'm motivated both by getting into a position where I can do some good (either actively or passively, e.g. by earning to give) and which I enjoy so much that it doesn't feel like "work". So far, I still don't feel like I've really managed to find out, despite quite a bit of reading, discussions with friends, reflection, and switching jobs twice.

Then again, after reading up a bit on stoicism (very superficially, I have to admit), I get the impression that it might be at least as much about your inner attitude as much as it is about external circumstances. If one is able to truly "love their fate", there is a fairly good chance they will excel in a lot of different jobs/tasks/positions, I suppose. For me, that poses the question "How much energy should you dedicate to finding your "calling" versus how much energy you should invest in developing a mindset that makes you accept the current conditions, and work towards becoming excellent at what you do now?"

"Will Adam be able to get back to cycling within a month [after a recent accident]?"

(Probably unnecessary word of caution) do not forecast your own behavior due to risk of reduced agency.