LessWrong 2.0 Reader

So in this position, you are correct that my arguments aren't strong enough for that, because while there is quite a bit of evidence for my thesis, it's not overwhelmingly so (though I do think that the evidence is enough to rule out MIRI's models of AI doom entirely).

However, there is one important implication, though, that has to do with AI control, and that's about which models can be safely used, and my answer here is that the ones trained on more human values data are better used as trusted AIs to monitor untrusted AIs, and this both raises the work that can be done under control by raising the threshold of intelligence that can be used, and also allows you to have stronger guarantees because there is at least 1 trusted superintelligent model.

While I agree my AI alignment arguments aren't strong enough to create an AGI, I think that the emerging AI control agenda is enough to justify creating an AGI with dangerous capabilities as long as you use control measures.

1. My current system

I check a couple of sources most days, at random times during the afternoon or evening. I usually do this on my phone, during breaks or when I'm otherwise AFK. My phone and laptop are configured to block most of these sources during the morning.

When I find something I want to engage with at length, I usually put it into my "Reading inbox" note in Obsidian, or into my weekly todo list if it's above the bar.

I check my reading inbox on evenings and weekends, and also during "open" blocks that I sometimes schedule as part of my work week. 

I read about 1/5 of the items that get into my reading inbox, either on my laptop or iPad. I read and annotate using PDF Expert, take notes in Obsidian, and use Mochi for flashcards. My reading inbox—and all my articles, highlights and notes—are synced between my laptop and my iPad.

2. Most useful sources

(~Daily)

• AI News (usually just to the end of the "Twitter recap" section). 
• Private Slack and WhatsApp groups.
• Twitter (usually just the home screen, sometimes my lists).
• Marginal Revolution.
• LessWrong and EA Forum (via the 30+ karma podcast feeds; I rarely check the homepages)

(~Weekly)

• Newsletters: Zvi, CAIS.
• Podcasts: The Cognitive Revolution, AXRP, Machine Learning Street Talk, Dwarkesh.

3. Problems

I've not given the top of the funnel—the checking sources bit—much thought. In particular, I've never sat down for an afternoon to ask questions like "why, exactly, do I try to follow AI news?", "what are the main ways this is valuable (and disvaluable)?" and "how could I make it easy to do this better?". There's probably a bunch of low-hanging fruit here.

Twitter is... twitter. I currently check the "For you" home screen every day. At least once a week I'm very glad that I checked Twitter—because I found something useful, that I plausibly wouldn't have found otherwise. But—I wish I had a good way to see just the best AI stuff, and then do the "having fun on Twitter" thing separately. In the past I tried to figure something out with Twitter lists and Tweetdeck (now "X Pro"), but I've not found something that sticks. So I spend most of my time with the "For you" screen, training the algorithm with "not interested" reports, an aggressive follow/unfollow/block policy, and liberal use of the "mute words" function. I'm sure I can do better...

My newsletter inbox is a mess. I filter newsletters into a separate folder, so that they don't distract me when I process my regular email. But I'm subscribed to way too many newsletters, many of which aren't focussed on AI, so when I do open the "Newsletters" folder, it's overwhelming. I don't reliably read the sources which I flagged above, even though I consider them fairly essential reading (and would prefer to read them to many of the things I do, in fact, read). 

I addictively over-consume podcasts, at the cost of "shower time" (diffuse/daydream mode). 

I don't make the most of LLMs. I have various ideas for how LLMs could improve my information discovery and consumption, but on my current setup—especially on mobile—the affordances for using LLMs are poor.

I miss things that I'd really like to know about. I very rarely miss a "big story", but I'd guess I miss several things that I'd really like to know about each week, given my particular interests.

I find out about many things I don't need to know about (or find out sooner than I need to).

I could go on...

Hubinger

I do not see how those are in conflict? Indeed, a core responsibility of being a good collaborator and IMO also to be a decision maker in EA is to make ethical choices even if they are socially difficult.

Yes, it will be judged negatively by alien ASIs, not based on ethical grounds, but based on their judgment of its trustworthiness as a potential negotiator. For example, if another billionaire learns that Arnault is inclined to betray people who did a lot of good for him in the past, they will be more cautious about trading with him.

The only way an ASI will not care about this is in a situation where it is sure that it is alone in the light cone and there are no peers. To become sure of this takes time, maybe millions of years, and the relative value of human atoms declines for the ASI over time as it will control more and more space.

+1. Concretely this means converting every probability p into p/(1-p), and then multiplying those (you can then convert back to probabilities)

Intuition pump: Person A says 0.1 and Person B says 0.9. This is symmetric, if we instead study the negation, they swap places, so any reasonable aggregation should give 0.5

Geometric mean does not, instead you get 0.3

Arithmetic gets 0.5, but is bad for the other reasons you noted

Geometric mean of odds is sqrt(1/9 * 9) = 1, which maps to a probability of 0.5, while also eg treating low probabilities fairly

Bernard Arnault?

If you parse this post as "attempting to impart a basic intuition that might let people (new to AI x-risk arguments) avoid certain classes of errors" rather than "trying to argue with the bleeding-edge arguments on x-risk", this post seems good

This seems reasonable in isolation, but it gets frustrating when the former is all Eliezer seems to do these days, with seemingly no attempt at the latter. When all you do is retread these dunks on "midwits" and show apathy/contempt for engaging with newer argumens, it makes it look like you don't actually have an interest in being maximally truth-seeking but instead like you want to just dig in and grandstand.

From what little engagement there is with novel criticisms of their arguments (like Nate's attempt to respond to Quintin/Nora's work), it seems like there's a cluster of people here who don't understand and don't particularly care about understanding some objections to their ideas and instead want to just focus on relitigating arguments they know they can win.

no longer possible

did you mean "no longer impossible"?

"2600:1f18:17c:2d43:338d:2669:3fa5:82f8" is an IPv6 address, which one reverse lookup site maps to an Amazon AWS server in Ashburn, Virginia. There could be anything on that machine, and no-one to connect it with. But the URL given does not work in my web browser or in curl. Looks sketchy to me.