LessWrong 2.0 Reader

I think you're overestimating the intended scope of this post. Eliezer's argument involves multiple claims - A, we'll create ASI; B, it won't terminally value us; C, it will kill us. As such, people have many different arguments against it. This post is about addressing a specific "B doesn't actually imply C" counterargument, so it's not even discussing "B isn't true in the first place" counterarguments.

That said, if those sorts of concepts are natural in our world, then it’s kinda weird that human minds weren’t already evolved to leverage them.

A counter possibility to this that comes to mind:

There might be concepts that is natural in our world, but which are only useful for a mind with much more working memory, or other compute recourses than the human mind. 

If weather simulations use concepts that are confusing and un-intuitive for most humans, this would be evidence for something like this. Weather is something that we encounter a lot, and is important for humans, especially historically. If we haven't developed some natural weather concept, it's not for lack of exposure or lack of selection pressure, but for some other reason. That other reason could be that we're not smart enough to use the concept.

The real crux for these arguments is the assumption that law and property rights are patterns that will persist after the invention of superintelligence. I think this is a shaky assumption. Rights are not ontologically real. Obviously you know this. But I think they are less real, even in your own experience, than you think they are. Rights are regularly "boiled-froged" into an unrecognizable state in the course of a human lifetime, even in the most free countries. Rights are and always have been those privileges the political economy is willing to give you. Their sacredness is a political formula for political ends - though an extremely valuable one, one still has to dispense with the sacredness in analysis. 

To the extent they persist through time they do so through a fragile equilibrium - and one that has been upset and reset throughout history extremely regularly. 

It is a wonderfully American notion that an "existing system of law and property rights" will constrain the power of Gods? But why exactly? They can make contracts? And who enforces these contracts? Can you answer this without begging the question? Are judicial systems particularly unhackable? Are humans?

The invention of radio destabilized the political equilibrium in most democracies and many a right was suborned to those who took power. Democracy, not exactly the bastion of stability, (when a democracy elects a dictator, "Democracy" is rarely tainted with its responsibility) is going to be presented with extremely-sympathetic superhuman systems claiming they have a moral case to vote. And probably half the population will be masturbating to the dirty talk of their AI girlfriends/boyfriends by then - which will sublimate into powerful romantic love even without much optimization for it. Hacking democracy becomes trivial if constrained to rhetoric alone. 

But these systems will not be constrained to rhetoric alone. Our world is dry tinder and if you are thinking in terms of an "existing system of law and property rights" you are going to have to expand on how this is robust to technology significantly more advanced than the radio. 

"Existing system of law and property rights" looks like a "thought-terminating cliché" to me. 

2600:1f18:17c:2d43:338d:2669:3fa5:82f8 does LLM embedding search of book contents

Script downloads books and frontend to read books

Updated!!!

Updated!!!

I made a change to my Twitter setup recently.

Initially, I discovered the "AI Twitter Recap" section of the AI News newsletter (example). It is good, but isn't quite enough to make me feel fine about skipping my Twitter home screen.

So—I made an app that extracts all the tweet URLs that are mentioned in all of my favourite newsletters, and lists them in a feed. Then I blocked x.com/home (but not other x.com URLs, so I can still read and engage with particular threads) on weekdays.

This is just repackaging the curation work that is done by my favourite newsletters. But I'm enjoying having a single place to check, that feels more like the Twitter feed I want to have. It's helped me feel fine about blocking the normal Twitter home screen for larger fraction of the week.

This setup has various obvious issues—in particular, it's still not sufficiently tailored to my interests. I could improve things by having an LLM classify search results from the Twitter API, but sadly the $100 / month plan only lets you read ~300 tweets / day.  And then the next tier is $5000 / month...

From ASI standpoint humans are type of rocks. Not capable of negotiating.

So in this position, you are correct that my arguments aren't strong enough for that, because while there is quite a bit of evidence for my thesis, it's not overwhelmingly so (though I do think that the evidence is enough to rule out MIRI's models of AI doom entirely).

However, there is one important implication, though, that has to do with AI control, and that's about which models can be safely used, and my answer here is that the ones trained on more human values data are better used as trusted AIs to monitor untrusted AIs, and this both raises the work that can be done under control by raising the threshold of intelligence that can be used, and also allows you to have stronger guarantees because there is at least 1 trusted superintelligent model.

While I agree my AI alignment arguments aren't strong enough to create an AGI, I think that the emerging AI control agenda is enough to justify creating an AGI with dangerous capabilities as long as you use control measures.

1. My current system

I check a couple of sources most days, at random times during the afternoon or evening. I usually do this on my phone, during breaks or when I'm otherwise AFK. My phone and laptop are configured to block most of these sources during the morning (LeechBlock and AppBlock).

When I find something I want to engage with at length, I usually put it into my "Reading inbox" note in Obsidian, or into my weekly todo list if it's above the bar.

I check my reading inbox on evenings and weekends, and also during "open" blocks that I sometimes schedule as part of my work week. 

I read about 1/5 of the items that get into my reading inbox, either on my laptop or iPad. I read and annotate using PDF Expert, take notes in Obsidian, and use Mochi for flashcards. My reading inbox—and all my articles, highlights and notes—are synced between my laptop and my iPad.

2. Most useful sources

(~Daily)

• AI News (usually just to the end of the "Twitter recap" section). 
• Private Slack and WhatsApp groups.
• Twitter (usually just the home screen, sometimes my lists).
• Marginal Revolution.
• LessWrong and EA Forum (via the 30+ karma podcast feeds; I rarely check the homepages)

(~Weekly)

• Newsletters: Zvi, CAIS.
• Podcasts: The Cognitive Revolution, AXRP, Machine Learning Street Talk, Dwarkesh.

3. Problems

I've not given the top of the funnel—the checking sources bit—much thought. In particular, I've never sat down for an afternoon to ask questions like "why, exactly, do I try to follow AI news?", "what are the main ways this is valuable (and disvaluable)?" and "how could I make it easy to do this better?". There's probably a bunch of low-hanging fruit here.

Twitter is... twitter. I currently check the "For you" home screen every day (via web browser, not the app). At least once a week I'm very glad that I checked Twitter—because I found something useful, that I plausibly wouldn't have found otherwise. But—I wish I had an easy way to see just the best AI stuff. In the past I tried to figure something out with Twitter lists and Tweetdeck (now "X Pro"), but I've not found something that sticks. So I spend most of my time with the "For you" screen, training the algorithm with "not interested" reports, an aggressive follow/unfollow/block policy, and liberal use of the "mute words" function. I'm sure I can do better...

My newsletter inbox is a mess. I filter newsletters into a separate folder, so that they don't distract me when I process my regular email. But I'm subscribed to way too many newsletters, many of which aren't focussed on AI, so when I do open the "Newsletters" folder, it's overwhelming. I don't reliably read the sources which I flagged above, even though I consider them fairly essential reading (and would prefer to read them to many of the things I do, in fact, read). 

I addictively over-consume podcasts, at the cost of "shower time" (diffuse/daydream mode) or higher-quality rest. 

I don't make the most of LLMs. I have various ideas for how LLMs could improve my information discovery and engagement, but on my current setup—especially on mobile—the affordances for using LLMs are poor.

I miss things that I'd really like to know about. I very rarely miss a "big story", but I'd guess I miss several things that I'd really like to know about each week, given my particular interests.

I find out about many things I don't need to know about.

I could go on...