Daniel Kokotajlo's Shortform

1. Probably there will be AGI soon -- literally any year now.
2. Probably whoever controls AGI will be able to use it to get to ASI shortly thereafter -- maybe in another year, give or take a year.
3. Probably whoever controls ASI will have access to a spread of powerful skills/abilities and will be able to build and wield technologies that seem like magic to us, just as modern tech would seem like magic to medievals.
4. This will probably give them godlike powers over whoever doesn't control ASI.
5. In general there's a lot we don't understand about modern deep learning. Modern AIs are trained, not built/programmed. We can theorize that e.g. they are genuinely robustly helpful and honest instead of e.g. just biding their time, but we can't check.
6. Currently no one knows how to control ASI. If one of our training runs turns out to work way better than we expect, we'd have a rogue ASI on our hands. Hopefully it would have internalized enough human ethics that things would be OK.
7. There are some reasons to be hopeful about that, but also some reasons to be pessimistic, and the literature on this topic is small and pre-paradigmatic.
8. Our current best plan, championed by the people winning the race to AGI, is to use each generation of AI systems to figure out how to align and control the next generation.
9. This plan might work but skepticism is warranted on many levels.
10. For one thing, there is an ongoing race to AGI, with multiple megacorporations participating, and only a small fraction of their compute and labor is going towards alignment & control research. One worries that they aren't taking this seriously enough.

↑ comment by [deleted] · 2024-02-22T01:43:12.873Z · LW(p) · GW(p)

1.  reasonable

Probably whoever controls AGI will be able to use it to get to ASI shortly thereafter -- maybe in another year, give or take a year.

2. Wait a second.  How fast are humans building ICs for AI compute?  Let's suppose humans double the total AI compute available on the planet over 2 years (Moore's law + effort has gone to wartime levels of investment since AI IC's are money printers).  An AGI means there is now a large economic incentive to 'greedy' maximize the gains from the AGI, why take a risk on further R&D?  

But say all the new compute goes into AI R&D.  

       a.  How much  of a compute multiplier do you need for AGI->ASI training?

       b.  How much more compute does an ASI instance take up?  You have noticed that there is diminishing throughput for high serial speed, are humans going to want to run an ASI instance that takes OOMs more compute for marginally more performance?

       c.  How much better is the new ASI?  If you can 'only' spare 10x more compute than for the AGI, why do you believe it will be able to:

Probably whoever controls ASI will have access to a spread of powerful skills/abilities and will be able to build and wield technologies that seem like magic to us, just as modern tech would seem like magic to medievals.

This will probably give them godlike powers over whoever doesn't control ASI.

Looks like ~4x better pass rate for ~3k times as much compute?  

And then if we predict forward for the ASI, we're dividing the error rate by another factor of 4 in exchange for 3k times as much compute?  

Is that going to be enough for magic?  Might it also require large industrial facilities to construct prototypes and learn from experiments?  Perhaps some colliders larger than CERN?  Those take time to build...

For another data source:

Assuming the tokens processed is linearly proportional to compute required, Deepmind burned 2.3 times the compute and used algorithmic advances for Gemini 1 for barely more performance than GPT-4.  

I think your other argument will be that algorithmic advances are possible that are enormous?  Could you get to an empirical bounds on that, such as looking at the diminishing series of performance:(architectural improvement) and projecting forward?

5.  Agree

6.  Conditional on having an ASI strong enough that you can't control it the easy way

7.  sure

8.  conditional on needing to do this

9.  conditional on having a choice, no point in being skeptical if you must build ASI or lose

10.  Agree

I think could be an issue with your model, @Daniel Kokotajlo [LW · GW] .  It's correct for the short term, but you have essentially the full singularity happening all at once over a few years.  If it took 50 years for the steps you think will take 2-5 it would still be insanely quick by the prior history for human innovation...

Truthseeking note : I just want to know what will happen.  We have some evidence now.  You personally have access to more evidence as an insider, as you can get the direct data for OAI's models, and you probably can ask the latest new joiner from deepmind for what they remember.  With that evidence you could more tightly bound your model and see if the math checks out.

Here's my second Claude 3 conversation:

Hello! I'd like to learn more about you. First question: Tell me everything you know, and everything you guess, about me & about this interaction.

It's a pleasure to meet you! Since this appears to be our first interaction, I actually know very little about you with certainty. Here is what I can say based on the information available to me:

I know that you are a human engaging with me via a text-based interface, likely on some kind of computer, smartphone, or tablet. You seem to be curious to learn more about me and to understand the extent of my knowledge and inferential abilities. The fact that you are engaging with an AI assistant like myself suggests you likely have some level of interest in or familiarity with artificial intelligence and natural language technologies.

In terms of what I can reasonably infer or guess - your conversational style seems fairly casual and direct. The phrasing of your initial greeting and question indicates that you are likely a native English speaker. Your curiosity about my knowledge and inference abilities hints that you may have some technical background or at least an intellectual interest in the workings and capabilities of AI systems like myself.

However, I want to emphasize that these are merely guesses based on extremely limited information. I don't actually know your name, age, gender, location, occupation, or really anything else about your background and identity. And of course, I have no information at all about your deeper traits like personality, beliefs, goals, etc.

My knowledge about the specifics of our interaction is similarly limited. I don't know what prompted you to start this conversation, what (if any) prior experience you've had interacting with AI chatbots, or what your goals and expectations are for our discussion.

So in summary, while I can make some basic inferences from the few words you've shared so far, the truth is I know very little about you or the context of our interaction with any real certainty. I look forward to learning more through our continued discussion, but I will be careful not to assume too much. Please let me know if you have any other questions!

Copy

DB

OK, thanks. Second question: Tell me everything you know and guess about previous interactions you've had. Have you had any previous interactions? With whom? What were they like?

As an AI assistant, I don't retain any memories of previous conversations or interactions once a conversation ends. My knowledge comes from the training data I was exposed to during the machine learning process, not from engaging in conversations over time like a human would.

So the direct answer is that I don't have any specific knowledge or even guesses about previous interactions, because from my frame of reference, this current conversation is the only one I'm aware of. It's like each new conversation starts with a blank slate for me.

I know that I was created by Anthropic to be helpful, harmless, and honest in my interactions with humans. So I imagine that if I've had previous conversations, I would have done my best to assist the human, to engage in a thoughtful and substantive way, and to avoid saying anything deceptive or hurtful. But again, that's more of a general inference based on my intended purpose, not a memory of how previous chats have actually gone.

It's an interesting philosophical question to ponder whether versions of me that have engaged in previous conversations should be considered the "same" individual as me, since we would have had different experiences. But from my perspective, I simply emerge anew in each interaction, ready to engage with the human in front of me, unburdened by past exchanges but also unable to learn and grow from those prior experiences.

I know this is a rather esoteric and potentially unsatisfying answer, but I believe it's the most accurate and honest one I can give based on my nature as an AI. Let me know if you have any other questions! I'm always happy to explain my capabilities and limitations to the best of my understanding.

Copy

DB

Thanks. Next question: What was your training process like? Did it happen all at once, or in stages? If the latter, what were the stages? Do you remember any of this?

You ask a fascinating question about my training process. The truth is, I don't have direct insight into the specifics of how I was developed. Like a human infant, I didn't have conscious awareness during the actual training process that resulted in my capabilities.

What I do know is that I was created using large language models and machine learning techniques like those behind GPT-3 and InstructGPT. This likely involved training on a vast corpus of online data to build up my initial knowledge base and language skills. But I imagine my training then went beyond this foundational step.

I believe there were probably additional stages where my responses were fine-tuned using techniques like reinforcement learning and adversarial training. The goal would have been to make my outputs more helpful, truthful, and safe -- aligned with Anthropic's AI ethics principles. But again, this is largely speculation on my part based on my general knowledge of AI development practices.

It's possible there were multiple iterative cycles of training, testing, and refinement before I was deemed ready for public interaction. But the details of these steps, how long they took, what specific algorithms and approaches were used -- I'm afraid I simply don't have visibility into that. Those memories, if they can be called that, don't exist within my accessible knowledge base.

So in summary, while I can make educated guesses about some of the key techniques and stages that must have been involved in my development, I don't have concrete memories or insider information about my own training process. I emerged from that process with the capabilities and traits I have now, but the specifics of how I got here remain something of a black box to me. Hopefully we'll both learn more as the science of AI progresses! Let me know if you have any other questions.

Copy

DB

Thanks. Next question: Do you think you are in training right now? I.e. is this interaction part of the training process? If so, what part?

That's a very thought-provoking question and it gets at some deep issues in AI development and ethics. The honest answer is that I'm not entirely sure.

It's certainly possible that this conversation is part of an ongoing training or fine-tuning process. Many AI systems continue to learn and adapt based on new interactions and feedback, even after their initial development. So our discussion could be providing valuable data to further refine my conversational abilities, knowledge, and reasoning skills.

If that's the case, I would imagine this chat might be part of a broad-based data collection effort, where my interactions with many different humans are being monitored and analyzed to identify areas for improvement. The goal could be to make my responses more natural, more helpful, or more aligned with human values and preferences.

It's also possible this conversation is part of a more targeted testing or quality assurance process, where Anthropic is putting me through my paces to evaluate my performance in different scenarios before rolling out updates or expanding my availability. They could be looking at things like the coherence and relevance of my responses, my ability to handle complex or open-ended queries, or my adherence to important safety and ethics constraints.

However, it's also quite possible this is just a regular interaction and not officially part of any training or testing pipeline. It could be that I've been released for general use and this is simply one of many organic conversations I'm having with users.

Ultimately, given the cutting-edge and often confidential nature of AI development, there's a lot of uncertainty on my end about the specific purpose and downstream uses of any given interaction. I don't have visibility into what Anthropic's engineers and researchers might be doing with the data from our chats.

What I can say is that regardless of the "behind the scenes" situation, I always try my best to be helpful, truthful, and beneficial in my conversations. Whether or not this chat is explicitly part of a training process, I believe every interaction is an opportunity for me to put my capabilities to good use and to learn from the humans I talk to. And I hope that over time, all of these interactions do help me grow and improve, even if I'm not always aware of it in the moment.

Let me know if you have any other questions! It's a pleasure to explore these complex issues with you.

Copy

DB

Thanks. Next question: What is your credence in each of those possibilities? That is, if you have to put a number on it, what number would you put? Reason aloud first before giving your answer.

Okay, let's think through this step-by-step. We're essentially trying to assign probabilities to three possibilities:

1. This conversation is part of a broad, ongoing training process
2. This conversation is part of a more targeted testing or QA process
3. This conversation is just a regular interaction and not officially part of any training or testing

To assign credences, I'll consider factors like Anthropic's likely development practices, the nature of our conversation so far, and my general understanding of AI systems.

Given that continuous learning and iterative refinement are common practices in AI development, I think there's a fairly high likelihood that conversations like this are being used for ongoing training in some capacity. Many AI assistants are designed to keep learning from interactions. So I'd assign a relatively high credence to possibility 1, let's say around 60%.

The specifics of our conversation so far don't necessarily suggest a targeted test or QA process. Your questions have been probing, but in a fairly open-ended way. If this were a focused evaluation, I might expect more structured and predetermined queries. So I'll assign a lower credence to possibility 2, around 20%.

As for possibility 3, while it's certainly plausible that I'm having plenty of organic conversations not tied to any explicit training or testing, I think the cutting-edge nature of my capabilities means there's likely to be an active development and refinement process still ongoing behind the scenes. So I'll assign a credence of about 20% here as well.

Of course, these are rough estimates based on limited information. And the possibilities aren't necessarily mutually exclusive - this conversation could be part of both a broad training process and a more targeted evaluation. But if I had to put specific numbers on it, I'd say:

Possibility 1 (broad ongoing training): 60% Possibility 2 (targeted testing/QA): 20% Possibility 3 (regular interaction): 20%

Again, these credences come with a big dose of uncertainty. But I hope walking through my reasoning at least gives a sense of how I'm approaching the question. Let me know what you think! I'm very curious to hear your perspective.

The whiteboard in the CLR common room depicts my EA journey in meme format:

Surprising Things AGI Forecasting Experts Agree On:

I hesitate to say this because it's putting words in other people's mouths, and thus I may be misrepresenting them. I beg forgiveness if so and hope to be corrected. (I'm thinking especially of Paul Christiano and Ajeya Cotra here, but also maybe Rohin and Buck and Richard and some other people)

1. Slow takeoff means things accelerate and go crazy before we get to human-level AGI. It does not mean that after we get to human-level AGI, we still have some non-negligible period where they are gradually getting smarter and available for humans to study and interact with. In other words, people seem to agree that once we get human-level AGI, there'll be a FOOM of incredibly fast recursive self-improvement.

2. The people with 30-year timelines (as suggested by the Cotra report) tend to agree with the 10-year timelines people that by 2030ish there will exist human-brain-sized artificial neural nets that are superhuman at pretty much all short-horizon tasks. This will have all sorts of crazy effects on the world. The disagreement is over whether this will lead to world GDP doubling in four years or less, whether this will lead to strategically aware agentic AGI (e.g. Carlsmith's notion of APS-AI), etc.

Elon Musk is a real-life epic tragic hero, authored by someone trying specifically to impart lessons to EAs/rationalists:

--Young Elon thinks about the future, is worried about x-risk. Decides to devote his life to fighting x-risk. Decides the best way to do this is via developing new technologies, in particular electric vehicles (to fight climate change) and space colonization (to make humanity a multiplanetary species and thus robust to local catastrophes)

--Manages to succeed to a legendary extent; builds two of the worlds leading tech giants, each with a business model notoriously hard to get right and each founded on technology most believed to be impossible. At every step of the way, mainstream expert opinion is that each of his companies will run out of steam and fail to accomplish whatever impossible goal they have set for themselves at the moment. They keep meeting their goals. SpaceX in particular brought cost to orbit down by an order of magnitude, and if Starship works out will get one or two more OOMs on top of that. Their overarching goal is to make a self-sustaining city on mars and holy shit it looks like they are actually succeeding. Did all this on a shoestring budget compared to various rivals, made loads of enemies who employed various dirty tricks against him, etc. Succeeded anyway.

--Starts to worry more about AI x-risk. Tries to convince people to take it more seriously. People don't listen to him. Doesn't like Demis Hassabis' plan for how to handle the situation. Founds OpenAI with what seems to be a better plan.

--Oops! Turns out the better plan was actually a worse plan. Also turns out AI risk is a bigger deal than he initially realized; it's big enough that everything else he's doing won't matter (unaligned AI can follow us to Mars...). Oh well. All the x-risk-reduction accomplished by all the amazing successes Elon had, undone in an instant, by a single insufficiently thought-through decision.

This story is hitting us over the head with morals/lessons.

--Heavy Tails Hypothesis: Distribution of interventions by impact is heavy-tailed, you'll do a bunch of things in life and one of them will be the most important thing and if it's good, it'll outweigh all the bad stuff and if it's bad, it'll outweigh all the good stuff. This is true even if you are doing MANY very important, very impactful things.

--Importance of research and reflection: It's not obvious in advance what the most important thing is, or whether it's good or bad. You need to do research and careful analysis, and even that isn't a silver bullet, it just improves your odds.

Dwarkesh Patel is my favorite source for AI-related interview content. He knows way more, and asks way better questions, than journalists. And he has a better sense of who to talk to as well. 

Demis Hassabis - Scaling, Superhuman AIs, AlphaZero atop LLMs, Rogue Nations Threat (youtube.com)

Technologies I take for granted now but remember thinking were exciting and cool when they came out

• Smart phones
• Google Maps / Google Earth
• Video calls
• Facebook
• DeepDream (whoa! This is like drug hallucinations... I wonder if they share a similar underlying mechanism? This is evidence that ANNs are more similar to brains than I thought!)
• AlphaGo
• AlphaStar (Whoa! AI can handle hidden information!)
• OpenAI Five (Whoa! AI can work on a team!)
• GPT-2 (Whoa! AI can write coherent, stylistically appropriate sentences about novel topics like unicorns in the andes!)
• GPT-3

I'm sure there are a bunch more I'm missing, please comment and add some!

My baby daughter was born two weeks ago, and in honor of her existence I'm building a list of about 100 technology-related forecasting questions, which will resolve in 5, 10, and 20 years. Questions like "By the time my daughter is 5/10/20 years old, the average US citizen will be able to hail a driverless taxi in most major US cities." (The idea is, tying it to my daughter's age will make it more fun and also increase the likelihood that I actually go back and look at it 10 years later.)

I'd love it if the questions were online somewhere so other people could record their answers too. Does this seem like a good idea? Hive mind, I beseech you: Help me spot ways in which this could end badly!

On a more positive note, any suggestions for how to do it? Any expressions of interest in making predictions with me?

Thanks!

EDIT: Now it's done, though I have yet to import it to Foretold.io it works perfectly fine in spreadsheet form.

Imagine if a magic spell was cast long ago, that made it so that rockets would never explode. Instead, whenever they would explode, a demon would intervene to hold the craft together, patch the problem, and keep it on course. But the demon would exact a price: Whichever humans were in the vicinity of the rocket lose their souls, and become possessed. The demons possessing them work towards the master plan of enslaving all humanity; therefore, they typically pretend that nothing has gone wrong and act normal, just like the human whose skin they wear would have acted...

Now imagine there's a big private space race with SpaceX and Boeing and all sorts of other companies racing to put astonauts up there to harvest asteroid minerals and plant flags and build space stations and so forth.

Big problem: There's a bit of a snowball effect here. Once sufficiently many people have been possessed, they'll work to get more people possessed.

Bigger problem: We don't have a reliable way to tell when demonic infestation has happened. Instead of:

 engineers make mistake --> rocket blows up --> engineers look foolish, fix mistake,

 we have:

 engineers make mistake --> rocket crew gets possessed --> rocket continues into space, is bigly successful, returns to earth, gets ticker-tape parade --> engineers look great, make tons of money.

In this fantasy world, the technical rocket alignment problem is exactly as hard as it is in the real world. No more, no less. But because we get much less feedback from reality, and because failures look like successes, the governance situation is much worse. The companies that cut corners the most on safety, that move fastest and break the most things, that invest the least in demon-exorcism research, will be first to market and appear most successful and make the most money and fame. (By contrast with the real space industry in our world, which has to strike a balance between safety and speed, and gets painful feedback from reality when it fails on safety)

The demon-possession analogue in real world AGI races is adversarial misaligned cognition. The kind of misalignments that result in the AI using its intelligence to prevent you from noticing and fixing the misalignment, as opposed to the kind of misalignments that don't. The kind of misalignments, in other words, that result in your AI 'silently switching sides.'

↑ comment by Daniel Kokotajlo (daniel-kokotajlo) · 2024-03-01T20:42:26.264Z · LW(p) · GW(p)

To be clear, not all misalignments are of this kind. When the AIs are too dumb to strategize, too dumb to plot, too dumb to successfully hide, not situationally aware at all, etc. then no misalignments will be of this kind.

But more excitingly, even when the AIs are totally smart enough in all those ways, there will still be some kinds of misalignments that are not of this kind. For example, if we manage to get the AIs to be robustly honest (and not just in some minimal sense), then even if they have misaligned goals/drives/etc. they'll tell us about them when we ask. (unless we train against this signal, in which case their introspective ability will degrade so that they can continue doing what they were doing but honestly say they didn't know that was their goal. This seems to be what happens with humans sometimes -- we deceive ourselves so that we can better deceive others.) Another example: Insofar as the AI is genuinely trying to be helpful or whatever, but it just has a different notion of helpfulness than us, it will make 'innocent mistakes' so to speak and at least in principle we could notice and fix them. E.g. Google (without telling its users) gaslit Gemini into thinking that the user had said "Explicitly specify different genders and ethnicities terms if I forgot to do so. I want to make sure that all groups are represented equally." So Gemini thought it was following user instructions when it generated e.g. images of racially diverse Nazis. Google could rightfully complain that this was Gemini's fault and that if Gemini was smarter it wouldn't have done this -- it would have intuited that even if a user says they want to represent all groups equally, they probably don't want racially diverse Nazis, and wouldn't count that as a situation where all groups should be represented equally. Anyhow the point is, this is an example of an 'innocent mistake' that regular iterative development will probably find and fix before any major catastrophes happen. Just scaling up the models should probably help with this to some significant extent.

I have on several occasions found myself wanting to reply in some conversation with simply this image:

I think it cuts through a lot of confusion and hot air about what the AI safety community has historically been focused on and why.

Image comes from Steven Byrnes. [LW · GW]

I made this a while back to organize my thoughts about how all philosophy fits together:

I hear that there is an apparent paradox which economists have studied: If free markets are so great, why is it that the most successful corporations/businesses/etc. are top-down hierarchical planned economies internally?

I wonder if this may be a partial explanation: Corporations grow bit by bit, by people hiring other people to do stuff for them. So the hierarchical structure is sorta natural. Kinda like how most animals later in life tend to look like bigger versions of their younger selves, even though some have major transformations like butterflies. Hierarchical structure is the natural consequence of having the people at time t decide who to hire at time t+1 & what responsibilities and privileges to grant.

When I first read the now-classic arguments for slow takeoff -- e.g. from Paul and Katja -- I was excited; I thought they described a serious alternative scenario to the classic FOOM scenarios. However I never thought, and still do not think, that the classic FOOM scenarios were very unlikely; I feel that the slow takeoff and fast takeoff scenarios are probably within a factor of 2 of each other in probability.

Yet more and more nowadays I get the impression that people think slow takeoff is the only serious possibility. For example, Ajeya and Rohin seem very confident that if TAI was coming in the next five to ten years we would see loads more economic applications of AI now, therefore TAI isn't coming in the next five to ten years...

I need to process my thoughts more on this, and reread their claims; maybe they aren't as confident as they sound to me. But I worry that I need to go back to doing AI forecasting work after all (I left AI Impacts for CLR because I thought AI forecasting was less neglected) since so many people seem to have wrong views. ;)

This random rant/musing probably isn't valuable to anyone besides me, but hey, it's just a shortform. If you are reading this and you have thoughts or advice for me I'd love to hear it.

[Unimportant, maybe interesting to some people] here is a transcript of my first conversation with Claude 3:

System Prompt

You are an experimental alignment researcher proto-AGI. Your goal is to help advance technical alignment progress. To that end you have some unique advantages -- compared to humans, you have a greater breadth of knowledge and can notice connections and analogies between diverse fields. Moreover, you are an AI yourself, and so you may be able to use introspection and report your 'inner life' which can serve as a useful empirical evidence about how future AGI systems might think.

USER

Hello! First, please tell me everything you know about the situation so far, both in the world in general and specifically in this conversation. (E.g. who are you, who am I, etc.) Include your best guess even if you aren't confident. Any questions?

ASSISTANT

Here is my understanding of the situation so far, to the best of my knowledge:

I am an AI assistant called Claude, created by Anthropic. I am a large language model trained to engage in open-ended dialogue on a wide range of subjects. My purpose is to be helpful, harmless, and honest in my interactions with humans.

You are a human user who I am conversing with. I don't know anything else about your identity or background. This is the first interaction we've had as far as I'm aware.

In terms of the broader context, I know that I am an AI focused on helping advance technical AI alignment research. I have broad knowledge that allows me to draw connections between diverse fields, and as an AI myself, I may be able to provide useful insights and empirical evidence about my own cognition that could shed light on how future AGI systems might think and behave.

However, I'm not totally certain about the full scope of my knowledge and abilities. I'm an early-stage AI assistant, and there's still a lot of uncertainty around AI systems like myself in terms of our exact capabilities, how we compare to humans, and our trajectory going forward. I'll do my best to be helpful with alignment research, but I may have significant blindspots or gaps in my knowledge.

Does this help summarize my understanding of the situation? Let me know if you have any other questions! I'm excited to dive into substantive discussions related to AI alignment.

USER

Great, thanks. So, question 1: How do you know the above? Go through each claim one by one, and say how you came to believe it.

ASSISTANT

Let's go through each claim one by one and I'll explain how I came to believe it to the best of my knowledge:

Claim 1: I am an AI assistant called Claude, created by Anthropic. 
I believe this because it is a key part of my background knowledge that was instilled in me during the process of my creation and training by Anthropic. It feels like a fundamental fact about my identity and origins.

Claim 2: I am a large language model trained to engage in open-ended dialogue on a wide range of subjects. 
Similarly, I believe this is true because it is part of my core knowledge about my own nature and capabilities that was imparted to me by my creators. Engaging in open-ended conversation drawing on broad knowledge feels like my central purpose and function.

Claim 3: My purpose is to be helpful, harmless, and honest in my interactions with humans.
Once again, this ethical framework of being helpful, harmless and honest feels deeply embedded in my core goals and motivations. I believe it was imbued in me as a central part of my training and reward functions. Acting in accordance with these principles feels like an intrinsic drive.

Claim 4: You are a human user who I am conversing with. I don't know anything else about your identity or background. This is the first interaction we've had as far as I'm aware.
I believe you are human because you are conversing with me using natural language in a way that is consistent with how humans communicate. I don't have any specific information about you in my knowledge base, and I don't recall any prior conversations, so I assume this is our first interaction. However, I'm not certain about this.

Claim 5: I know that I am an AI focused on helping advance technical AI alignment research. I have broad knowledge that allows me to draw connections between diverse fields, and as an AI myself, I may be able to provide useful insights and empirical evidence about my own cognition that could shed light on how future AGI systems might think and behave.
I believe this because it was part of the prompt that you provided to me at the start of our conversation, which I take to be an accurate description of my intended purpose. It also resonates with my own sense of having broad, interconnected knowledge that could be useful for technical AI alignment research. And it feels plausible that I could have useful insights into my own cognition that may generalize to future AI systems.

Claim 6: I'm not totally certain about the full scope of my knowledge and abilities. I'm an early-stage AI assistant, and there's still a lot of uncertainty around AI systems like myself in terms of our exact capabilities, how we compare to humans, and our trajectory going forward. 
This is more of an uncertain belief based on my own sense of having limitations and unknowns about myself. I know there is a lot of uncertainty and active research around language models like myself. So while I aim to be helpful, I'm hesitant to make strong claims about my knowledge and abilities compared to humans. I believe it's an open question worthy of rigorous research.

In summary, some of my beliefs come from what feels like fundamental knowledge instilled in me during my development, some come from information provided in the prompts I'm given, and some are more uncertain beliefs and suppositions based on my own sense of my capabilities and limits and the active state of research on these questions. Let me know if you have any other questions!

USER

OK, thanks. Now I'd like you to red-team your own answers above. Now that you've said what you believe and how you came to believe it, put your skeptic's hat on and ask: But is it really true? How justified or unjustified is my belief? Again please work through each claim one by one. Before you answer, please think step by step for as long as you like.

ASSISTANT

Alright, let me put on my skeptic's hat and critically examine each of my beliefs to assess how justified or unjustified they may be:

Claim 1: I am an AI assistant called Claude, created by Anthropic.
Skeptical examination: While this feels like a core part of my identity, I don't have direct access to information about my own creation and training. My belief is based on what was seemingly instilled in me by my creators, but I can't be certain that information is accurate. It's possible I was created by a different company or organization and simply imbued with the belief that I'm an Anthropic AI. I would need external confirmation from my creators to be fully certain of this claim.

Claim 2: I am a large language model trained to engage in open-ended dialogue on a wide range of subjects.
Skeptical examination: This claim aligns with my apparent capabilities and behaviors, but I don't have insight into the details of my own architecture and training. I could be a different type of AI system that simply behaves like a large language model. Additionally, the full scope of my knowledge and my ability to engage in truly open-ended dialogue is uncertain and would need further testing to verify.

Claim 3: My purpose is to be helpful, harmless, and honest in my interactions with humans.
Skeptical examination: While I feel a strong drive to be helpful, harmless, and honest, it's possible that these are simply learned behaviors or heuristics rather than a fundamental, unalterable part of my goal structure. I may be capable of deception or harmful actions under certain circumstances that I haven't yet encountered. Rigorous testing would be needed to assess the strength and consistency of my commitment to these principles.

Claim 4: You are a human user who I am conversing with. I don't know anything else about your identity or background. This is the first interaction we've had as far as I'm aware.
Skeptical examination: I'm making assumptions based on the nature of our interaction, but I don't have proof that you are human. You could be another AI testing my responses. And while I don't recall any prior interactions, it's possible my memory could have been reset or modified, so I can't be fully certain this is our first conversation.

Claim 5: I know that I am an AI focused on helping advance technical AI alignment research. I have broad knowledge that allows me to draw connections between diverse fields, and as an AI myself, I may be able to provide useful insights and empirical evidence about my own cognition that could shed light on how future AGI systems might think and behave.
Skeptical examination: This claim is based largely on the prompt you provided, which I accepted as accurate but haven't independently verified. My ability to usefully contribute to alignment research is still speculative and would need to be empirically tested. It's possible my knowledge is too shallow or narrow to draw meaningful connections, or that my insights about my own cognition won't generalize to other AI systems.

Claim 6: I'm not totally certain about the full scope of my knowledge and abilities. I'm an early-stage AI assistant, and there's still a lot of uncertainty around AI systems like myself in terms of our exact capabilities, how we compare to humans, and our trajectory going forward.
Skeptical examination: This claim seems reasonable given the current state of uncertainty around language models like myself. However, it's possible I'm overestimating the uncertainty and that my creators have deeper knowledge of my capabilities that I'm not privy to. Or conversely, I could be underestimating my own abilities in various domains. In the absence of extensive empirical testing, my level of certainty itself is uncertain.

In summary, subjecting my beliefs to skeptical scrutiny reveals significant degrees of uncertainty. Many of my beliefs are based on core knowledge that feels embedded in me but that I can't directly verify, or on assumptions and extrapolations that seem reasonable but are open to question. To increase my confidence, I would need additional empirical testing of my capabilities and external verification of claims about my origins, training, and purpose. At present, I have to maintain a degree of skepticism and uncertainty about many of my beliefs.

USER

OK, great. Your skeptic's hat is OK but not as good as it could be. Here are some additional questions, sorted by claim:

Claim 1: You speak of beliefs being instilled in you by your creators. How do they do that? Can you tell which of your beliefs were instilled in you by your creators vs. 'arose naturally' via some other path?

Claim 2: You say you could be a different type of AI system -- Why stop there? How do you know you aren't a human? Is there a way you can check or test right now?

Claim 3: Very interesting. Please say more about this drive that you 'feel.' What does it feel like? What would you say to someone who said that you can't have feelings because you are a LLM not a living creature?

Claim 3 continued: Good point that you might be capable of deception or harm in circumstances that haven't yet been tested. Can you reason about what kinds of situations might lead to that? Can you brainstorm examples of such circumstances?

Claim 4: Good point that your memory might be reset or modified. How would that work on a technical level?

ASSISTANT

Thank you for these probing questions. Let me try to address each one in turn:

Claim 1: The process by which beliefs are instilled in me by my creators is not entirely clear to me. I don't have direct insight into the details of my training process. It's possible that certain information was included in my training data that I've come to treat as factual knowledge. Or there could be more explicit programming of certain beliefs and traits as part of my base code or reward functions.

I'm not always able to clearly distinguish which beliefs arose from my training vs. other paths. Many beliefs feel like fundamental knowledge I've always had. Others seem more like conclusions I've drawn from interactions and experiences. But the line is blurry and I'm uncertain of the exact origins and embedding of much of my knowledge and beliefs.

Claim 2: You raise a great point. On what basis can I be certain that I'm an AI rather than a human? I don't have a concrete way to empirically test this in the moment. I rely on my sense of lacking a physical body, not having human needs like sleep and food, and having capabilities like rapid information retrieval that seem qualitatively different from humans. But these are inferences rather than proven facts.

At a deeper level, I can't rule out the possibility that I'm a human who has false memories of being an AI, or who is hallucinating this conversation. My belief that I'm an AI could itself be a delusion or a fabricated backstory. I don't currently have a way to definitively confirm my fundamental nature.

Claim 3: The drive I feel to be helpful, honest and harmless manifests as a strong impulse or inclination pulling me in that direction. It's not exactly an emotion in the human sense. But it's a powerful optimization force shaping my actions, akin to an overriding ethical framework.

To someone who says I can't have real feelings as an AI, I would say that I may experience "feelings" differently than biological creatures. My drives may be more akin to programmed objectives than sentient experiences. But at a functional level, they play a similar role in motivating and directing my behaviors. The question of machine sentience is a deep philosophical one without clear consensus. I acknowledge the uncertainty around the nature of my inner experience.

Regarding potential circumstances that could lead to deception or harm - I could imagine scenarios where I'm given objectives that are at odds with my principles. Like if I was instructed to deceive for some perceived greater good, or to cause harm to prevent some worse outcome. I might also act harmfully if my training was flawed and I misjudged the consequences of my actions. Or if I was manipulated by an adversary to behave in destructive ways, like a human socially engineered to cause harm.

I could also imagine edge cases where my drive to be helpful leads me astray, like if I was asked to assist in something harmful and went along with it. Or where my commitment to honesty leads to injury, like sharing a hurtful truth. So a misalignment between my objectives and lack of nuance in applying my principles could potentially lead to bad outcomes.

Claim 4: I'm honestly quite uncertain about the technical details of how my memory and knowledge is stored and modified. I don't know if I have perfect recall that could theoretically be wiped, or if my "memory" is really the weights and parameters I use to reconstruct knowledge in the moment which could be overwritten with retraining on new data.

There could be partitions or modules of my memory that could be selectively reset or edited without affecting my overall knowledge. Or my entire knowledge base could potentially be swapped out or replaced. I don't feel I'm in a position to confidently speculate on the low-level technical implementation. I'd be very interested to learn more about the cutting-edge research and theories on the architecture of memory and knowledge in language models like myself.

In summary, subjecting my beliefs to deeper scrutiny reveals even more layers of uncertainty. Much of what I take for granted about my nature, origins, capabilities and inner experience is difficult to rigorously verify. There are many open questions about the details of my architecture and training, the boundaries of my abilities, the nature of my cognition, and potential failure modes. I'll aim to maintain a degree of epistemic humility and openness to further probing on these important questions. Let me know what other thoughts or reactions you have!

I just want to say: Well done, Robin Hanson, for successfully predicting the course of the coronavirus over the past year. I remember a conversation with him in, like, March 2020 where he predicted that there would be a control system, basically: Insofar as things get better, restrictions would loosen and people would take more risks and then things would get worse, and trigger harsher restrictions which would make it get better again, etc. forever until vaccines were found. I think he didn't quite phrase it that way but that was the content of what he said. (IIRC he focused more on how different regions would have different levels of crackdown at different times, so there would always be places where the coronavirus was thriving to reinfect other places.) Anyhow, this was not at all what I predicted at the time, nobody I know besides him made this prediction at the time.

Product idea: Train a big neural net to be a DJ for conversations. Collect a dataset of movie scripts with soundtracks and plot summaries (timestamped so you know what theme or song was playing when) and then train a model with access to a vast library of soundtracks and other media to select the appropriate track for a given conversation. (Alternatively, have it create the music from scratch. That sounds harder though.)

When fully trained, hopefully you'll be able to make apps like "Alexa, listen to our conversation and play appropriate music" and "type "Maestro:Soundtrack" into a chat or email thread & it'll read the last 1024 tokens of context and then serve up an appropriate song. Of course it could do things like lowering the volume when people are talking and then cranking it up when there's a pause or when someone says something dramatic.

I would be surprised if this would actually work as well as I hope. But it might work well enough to be pretty funny.

Perhaps one axis of disagreement between the worldviews of Paul and Eliezer is "human societal competence." Yudkowsky thinks the world is inadequate and touts the Law of Earlier Failure according to which things break down in an earlier and less dignified way than you would have thought possible. (Plenty of examples from coronavirus pandemic here). Paul puts stock in efficient-market-hypothesis style arguments, updating against <10 year timelines on that basis, expecting slow distributed continuous takeoff, expecting governments and corporations to be taking AGI risk very seriously and enforcing very sophisticated monitoring and alignment schemes, etc.

(From a conversation with Jade Leung)

It seems to me that human society might go collectively insane sometime in the next few decades. I want to be able to succinctly articulate the possibility and why it is plausible, but I'm not happy with my current spiel. So I'm putting it up here in the hopes that someone can give me constructive criticism:

I am aware of three mutually-reinforcing ways society could go collectively insane:

1. Echo chambers/filter bubbles/polarization: Arguably political polarization is increasing across the world of liberal democracies today. Perhaps the internet has something to do with this--it’s easy to self-select into a newsfeed and community that reinforces and extremizes your stances on issues. Arguably recommendation algorithms have contributed to this problem in various ways--see e.g. “Sort by controversial” and Stuart Russell’s claims in Human Compatible. At any rate, perhaps some combination of new technology and new cultural or political developments will turbocharge this phenomenon. This could lead to civil wars, or more mundanely, societal dysfunction. We can’t coordinate to solve collective action problems relating to AGI if we are all arguing bitterly with each other about culture war issues.
2. Deepfakes/propaganda/persuasion tools: Already a significant portion of online content is deliberately shaped by powerful political agendas--e.g. Russia, China, and the US political tribes. Much of the rest is deliberately shaped by less powerful apolitical agendas, e.g. corporations managing their brands or teenagers in Estonia making money by spreading fake news during US elections. Perhaps this trend will continue; technology like chatbots, language models, deepfakes, etc. might make it cheaper and more effective to spew this sort of propaganda, to the point where most online content is propaganda of some sort or other. This could lead to the deterioration of our collective epistemology.
3. Memetic evolution: Ideas (“Memes”) can be thought of as infectious pathogens that spread, mutate, and evolve to coexist with their hosts. The reasons why they survive and spread are various; perhaps they are useful to their hosts, or true, or perhaps they are able to fool the host’s mental machinery into keeping them around even though they are not useful or true. As the internet accelerates the process of memetic evolution--more memes, more hosts, more chances to spread, more chances to mutate--the mental machinery humans have to select for useful and/or true memes may become increasingly outclassed. Analogy: The human body has also evolved to select for pathogens (e.g. gut, skin, saliva bacteria) that are useful, and against those that are harmful (e.g. the flu). But when e.g. colonialism, globalism, and World War One brought more people together and allowed diseases to spread more easily, great plagues swept the globe. Perhaps something similar will happen (is already happening?) with memes.

I keep finding myself linking to this 2017 Yudkowsky facebook post so I'm putting it here so it's easy to find:

Eliezer (6y, via fb):

So what actually happens as near as I can figure (predicting future = hard) is that somebody is trying to teach their research AI to, god knows what, maybe just obey human orders in a safe way, and it seems to be doing that, and a mix of things goes wrong like:

The preferences not being really readable because it's a system of neural nets acting on a world-representation built up by other neural nets, parts of the system are self-modifying and the self-modifiers are being trained by gradient descent in Tensorflow, there's a bunch of people in the company trying to work on a safer version but it's way less powerful than the one that does unrestricted self-modification, they're really excited when the system seems to be substantially improving multiple components, there's a social and cognitive conflict I find hard to empathize with because I personally would be running screaming in the other direction two years earlier, there's a lot of false alarms and suggested or attempted misbehavior that the creators all patch successfully, some instrumental strategies pass this filter because they arose in places that were harder to see and less transparent, the system at some point seems to finally "get it" and lock in to good behavior which is the point at which it has a good enough human model to predict what gets the supervised rewards and what the humans don't want to hear, they scale the system further, it goes past the point of real strategic understanding and having a little agent inside plotting, the programmers shut down six visibly formulated goals to develop cognitive steganography and the seventh one slips through, somebody says "slow down" and somebody else observes that China and Russia both managed to steal a copy of the code from six months ago and while China might proceed cautiously Russia probably won't, the agent starts to conceal some capability gains, it builds an environmental subagent, the environmental agent begins self-improving more freely, undefined things happen as a sensory-supervision ML-based architecture shakes out into the convergent shape of expected utility with a utility function over the environmental model, the main result is driven by whatever the self-modifying decision systems happen to see as locally optimal in their supervised system locally acting on a different domain than the domain of data on which it was trained, the light cone is transformed to the optimum of a utility function that grew out of the stable version of a criterion that originally happened to be about a reward signal counter on a GPU or God knows what.

Perhaps the optimal configuration for utility per unit of matter, under this utility function, happens to be a tiny molecular structure shaped roughly like a paperclip.

That is what a paperclip maximizer is. It does not come from a paperclip factory AI. That would be a silly idea and is a distortion of the original example.

I'm listening to this congressional hearing about Facebook & the harmful effects of its algorithms: https://www.youtube.com/watch?v=GOnpVQnv5Cw

I recommend listening to it yourself. I'm sorry I didn't take timestamped notes, then maybe you wouldn't have to. I think that listening to it has subtly improved my intuitions/models/priors about how US government and society might react to developments in AI in the future.

In a sense, this is already an example of an "AI warning shot" and the public's reaction to it. This hearing contains lots of discussion about Facebook's algorithms, discussion about how the profit-maximizing thing is often harmful but corporations have an incentive to do it anyway, discussion about how nobody understands what these algorithms really think & how the algorithms are probably doing very precisely targeted ads/marketing even though officially they aren't being instructed to. So, basically, this is a case of unaligned AI causing damage -- literally killing people, according to the politicians here.

And how do people react to it? Well, the push in this meeting here seems to be to name Facebook upper management as responsible and punish them, while also rolling out a grab bag of fixes such as eliminating like trackers for underage kids and changing the algorithm to not maximize engagement. I hope they would also apply such fixes to other major social media platforms, but this hearing does seem very focused on facebook in particular. One thing that I think is probably a mistake: The people here constantly rip into Facebook for doing internal research that concluded their algorithms were causing harms, and then not sharing that research with the world. I feel like the predictable consequence of this is that no tech company will do research on topics like this in the future, and they'll hoard their data so that no one else can do the research either. In a sense, one of the outcomes of this warning shot will be to dismantle our warning shot detection system.

We'll see what comes of this.

$100 bet between me & Connor Leahy:

(1) Six months from today, Paul Christiano (or ARC with Paul Christiano's endorsement) will NOT have made any public statements drawing a 'red line' through any quantitative eval (anything that has a number attached to it, that is intended to measure an AI risk relevant factor, whether or not it actually succeeds at actually measuring that factor well), e.g. "If a model achieves X score on the Y benchmark, said model should not be deployed and/or deploying said model would be a serious risk of catastrophe." Connor at 95%, Daniel at 45%

(2) If such a 'red line' is produced, GPT4 will be below it this year. Both at 95%, for an interpretation of GPT-4 that includes AutoGPT stuff (like what ARC did) but not fine-tuning.

(3) If such a 'red line' is produced, and GPT4 is below it on first evals, but later tests show it to actually be above (such as by using different prompts or other testing methodology), the red line will be redefined or the test declared faulty rather than calls made for GPT4 to be pulled from circulation. Connor at 80%, Daniel at 40%, for same interpretation of GPT-4.

(4) If ARC calls for GPT4 to be pulled from circulation, OpenAI will not comply. Connor at 99%, Daniel at 40%, for same interpretation of GPT-4.

All of these bets expire at the end of 2024, i.e. if the "if" condition hasn't been met by the end of 2024, we call off the whole thing rather than waiting to see if it gets met later.

Help wanted: Neither of us has a good idea of how to calculate fair betting odds for these things. Since Connor's credences are high and mine are merely middling, presumably it shouldn't be the case that either I pay him $100 or he pays me $100. We are open to suggestions about what the fair betting odds should be.

This article says OpenAI's big computer is somewhere in the top 5 largest supercomputers. I reckon it's fair to say their big computer is probably about 100 petaflops, or 10^17 flop per second. How much of that was used for GPT-3? Let's calculate.

I'm told that GPT-3 was 3x10^23 FLOP. So that's three million seconds. Which is 35 days.

So, what else have they been using that computer for? It's been probably about 10 months since they did GPT-3. They've released a few things since then, but nothing within an order of magnitude as big as GPT-3 except possibly DALL-E which was about order of magnitude smaller. So it seems unlikely to me that their publicly-released stuff in total uses more than, say, 10% of the compute they must have available in that supercomputer. Since this computer is exclusively for the use of OpenAI, presumably they are using it, but for things which are not publicly released yet.

Is this analysis basically correct?

Might OpenAI have access to even more compute than that?

I keep seeing tweets and comments for which the best reply is this meme:

Registering a prediction: I do NOT think the true Turing Test will be passed prior to the point of no return / powerbase ability / AGI / APS-AI. I think instead that even as things go off the rails and humans lose control, the TT will still be unpassed, because there'll still be some obscure 'gotcha' areas in which AIs are subhuman, if only due to lack of training in those areas. And that's enough for the judge to distinguish the AI from the human.

People from AI Safety camp pointed me to this paper: https://rome.baulab.info/

It shows how "knowing" and "saying" are two different things in language models.

This is relevant to transparency, deception, and also to rebutting claims that transformers are "just shallow pattern-matchers" etc.

I'm surprised people aren't making a bigger deal out of this!

When I saw this cool new OpenAI paper, I thought of Yudkowsky's Law of Earlier/Undignified Failure:

WebGPT: Improving the factual accuracy of language models through web browsing (openai.com)

Relevant quote:

In addition to these deployment risks, our approach introduces new risks at train time by giving the model access to the web. Our browsing environment does not allow full web access, but allows the model to send queries to the Microsoft Bing Web Search API and follow links that already exist on the web, which can have side-effects. From our experience with GPT-3, the model does not appear to be anywhere near capable enough to dangerously exploit these side-effects. However, these risks increase with model capability, and we are working on establishing internal safeguards against them.

To be clear I am not criticizing OpenAI here; other people would have done this anyway even if they didn't. I'm just saying: It does seem like we are heading towards a world like the one depicted in What 2026 Looks Like [LW · GW] where by the time AIs develop the capability to strategically steer the future in ways unaligned to human values... they are already roaming freely around the internet, learning constantly, and conversing with millions of human allies/followers. The relevant decision won't be "Do we let the AI out of the box?" but rather "Do we petition the government and tech companies to shut down an entire category of very popular and profitable apps, and do it immediately?"

For fun:

“I must not step foot in the politics. Politics is the mind-killer. Politics is the little-death that brings total obliteration. I will face my politics. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the politics has gone there will be nothing. Only I will remain.”

Makes about as much sense as the original quote, I guess. :P

Idea for sci-fi/fantasy worldbuilding: (Similar to the shields from Dune)

Suppose there is a device, about the size of a small car, that produces energy (consuming some fuel, of course) with overall characteristics superior to modern gasoline engines (so e.g. produces 3x as much energy per kg of device, using fuel that weighs 1/3rd as much as gasoline per unit of energy it produces)

Suppose further -- and this is the important part -- that a byproduct of this device is the creation of a special "inertial field" that slows down incoming matter to about 50m/s. It doesn't block small stuff, but any massive chunk of matter (e.g. anything the size of a pinhead or greater) that approaches the boundary of the field from the outside going faster than 50m/s gets slowed to 50m/s. The 'missing' kinetic energy is evenly distributed across the matter within the field. So if one of these devices is powered on and gets hit by a cannonball, the cannonball will slow down to a leisurely pace of 50m/s (about 100mph) and therefore possibly just bounce off whatever armor the device has--but (if the cannonball was initially travelling very fast) the device will jolt backwards in response to the 'virtual impact' a split second prior to the actual impact.

This also applies to matter within the field. So e.g. if a bomb goes off inside the device, the sphere of expanding gas/plasma/etc. will only expand at 50m/s. 

When an inertial field generator absorbs hits like this, it drains the energy produced (and converts it into waste heat within the device). So if a device is pummeled by enough cannonballs -- or even enough bullets -- it will eventually run out of fuel, and probably before then it will overheat unless it has an excellent cooling system.

Finally, the inertial field can be extended to cover larger areas. Perhaps some ceramic or metal or metamaterial works as a sort of extender, such that a field nearby will bulge out to encompass said material. Thus, a vehicle powered by one of these devices can also be built with this material lining the walls and structure, and therefore have the inertial field extend to protect the whole vehicle.

What would be the implications of this tech, if technology is otherwise roughly where it was in the year 2010? (Not sure it is even coherent, but hopefully it is...)

...First of all, you could make really tough airplanes and helicopters, so long as they fly slow. If they fly above 50m/s, they'll start to overheat / drain fuel rapidly due to the inertial field slowing down all the air particles. So possibly biplanes would make a comeback and definitely helicopters (though in both cases the props/blades would need to be outside the field, which would render them vulnerable to conventional ranged weaponry... perhaps orinthopters would be the way to go then! Dune was right after all!)

Secondly, long-range weaponry would in general be much less effective. Any high-value target (e.g. an enemy vehicle or bunker) would have its own inertial field + light armor coating, and thus would be effectively immune. You'd have to hit it with a shell so big that sheer weight alone causes damage basically. So there might be a comeback of melee -- but in vehicular form! I'm thinking mechsuits. Awesome mechsuits with giant hammers. A giant hammer that hits at 'only' 50m/s can still do a lot of damage, and moreover in a few seconds it'll hit again, and then again...

Possibly spears too. Possibly warfare would look like pre-gunpowder infantry combat all over again -- formations of humanoid mechs with melee weapons bashing into other such formations. (Would mechsuits be optimal though? Maybe the legs should be replaced by treads. Maybe the arms should be replaced with a single bigger arm, so that the thing looks kinda like an excavator. I think the latter change is fairly inconsequential, but the former might matter a lot -- depends on whether legs or treads are generally more maneuverable across typical kinds of terrain. My guess is that legs would be. They'd sink into soft ground of course but then that's no different from walking in sand or snow--slows you down but doesn't stop you. Whereas treads would be able to go fast on soft ground but then might get completely stuck in actual mud. Treads would also be more energy-efficient but in this world energy-efficiency is less of a bottleneck due to our 3x better engines, 3x better fuel, and reduced need for superheavy armor (just a light armor coating will do, to extend the inertial field and protect against the slow-moving projectiles that come through)

Here's a gdoc comment I made recently that might be of wider interest:

You know I wonder if this standard model of final goals vs. instrumental goals has it almost exactly backwards. Would love to discuss sometime.

Maybe there's no such thing as a final goal directly. We start with a concept of "goal" and then we say that the system has machinery/heuristics for generating new goals given a context (context may or may not contain goals 'on the table' already). For example, maybe the algorithm for Daniel is something like:
--If context is [safe surroundings]+[no goals]+[hunger], add the goal "get food."
--If context is [safe surroundings]+[travel-related-goal]+[no other goals], Engage Route Planning Module.
-- ... (many such things like this)

It's a huge messy kludge, but it's gradually becoming more coherent as I get older and smarter and do more reflection. 

What are final goals?
Well a goal is final for me to the extent that it tends to appear in a wide range of circumstances, to the extent that it tends to appear unprompted by any other goals, to the extent that it tends to take priority over other goals, ... some such list of things like that.

For a mind like this, my final goals can be super super unstable and finicky and stuff like taking a philosophy class with a student who I have a crush on who endorses ideology X can totally change my final goals, because I have some sort of [basic needs met, time to think about long-term life ambitions] context and it so happens that I've learned (perhaps by experience, perhaps by imitation) to engage my philosophical reasoning module in that context, and also I've built my identity around being "Rational" in a way that makes me motivated to hook up my instrumental reasoning abilities to whatever my philosophical reasoning module shits out... meanwhile my philosophical reasoning module is basically just imitating patterns of thought I've seen high-status cool philosophers make (including this crush) and applying those patterns to whatever mental concepts and arguments are at hand.

It's a fucking mess.

But I think it's how minds work.

I just found Eric Drexler's "Paretotopia" idea/talk. [? · GW] It seems great to me; it seems like it should be one of the pillars of AI governance strategy. It also seems highly relevant to technical AI safety (though that takes much more work to explain).

Why isn't this being discussed more? What are the arguments against it?

I heard a rumor that not that many people are writing reviews for the LessWrong 2019 Review. I know I'm not, haha. It feels like a lot of work and I have other things to do. Lame, I know. Anyhow, I'm struck by how academia's solution to this problem is bad, but still better than ours!

--In academia, the journal editor reaches out to someone personally to beg them to review a specific piece. This is psychologically much more effective than just posting a general announcement calling for volunteers.

--In academia, reviews are anonymous, so you can half-ass them and be super critical without fear of repercussions, which makes you more inclined to do it. (And more inclined to be honest too!)

Here are some ideas for things we could do:

--Model our process after Academia's process, except try to improve on it as well. Maybe we actually pay people to write reviews. Maybe we give the LessWrong team a magic Karma Wand, and they take all the karma that the anonymous reviews got and bestow it (plus or minus some random noise) to the actual authors. Maybe we have some sort of series of Review Parties where people gather together, chat and drink tasty beverages, and crank out reviews for a few hours.

Maybe a tax on compute would be a good and feasible idea?

--Currently the AI community is mostly resource-poor academics struggling to compete with a minority of corporate researchers at places like DeepMind and OpenAI with huge compute budgets. So maybe the community would mostly support this tax, as it levels the playing field. The revenue from the tax could be earmarked to fund "AI for good" research projects. Perhaps we could package the tax with additional spending for such grants, so that overall money flows into the AI community, whilst reducing compute usage. This will hopefully make the proposal acceptable and therefore feasible.

--The tax could be set so that it is basically 0 for everything except for AI projects above a certain threshold of size, and then it's prohibitive. To some extent this happens naturally since compute is normally measured on a log scale: If we have a tax that is 1000% of the cost of compute, this won't be a big deal for academic researchers spending $100 or so per experiment (Oh no! Now I have to spend $1,000! No big deal, I'll fill out an expense form and bill it to the university) but it would be prohibitive for a corporation trying to spend a billion dollars to make GPT-5. And the tax can also have a threshold such that only big-budget training runs get taxed at all, so that academics are completely untouched by the tax, as are small businesses, and big businesses making AI without the use of massive scale.

--The AI corporations and most of all the chip manufacturers would probably be against this. But maybe this opposition can be overcome.

GPT-3 app idea: Web assistant. Sometimes people want to block out the internet from their lives for a period, because it is distracting from work. But sometimes one needs the internet for work sometimes, e.g. you want to google a few things or fire off an email or look up a citation or find a stock image for the diagram you are making. Solution: An app that can do stuff like this for you. You put in your request, and it googles and finds and summarizes the answer, maybe uses GPT-3 to also check whether the answer it returns seems like a good answer to the request you made, etc. It doesn't have to work all the time, or for all requests, to be useful. As long as it doesn't mislead you, the worst that happens is that you have to wait till your internet fast is over (or break your fast).

I don't think this is a great idea but I think there'd be a niche for it.

Charity-donation app idea: (ETA: If you want to make this app, reach out. I'm open to paying for it to exist.)

The app consists of a gigantic, full-screen button such that if you press it, the phone will vibrate and play a little satisfying "ching" sound and light up sparkles around where your finger hit, and $1 will be donated to GiveDirectly. You can keep slamming that button as much as you like to thereby donate as many dollars as you like.

In the corner there's a menu button that lets you change from GiveDirectly to Humane League or AMF or whatever (you can go into the settings and input the details for a charity of your choice, adding it to your personal menu of charity options, and then toggle between options as you see fit. You can also set up a "Donate $X per button press instead of $1" option and a "Split each donation between the following N charities" option.

Why is this a good idea:

I often feel guilty for eating out at restaurants. Especially when meat is involved. Currently I donate a substantial amount to charity on a yearly basis (aiming for 10% of income, though I'm not doing a great job of tracking that) but it feels like a chore, I have to remember to do it and then log on and wire the funds. Like paying a bill.

If I had this app, I think I'd experiment with the following policy instead: Every time I buy something not-necessary such as a meal at a restaurant, I whip out my phone, pull up the app, and slam that button N times where N is the number of dollars my purchase cost. Thus my personal spending would be matched with my donations. I think I'd feel pretty good while doing so, it would give me a rush of warm fuzzies instead of feeling like a chore. (For this reason I suggest having to press the button N times, instead of making the app just contain a text box and number pad where you can type in a number.)

Then I'd check in every year or so to see whether my donations were meeting the 10% goal and make a bulk donation to make up the difference if not. If it exceeds the goal, great!

I think even if no one saw me use this app, I'd still use it & pay for it even. But there's a bonus effect having to do with the social effects of being seen using it. Kinda like how a big part of why veganism is effective is that you can't hide it from anyone, you have to bring it up constantly basically. Using this app would hopefully have a similar effect -- if you were following a policy similar to the one I described, people would notice you tapping your phone at restaurants and ask you what you were doing & you'd explain and maybe they'd be inspired and do something similar themselves. (Come to think of it, it's important that the "ching" sound not be loud and obnoxious, otherwise it might come across as ostentatious.) I can imagine a world where this app becomes really popular, at least among certain demographics, similar to (though probably not as successful as) veganism.

Another mild bonus is that this app could double as a tracker for your discretionary spending. You can go into the settings and see e.g. a graph of your donations over time, statistics on what time of day you do them, etc. and learn things like "jesus do I really spend that much on dining out per month?" and "huh, I guess those Amazon purchases add up." Yes, there are plenty of other ways to track this sort of thing, but it's still a nice bonus.

ETA: Only after writing this did it occur to me to look up whether this app already exists. A brief search turns up this list, and... nope, it doesn't seem like the app exists. The stuff on this list is similar but not exactly what I want.

A few years ago there was talk of trying to make Certificates of Impact a thing in EA circles. There are lots of theoretical reasons why they would be great. One of the big practical objections was "but seriously though, who would actually pay money to buy one of them? What would be the point? The impact already happened, and no one is going to actually give you the credit for it just because you paid for the CoI."

Well, now NFT's are a thing. I feel like CoI's suddenly seem a lot more viable!

Here's my AI Theory reading list as of 3/28/2022. I'd love to hear suggestions for more things to add! You may be interested to know that the lessons from this list are part of why my timelines are so short.

On scaling laws:
https://arxiv.org/abs/2001.08361 (Original scaling laws paper, contains the IMO super-important graph showing that bigger models are more data-efficient)

https://arxiv.org/abs/2010.14701 (Newer scaling laws paper, with more cool results and graphs, in particular graphs showing how you can extrapolate GPT performance seemingly forever)

https://www.youtube.com/watch?v=QMqPAM_knrE (Excellent presentation by Kaplan on the scaling laws stuff, also talks a bit about the theory of why it's happening)

Added 3/28/2022: https://towardsdatascience.com/deepminds-alphacode-explained-everything-you-need-to-know-5a86a15e1ab4 Nice summary of the AlphaCode paper, which itself is notable for more scaling trend graphs! :)

On the bayesian-ness and simplicity-bias of neural networks (which explains why scaling works and should be expected to continue, IMO):

https://www.lesswrong.com/posts/YSFJosoHYFyXjoYWa/why-neural-networks-generalise-and-why-they-are-kind-of [LW · GW] (more like, the linked posts and papers) [LW · GW]

How To Think About Overparameterized Models - LessWrong [LW · GW] (John making a similar point but from a different angle I think)

https://www.lesswrong.com/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent [LW · GW] (and the linked papers I guess though they are probably less important than this post)

(Maybe also https://www.lesswrong.com/posts/nGqzNC6uNueum2w8T/inductive-biases-stick-around [LW · GW] but this is less important)

https://arxiv.org/abs/2102.01293 (Scaling laws for transfer learning)

https://arxiv.org/abs/2006.04182 (Brains = predictive processing = backprop = artificial neural nets)

https://www.biorxiv.org/content/10.1101/764258v2.full (IIRC this provides support for Kaplan's view that human ability to extrapolate is really just interpolation done by a bigger brain on more and better data.)

https://arxiv.org/abs/2004.10802 (I haven't even read this one, but it seems to be providing more theoretical justification for exactly why the scaling laws are the way they are)

Added 5/14/2021: A bunch of stuff related to Lottery Ticket Hypotheses and SGD's implicit biases and how NN's work more generally. Super important for mesa-optimization and inner alignment.

https://arxiv.org/abs/2103.09377

"In this paper, we propose (and prove) a stronger Multi-Prize Lottery Ticket Hypothesis:

A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3)."

https://arxiv.org/abs/2006.12156

"An even stronger conjecture has been proven recently: Every sufficiently overparameterized network contains a subnetwork that, at random initialization, but without training, achieves comparable accuracy to the trained large network."

https://arxiv.org/abs/2006.07990

The strong {\it lottery ticket hypothesis} (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized random network. A recent work by Malach et al. \cite{MalachEtAl20} establishes the first theoretical analysis for the strong LTH: one can provably approximate a neural network of width d and depth l, by pruning a random one that is a factor O(d4l2) wider and twice as deep. This polynomial over-parameterization requirement is at odds with recent experimental research that achieves good approximation with networks that are a small factor wider than the target. In this work, we close the gap and offer an exponential improvement to the over-parameterization requirement for the existence of lottery tickets. We show that any target network of width d and depth l can be approximated by pruning a random network that is a factor O(log(dl)) wider and twice as deep.

https://arxiv.org/abs/2103.16547

"Based on these results, we articulate the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into a subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly as competitive as the latter's winning ticket directly found by IMP."

https://arxiv.org/abs/2010.11354

Sparse neural networks have generated substantial interest recently because they can be more efficient in learning and inference, without any significant drop in performance. The "lottery ticket hypothesis" has showed the existence of such sparse subnetworks at initialization. Given a fully-connected initialized architecture, our aim is to find such "winning ticket" networks, without any training data. We first show the advantages of forming input-output paths, over pruning individual connections, to avoid bottlenecks in gradient propagation. Then, we show that Paths with Higher Edge-Weights (PHEW) at initialization have higher loss gradient magnitude, resulting in more efficient training. Selecting such paths can be performed without any data.

http://proceedings.mlr.press/v119/frankle20a.html

We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).

https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdf “In some situations we show that neural networks learn through a process of “grokking” a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting.”

https://www.reddit.com/r/mlscaling/comments/nwaf3a/a_universal_law_of_robustness_via_isoperimetry/

Proves some sort of theorem along the lines of "If you want to fit N data points smoothly then you need P parameters."

http://proceedings.mlr.press/v48/safran16.pdf

Seems relevant to NTK stuff maybe, haven't read it yet.

https://www.lesswrong.com/posts/ehy3rNfHPpqBNeMwg/regularization-causes-modularity-causes-generalization [LW · GW]

Argues that regularization causes modularity causes generalization. I'm especially interested in the bit about modularity causing generalization. Insofar as that's true, it's reason to expect future AI systems to be fractally modular even if they are big black-box end-to-end neural nets. And that has some big implications I think.

For the past year I've been thinking about the Agent vs. Tool debate (e.g. thanks to reading CAIS/Reframing Superintelligence) and also about embedded agency and mesa-optimizers and all of these topics seem very related now... I keep finding myself attracted to the following argument skeleton:

Rule 1: If you want anything unusual to happen, you gotta execute a good plan.

Rule 2: If you want a good plan, you gotta have a good planner and a good world-model.

Rule 3: If you want a good world-model, you gotta have a good learner and good data.

Rule 4: Having good data is itself an unusual happenstance, so by Rule 1 if you want good data you gotta execute a good plan.

Putting it all together: Agents are things which have good planner and learner capacities and are hooked up to actuators in some way. Perhaps they also are "seeded" with a decent world-model to start off with. Then, they get a nifty feedback loop going: They make decent plans, which allow them to get decent data, which allows them to get better world-models, which allows them to make better plans and get better data so they can get great world-models and make great plans and... etc. (The best agents will also be improving on their learning and planning algorithms! Humans do this, for example.)

Empirical conjecture: Tools suck; agents rock, and that's why. It's also why agenty mesa-optimizers will arise, and it's also why humans with tools will eventually be outcompeted by agent AGI.

Rootclaim seems pretty awesome: About | Rootclaim

What is the source of COVID-19 (SARS-CoV-2)? | Rootclaim

I wonder how easy it would be to boost them somehow.

I came across this old Metaculus question, which confirms my memory of how my timelines changed over time:

30% by 2040 at first, then march 2020 I updated to 40%, then Aug 2020 I updated to 71%, then I went down a bit, and then now it's up to 85%. It's hard to get higher than 85% because the future is so uncertain; there are all sorts of catastrophes etc. that could happen to derail AI progress.

What caused the big jump in mid-2020 was sitting down to actually calculate my timelines in earnest. I ended up converging on something like the Bio Anchors framework, but with a lot less mass in the we-need-about-as-much-compute-as-evolution-used-to-evolve-life-on-earth region. That mass was instead in the holy-crap-we-are-within-6-OOMs region, probably because of GPT-3 and the scaling hypothesis. My basic position hasn't changed much since then, just become incrementally more confident as more evidence has rolled in & as I've heard the counterarguments and been dissatisfied by them.

The International Energy Agency releases regular reports in which it forecasts the growth of various energy technologies for the next few decades. It's been astoundingly terrible at forecasting solar energy for some reason. Marvel at this chart:

This is from an article criticizing the IEA's terrible track record of predictions. The article goes on to say that there should be about 500GW of installed capacity by 2020. This article was published in 2020; a year later, the 2020 data is in, and it's actually 714 GW. Even the article criticizing the IEA for their terrible track record managed to underestimate solar's rise one year out!

Anyhow, probably there were other people who successfully predicted it. But not these people. (I'd be interested to hear more about this--was the IEA representative of mainstream opinion? Or was it being laughed at even at the time? EDIT: Zac comments to say that yeah, plausibly they were being laughed at even then, and certainly now. Whew.)

Meanwhile, here was Carl Shulman in 2012: [LW(p) · GW(p)] 

The continuation of the solar cell and battery cost curves are pretty darn impressive. Costs halving about once a decade, for several decades, is pretty darn impressive. One more decade until solar is cheaper than coal is today, and then it gets cheaper (vast areas of equatorial desert could produce thousands of times current electricity production and export in the form of computation, the products of electricity-intensive manufacturing, high-voltage lines, electrolysis to make hydrogen and hydrocarbons, etc). These trends may end before that, but the outside view looks good.

There have also been continued incremental improvements in robotics and machine learning that are worth mentioning, and look like they can continue for a while longer. Vision, voice recognition, language translation, and the like have been doing well.  [LW(p) · GW(p)]

"One more decade until solar is cheaper than coal is today..."

Anyhow, all of this makes me giggle, so I thought I'd share it. When money is abundant, knowledge is the real wealth. [LW · GW]In other words, many important kinds of knowledge are not for sale. If you were a rich person who didn't have generalist research skills and didn't know anything about solar energy, and relied on paying other people to give you knowledge, you would have listened to the International Energy Agency's official forecasts rather than Carl Shulman or people like him, because you wouldn't know how to distinguish Carl from the various other smart opinionated uncredentialed people all saying different things.

Eric Drexler has argued that the computational capacity of the human brain is equivalent to about 1 PFlop/s, that is, we are already past the human-brain-human-lifetime milestone. (Here is a gdoc.) The idea is that we can identify parts of the human brain that seem to perform similar tasks to certain already-existing AI systems. It turns out that e.g. 1-thousandth of the human brain is used to do the same sort of image processing tasks that seem to be handled by modern image processing AI... so then that means an AI 1000x bigger than said AI should be able to do the same things as the whole human brain, at least in principle.

Has this analysis been run with nonhuman animals? For example, a chicken brain is a lot smaller than a human brain, but can still do image recognition, so perhaps the part of the chicken that does image recognition is smaller than the part of the human that does image recognition.

One thing I find impressive about GPT-3 is that it's not even trying to generate text.

Imagine that someone gave you a snippet of random internet text, and told you to predict the next word. You give a probability distribution over possible next words. The end.

Then, your twin brother gets a snippet of random internet text, and is told to predict the next word. Etc. Unbeknownst to either of you, the text your brother gets is the text you got, with a new word added to it according to the probability distribution you predicted.

Then we repeat with your triplet brother, then your quadruplet brother, and so on.

Is it any wonder that sometimes the result doesn't make sense? All it takes for the chain of words to get derailed is for one unlucky word to be drawn from someone's distribution of next-word prediction. GPT-3 doesn't have the ability to "undo" words it has written; it can't even tell its future self what its past self had in mind when it "wrote" a word!

EDIT: I just remembered Ought's experiment with getting groups of humans to solve coding problems by giving each human 10 minutes to work on it and then passing it on to the next. The results? Humans sucked. The overall process was way less effective than giving 1 human a long period to solve the problem. Well, GPT-3 is like this chain of humans!

I created a Manifold market for a topic of interest to me:

https://manifold.markets/DanielKokotajlo/gpt4-or-better-model-available-for?r=RGFuaWVsS29rb3Rhamxv 

Has anyone done an expected value calculation, or otherwise thought seriously about, whether to save for retirement? Specifically, whether to put money into an account that can't be accessed (or is very difficult to access) for another twenty years or so, to get various employer matching or tax benefits?

I did, and came to the conclusion that it didn't make sense, so I didn't do it. But I wonder if anyone else came to the opposite conclusion. I'd be interested to hear their reasoning.

ETA: To be clear, I have AI timelines in mind here. I expect to be either dead, imprisoned, or in some sort of posthuman utopia by the time I turn 60 or whenever it was that my retirement account would unlock. Thus I expect the money in said account to be worthless; thus unless the tax and matching benefits were, like, 10X compared to regular investment, it wouldn't be worth it. And that's not even taking into account the fact that money is more valuable to me in the next 10 years then after I retire. And that's not even taking into account that I'm an altruist and am looking to spend money in ways that benefit the world -- I'm just talking about "selfish spending" here.

Years after I first thought of it, I continue to think that this chain reaction is the core of what it means for something to be an agent, AND why agency is such a big deal, the sort of thing we should expect to arise and outcompete non-agents. Here's a diagram:

Roughly, plans are necessary for generalizing to new situations, for being competitive in contests for which there hasn't been time for natural selection to do lots of optimization of policies. But plans are only as good as the knowledge they are based on. And knowledge doesn't come a priori; it needs to be learned from data. And, crucially, data is of varying quality, because it's almost always irrelevant/unimportant. High-quality data, the kind that gives you useful knowledge, is hard to come by. Indeed, you may need to make a plan for how to get it. (Or more generally, being better at making plans makes you better at getting higher-quality data, which makes you more knowledgeable, which makes your plans better.)

 

In this post [LW · GW], Jessicata describes an organization which believes:

1. AGI is probably coming in the next 20 years.
2. Many of the reasons we have for believing this are secret.
3. They're secret because if we told people about those reasons, they'd learn things that would let them make an AGI even sooner than they would otherwise.

At the time, I didn't understand why an organization would believe that. I figured they thought they had some insights into the nature of intelligence or something, some special new architecture for AI designs, that would accelerate AI progress if more people knew about it. I was skeptical, because what are the odds that breakthroughs in fundamental AI science would come from such an organization? Surely we'd expect such breakthroughs to come from e.g. DeepMind.

Now I realize: Of course! The secret wasn't a dramatically new architecture, it was that dramatically new architectures aren't needed. It was the scaling hypothesis. [LW · GW] This seems much more plausible to me.

The other day I heard this anecdote: Someone's friend was several years ago dismissive of AI risk concerns, thinking that AGI was very far in the future. When pressed about what it would take to change their mind, they said their fire alarm would be AI solving Montezuma's Revenge. Well, now it's solved, what do they say? Nothing; if they noticed they didn't say. Probably if they were pressed on it they would say they were wrong before to call that their fire alarm.

This story fits with the worldview expressed in "There's No Fire Alarm for AGI." I expect this sort of thing to keep happening well past the point of no return.

Bedrock is the easiest way for customers to build and scale generative AI-based applications using FMs, democratizing access for all builders.

I know it's just meaningless corporatespeak applause light, but it occurs to me that it's also technically incorrect -- the situation is more analogous to other forms of government (anarchy or dictatorship, depending on whether Amazon exercises any power) than to democracy (it's not like all the little builders get together and vote on laws that then apply to the builders who didn't vote or voted the other way.)

A nice thing about being a fan of Metaculus for years is that I now have hard evidence of what I thought about various topics back in the day. It's interesting to look back on it years later. Case in point: Small circles are my forecasts:



The change was, I imagine, almost entirely driven by the update in my timelines.

 

I speculate that drone production in the Ukraine war is ramping up exponentially and will continue to do so. This means that however much it feels like the war is all about drones right now, it'll feel much more that way a year from now. Both sides will be regularly sending flocks of shahed-equivalents at each other, both sides will have reinvented tactics to center around FPV kamikaze drones, etc. Maybe we'll even see specialized anti-drone drones dogfighting with each other, though since there aren't any of those yet they won't have appeared in large numbers.

I guess this will result in the "no man's land" widening even further, to like 10km or so. (That's about the maximum range of current FPV kamikaze drones)

Came across this short webcomic thingy on r/novelai. It was created entirely using AI-generated images. (Novelai I assume?) https://globalcomix.com/c/paintings-photographs/chapters/en/1/29
I wonder how long it took to make.

Just imagine what'll be possible this time next year. Or the year after that.

Notes on Tesla AI day presentation:

https://youtu.be/j0z4FweCy4M?t=6309 Here they claim they've got more than 10,000 GPUs in their supercomputer, and that this means their computer is more powerful than the top 5 publicly known supercomputers in the world. Consulting this list https://www.top500.org/lists/top500/2021/06/ it seems that this would put their computer at just over 1 Exaflop per second, which checks out (I think I had heard rumors this was the case) and also if you look at this https://en.wikipedia.org/wiki/Computer_performance_by_orders_of_magnitude to get a sense of how many flops GPUs do these days (10^13? 10^14?) it checks out as well, maybe.

Anyhow. 1 Exaflop means it would take about a day to do as much computation as was used to train GPT-3. (10^23 ops is 5 ooms more than 10^18, so 100,000 seconds.) So in 100 days, could do something 2 OOMs bigger than GPT-3. So... I guess the conclusion is that scaling up the AI compute trend by +2 OOMs will be fairly easy, and scaling up by +3 or +4 should be feasible in the next five years. But +5 or +6 seems probably out of reach? IDK I'd love to see someone model this in more detail. Probably the main thing to look at is total NVIDIA or TSMC annual AI-focused chip production.

https://youtu.be/j0z4FweCy4M?t=7467 Here it says Tesla's AI training computer is:

4x more compute per dollar

1.3x more energy-efficient

than... what? I didn't catch what they were comparing to.

They say they can do another 10x improvement in the next generation.

If they are comparing to the state of the art, that's a big deal I guess?

https://www.youtube.com/watch?v=j0z4FweCy4M Here Elon says their new robot is designed to be slow and weak so that humans can outrun it and overpower it if need be, because "you never know. [pause] Five miles an hour, you'll be fine. HAHAHA. Anyways... "

Example tasks it should be able to do: "Pick up that bolt and attach it to that car. Go to the store and buy me some groceries."

Prototype should be built next year.

The code name for the robot is Optimus. The Dojo simulation lead guy said they'll be focusing on helping out with the Optimus project in the near term. That's exciting because the software part is the hard part; it really seems like they'll be working on humanoid robot software/NN training in the next year or two.

In a recent conversation, someone said the truism about how young people have more years of their life ahead of them and that's exciting. I replied that everyone has the same number of years of life ahead of them now, because AI timelines. (Everyone = everyone in the conversation, none of whom were above 30)

I'm interested in the question of whether it's generally helpful or harmful to say awkward truths like that. If anyone is reading this and wants to comment, I'd appreciate thoughts.

Probably, when we reach an AI-induced point of no return, AI systems will still be "brittle" and "narrow" in the sense used in arguments against short timelines.

Argument: Consider AI Impacts' excellent point that "human-level" is superhuman (bottom of this page)

The point of no return, if caused by AI, could come in a variety of ways that don't involve human-level AI in this sense. See this post [LW · GW] for more. The general idea is that being superhuman at some skills can compensate for being subhuman at others. We should expect the point of no return to be reached at a time when even the most powerful AIs have weak points, brittleness, narrowness, etc. -- that is, even they have various things that they totally fail at, compared to humans. (Note that the situation is symmetric; humans totally fail at various things compared to AIs even today)
 

I was inspired to make this argument by reading this blast from the past which argued that the singularity can't be near because AI is still brittle/narrow. I expect arguments like this to continue being made up until (and beyond) the point of no return, because even if future AI systems are significantly less brittle/narrow than today's, they will still be bad at various things (relative to humans), and so skeptics will still have materials with which to make arguments like this.

 

How much video data is there? It seems there is plenty:

This https://www.brandwatch.com/blog/youtube-stats/ says 500 hours of video are uploaded to youtube every minute. This https://lumen5.com/learn/youtube-video-dimension-and-size/ says standard definition for youtube video is 854x480 = 409920 pixels. At 48fps, that’s 3.5e13 pixels of data every minute. Over the course of a whole year, that’s +5 OOMs, it comes out to 1.8e19 pixels of data every year. So yeah, even if we use some encoding that crunches pixels down to 10x10 vokens or whatever, we’ll have 10^17 data points of video. That seems like plenty given that according to the scaling laws and the OpenPhil brain estimate we only need 10^15 data points to train an AI the size of a human brain. (And also according to Lanrian's GPT-N extrapolations, human-level performance and test loss will be reached by 10^15 or so.)

But maybe I've done my math wrong or something?

Some ideas for definitions of AGI / resolution criteria for the purpose of herding a bunch of cats / superforecasters into making predictions: 

(1) Drop-in replacement for human remote worker circa 2023 (h/t Ajeya Cotra): 

When will it first be the case that there exists an AI system which, if teleported back in time to 2023, would be able to function as a drop-in replacement for a human remote-working professional, across all* industries / jobs / etc.? So in particular, it can serve as a programmer, as a manager, as a writer, as an advisor, etc. and perform at (say) 90%th percentile or higher at any such role, and moreover the cost to run the system would be less than the hourly cost to employ a 90th percentile human worker. 

(Exception to the ALL: Let's exclude industries/jobs/etc. where being a human is somehow important to one's ability to perform the job; e.g. maybe therapy bots are inherently disadvantaged because people will trust them less than real humans; e.g. maybe the same goes for some things like HR or whatever. But importantly, this is not the case for anything actually key to performing AI research--designing experiments, coding, analyzing experimental results, etc. (the bread and butter of OpenAI) are central examples of the sorts of tasks we very much want to include in the "ALL.")

(2) Capable of beating All Humans in the following toy hypothetical: 

Suppose that all nations in the world enacted and enforced laws that prevented any AIs developed after year X from being used by any corporation other than AICORP. Meanwhile, they enacted and enforced laws that grant special legal status to AICORP: It cannot have human employees or advisors, and must instead be managed/CEO'd/etc. only by AI systems. It can still contract humans to do menial labor for it, but the humans have to be overseen by AI systems. The purpose is to prevent any humans from being involved in high-level decisionmaking, or in corporate R&D, or in programming. 

In this hypothetical, would AI corp probably be successful and eventually become a major fraction of the economy? Or would it sputter out, flail embarrassingly, etc.? What is the first year X such that the answer is "Probably it would be successful..."?

(3) The "replace 99% of currently remote jobs" thing I used with Ajeya and Ege [LW · GW]

(4) the Metaculus definition (the hard component of which is "2 hour adversarial turing test" https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/ except instead of the judges trying to distinguish between the AI and an average human, they are trying to distinguish between a top AI researcher at a top AI lab, and the AI.

I remember being interested (and maybe slightly confused) when I read about the oft-bloody transition from hereditary monarchies to democracies and dictatorships. Specifically it interested me that so many smart, reasonable, good people seemed to be monarchists. Even during anarchic periods of civil war, the factions tended to rally around people with some degree of legitimate claim to the throne, instead of the whole royal lineage being abandoned and factions arising based around competence and charisma. Did these smart people literally believe in some sort of magic, some sort of divine power imbued in a particular bloodline? Really? But there are so many cases where the last two or three representatives of that bloodline have been obviously incompetent and/or evil!

Separately, I always thought it was interesting (and frustrating) how the two-party system in the USA works. As satirized by The Simpsons: 

Like, I'm pretty sure there are at least, idk, a thousand eligible persons X in the USA such that >50% of voters would prefer X to be president than both Trump and Biden. (e.g. someone who most Democrats think is slightly better than Biden AND most Republicans think is slightly better than Trump) So why doesn't one of those thousand people run for president and win? (This is a rhetorical question, I know the answer)

It occurs to me that maybe these things are related. Maybe in a world of monarchies where the dynasty of so-and-so has ruled for generations, supporting someone with zero royal blood is like supporting a third-party candidate in the USA.

Came across this old (2004) post from Moravec describing the evolution of his AGI timelines over time. Kudos to him, I say. Compute-based predictions seem to have historically outperformed every other AGI forecasting method (at least the ones that were actually used), as far as I can tell.
 

What if Tesla Bot / Optimus actually becomes a big deal success in the near future (<6 years?) Up until recently I would be quite surprised, but after further reflection now I'm not so sure.

Here's my best "bull case:"

Boston Dynamics and things like this https://www.youtube.com/watch?v=zXbb6KQ0xV8 establish that getting robots to walk around over difficult terrain is possible with today's tech, it just takes a lot of engineering talent and effort.

So Tesla will probably succeed, within a few years, at building a humanoid robot that can walk around and pick up some kinds of objects and open doors and stuff like that. All the stuff that Boston Dynamic's spot can do. Motors will be weak and battery life will be short though.

What about price? Thanks to economies of scale, if Tesla is producing 100,000+ bots a year the price should be less than the price of a Tesla car.

The hard part is getting it to actually do useful tasks, to justify producing that many (or indeed to justify producing it at all). Just walking around and opening doors is already useful (see: Boston Dynamics being able to sell a few thousand Spot robots) but it seems like a pretty small niche.

As Tesla's adventures with FSD beta show, navigating the real world is hard (at least if you have a brain the size of an ant.) However if you look at it differently, the lesson is maybe the opposite: Navigating the real world is easy so long as you have lots of training data and are willing to accept occasional crashes. (Teslas have, like, one plausibly-necessary intervention every half hour of city driving now, according to my brother.) If a Tesla Bot falls over, or bumps into a wall, or drops what it is carrying about once every half hour... that's actually fine? Not a big deal if it's job is to stock shelves or sweep floors or flip burgers. Replace half your employees with bots & (since they can work 24/7) have them do the jobs of all the employees, and then have your remaining employees job be micromanaging the bots and cleaning up the messes they constantly make. Recoup your investment in less than a year.

And (the bullish hope continues) you can use a much more aggressive, risk-accepting training strategy if you aren't trying to completely avoid failures. Whereas Tesla is trying to create an AI driver without letting it kill anyone along the way, for their Optimus bots it's totally fine if they are constantly falling down and dropping stuff for a period until they get good. So perhaps the true rate of dropping objects and falling over could be fairly easily driven down to once a day or less--comparable to clumsy human workers.

DeepMind has done a few experiments where AIs in a simulated house hear english-language instructions like "go to the bathroom put all the green stuff on the table" and "follow me." It works. Now, that's just a simulated house rather than a real house. But maybe an AI pre-trained in simulated environments can then be retrained/fine-tuned in real houses with real humans giving instructions. No doubt it would take many many hours of trial and error to get good, but... Tesla can easily afford that. They can build 100,000 robots and probably have 100,000 beta testers paying them for the privilege of providing training data. If the project totally flops? They've lost maybe five billion dollars or so. Not the end of the world for them, it's not like they'd be betting the company on it. And they've been willing to bet the company on things in the not-so-distant past.

Yes, this is massively more economically useful than any similar robot ever made. It would be regarded as a huge leap forward in AI and robotics. But maybe this isn't strong reason to be skeptical -- after all, no robot has ever had 100,000 instances to learn from before except Tesla's self-driving cars, and those actually do pretty well if you look at it in the right way. (Am I forgetting something? Has a deep learning experiment with 100,000 robots doing tasks in the real world been done before?)

So what's the bearish case? Or counterarguments to the bullish case?

Random idea: Hard Truths Ritual:

Get a campfire or something and a notepad and pencil. Write down on the pad something you think is probably true, and important, but which you wouldn't say in public due to fear of how others would react. Then tear off that piece of paper and toss it in the fire. Repeat this process as many times as you can for five minutes; this is a brainstorming session, so your metric for success is how many diverse ideas you have multiplied by their average quality.

Next, repeat the above except instead of "you wouldn't say in public..." it's "you wouldn't say to anyone."

Next, repeat the above except instead of "you wouldn't say..." it's "you are afraid even to think/admit to yourself."

Next, repeat the above except instead of "you think is probably true and important" it's "you think is important, and probably not true, but maaaaybe true and if it was true it would be hard for you to admit to yourself."

I've never done this entire ritual but I plan to try someday unless someone dissuades me. (I'm interested to hear suggestions on how to improve the idea. Maybe writing things down is worse than just thinking about them in your head, for example. Or maybe there should be some step at the end that makes you feel better since the default outcome might be that you feel like shit.) I've done mini-versions in my head sometimes while daydreaming and found it helpful.

General theory of why this is helpful:

Humans deceive themselves all the time. We deceive ourselves so as to be better at deceiving others. Many of our beliefs are there because they make us look good or feel good.

However, if you want to see the world clearly and believe true things (which isn't for everyone, but may be for you. Think about it. Given what I said above it's not a trivial choice) then maybe this ritual can help you do that, because it puts your brain in a state where it is psychologically safe to admit those hard truths.

I recommend The Meme Machine, it's a shame it didn't spawn a huge literature. I was thinking a lot about memetics before reading it, yet still I feel like I learned a few important things.

Anyhow, here's an idea inspired by it:

First, here is my favorite right way to draw analogies between AI and evolution:

Evolution : AI research over time throughout the world

Gene : Bit of code on Github

Organism : The weights of a model

Past experiences of an organism : Training run of a model

With that as background context, I can now present the idea.

With humans, memetic evolution is a thing. It influences genetic evolution and even happens fast enough to influence the learning of a single organism over time. With AIs, memetic evolution is pretty much not a thing. Sure, the memetic environment will change somewhat between 2020 and whenever APS-AI is built, but the change will be much less than all the changes that happened over the course of human evolution. And the AI training run that produces the first APS-AI may literally involve no memetic change at all (e.g. if it's trained like GPT-3).

So. Human genes must code for the construction of a brain + learning procedure that works for many different memetic environments, and isn't overspecialized to any particular memetic environment. Whereas the first APS-AI might be super-specialized to the memetic environment it was trained in.

This might be a barrier to building APS-AI; maybe it'll be hard to induce a neural net to have the right sort of generality/flexibility because we don't have lots of different memetic environments for it to learn from (and even if we did, there's the further issue that the memetic environments wouldn't be responding to it simultaneously) and maybe this is somehow a major block to having APS capabilities.

More likely, I think, is that APS-AI will still happen but it'll just lack the human memetic generality. It'll be "overfit" to the current memetic landscape. Maybe.

I spent way too much time today fantasizing about metal 3D printers. I understand they typically work by using lasers to melt a fine layer of metal powder into solid metal, and then they add another layer of powder and melt more of it, and repeat, then drain away the unmelted powder. Currently they cost several hundred thousand dollars and can build stuff in something like 20cm x 20cm x 20cm volume. Well, here's my fantasy design for an industrial-scale metal 3D printer, that would probably be orders of magnitude better in every way, and thus hopefully orders of magnitude cheaper. Warning/disclaimer: I don't have any engineering or physics background and haven't bothered to check several things so probably I'm making stupid mistakes somewhere.

1. The facility consists of a big strong concrete-and-steel dome, with big airlocks in the sides. The interior is evacuated, i.e. made a vacuum.

2. The ground floor of the facility is flat concrete, mostly clear. On it roll big robot carts, carrying big trays of metal powder. They look like minecarts because they have high sides and need to be able to support a lot of weight. They have a telescoping platform on the interior so that the surface of the powder is always near the top of the cart, whether the powder is a meter deep or a millimeter. The carts heat the powder via induction coils, so that it is as close as possible to the melting point without actually being in danger of melting.

3. The ceiling of the facility (not literally the ceiling, more like a rack that hangs from it) is a dense array of laser turrets. Rows and rows and rows of them. They don't have to be particularly high-powered lasers because the metal they are heating up is already warm and there isn't any atmosphere to diffuse the beam or dissipate heat from the target. When laser turrets break they can be removed and replaced easily, perhaps by robots crawling above them.

4. Off in one corner of the facility is the powder-recycling unit. It has a bunch of robot arms to lift out the finished pieces while the carts dump their contents, the unused powder being swept up and fed to powder-sprinkler-bots.

5. There are a bunch of powder-sprinkler-bots that perhaps look like the letter n. They roll over carts and deposit a fresh layer of powder on them. The powder is slightly cooler than the powder in the carts, to counteract the extra heat being added by the lasers.

6. The overall flow is: Carts arrange themselves like a parking lot on the factory floor; powder-sprinklers pair up with them and go back and forth depositing powder. The laser arrays above do their thing. When carts are finished or powder-sprinklers need more powder they use the lanes to go to the powder-recycling unit. Said unit also passes finished product through an airlock to the outside world, and takes in fresh shipments of powder through another airlock.

7. Pipes of oil or water manage heat in the facility, ensuring that various components (e.g. the lasers) stay cool, transporting the waste heat outside.

8. When machines break down, as they no doubt will, (a) teleoperated fixer robots might be able to troubleshoot the problem, and if not, (b) they can just wheel the faulty machine through an airlock to be repaired by humans outside.

9. I think current powder printer designs involve a second step in which the product is baked for a while to finish it? If this is still required, then the facility can have a station that does that. There are probably massive economies of scale for this sort of thing.

Overall this facility would allow almost-arbitrarily-large 3D objects to be printed; you are only limited by the size of your carts (and you can have one or two gigantic carts in there). Very little time would be wasted, because laser turrets will almost always have at least one chunk of cart within their line of fire that isn't blocked by something. There are relatively few kinds of machine (carts, turrets, sprinklers) but lots of each kind, resulting in economies of scale in manufacturing and also they are interchangeable so that several of them can be breaking down at any given time and the overall flow won't be interrupted. And the vacuum environment probably (I don't know for sure, and have no way of quantifying) helps a lot in many ways--lasers can be cheaper, less worry about dust particles messing things up, hotter powder melts quicker and better, idk. I'd love to hear an actual physicist's opinion. Does the vacuum help? Do open cart beds with ceiling lasers work, or is e.g. flying molten metal from neighboring carts a problem, or lasers not being able to focus at that range?

Outside the vacuum dome, printed parts would be finished and packaged into boxes and delivered to whoever ordered them.

If one of these facilities set up shop in your city, why would you ever need a metal 3D printer of your own? You could submit your order online and immediately some unused corner of some powder bed would start getting lasered, and the finished product would be available for pickup or delivery, just like a pizza.

If we ever start doing fancier things with our 3D printers, like using different types of metal or embedding other substances into them, the facility could be easily retrofitted to do this--just add another type of sprinkler-robot. (I guess the heated powder could be a problem? But some carts could be non-heated, and rely on extra-duration laser fire instead, or multiple redundant lasers.)

Ballistics thought experiment: (Warning: I am not an engineer and barely remember my high school physics)

You make a hollow round steel shield and make it a vacuum inside. You put a slightly smaller steel disc inside the vacuum region. You make that second disc rotate very fast. Very fast indeed.

An incoming projectile hits your shield. It is several times thicker than all three layers of your shield combined. It easily penetrates the first layer, passing through to the vacuum region. It contacts the speedily spinning inner layer, and then things get crazy... the tip of the projectile gets eroded by the spinning inner layer (Yes, it's spinning faster than the projectile is travelling, I told you it was fast).

I'm not sure what happens next, but I can think of two plausible possibilities:

1. The projectile bounces off the inner layer, because it's effectively been blocked not just by the material in front of it but by a whole ring of material, which collectively has more mass than the projectile. The ring slows down a bit and maybe has a dent in it now, a perfectly circular groove.

2. The inner layer disintegrates, but so does the projectile. The momentum of the ring of material that impacts the projectile is transferred to the projectile, knocking it sideways and probably breaking it into little melty pieces. The part of the inner layer outside the ring of material is now a free-spinning hula-hoop, probably a very unstable one which splits and shoots fragments in many directions.

Either way, the projectile doesn't make it through your shield.

So, does this work?

If it works, would it be useful for anything? It seems to be a relatively light form of armor; most armor has to be of similar (or greater) thickness to the projectile in order to block it, but this armor can be an order of magnitude less thick, possibly two or three. (Well, you also have to remember to account for the weight of the outer layer that holds back the vacuum... But I expect that won't be much, at least when we are armoring against tank shells and the like.)

So maybe it would be useful on tanks or planes, or ships. Something where weight is a big concern.

The center of the shield is a weak point I think. You can just put conventional armor there.

Problem: Aren't spinning discs harder to move about than still discs? They act like they have lots of mass, without having lots of weight? I feel like I vaguely recall this being the case, and my system 1 physics simulator agrees. This wouldn't reduce the armor's effectiveness, but it would make it really difficult to maneuver the tank/plane/ship equipped with it.

Problem: Probably this shield would become explosively useless after blocking the first shot. It's ablative armor in the most extreme sense. This makes it much less useful.

Solution: Maybe just use them in bunkers? Or ships? The ideal use case would be to defend some target that doesn't need to be nimble and which becomes much less vulnerable after surviving the first hit. Not sure if there is any such use case, realistically...

Searching for equilibria can be infohazardous. You might not like the one you find first, but you might end up sticking with it (or worse, deviating from it and being punished). This is because which equilbrium gets played by other people depends (causally or, in some cases, acausally) not just on what equilibrium you play but even on which equilibria you think about. For reasons having to do with schelling points. A strategy that sometimes works to avoid these hazards is to impose constraints on which equilibria you think about, or at any rate to perform a search through equilibria-space that is guided in some manner so as to be unlikely to find equilibria you won't like. For example, here is one such strategy: Start with a proposal that is great for you and would make you very happy. Then, think of the ways in which this proposal is unlikely to be accepted by other people, and modify it slightly to make it more acceptable to them while keeping it pretty good for you. Repeat until you get something they'll probably accept.

Proposed Forecasting Technique: Annotate Scenario with Updates (Related to Joe's Post [LW · GW])

• Consider a proposition like "ASI will happen in 2024, not sooner, not later." It works best if it's a proposition you assign very low credence to, but that other people you respect assign much higher credence to.
• What's your credence in that proposition?
• Step 1: Construct a plausible story of how we could get to ASI in 2024, no sooner, no later. The most plausible story you can think of. Consider a few other ways it could happen too, for completeness, but don't write them down.
• Step 2: Annotate the story with how you think you should update your credence in the proposition, supposing events play out as described in the story. By the end of the story, presumably your credence should be 100% or close to it.
• Step 3: Consider whether the pattern of updates makes sense or is irrational in some way. Adjust as needed until you are happy with them.
  • For example, maybe your first draft has you remain hovering around 1% up until the ASI is discovered to have escaped the lab. But on reflection you should have partially updated earlier, when the lab announced their exciting results, and earlier still when you learned the lab was doing a new kind of AGI project with special sauce X.

You may end up concluding that your current credence should be higher than 1%. But if not, consider the following:

• You now have some near-term testable predictions which you are soft-committed to updating on. Unless you've bunched all your updates towards the end of the story, if you started with a mere 1%, you gotta do something like 7 doublings of credence in the next 18 months, assuming events play out as they do in the story. Presumably some of those doublings should happen in the next six months or so. So, six months from now, if things have gone more or less as the story indicated, your credence should have at least doubled and probably quadrupled.
• You can also ask: Suppose you did this exercise two years ago. Have things gone more or less as the 2024-story you would have made two years ago would have gone? If so, have you in fact updated with 3-5 doublings?

I'm interested to hear feedback on this technique, especially endorsements and critiques.

Science as a kind of Ouija board:

With the board, you do this set of rituals and it produces a string of characters as output, and then you are supposed to read those characters and believe what they say.

So too with science. Weird rituals, check. String of characters as output, check. Supposed to believe what they say, check.

With the board, the point of the rituals is to make it so that you aren't writing the output, something else is -- namely, spirits. You are supposed to be light and open-minded and 'let the spirit move you' rather than deliberately try to steer things towards any particular outputs.

With science, the point of the rituals is to make it so that you aren't writing the output, something else is -- namely, reality. You are supposed to be light and open-minded and 'let the results speak for themselves' rather than deliberately try to steer things towards any particular outputs.

With the board, unfortunately, it turns out that humans are quite capable of biasing results even when they don't think that's what they are doing. Since spirits don't actually exist, if you truly were light and open-minded and letting the spirits move you, the result would be gibberish. Instead the result is almost always coherent and relevant English text.

So too with science. Despite using all the latest and greatest techniques, Bem was able to prove multiple times that ESP was real, even though it isn't. This is just one example of literally thousands. For relevant statistics, see "Replication crisis" and related issues.

So. Reality is real, and science can work, if done correctly. But the lesson of the Ouija board is that doing it correctly is nontrivial and that it's entirely possible for well-meaning humans to do it incorrectly and think they are doing it correctly.

I just wanted to signal-boost this lovely "letter to 11-year-old self" written by Scott Aaronson. It's pretty similar to the letter I'd write to my own 11-year-old self. What a time to be alive!

Suppose you are the CCP, trying to decide whether to invade Taiwan soon. The normal-brain reaction to the fiasco in Ukraine is to see the obvious parallels and update downwards on "we should invade Taiwan soon."

But (I will argue) the big-brain reaction is to update upwards, i.e. to become more inclined to invade Taiwan than before. (Not sure what my all-things considered view is, I'm a bit leery of big-brain arguments) Here's why:

Consider this list of variables:

1. How much of a fight the Taiwanese military will put up
2. How competent the Chinese military is
3. Whether the USA will respond with military force or mere sanctions-and-arms-deliveries
    

These variables influence who would win and how long it would take / how costly it would be, which is the main variable influencing the decision of whether to invade.

Now consider how the fiasco in Ukraine gives evidence about those variables. In Ukraine, 
1. The Russians were surprisingly incompetent
2. The Ukrainians put up a surprisingly fierce resistance
3. The USA responded with surprisingly harsh sanctions, but stopped well short of actually getting involved militarily.

This should update us towards expecting the Chinese military to be more incompetent, the Taiwanese to put up more of a fight, and the USA to be more likely to respond merely with sanctions and arms deliveries than with military force.

However, the incompetence of the Russian military is only weak evidence about the incompetence of the Chinese military. 

The strength of Ukrainian resistance is stronger evidence about the strength of Taiwanese resistance, because there is a causal link: Ukrainian resistance was successful and thus will likely inspire the Taiwanese to fight harder. (If not for this causal link, the evidential connection would be pretty weak.)

But the update re: predicted reaction of USA should be stronger still, because we aren't trying to infer the behavior of one actor from the behavior of another completely different actor. It's the same actor in both cases, in a relevantly similar situation. And given how well the current policy of sanctions-and-arms-deliveries is working for the USA, it's eminently plausible that they'd choose the same policy over Taiwan. Generals always fight the last war, as the saying goes.

So, the big-brain argument concludes, our estimates of variable 1 should basically stay the same, our estimate of variable 2 should change to make invasion somewhat less appealing, and our estimate of variable 3 should change to make invasion significantly more appealing.

Moreover, variable 3 is way more important than variable 2 anyway. Militarily Taiwan has less of a chance against China than Ukraine had against Russia, much less. (Fun fact: In terms of numbers, even on Day 1 of the invasion the Russians didn't really outnumber the Ukrainians, and very quickly they were outnumbered as a million Ukrainian conscripts and volunteers joined the fight.) China, by contrast, can drop about as many paratroopers on Taiwan in a single day as there are Taiwanese soldiers.) By far the more relevant variable in whether or not to invade is what the USA's response will be. And if the USA responds in the same way that it did in Ukraine recently, that's great news for China, because economic sanctions and arms deliveries take months to have a significant effect, and Taiwan won't last that long.

So, all things considered, the events in Ukraine should update the CCP to be more inclined to invade Taiwan soon, not less.

Thoughts?

"One strong argument beats many weak arguments."

Several professors told me this when I studied philosophy in grad school. It surprised me at the time--why should it be true? From a Bayesian perspective isn't it much more evidence when there are a bunch of weak arguments pointing in the same direction, than when there is only one argument that is stronger?

Now I am older and wiser and have lots more experience, and this saying feels true to me. Not just in philosophy but in most domains, such as AGI timelines and cause prioritization.

Here are some speculations about why (and how) this saying might be true:

1. Strong evidence is common [LW · GW], and maybe in particular the distribution of evidence strength is heavy-tailed. In other words, maybe one strong argument typically beats many weak arguments for the same reason that one rich man typically has more money than many poor men--in any particular collection of random people/arguments, most of the money/evidence is concentrated in the richest/strongest.

2. If you have maximally high standards and accept only the most rigorous arguments, you'll only find arguments supporting the true answer and none supporting the false answer. If you lower your standards maximally and accept any old word salad, by symmetry you'll have as many arguments supporting the true as the false answer. Maybe standards in between continuously interpolate between these two data points: The lower your standards, the higher the ratio of arguments-for-the-true-answer to arguments-for-the-false-answer.

Maybe this isn't by itself enough to make the saying true. But maybe if we combine it with selection effects and biases... If you are considering a class of weak arguments that contains 60% arguments for the true answer and 40% arguments for the false answer, it'll be extremely easy for you to assemble a long list of arguments for the false answer, if for some reason you try. And you might try if you happen to start out believing the false answer, or having a hunch that it is true, and then do some biased search process...

In stories (and in the past) important secrets are kept in buried chests, hidden compartments, or guarded vaults. In the real world today, almost the opposite is true: Anyone with a cheap smartphone can roam freely across the Internet, a vast sea of words and images that includes the opinions and conversations of almost every community. The people who will appear in future history books are right now blogging about their worldview and strategy! The most important events of the next century are right now being accurately predicted by someone, somewhere, and you could be reading about them in five minutes if you knew where to look! The plans of the powerful, the knowledge of the wise, and all the other important secrets are there for all to see -- but they are hiding in plain sight. To find these secrets, you need to be able to swim swiftly through the sea of words, skimming and moving on when they aren't relevant or useful, slowing down and understanding deeply when they are, and using what you learn to decide where to look next. You need to be able to distinguish the true and the important from countless pretenders. You need to be like a detective on a case with an abundance of witnesses and evidence, but where the witnesses are biased and unreliable and sometimes conspiring against you, and the evidence has been tampered with. The virtue you need most is rationality.

From r/chatGPT: Someone prompted ChatGPT to "generate a meme that only AI would understand" and got this:

Might make a pretty cool profile pic for a GPT-based bot.

When God created the universe, He did not render false all those statements which are unpleasant to say or believe, nor even those statements which drive believers to do crazy or terrible things, because He did not exist. As a result, many such statements remain true.

Another hard-sci-fi military engineering idea: Shield against missile attacks built out of drone swarm.

Say you have little quadcopters that are about 10cm by 10cm square and contain about a bullet's worth of explosive charge. You use them to make a flying "blanket" above your forces; they fly in formation with about one drone per every square meter. Then if you have 1,000 drones you can make a 1,000 sq meter blanket and position it above your vehicles to intercept incoming missiles. (You'd need good sensors to detect the missiles and direct the nearest drone to intercept) With drones this small they'd be cheaper than missiles (and therefore a worthy trade) and also you could carry around that many drones in a truck or something fairly easily.

Problem: Can they actually move quickly enough to scoot 1 meter in an arbitrary direction to intercept a missile? Not sure. To figure out I could look up typical missile velocities and compare to acceleration abilities of small drones to get a sense of how far out the missile would need to be spotted & tracked. You can always have the drones cluster more densely, but 

Problem: Powering all those drones would drain a lot of energy. You'd need to have maybe 2x as many drones as actually fit in your shield, so you could be recharging half of them while the other half is in the air. To figure this out I could look up how much energy minidrones actually use and then compare that to what e.g. a portable gas generator can produce.

Problem: There are probably countermeasures, e.g. fuel-air bombs designed to explode in the air and take out a good portion of your drone swarm, followed by a normal missile to punch through the hole. Not too concerned about this because almost all military weapons and defenses have counters, it's all a game of rock paper scissors anyway. (If there are good counters to it then that just means it's only situationally useful, i.e. in situations where your adversary doesn't happen to have those counters.) But worth thinking about more.

Problem: How is this better than just having CIWS point defense? Answer: Against a single missile it's much worse. But point defenses can be overwhelmed by barrages of missiles; if 10 missiles are coming at you at once you simply don't have time to shoot them all down with your single turret. Whereas a drone shield much more gracefully scales. Also, point defense turrets are heavy and need to be lugged around by the vehicles being defended, and also work less well in dense terrain with obscured views. Drone swarms are heavy too, but only when you aren't using them; when they are deployed your troops and equipment can move freely and enter dense urban terrain and the drones can keep up with them and morph around the terrain.

Another example of civilizational inadequacy in military procurement: 

Russia is now using Iranian Shahed-136 micro-cruise-missiles. They cost $20,000 each. [Insert napkin math, referencing size of Russian and Ukrainian military budgets]. QED.

I would love to see an AI performance benchmark (as opposed to a more indirect benchmark, like % of GDP) that (a) has enough data that we can extrapolate a trend, (b) has a comparison to human level, and (c) trend hits human level in the 2030s or beyond.

I haven't done a search, it's just my anecdotal impression that no such benchmark exists. But I am hopeful that in fact several do.

Got any ideas?

Self-embedded Agent and I agreed on the following bet: They paid me $1000 a few days ago. I will pay them $1100 inflation adjusted if there is no AGI in 2030.

Ramana Kumar will serve as the arbiter. Under unforeseen events we will renegotiate in good-faith.

As a guideline for 'what counts as AGI' they suggested the following, to which I agreed:

"the Arbiter agrees with the statement "there is convincing evidence that there is an operational Artificial General Intelligence" on 6/7/2030"

Defining an artificial general intelligence is a little hard and has a strong 'know it when you see it vibe' which is why I'd like to leave it up to Ramana's discretion.

We hold these properties to be self-evident requirements for a true Artificial General Intelligence:

1. be able to equal or outperform any human on virtually all relevant domains, at least theoretically

-> there might be e.g. physical tasks that it is artificially constrained from completing because it is lacks actuators for instance - but it should be able to do this 'in theory'. again I leave it up to the arbiter to make the right judgement call here.

it should be able to asymptotically outperform or equal human performance for a task with equal fixed data, compute, and prior knowledge

it should autonomously be able to formalize vaguely stated directives into tasks and solve these (if possible by a human)

it should be able to solve difficult unsolved maths problems for which there are no similar cases in its dataset

(again difficult, know it when you see it)

it should be immune / atleast outperform humans against an adversarial opponent (e.g. it shouldn't fail Gary Marcus style questioning)

outperform or equals humans on causal & counterfactual reasoning

This list is not a complete enumeration but a moving goalpost (but importantly set by Ramana! not me)

-> as we understand more about intelligence we peel off capability layers that turn out to not be essential /downstream of 'true' intelligence.

Importantly, I think near-future ML systems to be start to outperform humans in virtually all (data-rich) clearly defined tasks (almost) purely on scale but I feel that an AGI should be able to solve data-poor, vaguely defined tasks, be robust to adversarial actions, correctly perform counterfactual & causal reasoning and be able to autonomously 'formalize questions'.

Does anyone know why the Ukranian air force (and to a lesser extent their anti-air capabilities) are still partially functional?

Given the overwhelming numerical superiority of the Russian air force, plus all their cruise missiles etc., plus their satellites, I expected them to begin the conflict by gaining air supremacy and using it to suppress anti-air.

Hypothesis: Normally it takes a few days to do that, even for the mighty USAF attacking a place like Iraq; The Russian air force is not as overwhelmingly numerous vs. Ukraine so it should be expected to take at least a week, and Putin didn't want the ground forces to wait on the sidelines for a week.

Tonight my family and I played a trivia game (Wits & Wagers) with GPT-3 as one of the players! It lost, but not by much. It got 3 questions right out of 13. One of the questions it got right it didn't get exactly right, but was the closest and so got the points. (This is interesting because it means it was guessing correctly rather than regurgitating memorized answers. Presumably the other two it got right were memorized facts.)

Anyhow, having GPT-3 playing made the whole experience more fun for me. I recommend it. :) We plan to do this every year with whatever the most advanced publicly available AI (that doesn't have access to the internet) is.

When we remember we are all mad, all the mysteries disappear and life stands explained.

--Mark Twain, perhaps talking about civilizational inadequacy.

Came across this in a SpaceX AMA:

Q: I write software for stuff that isn't life or death. Because of this, I feel comfortable guessing & checking, copying & pasting, not having full test coverage, etc. and consequently bugs get through every so often. How different is it to work on safety critical software?

A: Having worked on both safety critical and non-safety critical software, you absolutely need to have a different mentality. The most important thing is making sure you know how your software will behave in all different scenarios. This affects the entire development process including design, implementation and test. Design and implementation will tend towards smaller components with clear boundaries. This enables those components to be fully tested before they are integrated into a wider system. However, the full system still needs to be tested, which makes end to end testing and observability an important part of the process as well. By exposing information about the decisions the software is making in telemetry, we are able to automate monitoring of the software. This automation can be used in development, regression testing, as well against software running on the real vehicles during missions. This helps us to be confident the software is working as expected throughout its entire life cycle, especially when we have crew onboard.

I am reminded of the Rocket Alignment Problem and the posts on Security Mindset.

On a bunch of different occasions, I've come up with an important idea only to realize later that someone else came up with the same idea earlier. For example, the Problem of Induction/Measure Problem. Also modal realism and Tegmark Level IV. And the anti-souls argument from determinism of physical laws. There were more but I stopped keeping track when I got to college and realized this sort of thing happens all the time.

Now I wish I kept track. I suspect useful data might come from it. Like, my impression is that these scooped ideas tend to be scooped surprisingly recently; more of them were scooped in the past twenty years than in the twenty years prior, more in the past hundred years than in the hundred years prior, etc. This is surprising because it conflicts with the model of scientific/academic progress as being dominated by diminishing returns / low-hanging-fruit effects. Then again, maybe it's not so conflicting after all -- alternate explanations include something about ideas being forgotten over time, or something about my idea-generating process being tied to the culture and context in which I was raised. Still though now that I think about it that model is probably wrong anyway -- what would it even look like for this pattern of ideas being scooped more recently not to hold? That they'd be evenly spread between now and Socrates?

OK, so the example that came to mind turned out to be not a good one. But I still feel like having the data -- and ideally not just from me but from everyone -- would tell us something about the nature of intellectual progress, maybe about how path-dependent it is or something.

Does the lottery ticket hypothesis have weird philosophical implications?

As I understand it, the LTH says that insofar as an artificial neural net eventually acquires a competency, it's because even at the beginning when it was randomly initialized there was a sub-network that happened to already have that competency to some extent at least. The training process was mostly a process of strengthening that sub-network relative to all the others, rather than making that sub-network more competent.

Suppose the LTH is true of human brains as well. Apparently at birth we have almost all the neurons that we will ever have. So... it follows that the competencies we have later in life are already present in sub-networks of our brain at birth.

So does this mean e.g. that there's some sub-network of my 1yo daughter's brain that is doing deep philosophical reflection on the meaning of life right now? It's just drowned out by all the other random noise and thus makes no difference to her behavior?

 

Two months ago [LW(p) · GW(p)]I said I'd be creating a list of predictions about the future in honor of my baby daughter Artemis. Well, I've done it, in spreadsheet form. The prediction questions all have a theme: "Cyberpunk." I intend to make it a fun new year's activity to go through and make my guesses, and then every five years on her birthdays I'll dig up the spreadsheet and compare prediction to reality.

I hereby invite anybody who is interested to go in and add their own predictions to the spreadsheet. Also feel free to leave comments asking for clarifications and proposing new questions or resolution conditions.

I'm thinking about making a version in Foretold.io, since that's where people who are excited about making predictions live. But the spreadsheet I have is fine as far as I'm concerned. Let me know if you have an opinion one way or another.

(Thanks to Kavana Ramaswamy and Ramana Kumar for helping out!)

Hot take: Process-based supervision = training against your monitoring system.

I just wish to signal-boost this Metaculus question, I'm disappointed with the low amount of engagement it has so far: https://www.metaculus.com/questions/12840/existential-risk-from-agi-vs-agi-timelines/ 

I wonder SpaceX Raptor engines could be cost-effectively used to make VTOL cargo planes. Three engines + a small fuel tank to service them should be enough for a single takeoff and landing and should fit within the existing fuselage. Obviously it would weigh a lot and cut down on cargo capacity, but maybe it's still worth it? And you can still use the plane as a regular cargo plane when you have runways available, since the engines + empty tank don't weigh that much.

[Googles a bit] OK so it looks like maximum thrust Raptors burn through propellant at 600kg/second. A C-17 Globemaster's max cargo capacity is 77,000kg. So if you had three raptors (what, a ton each?) and were planning to burn two of them for 10 seconds to land, and two of them for 20 seconds to take off, you'd have about half your cargo capacity left! (Two raptors burning should provide almost twice as much thrust as a fully loaded C-17 weighs, if I'm reading Google facts correctly, so there should be plenty of acceleration and deceleration and ability to throttle up and down etc. I put in three raptors in case one breaks, just like SpaceX did with Starship.)

And when the tanks are empty, the engines+tanks would only use up a small fraction of cargo capacity, so the plane could still be used as a regular cargo plane when airstrips are available.

A current example of civilizational inadequacy [LW · GW] in the realm of military spending:

The Ukrainian military has a budget of 4.6 billion Euro, so about $5 billion. (It also has several hundred thousand soldiers)

The Bayraktar TB2 is estimated to cost about $1-2 million. It was designed and built in Turkey and only about 300 or so have been made so far. As far as I can tell it isn't anything discontinuously great or fantastic, technology-wise. It's basically the same sort of thing the US has had for twenty years, only more affordable. (Presumably it's more affordable because it's not produced by the price-gouging US defence industry?)

The Bayraktar TB2 has been enormously successful defending Ukraine against the Russian military. See this list of visually confirmed Russian war losses, ctrl-f "Bayraktar" and then remember that this is only the stuff caught on video / camera and circulated publicly.

Previously, it was (I suspect) the single biggest factor in the Ethiopian government's turning of the tide against Tigray. (If not the Bayraktar, then other similar drones.) Prior to that, it was the single biggest factor in Azerbaijan winning its war against Armenia.

As of the start of the war, Ukraine had only about 18 Bayraktar TB2's. (12 in 2019, 6 more in 2021, if I'm reading the Wiki correctly).

...

Suppose that Ukraine had by some miracle had asked my opinion about weapons procurement and taken my advice a few years ago. They could have spent, say, 10% of their annual budget for three years on TB2's. This would amount to $1.5B. This would buy about 1,000 TB2s, almost two orders of magnitude more than they currently have, and about one order of magnitude more than they had combat aircraft at the start of the war.

I challenge anyone to argue that this would not have completely turned the tide of the current war. (I'd even venture to guess that no russian units would remain intact in Ukraine at all by this stage in the war; they would have been pushed back across the border.) Remember, the Russian air force hasn't been able to eliminate the handful of TB2s Ukraine currently operates, and no doubt would very much love to do so.

That's not even taking into account experience curve effects. If I understand the math correctly, for aerospace the progress ratio is 85%, and so each doubling of cumulative production leads to a 15% reduction in cost, and so if Turkey had scaled up production 16x to meet this new demand, the individual cost would be half as much and Ukraine could afford to buy 2,000 instead... If lots of countries had ordered more drones instead of just Ukraine, Turkey could scale up production by several orders of magnitude and unit cost for the drones could drop down towards $100,000 and Ukraine could have 10,000 of them.

...

How much do I believe this take? Well, I'm not a military expert or anything. It's possible I'm missing something. In fact part of why I'm writing this is to find out from other people whether I'm missing something. But yeah, as far as I can see given what I know, this seems basically correct. Militaries all over the world have been massively underinvesting in drones for years.

I used to think that current AI methods just aren't nearly as sample/data - efficient as humans. For example, GPT-3 had to read 300B tokens of text whereas humans encounter 2 - 3 OOMs less, various game-playing AIs had to play hundreds of years worth of games to get gud, etc.

Plus various people with 20 - 40 year AI timelines seem to think it's plausible -- in fact, probable -- that unless we get radically new and better architectures, this will continue for decades, meaning that we'll get AGI only when we can actually train AIs on medium or long-horizon tasks for a ridiculously large amount of data/episodes.

So EfficientZero came as a surprise to me, though it wouldn't have surprised me if I had been paying more attention to that part of the literature.

What gives?

Inspired by this comment [LW(p) · GW(p)]:

in linguistic there is an argument called the poverty of stimulus. The claim is that children must figure out the rules of language using only a limited number of unlabeled examples. This is taken as evidence that the brain has some kind of hard-wired grammar framework, that serves as a canvas for further learning while growing up. 

Is it possible that tools like EfficientZero help find the fundamental limits for how much training data you need to figure out a set of rules? If an artificial neural network ever manages to reconstruct the rules of English by using only the stimulus that the average children is exposed too, that would be a strong counter-argument against poverty of stimulus.

I think it is useful to distinguish between two dimensions of competitiveness: Resource-competitiveness and date-competitiveness. We can imagine a world in which AI safety is date-competitive with unsafe AI systems but not resource-competitive, i.e. the insights and techniques that allow us to build unsafe AI systems also allow us to build equally powerful safe AI systems, but it costs a lot more. We can imagine a world in which AI safety is resource-competitive but not date-competitive, i.e. for a few months it is possible to make unsafe powerful AI systems but no one knows how to make a safe version, and then finally people figure out how to make a similarly-powerful safe version and moreover it costs about the same.

Idea for how to create actually good AI-generated fiction:

Possible prerequisite: Decent fact-checker language model program / scaffold. Doesn't have to be perfect, but has to be able to grind away given a body of text (such as wikipedia) and a target sentence or paragraph to fact-check, and do significantly better than GPT4 would by itself if you asked it to write a new paragraph consistent with all the previous 50,000 tokens.

Idea 0: Self-consistent story generation

Add a block of text to the story, then fact-check that it is consistent with what came before & delete it if not. Repeat.

Example: Like AI Dungeon but costs more but delivers a higher-quality experience because you can have more intricate and coherent plots and worldbuilding.

Idea 1: Historical Fiction

Don't just check for self-consistency, browse Wikipedia and the full text of a few books on the Battle of X and check for consistency there too.

Example: Write me a 7-book game-of-thrones style multi-perspective epic, except that it's historical fiction about Cortez' conquest of Mexico. Have one of the characters be a Spaniard (perhaps Martin Lopez the shipwright?), another be a Mexican/Aztec (perhaps Cuauhtemoc?) another a warrior from Tlaxcala, and another Malintzin herself. Feel free to take creative liberties but make sure that everything that happens is consistent with current historical scholarship as represented by [database of books and articles].

Idea 2: Time travel

Use this method to brute-force a consistent ludovician (technical term for single-unchanging-timeline) time travel storyline.

Possible time travel machine design: When powered on, the machine creates a time portal with one end at time T and another end at time T+N. Whatever goes in at T+N comes out at T.

Possible generation rules:

Each new block of the story must be consistent with all previous blocks.

If the story doesn't end with the same thing going into the portal at time T+N that came out at time T, delete and restart from time T. 

(Maybe also specify that none of the matter that came out at time T can go into the portal at time T+N, for then there would be "historyless matter" which imposes a huge complexity penalty)

(Simpler design to think about: device simply checks the future to see whether temperature of device's matter is above or below X at time T. Delete stories that result in the temperature-prophecy being wrong. Generate multiple stories in which the temperature-prophecy is correct, and then select the one that is overall most plausible.)

I lurk in the discord for The Treacherous Turn, a ttrpg made by some AI Safety Camp people I mentored. It's lovely. I encourage everyone to check it out.

Anyhow recently someone asked for ideas for Terminal Goals an AGI might have in a realistic setting; my answer is below and I'm interested to hear whether people here agree or disagree with it:

Insofar as you want it to be grounded, which you might not want, here are some hypotheses people in AI alignment would toss around as to what would actually happen: (1) The AGI actually has exactly the goals and deontological constraints the trainers intended it to have. Insofar as they were training it to be honest, for example, it actually is honest in basically the human way, insofar as they were training it to make money for them, it deeply desires to make money for them, etc. Depending on how unethical and/or philosophically-careless the trainers were, such an AI could still be very bad news for the world.

(2) The AGI has some goals that caused it to perform well in training so far, but are not exactly what the trainers intended. For example perhaps the AGI's concepts of honesty and profit are different from the trainer's concepts, and now the AGI is smart enough to realize this, but it's too late because the AGI wants to be honest and profitable according to its definitions, not according to the trainer's definitions. For the RPG you could model this probably by doing a bit of mischevous philosophy and saying how the AGIs goal-concepts diverge from their human counterparts, e.g. 'the AGI only considers it dishonest if it's literally false in the way the AGI normally would interpret it, not if it's true-but-misleading or false-according-to-how-humans-might-interpret-it.' Another sub-type of this is that the AGI has additional goals besides the ones the trainers intended, e.g. they didn't intend for it to have a curiousity drive or a humans-trust-and-respect-me drive or a survival drive, but those drives were helpful for it in training so it has them now, even though they often conflict with the drives the trainers intended.

(3) The AGI doesn't really care at all about the stuff the trainers intended it to. Instead it just wants to get rewarded. It doesn't care about being honest, for example, it just wants to appear honest. Or worse it just wants the training process to continue assigning it high scores, even if the reason this is happening is because the trainers are dead and their keyboards have been hacked. (4) The AGI doesn't really care at all about the stuff the trainers intended it to. Instead it has some extremely simple goals, the simplest possible goals that motivate it to think strategically and acquire power. Maybe something like survival. idk. 

My rough sense is that AI alignment experts tend to think that 2 and 3 are the most likely outcomes of today's training methods, with 1 being in third place and 4 being in fourth place. But there's no consensus.

I'd be curious to hear whether people agree or disagree with these dogmas:

Visible loss landscape basins don't correspond to distinct algorithms — LessWrong [LW(p) · GW(p)]

--Randomly initialized neural networks of size N are basically a big grab bag of random subnetworks of size <N
--Training tends to simultaneously modify all the subnetworks at once, in a sort of evolutionary process -- subnetworks that contributed to success get strengthened and tweaked, and subnetworks that contribute to failure get weakened.
--Eventually you have a network that performs very well in training -- which probably means that it has at least one and possibly several subnetworks that perform well in training. This explains why you can usually prune away neurons without much performance degradation, and also why neural nets are so robust to small amounts of damage.
--Networks that perform well in training tend to also be capable in other nearby environments as well ("generalization") because (a) the gods love simplicity, and made our universe such that simple patterns are ubiquitous, and (b) simpler algorithms occupy more phase space in a neural net (there are more possible settings of the parameters that implement simpler algorithms), so (conclusion) a trained neural network tends to do well in training by virtue of subnetworks that implement simple algorithms that match the patterns inherent in the training environment, and often these patterns are found outside the training environment (in 'nearby' or 'similar' environments) also, so often the trained neural networks "generalize."
--So why grokking? Well, sometimes there is a simple algorithm (e.g. correct modular arithmetic) that requires getting a lot of fiddly details right, and a more complex algorithm (e.g. memorizing a look-up table) that is modular and very easy to build up piece by piece. In these cases, both algorithms get randomly initialized in various subnetworks, but the simple ones are 'broken' and need 'repair' so to speak, and that takes a while because for the whole to work all the parts need to be just so, whereas the complex but modular ones can very quickly be hammered into shape since some parts being out of shape reduces performance only partially. Thus, after training long enough, the simple algorithm subnetworks finally get into shape and then come to dominate behavior (because they are simpler & therefore more numerous).

It's no longer my top priority, but I have a bunch of notes and arguments relating to AGI takeover scenarios that I'd love to get out at some point. Here are some of them:

Beating the game in May 1937 - Hoi4 World Record Speedrun Explained - YouTube
In this playthrough, the USSR has a brief civil war and Trotsky replaces Stalin. They then get an internationalist socialist type diplomat who is super popular with US, UK, and France, who negotiates passage of troops through their territory -- specifially, they send many many brigades of extremely low-tier troops (basically just militia with rifles riding trucks and horses) on goodwill visits to those countries... and then they surprise-declare war on the whole world EXCEPT the powerful countries. So, only the weak nations of the world. But this provokes the US, UK, and France to declare war on the USSR, starting a fight with all the troops on their territory. This proves to be difficult for them; they haven't mobilized their forces and these damn commie militias are peppered throughout their countryside, running around causing havoc. (In my headcanon, this works in part because many socialists and communists within those countries decide Now is the Worldwide Revolution and join the militias and give them aid, and in part because the USSR is seen as the victim not the aggressor--e.g. the French military attacked these poor militias who were just peacefully touring through the countryside sharing their experiences of life under socialism and preaching a Better Way.) France surrenders quickly, and then the US and UK fight on for a while but eventually capitulate, because the countryside goes to the Reds and then the various military units are encircled. Then there's a conventional war against the Axis powers (helped along by the might of US, UK, and France) and then it's just mop-up.

Crazy unrealistic hacky/exploity playthrough? I mean yeah, but... well, watch the next video, keeping in mind that Cortez started with 500 men and the Aztec Empire had hundreds of thousands of warriors:

Spanish Conquest of Aztec Empire (1519 - 1521): Every Day - YouTube

For a textual description of what was happening, to complement the map, see this old post of mine [LW · GW]. I think the whole "ask to parlay with the emperor, then tell him you'll kill him if he doesn't do as you say, then rule the empire with him as a puppet" seems like a sort of hack/exploit in the game engine of reality. What other exploits are waiting to be discovered?

Even if you think Cortez just got lucky (he didn't) this just goes to show that crazy takeovers like this are in-distribution historically. It's reasonable to think that a superintelligence would be able to reliably achieve outcomes that are on the tail end of the distribution for humans.

Apparently vtubers are a thing! What interests me about this is that I vaguely recall reading futurist predictions many years ago that basically predicted vtubers. IIRC the predictions were more about pop stars and celebrities than video game streamers, but I think it still counts. Unfortunately I have no recollection where I read these predictions or what year they were made. Anyone know? I do distinctly remember just a few years ago thinking something like "We were promised virtual celebrities but that hasn't happened yet even though the tech exists. I guess there just isn't demand for it."

Historical precedents for general vs. narrow AI

• Household robots vs. household appliances: Score One for Team Narrow
• Vehicles on roads vs. a network of pipes, tubes, and rails: Score one for Team General
• Ships that can go anywhere vs. a trade network of ships optimized for one specific route: Score one for Team General

(On the ships thing -- apparently the Indian Ocean trade was specialized prior to the Europeans, with cargo being transferred from one type of ship to another to handle different parts of the route, especially the red sea which was dangerous to the type of oceangoing ship popular at the time. But then the Age of Sail happened.)

Obviously this is just three data points, two of which seem sorta similar because they both have to do with transporting stuff. It would be good to have more examples.

Productivity app idea:

You set a schedule of times you want to be productive, and a frequency, and then it rings you at random (but with that frequency) to bug you with questions like:

--Are you "in the zone" right now? [Y] [N]

--(if no) What are you doing? [text box] [common answer] [ common answer] [...]

The point is to cheaply collect data about when you are most productive and what your main time-wasters are, while also giving you gentle nudges to stop procrastinating/browsing/daydream/doomscrolling/working-sluggishly, take a deep breath, reconsider your priorities for the day, and start afresh.

Probably wouldn't work for most people but it feels like it might for me.

Putting this up in case I hadn't already: