The below is the draft of a blog post I have about why I like AI doom liability. My dream is that people read it and decide "ah yes this is the main policy we will support" or "oh this is bad for a reason Daniel hasn't noticed and I'll tell him why". I think usually you're supposed to flesh out posts, but I'm not sure that adds a ton of information in this case.

Why I like AI doom liability

• AI doom liability is my favourite approach to AI regulation. I want to sell you all on it.

• the basic idea

  • general approach to problems: sue people for the negative impacts
    • internalizes externalities
    • means that the people figuring out how to avoid are informed and aligned (rather than bureaucrats less aware of on-the-ground conditions / trying to look good / seeking power)
    • less fucked than criminal law, regulatory law
      • look at what hits the supreme court, which stuff ends up violating people's rights the worst, what's been more persistent over human history, what causes massive protests, etc.
  • first-pass approach to AI: sue for liabilities after AI takes over
    • can't do that
  • so sue for intermediate disasters, get punitive damages for how close you were to AI takeover
    • intuition: pulling liability forward into places it can be paid, for same underlying conduct.
    • also mention strict liability, liability insurance
  • See Foom Liability (Hanson, 2023), Tort Law as a Tool for Mitigating Catastrophic Risk from Artificial Intelligence (Weil, 2024).

• why it's nice

  • liability when you're more informed of risks, vs regulation now, when we know less
  • doesn't require the right person in the right position
    • judged by juries informed by lawyers on both sides, not power-hungry politically constrained
  • we don't really know what the right way to make safe AI is right now
  • good in high-risk worlds or low-risk worlds - as long as you believe in intermediate disasters
    • intermediate disasters seem plausible because slow takeoff
  • more fair: ai companies can't get away with making egregiously unsafe AI, but they're not penalized for doing stuff that is actually harmless.

• difficulties with the proposal:

  • jury discretion
    • you could give the jury the optimal formula, which isn't easy to plug numbers in, and give them a bunch of discretion how to apply it
    • or you could give them a more plug-and-play formula which sort of approximates the optimal formula, making things more predictable but less theoretically optimal.
    • it's not clear how you want to trade off predictability with theoretical optimality, or what the trade-off even looks like (Hanson's post is a bit more predictable but it's unclear how predictable it actually is).
  • positive externalities
    • in a world where research produces positive externalities, it's a really bad idea to force people to internalize all negative externalities
    • one way this is clear: open source AI. tons of positive externalities - people get to use AI to do cool stuff, and you can do research on it, maybe helping you figure out how to make AI more safely.
    • this regime, without tweaks, would likely make it economically unviable to open source large SOTA models. it's unclear whether this is optimal.
    • I don't know a principled way to deal with this.

Hot take: if you think that we'll have at least 30 more years of future where geopolitics and nations are relevant, I think you should pay at least 50% as much attention to India as to China. Similarly large population, similarly large number of great thinkers and researchers. Currently seems less 'interesting', but that sort of thing changes over 30-year timescales. As such, I think there should probably be some number of 'India specialists' in EA policy positions that isn't dwarfed by the number of 'China specialists'.

The Indian grammarian Pāṇini wanted to exactly specify what Sanskrit grammar was in the shortest possible length. As a result, he did some crazy stuff:

Pāṇini's theory of morphological analysis was more advanced than any equivalent Western theory before the 20th century. His treatise is generative and descriptive, uses metalanguage and meta-rules, and has been compared to the Turing machine wherein the logical structure of any computing device has been reduced to its essentials using an idealized mathematical model.

There are two surprising facts about this:

1. His grammar was written in the 4th century BC.
2. People then failed to build on this machinery to do things like formalise the foundations of mathematics, formalise a bunch of linguistics, or even do the same thing for languages other than Sanskrit, in a way that is preserved in the historical record.

I've been obsessing about this for the last few days.

A complaint about AI pause: if we pause AI and then unpause, progress will then be really quick, because there's a backlog of improvements in compute and algorithmic efficiency that can be immediately applied.

One definition of what an RSP is: if a lab makes observation O, then they pause scaling until they implement protection P.

Doesn't this sort of RSP have the same problem with fast progress after pausing? Why have I never heard anyone make this complaint about RSPs? Possibilities:

• They do and I just haven't seen it
• People expect "AI pause" to produce longer / more serious pauses than RSPs (but this seems incidental to the core structure of RSPs)

A theory of how alignment research should work

(cross-posted from danielfilan.com)

Epistemic status:

• I listened to the Dwarkesh episode with Gwern and started attempting to think about life, the universe, and everything
• less than an hour of thought has gone into this post
• that said, it comes from a background of me thinking [LW(p) · GW(p)] for a while about how the field of AI alignment should relate to agent foundations research

Maybe obvious to everyone but me, or totally wrong (this doesn't really grapple with the challenges of working in a domain where an intelligent being might be working against you), but:

• we currently don't know how to make super-smart computers that do our will
  • this is not just a problem of having a design that is not feasible to implement: we do not even have a sense of what the design would be
  • I'm trying to somewhat abstract over intent alignment vs control approaches, but am mostly thinking about intent alignment
  • I have not thought that much about societal/systemic risks very much, and this post doesn't really address them.
• ideally we would figure out how to do this
• the closest traction that we have: deep learning seems to work well in practice, altho our theoretical knowledge of why it works so well or how capabilities are implemented is lagging
• how should we proceed? Well:
  • thinking about theory alone has not been practical
  • probably we need to look at things that exhibit alignment-related phenomena and understand them, and that will help us develop the requisite theory
    • said things are probably neural networks
  • there are two ways we can look at neural networks: their behaviour, and their implementation.
  • looking at behaviour is conceptually straightforward, and valuable, and being done
  • looking at their implementation is less obvious
  • what we need is tooling that lets us see relevant things about how neural networks are working
  • such tools (e.g. SAEs) are not impossible to create, but it is not obvious that their outputs tell us quantities that are actually of interest
  • in order to discipline the creation of such tools, we should demand that they help us understand models in ways that matter
    • see Stephen Casper's engineer's interpretability sequence [? · GW], Jason Gross on compact proofs
  • once we get such tools, we should be trying to use them to understand alignment-relevant phenomena, to build up our theory of what we want out of alignment and how it might be implemented
    • this is also a thing that looking at the external behaviour of models in alignment-relevant contexts should be doing
• so should we be just doing totally empirical things? No.
  • firstly, we need to be disciplined along the way by making sure that we are looking at settings that are in fact relevant to the alignment problem, when we do our behavioural analysis and benchmark our interpretability tools. This involves having a model of what situations are in fact alignment-relevant, what problems we will face as models get smarter, etc
  • secondly, once we have the building blocks for theory, ideally we will put them together and make some actual theorems like "in such-and-such situations models will never become deceptive" (where 'deceptive' has been satisfactorily operationalized in a way that suffices to derive good outcomes from no deception and relatively benign humans)
• I'm imagining the above as being analogous to an imagined history of statistical mechanics (people who know this history or who have read "inventing temperature" should let me know if I'm totally wrong about it):
  • first we have steam engines etc
  • then we figure out that 'temperature' and 'entropy' are relevant things to track for making the engines run
  • then we relate temperature, entropy, and pressure
  • then we get a good theory of thermodynamics
  • then we develop statistical mechanics
• exceptions to "theory without empiricism doesn't work":
  • thinking about deceptive mesa-optimization
  • RLHF failures [LW · GW]
  • CIRL analysis
• lesson of above: theory does seem to help us analyze some issues and raise possibilities

I continue to think that agent foundations research is kind of underrated. Like, we're supposed to do mechinterp to understand the algorithm models implement - but how do we know what algorithms are good?

Shower thought[*]: the notion of a task being bounded doesn't survive composition. Specifically, say a task is bounded if the agent doing it is only using bounded resources and only optimising a small bit of the world to a limited extent. The task of 'be a human in the enterprise of doing research' is bounded, but the enterprise of research in general is not bounded. Similarly, being a human with a job vs the entire human economy. I imagine keeping this in mind would be useful when thinking about CAIS.

Similarly, the notion of a function being interpretable doesn't survive composition. Linear functions are interpretable (citation: the field of linear algebra), as is the ReLU function, but the consensus is that neural networks are not, or at least not in the same way.

I basically wish that the concepts that I used survived composition.

[*] Actually I had this on a stroll.

Frankfurt-style counterexamples for definitions of optimization

In "Bottle Caps Aren't Optimizers", I wrote about a type of definition of optimization that says system S is optimizing for goal G iff G has a higher value than it would if S didn't exist or were randomly scrambled. I argued against these definitions by providing a examples of systems that satisfy the criterion but are not optimizers. But today, I realized that I could repurpose Frankfurt cases to get examples of optimizers that don't satisfy this criterion.

A Frankfurt case is a thought experiment designed to disprove the following intuitive principle: "a person is morally responsible for what she has done only if she could have done otherwise." Here's the basic idea: suppose Alice is considering whether or not to kill Bob. Upon consideration, she decides to do so, takes out her gun, and shoots Bob. But little-known to her, a neuroscientist had implanted a chip in her brain that would have forced her to shoot Bob if she had decided not to. That said, the chip didn't activate, because she did decide to shoot Bob. The idea is that she's morally responsible, even tho she couldn't have done otherwise.

Anyway, let's do this with optimizers. Suppose I'm playing Go, thinking about how to win - imagining what would happen if I played various moves, and playing moves that make me more likely to win. Further suppose I'm pretty good at it. You might want to say I'm optimizing my moves to win the game. But suppose that, unbeknownst to me, behind my shoulder is famed Go master Shin Jinseo. If I start playing really bad moves, or suddenly die or vanish etc, he will play my moves, and do an even better job at winning. Now, if you remove me or randomly rearrange my parts, my side is actually more likely to win the game. But that doesn't mean I'm optimizing to lose the game! So this is another way such definitions of optimizers are wrong.

That said, other definitions treat this counter-example well. E.g. I think the one given in "The ground of optimization" says that I'm optimizing to win the game (maybe only if I'm playing a weaker opponent).

Live in Berkeley? I think you should consider running for the city council. Why?

• 4 seats are going to be open with no incumbents:
  • District 4: the area between Sacramento, Blake, Fulton, and University, plus the area between University, Cedar, MLK, and Fulton. Lots of rationalists live in this area. This will be a special election that's yet to be scheduled, but I imagine it will be held in April or May, with a filing deadline in late Feb / early Mar. (Or maybe it will be held at the same time as District 7, on April 16, filing deadline on EOD Feb 16)
  • District 5: north of Cedar, between Spruce and Sacramento/Tulare/Nelson. Election in November.
  • District 6: north of Hearst, between Oxford/Spruce and Wildcat Canyon Road. Election in November.
  • District 7: campus and the couple blocks immediately south of it. Borders are hard to describe, check here. Special election: filing deadline is EOD Feb 16, election is April 16.
• Nobody is running in those races yet.
• You probably have gripes with how the city is running: maybe you wish policing were different, or there were more permissive zoning, or better education.
• You probably have a bunch of friends who feel similarly who maybe would want to vote for you or support your campaign.

Here is a candidate handbook for the District 7 election, I imagine running for the other districts is similar (but with different relevant dates).

On the most recent episode of the podcast Rationally Speaking, David Shor discusses how members of the USA's Democratic Party could perform better electorally by not talking about their unpopular extreme views, but notes that many individual Democrats have better lives by talking about their unpopular extreme views that are popular with left-wing activists (e.g. because they become more prominent and get to feel good about themselves), which cause some voters to associate those unpopular extreme views with the Democratic Party and not vote for them.

This is discussed as a sad irrationality that constitutes a coordination failure among Democrats, but I found that an odd tone. Part of the model in the episode is that Democratic politicians in fact have these unpopular extreme views, but it would hurt their electoral chances if that became known. From a non-partisan perspective, you'd expect it to be a good thing to know what elected officials actually think. Now, you might think that elected officials shouldn't enact the unpopular policies that they in fact believe in, but it's odd to me that they apparently can't credibly communicate that they won't enact those policies. At any rate, I'm a bit bothered by the idea of coordinated silence to ensure that people don't know what powerful people actually think being portrayed as good.

I get to nuke LW today AMA.

I think the use of dialogues to illustrate a point of view is overdone on LessWrong. Almost always, the 'Simplicio' character fails to accurately represent the smart version of the viewpoint he stands in for, because the author doesn't try sufficiently hard to pass the ITT of the view they're arguing against. As a result, not only is the dialogue unconvincing, it runs the risk of misleading readers about the actual content of a worldview. I think this is true to a greater extent than posts that just state a point of view and argue against it, because the dialogue format naively appears to actually represent a named representative of a point of view, and structurally discourages disclaimers of the type "as I understand it, defenders of proposition P might state X, but of course I could be wrong".

A bunch of my friends are very skeptical of the schooling system and promote homeschooling or unschooling as an alternative. I see where they're coming from, but I worry about the reproductive consequences of stigmatising schooling in favour of those two alternatives. Based on informal conversations, the main reason why people I know aren't planning on having more children is the time cost. A move towards normative home/unschooling would increase the time cost of children, and as such make them less appealing to prospective parents[*]. This in turn would reduce birth rates, worsening the problem that first-world countries face in the next couple of decades of a low working-age:elderly population ratio [EDIT: also, low population leading to less innovation, also low population leading to fewer people existing who get to enjoy life]. As such, I tentatively wish that home/unschooling advocates would focus on more institutional ways of supervising children, e.g. Sudbury schools, community childcare, child labour [EDIT: or a greater emphasis on not supervising children who don't need supervision, or similar things].

[*] This is the weakest part of my argument - it's possible that more people home/unschooling their kids would result in cooler kids that were more fun to be around, and this effect would offset the extra time cost (or kids who are more willing to support their elderly parents, perhaps). But given how lucrative the first world labour market is, I doubt it.

I often see (and sometimes take part in) discussion of Facebook here. I'm not sure whether when I partake in these discussions I should disclaim that my income is largely due to Good Ventures, whose money largely comes from Facebook investments. Nobody else does this, so shrug.

Why I am less than infinitely hostile to the time / bloomberg pieces:

• They are kinda informative about the way the scene has been in the past
• Much of the behaviour described in them is pretty fucked up
• It is relevant for people to know that EA/rationality is not an abuse-free zone
• The reported people have faced professional consequences, including being expelled from the community for the most serious offenders, but given that there were several, it's plausible that others will crop up.
• They point to a dynamic of "single-mindedness on AI stuff / extremely bad at normal human relationships" that is real and kinda bad.

One result that's related to Aumann's Agreement Theorem is that if you and I alternate saying our posterior probabilities of some event, we converge on the same probability if we have common priors. You might therefore wonder why we ever do anything else. The answer is that describing evidence is strictly more informative than stating one's posterior. For instance, imagine that we've both secretly flipped coins, and want to know whether both coins landed on the same side. If we just state our posteriors, we'll immediately converge to 50%, without actually learning the answer, which we could have learned pretty trivially by just saying how our coins landed. This is related to the original proof of the Aumann agreement theorem in a way that I can't describe shortly.

Models and considerations.

There are two typical ways of deciding whether on net something is worth doing. The first is to come up with a model of the relevant part of the world, look at all the consequences of doing the thing in the model, and determine if those consequences are net positive. When this is done right, the consequences should be easy to evaluate and weigh off against each other. The second way is to think of a bunch of considerations in favour of and against doing something, and decide whether the balance of considerations supports doing the thing or not.

I prefer model-building to consideration-listing, for the following reasons:

• By building a model, you're forcing yourself to explicitly think about how important various consequences are, which is often elided in consideration-listing. Or rather, I don't know how to quantitatively compare importances of considerations without doing something very close to model-building.
• Building a model lets you check which possible consequences are actually likely. This is an improvement on considerations, which are often of the form "such-and-such consequence might occur".
• Building a model lets you notice consequences which you might not have immediately thought of. This can either cause you to believe that those consequences are likely, or look for a faulty modelling assumption that is producing those assumptions within the model.
• Building a model helps you integrate your knowledge of the world, and explicitly enforces consistency in your beliefs about different questions.

However, there are also upsides to consideration-listing:

• The process of constructing a model is pretty similar to consideration-listing: specifically, the part where one uses one's judgement to determine which aspects of reality are important enough to include.
• Consideration-listing is much easier to do, which is why it's the form that this hastily-written shortform post takes.

Hot take: the norm of being muted on video calls is bad. It makes it awkward and difficult to speak, clap, laugh, or make "I'm listening" sounds. A better norm set would be:

• use zoom in gallery mode, so somebody making noise doesn't make them more focussed than they were before
• call from a quiet room
• be more tolerant of random background sounds, the way we are IRL

As far as I can tell, people typically use the orthogonality thesis to argue that smart agents could have any motivations. But the orthogonality thesis is stronger than that, and its extra content is false - there are some goals that are too complicated for a dumb agent to have, because the agent couldn't understand those goals. I think people should instead directly defend the claim that smart agents could have arbitrary goals.

A rough and dirty estimate of the COVID externality of visiting your family in the USA for Christmas when you don't feel ill [EDIT: this calculation low-balls the externality, see below]:

You incur some number of μCOVIDs[*] a week, let's call it x. Since the incubation time is about 5 days, let's say that your chance of having COVID is about 5x/7,000,000 when you arrive at the home of your family with n other people. In-house attack rate is about 1/3, I estimate based off hazy recollections, so in expectation you infect 5xn/21,000,000 people, which is about xn/4,000,000 people.

How bad is it to infect one family member? Well, people tend to be most infectious about 1.5 days before symptoms show, which is about 3.5 days after they get infected. Furthermore, we empirically see that R is about 1 on average, so the people you infect each infect one person, who goes on to infect one other person, etc. How long until the chain ends? It looks like vaccines will be widely distributed in the USA some time between the 1st of April and the 31st of December 2021. Median date looks kinda like the 1st of September. So let's say that there's 8 months of transmission. A month has about 30.5 days, so that's 244 days of transmission, which is 244/3.5=70 people. IFR is about 0.5%, so you get about 70 × 0.5% = 0.35 deaths. Each death loses maybe 13 life-years, altho that's not quality-adjusted. Since I don't want to quality-adjust that number, that's 13 × 0.35 = 4.55 life-years lost. But some infections result in bad disability but not death. I estimate the disability burden at about equal to the mortality burden, so that's 4.55 × 2 = 9.1 QALYs lost. A year is 365 days, so that's 9.1 × 365 = 3321.5 QALDs (Quality-Adjusted Life-Days) lost.

So: when you travel to visit family, the rest of the world loses about 33xn/40,000 QALDs, where n is the number of family members and x is how many μCOVIDs you incurred over the last week. If you follow microcovid.org's advice for healthy people and incur 200 μCOVIDs per week, and your family has 4 other people, that's about two thirds of a healthy life-day lost by strangers. There are a bunch of estimates in this calculation, so this number might be off by an order of magnitude.

[*] A μCOVID is a one in one million chance of catching COVID-19.

Better to concretise 3 ways than 1 if you have the time.

Here's a tale I've heard but not verified: in the good old days, Intrade had a prediction market on whether Obamacare would become law, which resolved negative, due to the market's definition of Obamacare.

Sometimes you're interested in answering a vague question, like 'Did Donald Trump enact a Muslim ban in his first term' or 'Will I be single next Valentine's day'. Standard advice is to make the question more specific [LW · GW] and concrete into something that can be more objectively evaluated. I think that this is good advice. However, it's inevitable that your concretisation may miss out on aspects of the original vague question that you cared about. As such, it's probably better to concretise the question multiple ways which have different failure modes. This is sort of obvious for evaluating questions about things that have already happened, like whether a Muslim ban was enacted, but seems to be less obvious or standard in the forecasting setting. That being said, sometimes it is done - OpenPhil's animal welfare series of questions seems to me to basically be an example - to good effect.

This procedure does have real costs. Firstly, it's hard to concretise vague questions, and concretising multiple times is harder than concretising once. It's also hard to predict multiple questions, especially if they're somewhat independent as is necessary to get the benefits, meaning that each question will be predicted less well. In a prediction market context, this may well manifest in having multiple thin, unreliable markets instead of one thick and reliable one.

This weekend, I looked up Benquo's post on zetetic explanation [LW · GW] in order to nominate it for the 2019 review [LW · GW]. Alas, it was posted in 2018, and wasn't nominated for that year's review. Nevertheless, I've recently gotten interested in amateur radio, and have noticed that the mechanistic/physical explanations of radio waves and such that I've come across while studying for exams are not really sufficient to empower me to actually get on the radio, and more zetetic explanations are useful, altho harder to test. Anyway, I recommend re-reading the post.

My bid for forecasters: come up with conditional prediction questions to forecast likely impacts of potential US policies towards Ukraine. See this thread where I brainstorm potential such questions.

Challenges as I see it: figuring out which policies are live options, operationalizing, and figuring out good success/failure metrics.

Benefits: potentially make policy more sane, or more realistically practice doing the sort of thing that might one day make policy more sane.

Ted Kaczynski as a relatively apolitical test case for cancellation norms:

Ted Kaczynski was a mathematics professor who decided that industrial society was terrible, and waged a terroristic bombing campaign to foment a revolution against technology. As part of this campaign, he wrote a manifesto titled "Industrial Society and Its Future" and said that if a major newspaper printed it verbatim he would desist from terrorism. He is currently serving eight life sentences in a "super-max" security prison in Colorado.

My understanding is that his manifesto (which, incidentally, has been updated and given a new title "Anti-Tech Revolution: Why and How", the second edition of which was released this year) is lucid and thought-out. Here are some questions the answers to which are not obvious to me:

• Should anybody read "Industrial Society and Its Future", given its origin?
• Suppose an EA group wrote to Kaczynski in prison, asking him to write a letter about opposition to technology to be read aloud and discussed in an EA meetup, and he complied. Would it have been unacceptable for the EA group to do this, and should it be unacceptable for the EA group to hold this meetup?

I made this post with the intent to write a comment, but the process of writing the comment out made it less persuasive to me. The planning fallacy?

Here's a script I wrote to analyze how good Manifold Markets is at predicting Ukraine stuff. Basically: it's about as good as you would be if you were calibrated at 80% accuracy if you average market prices over the life of the market, and if you take the probabilities at the mid-point of the market, it's about as good as you would be if you were calibrated at 72% accuracy.

Some puzzles:

• rubber ducking is really effective
• it's very difficult to write things clearly, even if you understand them clearly

These seem like they should be related, but I don't quite know how. Maybe if someone thought about it for an hour they could figure it out.

Quantitative claims about code maintenance from Working in Public, plausibly relevant to discussion of code rot and machine intelligence:

• "most computer programmers begin their careers doing software maintenance, and many never do anything but", attributed to Nathan Ensmenger, professor at Indiana University.
• "most software at Google gets rewritten every few years", attributed to Fergus Henderson of Google.
• "A 2018 Stripe survey of software developers suggested that developers spend 42% of their time maintaining code" - link
• "Nathan Ensmenger, the informatics professor, notes that, since the early 1960s, maintenance costs account for 50% to 70% of total expenditures on software development" - paper

Here's a project idea that I wish someone would pick up (written as a shortform rather than as a post because that's much easier for me):

• It would be nice to study competent misgeneralization empirically, to give examples and maybe help us develop theory around it.
• Problem: how do you measure 'competence' without reference to a goal??
• Prior work has used the 'agents vs devices' framework, where you have a distribution over all reward functions, some likelihood distribution over what 'real agents' would do given a certain reward function, and do Bayesian inference on that vs choosing actions randomly. If conditioned on your behaviour you're probably an agent rather than a random actor, then you're competent.
• I don't like this:
  • Crucially relies on knowing the space of reward functions that the learner in question might have.
  • Crucially relies on knowing how agents act given certain motivations.
  • A priori it's not so obvious why we care about this metric.
• Here's another option: throw out 'competence' and talk about 'consequential'.
  • This has a name collision with 'consequentialist' that you'll probably have to fix but whatever.
• The setup: you have your learner do stuff in a multi-agent environment. You use the AUP metric on every agent other than your learner. You say that your learner is 'consequential' if it strongly affects the attainable utility of other agents.
• How good is this?
  • It still relies on having a space of reward functions, but there's some more wiggle-room: you probably don't need to get the space exactly right, just to have goals that are similar to yours.
    • Note that this would no longer be true if this were a metric you were optimizing over.
  • You still need to have some idea about how agents will act realistically, because if you only look at the utility attainable by optimal policies, that might elide the fact that it's suddenly gotten much computationally harder to achieve that utility.
    • That said, I still feel like this is going to degrade more gracefully, as long as you include models that are roughly right. I guess this is because this model is no longer a likelihood ratio where misspecification can just rule out the right answer.
  • It's more obvious why we care about this metric.
• Bonus round: you can probably do some thinking about why various setups would tend to reduce other agents' attainable utility, prove some little theorems, etc., in the style of the power-seeking paper.
  • Ideally you could even show a relation between this and the agents vs devices framing.
• I think this is the sort of project a first-year PhD student could fruitfully make progress on.

This is a fun Aumann paper that talks about what players have to believe to be in a Nash equilibrium. Here, instead of imagining agents randomizing, we're instead imagining that the probabilities over actions live in the heads of the other agents: you might well know exactly what you're going to do, as long as I don't. It shows that in 2-player games, you can write down conditions that involve mutual knowledge but not common knowledge that imply that the players are at a Nash equilibrium: mutual knowledge of player's conjectures about each other, players' rationality, and players' payoffs suffices. On the contrary, in 3-player games (or games with more players), you need common knowledge: common priors, and common knowledge of conjectures about other players.

The paper writes:

One might suppose that one needs stronger hypotheses in Theorem B [about 3-player games] than in Theorem A [about 2-player games] only because when $n\ge 3$, the conjectures of two players about a third one may disagree. But that is not so. One of the examples in Section 5 shows that even when the necessary agreement is assumed outright, conditions similar to those of Theorem A do not suffice for Nash equilibrium when $n\ge 3$.

This is pretty mysterious to me and I wish I understood it better. Probably it would help to read more carefully thru the proofs and examples.

Let it be known: I'm way more likely to respond to (and thereby algorithmically signal-boost) criticisms of AI doomerism that I think are dumb than those that I think are smart, because the dumb objections are easier to answer. Caveat emptor.

An attempt at rephrasing a shard theory critique of utility function reasoning, while restricting myself to things I basically agree with:

Yes, there are representation theorems that say coherent behaviour is optimizing some utility function. And yes, for the sake of discussion let's say this extends to reward functions in the setting of sequential decision-making (even tho I don't remember seeing a theorem for that). But: just because there's a mapping, doesn't mean that we can pull back a uniform measure on utility/reward functions to get a reasonable measure on agents - those theorems don't tell us that we should expect a uniform distribution on utility/reward functions, or even a nice distribution! They would if agents were born with utility functions in their heads represented as tables or something, where you could swap entries in different rows, but that's not what the theorems say!

Here are two EA-themed podcasts that I think someone could make. Maybe that someone is you!

1. More or Less, but EA (or for forecasting)

More or Less is a BBC Radio program. They take some number that's circulating around the news, and provide context like "Is that literally true? How could someone know that? What is that actually measuring? Is that a big number? Does that mean what you think it means?" - stuff like that. They spend about 10 minutes on each number, and usually include interviews with experts in the field. IMO, someone could do this for numbers that circulate around in the EA space. Another variant is to focus on forecasts - what factors are going in, what's the reasoning for those guesses, etc.

This could be pretty easy to listen to, but moderately hard to make - requires research, editing conversations down, etc.

2. AI Safety Fellowship / Course thing - the podcast.

Get someone who's doing something like the AGI Safety Fundamentals course or the Center for AI Safety's thing like that. Each week, they make a podcast episode about what they think of the week's readings - what seemed persuasive, what didn't, what was interesting, what was novel. For a long version, you could make an episode about each reading.

If someone's already doing one of these courses, I think it wouldn't be much extra work to make this podcast (after the set cost of learning how you make a podcast). It would end up having an inherently limited run (but maybe you could do future seasons about reading thru Superintelligence / MIRI chat logs / various agendas?).

A sad fact is that good methods to elicit accurate probabilities of the outcome of some future process, e.g. who will win the next election, give you an incentive to influence that outcome, e.g. by campaigning and voting for the candidate you said was more likely to win. But with mind uploading and the 'right' theory of personal identity, we can fix this!

First, suppose that you think of all psychological descendants of your current self as 'you', but you don't think of descendants of your past self as 'you'. So, if you were to make a copy of yourself tomorrow, today you would think of your copy as fully you, but your 'main self' and the 'copy' would think of themselves as totally different people, and not care particularly much about the other one winning money.

Once you're happy with that, here's what you do: first, make your prediction about who wins the next election. Then, save your brain state right after making the prediction. Then wait for the election to happen, and after the result is known, instantiate a copy of you from that saved brain state, and reward or punish them according to how good the prediction was. At the time of making the prediction, you're incentivized to be right so that your future self gets rewarded, but after the prediction is made but before the election, you think of the person who gets rewarded/punished as not you, and therefore don't want to influence the election (any more than you already did).

(NB: this assumes that acausal trade isn't a thing.)

Suppose there are two online identities, and you want to verify that they're associated with the same person. It's not too hard to verify this: for instance, you could tell one of them something secretly, and ask the other what you told the first. But how do you determine that two online identities are different people? It's not obvious how you do this with anything like cryptographic keys etc.

One way to do it if the identities always do what's causal-decision-theoretically correct is to have the two identities play a prisoner's dilemma with each other, and make it impossible to enforce contracts. If you're playing with yourself, you'll cooperate, but if you're playing with another person you'll defect.

That being said, this only works if the payoff difference between both identities cooperating and both identities defecting is greater than the amount a single person controlling both would pay to convince you that they're actually two people. Which means it only works if the amount you're willing to pay to learn the truth is greater than the amount they're willing to pay to deceive you.

Blog post request: a summary of all the UFO stuff and what odds I should put on alien visitations of earth.

'Seminar' announcement: me talking quarter-bakedly about products, co-products, deferring, and transparency. 3 pm PT tomorrow (actually 3:10 because that's how time works at Berkeley).

I was daydreaming during a talk earlier today (my fault, the talk was great), and noticed that one diagram in Dylan Hadfield-Menell's off-switch paper looked like the category-theoretic definition of the product of two objects. Now, in category theory, the 'opposite' of a product is a co-product, which in set theory is the disjoint union. So if the product of two actions is deferring to a human about which action to take, what's the co-product? I had an idea about that which I'll keep secret until the talk, when I'll reveal it (you can also read the title to figure it out). I promise that I won't prepare any slides or think very hard about what I'm going to say. I also won't really know what I'm talking about, so hopefully one of you will. The talk will happen in my personal zoom room. Message me for the passcode.

Avoid false dichotomies when reciting the litany of Tarski.

Suppose I were arguing about whether it's morally permissible to eat vegetables. I might stop in the middle and say:

If it is morally permissible to eat vegetables, I desire to believe that it is morally permissible to eat vegetables. If it is morally impermissible to eat vegetables, I desire to believe that it is morally impermissible to eat vegetables. Let me not become attached to beliefs I may not want.

But this ignores the possibility that it's neither morally permissible nor morally impermissible to eat vegetables, because (for instance) things don't have moral properties, or morality doesn't have permissible vs impermissible categories, or whether or not it's morally permissible or impermissible to eat vegetables depends on whether or not it's Tuesday.

Luckily, when you're saying the litany of Tarski, you have a prompt to actually think about the negation of the belief in question. Which might help you avoid this mistake.

An interesting tension: it's kind of obvious from a micro-econ view that group houses should have Pigouvian taxes on uCOVIDs[*] (where I pay housemates for the chance I get them sick) rather than caps on how many uCOVIDs everyone can incur per week - and of course both of these are better than "just sort of be reasonable" or having no system. But uCOVID caps are nice in that they make it significantly easier to coordinate with other houses - it's much easier to figure out how risky interacting with somebody is when they can just tell you their cap, rather than having to guess how much they'll interact with others between now and when you'll meet them. It's not totally clear to me which system ends up being better, although I think for people who have stable habits Pigouvian taxes end up better.

[*] a uCOVID, short for microCOVID, is a one in a million chance of catching COVID-19.

FYI: I am not using the dialogue matching feature. If you want to dialogue with me, your best bet is to ask me. I will probably say no, but who knows.

Research project idea: formalize a set-up with two reinforcement learners, each training the other. I think this is what's going on in baby care. Specifically: a baby is learning in part by reinforcement learning: they have various rewards they like getting (food, comfort, control over environment, being around people). Some of those rewards are dispensed by you: food, and whether you're around them, smiling and/or mimicking them. Also, you are learning via RL: you want the baby to be happy, nourished, rested, and not cry (among other things). And the baby is dispensing some of those rewards.

Questions:

• What even happens? (I think in many setups you won't get mutual wireheading)
• Do you get a nice equilibrium?
• Is there some good alignment property you can get?
  • Maybe a terrible alignment property?

This could also be a model of humans and advanced algorithmic timelines or some such thing.

Blog posts I could write up in the next few days:

• My EDC as of late Dec 2022 [EDIT: here]
• thoughts on media I consumed in 2022 (would include Kindle books + stuff I watched on Netflix and Amazon Prime Video)

An argument for stock-picking:

I'm not sure whether I can pick stocks better than the market. But if I can, then money is more valuable to me in that world, since I have better-than-market opportunities in that world but only par-with-market opportunities in the EMH world. So I should buy stocks that look good to me, at least for a while, to check whether I'm in the world where I can do that, because it's a transfer from a world where money is less valuable to me to one where money is more valuable.

I think this argument goes thru if you assume market returns are equal in both worlds, which I think I think.

Results from an experiment I just found about inside vs outside view thinking (but haven't read the actual study, just the abstract: beware!)

Contrary to expectation, participants who assigned more importance to inside factors estimated longer completion times, and participants who gave greater weight to outside factors showed higher degrees of confidence in their estimates.

Excerpts from a FB comment I made about the principle of charity. Quote blocks are a person that I'm responding to, not me. Some editing for coherence has been made. tl;dr: it's fine to conclude that people are acting selfishly, and even to think that it's likely that they're acting selfishly on priors regarding the type of situation they're in.

The essence of charitable discourse is assuming that even your opponents have internally coherent and non-selfish reasons for what they do.

If this were true, then one shouldn't engage in charitable discourse. People often do things for entirely selfish reasons. I can determine this because I often do things for entirely selfish reasons, and in general things put under selection pressure will behave in accordance with that pressure. I could elaborate or develop this point further, but I'd be surprised to learn that you disagreed. I further claim that you shouldn't assume that something isn't the case if it is often the case.

That being said, the "non-selfish" qualifier doesn't appear in what Wikipedia thinks the principle of charity is, nor does it appear in r/slatestarcodex's sidebar description of what it means to be charitable, and I don't understand why you included it. In fact, in all of these cases, the principle of charity seems like it's meant to apply to arguments or stated beliefs rather than actions in general...

Tech people don't like it when the media assumes tech companies are all in it just for the money, and have no principles. We shouldn't do the same.

You should in fact assume that tech companies are in it for the money and have no principles, at least until seeing contrary evidence, since that's the standard and best model of corporations (although you shouldn't necessarily assume the same of their employees). Regarding "we shouldn't do the same", I wholeheartedly reject the implication that if people don't like having certain inferences drawn about them, one shouldn't draw those inferences. Sometimes the truth is unflattering!

A failure of an argument against sola scriptura (cross-posted from Superstimulus)

Recently, Catholic apologist Joe Heschmeyer has produced a couple of videos arguing against the Protestant view of the Bible - specifically, the claims of Sola Scriptura and Perspicuity (capitalized because I'll want to refer to them as premises later). "Sola Scriptura" has been operationalized a few different ways, but one way that most Protestants would agree on is (taken from the Westminster confession):

The whole counsel of God, concerning all things necessary for [...] man’s salvation [...] is either expressly set down in Scripture, or by good and necessary consequence may be deduced from Scripture

"Perspicuity" means clarity, and is propounded in the Westminster confession like this:

[T]hose things which are necessary to be known, believed, and observed, for salvation, are so clearly propounded and opened in some place of Scripture or other, that not only the learned, but the unlearned, in a due use of the ordinary means, may attain unto a sufficient understanding of them.

So, in other words, Protestants think that everything you need to know to be saved is in the Bible, and is expressed so obviously that anyone who reads it and thinks about it in a reasonable way will understand it.

I take Heschmeyer's argument to be that if Sola Scriptura and Perspicuity were true, then all reasonable people who have read the Bible and believe it would agree on which doctrines were necessary for salvation - in other words, you wouldn't have a situation where one person thinks P and P is necessary for salvation, while another thinks not-P, or a third thinks that P is not necessary for salvation. But in fact this situation happens a lot, even among seemingly sincere followers of the Bible who believe in Sola Scriptura and Perspicuity. Therefore Sola Scriptura and Perspecuity are false. (For the rest of this post, I'll write Nec(P) for the claim "P is necessary for salvation" to save space.)

I think this argument doesn't quite work. Here's why:

It can be the case that the Bible clearly explains everything that you need to believe, but it doesn't clearly explain which things you need to believe. In other words, Sola Scriptura and Perspicuity say that for all P such that Nec(P), the Bible teaches P clearly - but they don't say that for such P, the Bible teaches P clearly, and also clearly teaches Nec(P). Nor do they say that the only things that are taught clearly in the Bible are things you need to believe (otherwise you could figure out which doctrines you had to believe by just noticing what things the Bible clearly teaches).

For example, suppose that the Bible clearly teaches that Jesus died for at least some people, and that followers of Jesus should get baptized, and in fact, the only thing you need to believe to be saved is that Jesus died for at least some people. In that world, people of good faith could disagree about whether you need to believe that Jesus died for at least some people, and this would be totally consistent with Sola Scriptura and Perspicuity.

Furthermore, suppose that it's not clear to people of good faith whether or not something is clear to people of good faith. Perhaps something could seem clear to you but not be clear to others of good faith, or also something could be clear but others could fail to understand it because they're not actually of good faith (you need this part otherwise you can tell if something's clear by noticing if anyone disagrees with you). Then, you can have one person who believes P and Nec(P), and another who believes not-P and Nec(not-P), and that be consistent with Sola Scriptura and Perspicuity.

For example, take the example above, and suppose that some people read the Bible as clearly saying that Jesus died for everyone (aka Unlimited Atonement), and others read the Bible as clearly saying that Jesus only died for his followers (aka Limited Atonement). You could have that disagreement, and if the two groups think the others are being disingenuous, they could both think that you have to agree with them to be saved, while still having Sola Scriptura and Perspicuity being true.

That said, Heschmeyer's argument is still going to limit the kinds of Protestantism you can adopt. In the above example, if we suppose that you can tell that neither group is in fact being disingenuous, then his argument rules out the combination of Sola Scriptura, Perspicuity, and Nec(Limited Atonement) (as well as Sola Scriptura, Perspicuity, and Nec(Unlimited Atonement)). In this way, applied to the real world, it's going to rule out versions of Protestantism that claim that you have to believe a bunch of things that sincere Christians who are knowledgeable about the Bible don't agree on. That said, it won't rule out Protestantisms that are liberal about what you can believe while being saved.