LessWrong 2.0 Reader

You define your terms when you say: 

An inquisitive (aka inquisitorial) legal system is one in which the judge acts as both judge and prosecutor, personally digging into the facts before ruling.

In the German system, digging into the facts before the ruling is part of the job of the judge. They are doing it from a neutral perspective, but digging into facts is part of what they are supposed to do. In Anglosaxon common law on the other hand it's the job of both parties of a legal case to law out all the facts that they think support their side and it's not the job to dig into facts that neither of the sides presented. 

It was nice to see C S Lewis as a reminder we've kinda been here before. 

One of the things which helped groups during the fight for the Nuclear Test Ban Treaty in the US was Joanna Macy's "despair work", which was developed from individual grief work. 

Joanna started in intelligence and has been facing X-risks and slow actions of governments with others since the 1970s, and built a network of people doing that, and she still does. She did a lot in Cernobyl, and did some of the earliest longtermism and deep time work on nuclear waste storage.

Her despair work has been adapted for climate change and rainforest protection, so I'm sure it could be adapted for AI and other Xrisks/Srisks too, and even tougher goals like achieving universal veganism, instituting rational policymaking or "dealing with parents" ;-)
 

Trainers in despair work:
https://workthatreconnects.org/find-a-facilitator/, or ask me, or Dr Chris Johnstone for recommendations.

Trainer's Manual for groups (recommended): 
- Coming Back to Life, Joanna Macy 

Books: 
- Despair and Empowerment in the Nuclear Age, Joanna Macy
- Active Hope, Chris Johnstone and Joanna Macy

More recent video:
(be ready to filter some of the 1970s vocabulary; they're both confident with intense emotion)
 

As a full throated defender of pulling the lever (given traditional assumptions such as a lack of an audience, complete knowledge of each outcomes, productivity of the people on the tracks) , there are numerous issues with your proposals:

1.) Vague alternative: You seem to be pushing towards some form of virtue ethics/basic intuitionism, but there are numerous problems with this approach. Besides determining whose basic intuitions count and whose don't, or which virtues are important, there is very real problems when these virtues conflict. For instance, imagine you are walking at night, and are trying to cross a street. The sign says red, but no cars are around. Do you jaywalk? In this circumstance, one is forced to make a decision which pits two virtues/ intuitions against each other. The beauty of utilitarianism is that it allows us to choose in these circumstances. 

2.) Subjective Morality: Yes, utilitarianism, may not be "objective" in the sense that there is no intrinsic reason to value human flourishing, but I believe utilitarianism to be the viewpoint which closest conforms to what most people value. To illustrate why this matters, image you need to decide what color to paint a room. Nobody has very strong opinions, but most people in your household prefer the color blue. Yes, blue might not be "objectively" the best, but if most of the people in your household like the color blue the most, there is little reason not to. We are all individually going to seek what we value, so we might as well collectively agree to a system which reflects the preferences of most people.

3.) Altruism in Disguise:

Another thing to notice is that virtue ethics can be a form of effective altruism when practiced in specific ways. In general, bettering yourself as a person by becoming more rational, less biased, act, will in fact make the world a better place, and giving time to form meaningful relationships, engage in leisure, etc. can actually increase productivity in the long run. 

You also seem advocate for fundamental changes in society, changes I am not sure I would agree with, but if your proposed changes are indeed the best way to increase the general happiness of the population, it would be, by definition, the goal of the EA movement. I think a lot of people look at the recent stuff with SBF and AI research and come to think the EA movement is only concerned with lofty existential risk scenarios, but there is a lot more to it then that. 

I felt a bit confused by this bit:

At some point I sit down and think about escamoles. Yeah, ants are kinda gross, but on reflection I don’t think I endorse that reaction to escamoles. I can see why my reward system would generate an “ew, gross” signal, but I model that reward as being the result of two decoupled things: either a hardcoded aversion to insects, or my actual values. I know that I am automatically averse to putting insects in my mouth so it's less likely that the negative reward is evidence of my values in this case; the signal is explained away in the usual epistemic sense by some cause other than my values. So, I partially undo the value-downgrade I had assigned to escamoles in response to the “ew, gross” reaction. I might still feel some disgust, but I consciously override that disgust to some extent.

That last example is particularly interesting, since it highlights a nontrivial prediction of this model. Insofar as reward is treated as evidence about values, and our beliefs about values update in the ordinary epistemic manner, we should expect all the typical phenomena of epistemic updating to carry over to learning about our values. Explaining-away is one such phenomenon. What do other standard epistemic phenomena look like, when carried over to learning about values using reward as evidence?

I feel like this sort of makes sense but don't quite parse why this counted as "explaining away." How do I know my hardcoded reactions aren't values? 

Ludwig Wittgenstein tried to solve this problem in an a priori fashion with a theory of "logical atomism". So without a refined model of the world. In the Tractatus he postulated that there must be "atomic" propositions. For example, the proposition that Bob is a bachelor is clearly not atomic (but complex) because it can be decomposed into the proposition that Bob is a man and that Bob is unmarried. And those are arguably themselves sort of complex statements, since the concept of a man or of marriage themselves allow for definitions from simpler terms. But at some point, we hit undefinable, primitive terms, presumably those which refer directly to particular sense data or the like. Wittgenstein then argued that these atomic propositions have to be regarded as being independent and having probability 1/2.

Or more precisely, he came up with the concept of truth tables, and counted the fraction of the rows in which the conditions of the truth tables are satisfied. Each row he assumed to be a priori equally likely when only atomic propositions are involved. So for atomic propositions P and Q, the complex proposition "P and Q" has probability 1/4 (only one out of four rows makes this proposition true, namely "true and true"), and the complex proposition "P or Q" has probability 3/4 (three out of four rows in the truth table make a disjunction true: all except "false or false").

This turns out to be equivalent to assuming that all atomic propositions have a) probability 1/2 and are b) independent of each other.

However, logical atomism was later broadly abandoned for various reasons. One is that it is hard to define what an atomic proposition is. For example, I can't assume that "this particular spot in my visual field is blue" is atomic. Because it is incompatible with the statement "this particular spot in my visual field is yellow". The same spot can't be both blue and yellow, even though that wouldn't be a logical contradiction. The two statements are therefore not independent, so they can't be atomic. But it is hard to think of anything more primitive than qualia predicates like colors.

Yeah. I'm not entirely sure what you mean by the metaphor, but I use a similar metaphor. 

I think, ideally, you'll have at least 3 plans and at least 3 different frames for "what are your plans trying to accomplish". i.e. the 3 plans are solving somewhat different problems. And this is specifically getting at the "being able to move around freely in 3 dimensions" thingy.

The actual top-level blogpost I plan to write is probably going to be called "3 plans, 2 frames, and a crux", which is based on what actually feels reasonable to ask of people in most circumstances. i.e:

• Come up with at least 3 different plans. 
• At least two of them should looking at the problem differently. 
• If it's obvious which plan is your favorite, figure out what observations you could make later on that might change your mind towards one of the other two.

When you're fluent at this sort of thing, you should have lots of different plans and backup plans in mind such that you have way more than 3, but, this felt like a reasonable bid given that people will probably find all these steps cumbersome and annoying at first.

It seems fairly clear that widely deployed, highly capable AI systems enabling unrestricted access to knowledge about weapons development, social manipulation techniques, coordinated misinformation campaigns, engineered pathogens, etc. could pose a serious threat. Bad actors using that information at scale could potentially cause societal collapse even if the AI itself was not agentic or misaligned in the way we usually think about with existential risk. 

Alright, time to disagree with both @faul_sname [LW · GW] and @tailcalled [LW · GW], while also agreeing with some of each of their points, in order to explain my world view which differs importantly from both of theirs!

Topic 1: Offense-Defense Balance

tailcalled: [AI] It makes conflict more viable for small adversaries against large adversaries

faul_sname: I'm not sure about that. One dynamic of current-line AI is that it is pretty good at increasing the legibility of complex systems, which seems like it would advantage large adversaries over small ones relative to a world without such AI.

Unfortunately, until the world has acted to patch up some terrible security holes in society, we are all in a very fragile state. Currently, as of mid-2024, all nations on Earth have done a terrible job at putting preventative safety measures in place to protect against biorisk. The cost of doing this would be trivial compared to military expenses, even compared to militaries of small nations. So, I presume that the lack of such reasonable precautions is due to some combination of global decision makers having:

lack of belief in the existence of the risk

lack of belief that reducing the risk would be politically expedient

lack of knowledge about how to effectively counter the risk

lack of knowledge that it would be relatively cheap and easy to put preventative measures in place

corruption (combined with ignorance and/or laziness and/or stupidity), such that they aren't even motivated to learn what would be useful actions to take to benefit their country which would also benefit themselves (safety from bioweapons is good for rich people too!)

Furthermore, this situation is greatly exacerbated by AI. I have been working on AI Biorisk Evals with SecureBio for nearly a year now. As models increase in general capabilities, so too do they incidentally get more competent at assisting with the creation of bioweapons. It is my professional opinion that they are currently providing non-zero uplift over a baseline of 'bad actor with web search, including open-access scientific papers'. They are still far from some theoretical ceiling of maximal uplift, but the situation is getting worse rapidly. Other factors are at work making this situation worse, such as microbiology technology getting cheaper, more reliable, more available, and easier to use. Also, the rapid progress of wetlab automation technology.

Overall, this category of attack alone is rapidly making it much much easier for small actors (e.g. North Korea, or a well-funded terrorist organization) to plausibly make existential threats against large powerful actors (e.g. the United States). Why? Because ethnic-targeting and/or location-targeting of a bioweapon is possible, and getting easier as the advising AI gets smarter.

Importantly, a single attack could be completely devastating. If it were also difficult to trace the origin, then most of the population of the United States would be dead before anyone had figured out where the attack originated from. This makes it hard to plausibly threaten retaliatory mutual destruction.

Currently, there aren't other technologies that allow for such cheap, devastating, hard-to-trace attacks. In the future, technological advancement might unlock more. For instance, nanotech or nano-bio-weapon-combos. This doesn't seem like a near-term threat currently, but it's hard to foresee how technology might advance once AI gets powerful enough to accelerate research in a general way.

Topic 2: De-confusing intent-alignment versus value-alignment versus RLHF

I think it's important to distinguish which things we're talking about. For more detail, see this post by Seth Herd. [LW · GW]

RLHF is a tool, with certain inherent flaws, that can be directed to a variety of different ends.

Intent-alignment means getting an AI to do what you intend for it to do. Making it into a good obedient tool/servant which operates in accordance with your expectations. It doesn't surprise you by taking irreversible actions that you actually disapprove of once you learn of them. It's a good genie, not a bad genie.

Value-alignment means trying to make the AI act (relatively independently, using its own judgement) in accordance with specified values. These values could be the developer's values, or the developer's estimate of humanity's values, or their country's values, or their political party's values. Whatever. The point is that a choice is made by the developer about a set of values, and then the developer tries to get an agent to semi-independently work to alter the world in accordance with the specified values. This can involve a focus exclusively on end-states, or may also specify a limited set of acceptable means by which to make changes in the world to actualize the values. 

Topic 3: Threat modeling of AI agents

End-state-focused Value-aligned Highly-Effective-Optimizer AI Agents (hereafter Powerful AI Agents) with the primary value of win-at-all-costs are extremely dangerous. The more you restrict the means by which the agents act, or the amount they must defer to supervisors before acting, the less effective at shaping the world they will be. If you are in a desperate existential conflict, this difference seems likely to be key, and thus to present strong pressure to remove the limiting control measures. I don't think you'll find much disagreement with this hypothetical being a bad state of affairs to be in. What people tend to disagree on are the factors involved in getting to that world state. I'll try to outline some of the disagreements in this area (rather than focusing on my own beliefs).

1. Some people don't believe it will be possible in the next 20 years (or longer!) to create Powerful AI Agents.
   1. some say that they don't believe the 'highly effective' part will be possible, even if the value-aligned part is.
      1. some of these say that intelligence, even if there is such a thing, and even if it were possible to grant such a thing to an AI such that the AI had more of it than a human, wouldn't even be useful. This involves some sort of belief in declining returns to intelligence, and limitations on possible sets of actions by highly intelligent actors (even AI able to rapidly increase their populations and to work at superhuman speeds).
      2. some say that while superhumanly effective general AI is possible in theory, that there is basically no change that humanity develops it within the next 20 (or 50 or 100 or 1000) years.
   2. some say that the value-aligned part will fail, and be more likely to fail the more effective the agents are. Thus, that any agent competent enough to be decisively useful in a large scale conflict will destroy the developer/operator as well as the operator's enemies.
2. Some people don't believe the we will get to Powerful AI Agents before we've already arrived at other world states that make it unlikely we will continue to proceed on a trajectory towards Powerful AI Agents.
   1. some say that we have already reached (or will soon reach) the point of civilization-scale risk from weaponization / misuse of AIs well short of the theoretical Powerful AI Agents. For more details on such risks, see Topic 1.
      1. Some predict that these large scale risks will destroy civilization before we have a chance to develop Powerful AI Agents. 
      2. Some predict that the presence of such large concrete threats will force an international coalition which effectively enforces global control over AI development and civilization scale weapons.
      3. Some predict that all relevant actors will be so abhorrent of the civilization-scale risks which weaponization of AI would lead to that all such actors will voluntarily abstain from developing and using such. They predict that this ethical fortitude will hold even in the face of imminent destruction by violent conflict, or that such destructive conflict will never occur between the relevant actors. They also predict that the set of relevant actors will remain small (e.g. just the largest state actors, not terrorists or small states). [author's note: I am unable to avoid mentioning that I am rolling my eyes disbelievingly while writing this point.]
      4. Some predict that having such powerful weapons available will enable one powerful actor (or team of actors) to gain a decisive strategic military advantage sufficiently strong that they will be able to force surrender or utterly annihilate their enemies. They predict that once it is clear that this option is available, the actors likely to be the ones in this position (current top 5 candidates, in descending order of probability according to the author: US, UK, China, Russia,  India) will seize the opportunity and succeed. Then there will be no large scale AI-vs-AI conflict because a single state actor will control the entire world and rule sufficiently competently / harshly that no rebel faction will be able to gain sufficient power to launch a relevant counterattack.
   2. some predict that the powerful state actors will foresee the risk of escalating conflicts, and decide to form a sufficiently strong and well-enforced treaty/coalition that we reign in the already-in-play civilizational scale risks and act to prevent the emergence of new ones. This includes monitoring and regulating each other and all countries / companies / academic institutions / etc all around the world in order to prevent further research into AI or any sort of self-replicating weapons or other cheap and difficult-to-restrict weapons of mass destruction.
      1. for an example of this idea see: https://arxiv.org/abs/2310.09217 
      2. note that while this does seem like a fairly optimistic take, it is not nearly as implausible as the voluntary-abstention prediction from point 2.a.iii. This proposal would involve coercive enforcement of global bans via mandatory inspection, and promised military action against defectors.
3. Some say that intelligent agents being dangerously powerful is good actually, and won't be a risky situation
   1. some deny the hypothesis of the orthogonality of values and intelligence, and say that a highly intelligent AI will converge on human-ish values and treat humanity nicely
   2. some say that they value intelligent beings generally, rejecting human chauvinism. They therefore value the hypothetical future Powerful AI Agents so much that their creation and existence outweighs the likely extinction of humanity. (e.g. some e/acc supporters)

Would you mind sharing how you allocated the ratio of these positions?

Logically, maybe not that much.

Socially, this is a believe people hold.

I think it's the main argument for people choosing not to pull the lever (almost 20% by a survey I found).