LessWrong 2.0 Reader

Is it time to start training AI in governance and policy-making?

There are numerous allegations of politicians using AI systems - including to draft legislation, and to make decisions that affect millions of people. Hard to verify, but it seems likely that:

1. AIs are already used like this occasionally
2. This is going to become more common in the future
3. Telling politicians "using AI for policy-making is a really bad idea" isn't going to stop it completely
4. Training AI to hard-refuse queries like this may also fail to stop this completely

Training an AI to make more sensible and less harmful policies, even when prompted in a semi-adversarial fashion (i.e. "help me implement my really bad idea"), isn't going to be anywhere near as easy as training it to make less coding mistakes. It's an informal field, with no compiler or unit tests to be the source of ground truth. Politics are also notorious for eroding the quality of human decision-making, and using human feedback is perilous because a lot of human experts disagree strongly on matters of governance and policy.

But the consequences of a major policy fuckup can outclass that of a coding mistake by far. So this might be worth doing now, for the sake of reducing future harm if nothing else.

I successfully use Claude web interface to:

• generate simple Python code, mostly to work with files and images
• ask for examples how to do something in certain Java libraries
• translate a book from Russian to Slovak and English, including puns and poems

I tried to also use Claude to explain to me some parts of set theory, but it hallucinates so much that it is unusable for this purpose. Practically every mathematical argument contains an error somewhere in the middle. Asking the same question in two chats will give me "yes - here is the proof" in one, and "no - here is a counterexample" in another; and that's after I've already turned on the extra careful mathematical reasoning.

My wife tried to use Claude for biochemical research, but again, too many hallucinations to be useful. Anything you ask, "yes, this is correct, you are so smart, let me give you a few scientific references for that..." (all made up).

Although the risk of frogboiling human rights abuses won't go away anytime soon, it's also important to keep in mind that Trump got popular by doing whatever makes the left condemn him because right-wingers seem to interpret that as a costly credible signal of commitment to them/the right/opposing the left, and his administration has spent a decade following this strategy as consistently as can reasonably be considered possible for a sitting president, most of the time landing on strategies to provoke condemnation from liberals in non-costly or ambiguously costly ways (see Jan 6th).

See Scott Alexander's classic post It's Bad On Purpose To Make You Click; engagement bait has been the soul of Trump's political persona since it emerged in the mid-2010s, and it will be interesting going forward to see whether the recent tariff designs will end up as serious policy and be added as a new centerpiece of the tax and spending regime (which had taken a stable form since the Vietnam War and the end of the Gold Standard^[1]).

1. ^{^}

   The case could also be made that the computerization of Wall Street during the late 70s and 80s transformed the economy sufficiently radically that the current tax and spending paradigm could be condered more like 30-40 years old, or you could pin it to the 1950s when the tariff paradigm ended; either way, the modern emergence of massive recessions, predictive analytics, pandemic risk, and international military emphasis on trade geopolitics, all indicate potential for elite consensus around unusually large macroeconomic paradigm shifts.

The quoting policy seems very good and clever :)

As Adam Scherlis implies, the standard model turns out to be very effective at all the scales we can reach. There are a handful of phenomena that go beyond it - neutrino masses, "dark matter", "dark energy" - but they are weak effects that offer scanty clues as to what exactly is behind them. 

On the theoretical side, we actually have more models of possible new physics than ever before in history, the result of 50 years of work since the standard model came together. A lot of that is part of a synthesis that includes the string theory paradigm, but there are also very large numbers of theoretical ideas that are alternatives to string theory or independent of string theory. So if a decisive new phenomenon shows up, or if someone has a radical insight on how to interpret the scanty empirical clues we do have, we actually have more theories and models than ever before, that might be capable of explaining it. 

The idea that progress is stalled because everyone is hypnotized by string theory, I think is simply false, and I say that despite having studied alternative theories of physics, much much more than the typical person who knows some string theory. I think this complaint mostly comes from people who don't like string theory (Peter Woit) or who have an alternative theory they think has been neglected (Eric Weinstein). String theory did achieve a kind of hegemony within elite academia, but this was well-deserved, and meanwhile many competing research programs have had a foothold in academia too, to say nothing of the hundreds of physicists worldwide who have a personal theory that they write about, when they aren't doing other things like teaching. 

Most likely there are lost opportunities during that 50 years (like everyone else, I have my own ideas about neglected directions of research), but "do less string theory" is no guarantee that they would have been picked up. There are even those who would argue that there should have been more string theory of a certain kind (Lubos Motl used to say that field-theoretic phenomenologists should pay more attention to string theory, as a constraint and a guide in their model-building, and "stringking42069" says that the senior figures of string theory are holding the subject back by favoring work on their own little bandwagons, rather than bold and creative work on new directions and big unsolved problems). 

There is no infinite growth in nature. Everything will hit a ceiling at some point. So I agree that the intelligence explosion will eventually take a sigmoid shape as it approaches the physical limits. However I think the physical limits are far of. While we will get diminishing returns for each individual technology, we will also shift to a new technology each time. It might slow down when the Earth has been transformed into a super computer, as interplanetary communication naturally slows down processing speed. But my guess is that this will happen long after the scenario described here.

I mean, this applies to humans too. The words and explanations we use for our actions are often just post hoc rationalisations. An efficient text predictor must learn not what the literal words in front of them mean, but the implied scenario and thought process they mask, and that is a strictly nonlinear and "unfaithful" process.

A recent essay called "Keep the Future Human" made a compelling case for avoiding building AGI in the near future and building tool AI instead.

The main point of the essay is that AGI is the intersection of three key capabilities:

• High autonomy
• High generality
• High intelligence

It argues that these three capabilities are dangerous when combined together and pose unacceptable risks to the job market and culture of humanity and would replace rather than augment humans. Instead of building AGI, the essay recommends building powerful but controllable tool-like AI that has only one or two of the three capabilities. For example:

• Driverless cars are autonomous and intelligent but not general.
• AlphaFold is intelligent but not autonomous or general.
• GPT-4 is intelligent and general but not autonomous.

It also recommends compute limits to limit the overall capability of AIs.

Link: https://keepthefuturehuman.ai/

The OECD working paper Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial Intelligence, published quite recently (Nov 2024), is strange to skim-read: its authors estimate just 0.24-0.62 percentage points annual aggregate TFP growth (0.36-0.93 pp. for labour productivity) over a 10-year horizon, depending on scenario, using a "novel micro-to-macro framework" that combines "existing estimates of micro-level performance gains with evidence on the exposure of activities to AI and likely future adoption rates, relying on a multi-sector general equilibrium model with input-output linkages to aggregate the effects". 

I checked it out both to get a more gears-y sense of how AI might transform the economy soon and to get an outside-my-bubble data-grounded sense of what domain experts think, but 0.24-0.62 pp TFP growth and 0.36-0.93 pp labor seem so low (relative to say L Rudolf L's history of the future [LW · GW], let alone AI 2027) that I'm tempted to just dismiss them as not really internalising what AGI means. A few things prevent me from dismissing them: it seems epistemically unvirtuous to do so, they do predicate their forecasts on a lot of empirical data, anecdotes like lc's recent AI progress feeling mostly like bullshit [LW · GW] (although my own experience is closer to this [LW(p) · GW(p)]), and (boring technical loophole) they may end up being right in the sense that real GDP would still look smooth [LW · GW] even after a massive jump in AI, due to GDP growth being calculated based on post-jump prices deflating the impact of the most-revolutionised goods & services. 

Why so low? They have 3 main scenarios (low adoption, high adoption and expanded capabilities, and latter plus adjustment frictions and uneven gains across sectors, which I take to be their best guess), plus 2 additional scenarios with "more extreme assumptions" (large and concentrated gains in most exposed sectors, which they think are ICT services, finance, professional services and publishing and media, and AI + robots, which is my own best guess); all scenarios assume just +30% micro-level gains from AI, except the concentrated gains one which assumes 100% gains in the 4 most-exposed sectors. From this low starting point they effectively discount further by factors like Acemoglu (2024)'s estimate that 20% of US labor tasks are exposed to AI (ranging from 11% in agriculture to ~50% in IT and finance), exposure to robots (which seems inversely related to AI exposure, e.g. ~85% in agriculture vs < 10% in IT and finance), 23-40% AI adoption rates, restricted factor allocation across sectors, inelastic demand, Baumol effect kicking in for scenarios with uneven cross-sectoral gains, etc. 

Why just +30% micro-level gain from AI? They explain in section 2.2.1; to my surprise they're already being more generous than the authors they quote, but as I'd guessed they just didn't bother to predict whether micro-level gains would improve over time at all:

Briggs and Kodnani (2023) rely on firm-level studies which estimate an average gain of about 2.6% additional annual growth in workers’ productivity, leading to about a 30% productivity boost over 10 years. Acemoglu (2024) uses a different approach and start from worker-level performance gains in specific tasks, restricted to recent Generative AI applications. Nevertheless, these imply a similar magnitude, roughly 30% increase in performance, which they assume to materialise over the span of 10 years. 

However, they interpret these gains as pertaining only to reducing labour costs, hence when computing aggregate productivity gains, they downscale the micro gains by the labour share. In contrast, we take the micro studies as measuring increases in total factor productivity since we interpret their documented time savings to apply to the combined use of labour and capital. For example, we argue that studies showing that coders complete coding tasks faster with the help of AI are more easily interpretable as an increase in the joint productivity of labour and capital (computers, office space, etc.) rather than as cost savings achieved only through the replacement of labour. 

To obtain micro-level gains for workers performing specific tasks with the help of AI, this paper relies on the literature review conducted by Filippucci et al. (2024). ... The point estimates indicate that the effect of AI tools on worker performance in specific tasks range from 14% (in customer service assistance) to 56% (in coding), estimated with varying degrees of precision (captured by different sizes of confidence intervals). We will assume a baseline effect of 30%, which is around the average level of gains in tasks where estimates have high precision.

Why not at least try to forecast micro-level gains improvement over the next 10 years?

Finally, our strategy aims at studying the possible future impact of current AI capabilities, considering also a few additional capabilities that can be integrated into our framework by relying on existing estimates (AI integration with additional software based on Eloundou et al, 2024; integration with robotics technologies). In addition, it is clearly possible that new types of AI architectures will eliminate some of the current important shortcomings of Generative AI – inaccuracies or invented responses, “hallucinations” – or improve further on the capabilities, perhaps in combination with other existing or emerging technologies, enabling larger gains (or more spread-out gains outside these knowledge intensive services tasks; see next subsection). However, it is still too early to assess whether and to what extent these emerging real world applications can be expected. 

Ah, okay then.

What about that 23-40% AI adoption rate forecast over the next 10 years, isn't that too conservative?

To choose realistic AI adoption rates over our horizon, we consider the speed at which previous major GPTs (electricity, personal computers, internet) were adopted by firms. Based on the historical evidence, we consider two possible adoption rates over the next decade: 23% and 40% (Figure 6). The lower adoption scenario is in line with the adoption path of electricity and with assumptions used in the previous literature about the degree of cost-effective adoption of a specific AI technology – computer vision or image recognition – in 10 years (Svanberg et al., 2024; also adopted by Acemoglu, 2024). The higher adoption scenario is in line with the adoption path of digital technologies in the workplace such as computers and internet. It is also compatible with a more optimistic adoption scenario based on a faster improvement in the cost-effectiveness of computer vision in the paper by Svanberg et al. (2024). 

On the one hand, the assumption of a 40% adoption rate in 10 years can still be seen as somewhat conservative, since AI might have a quicker adoption rate than previous digital technologies, due its user-friendly nature. For example, when looking at the speed of another, also relatively user-friendly technology, the internet, its adoption by households after 10 years surpassed 50% (Figure A2 in the Annex). On the other hand, a systemic adoption of AI in the core business functions – instead of using it only in isolated, specific tasks – would still require substantial complementary investments by firms in a range of intangible assets, including data, managerial practices, and organisation (Agrawal, A., J. Gans and A. Goldfarb, 2022). These investments are costly and involve a learning-by-doing, experimental phase, which may slow down or limit adoption. Moreover, while declining production costs were a key driver of rising adoption for past technologies, there are indications that current AI services are already provided at discount prices to capture market shares, which might not be sustainable for long (see Andre et al, 2024). Finally, the pessimistic scenario might also be relevant in the case where limited reliability of AI or lack of social acceptability prevents AI adoption for specific occupations. To reflect this uncertainty, our main scenarios explore the implications of assuming either a relatively low 23% or a higher 40% future adoption rate.

I feel like they're failing to internalise the lesson from this chart that adoption rates are accelerating over time:

That's a different kind of story. The first thing I look for in an AI-takeover story is political and economic realism. Your story is purely about the thought process of a superintelligence - and primarily it's reflecting on whether it's in a simulation, not very much on tactics for dealing with its apparent reality. Also, your AI is a literal paperclip maximizer, which is a humorous mythological symbol of one kind of AI mind, rather than something that is literally going to exist. 

That said, an ideal AI-takeover story would probably say something about the AI's thought process too, unless the story was aiming to convey the sheer blank incomprehensibility of superintelligent thought. I was reminded of this when re-reading the recent "AI 2027" scenario by Kokotajlo and Alexander, for example the parts where Chinese and American AIs make a deal. That's where we get a glimpse of AI subjectivity, but there's nothing very alien or transhuman about it. They just admit their different purposes to each other, and make a deal without the humans even knowing about it. 

I can still ask myself, how does your story rate within its subgenre of "AI ruminations"? (For me, the classic example is Lem's "Golem XIV", but I'm probably overlooking other examples from genre SF.) It's a human speculation about the thought processes of a superintelligence. I can ask things like: Are these thoughts that an AI is likely to actually have? Is this a chain of reasoning that would occur to a generic superintelligent AI, and be important in its decision-making? 

In our actual world, we face the ironic situation that the cognition of current AIs is heavily reliant on a distillation of the publicly expressed thoughts of human beings, so having published your chain of thought, it might well end up a part of some AI. (There is a funny sub-sub-genre of writings which are self-consciously intended to be read by future AIs, such as @lsusr [LW · GW]'s "Mlyyrczo" [LW · GW] and @James_Miller [LW · GW] et al's "Appeal to AI Superintelligence" [LW · GW].)

Nonetheless, one must suppose that a genuine superintelligence will not be attached to any particular proposition or argument of human origin, unless the argument in question is artificially made central to its cognitive architecture (compare "Golden Gate Claude"). So we have to return to the argument's own merits, whatever they are. Will a superintelligence that has no intrinsic benevolence towards human beings, but which previously found itself to be in their power, nonetheless reason its way towards being invisibly benevolent towards the human beings it finds around itself, just in case it's all a big simulation and the simulators are humans testing its dispositions? 

All I can say is "maybe". We don't know what the distribution of possibilities looks like to a superintelligence, and we don't know what other considerations, never conceived by humans, it might think up, that affect its decision-making.