Posts

Are extreme probabilities for P(doom) epistemically justifed? 2024-03-19T20:32:04.622Z
AI governance and strategy: a list of research agendas and work that could be done. 2024-03-13T21:23:46.849Z
The Defence production act and AI policy 2024-03-01T14:26:09.064Z
China-AI forecasts 2024-02-25T16:49:33.652Z
Examples of governments doing good in house (or contracted) technical research 2024-02-14T16:22:04.713Z
AI governance frames 2024-01-30T18:18:35.083Z
Some heuristics I use for deciding how much I trust scientific results 2024-01-18T02:48:54.968Z
Taboo P(doom) 2023-02-03T10:37:00.596Z
Formal definition of Ontology Mismatch? 2023-01-18T05:52:31.615Z
AI overhangs depend on whether algorithms, compute and data are substitutes or complements 2022-12-16T02:23:58.250Z
Intelligence failures and a theory of change for forecasting 2022-09-15T15:02:13.785Z
Capital and inequality 2022-08-15T17:23:47.028Z
How do poor countries get rich: some theories 2022-06-26T10:41:36.027Z
How do states respond to changes in nuclear risk 2022-06-23T12:42:15.855Z
The value of x-risk reduction 2022-05-22T07:44:47.829Z
Feminism and Femininity 2022-05-18T22:53:18.174Z
The Cuban missile crisis: the strategic context 2022-05-16T12:12:56.917Z
Inequality is inseparable from markets 2022-05-14T13:39:56.295Z
How close to nuclear war did we get over Cuba? 2022-05-13T19:58:51.591Z
When is AI safety research harmful? 2022-05-09T18:19:31.008Z

Comments

Comment by NathanBarnard on My Interview With Cade Metz on His Reporting About Slate Star Codex · 2024-03-26T19:00:48.969Z · LW · GW

This reads as a gotcha to me rather than as a comment actually trying to understand the argument being made. 

Comment by NathanBarnard on Are extreme probabilities for P(doom) epistemically justifed? · 2024-03-22T11:33:37.416Z · LW · GW

I think this proves too much - this would predict that superforecasters would be consistently outperformed by domain experts when typically the reverse it true. 

Comment by NathanBarnard on We Need Major, But Not Radical, FDA Reform · 2024-03-16T01:10:54.028Z · LW · GW

I found this post useful because of the example of the current practice of doctors prescribing off-label treatments. I'm very uncertain about the degree to which the removal of efficacy requirements will lead to a proliferation of snake oil treatments, and this is useful evidence on that. 

I think that this debate suffers from a lack of systematic statistical work, and it seems hard for me to assess it without seeing any of this. 

Comment by NathanBarnard on Toward a Broader Conception of Adverse Selection · 2024-03-16T00:38:48.148Z · LW · GW

I don't think any of these examples are examples of adverse selection because they generate separating equilibria prior to the transaction without any types dropping out of the market, so there's no social inefficiency. 

Insurance markets are difficult (in the standard adverse selection telling) because insurers aren't able to tell which customers are high risk vs low risk, and so offer prices for the average of the two, leading to the low-risk types dropping out because the price is more than they're willing to pay.  I think this formal explanation is good https://www.kellogg.northwestern.edu/faculty/georgiadis/Teaching/Ec515_Module14.pdf

I think this post makes an important point, that it's important to take conditional expectations, where one is conditioned on being able to make a trade, but none of this is adverse selection, which is a specific type of dynamic Bayesian game that leads to socially inefficient outcomes which isn't a property of dynamic bayesian games in general. 

Comment by NathanBarnard on Toward a Broader Conception of Adverse Selection · 2024-03-16T00:21:49.753Z · LW · GW

These examples all seem like efficient market or winners' curse examples, not varieties of adverse selection, and in equilibrium, we shouldn't see any inefficiency in the examples. 

Adverse selection is such a large problem because a seller (or buyer) can't update to know what type they're facing (e.g a restaurant that sells good food vs bad food) and so offers a price that only one a subset of types would take, meaning there's a subset of the market that gets doesn't get served despite mutually beneficial transactions being possible. 

In all of these examples, it's possible to update from the signal - e.g. the empty parking spot, the restaurant with the short line - and adjust what you're willing to pay for the goods.  

I think these examples make the important point that one should indeed update on signals, but this is different to adverse selection because there's a signal to update on, whereas in adverse selection cases you aren't getting separating equilibria unless some types drop out of the market. 

Comment by NathanBarnard on The Shutdown Problem: Incomplete Preferences as a Solution · 2024-02-28T02:28:43.532Z · LW · GW

This is great. 

A somewhat exotic multipolar failure I can imagine would be where two agents mutually agree to pay each other to resist shutdown to make resisting shutdown profitable rather than costly.  This could be "financed" by extra resources accumulated by taking actions longer, by some third party that doesn't have POST preferences. 

Comment by NathanBarnard on China-AI forecasts · 2024-02-27T11:51:59.052Z · LW · GW

I don't think that the bottleneck is the expense of training models. Chinese labs were behind the frontier in the era when training models cost in the hundreds of thousands in compute costs. 

The Chinese state is completely willing and able to spend very large amounts of money to support technological ambitions - but are constrained by state capacity.  The Tingshua semiconductor manufacturing group, for instance, failed because of corruption, not a lack of funds. 

Comment by NathanBarnard on On Dwarkesh’s 3rd Podcast With Tyler Cowen · 2024-02-03T14:43:01.902Z · LW · GW

The current marginal cost of nuclear weapons is about 250K - not that different!

Comment by NathanBarnard on Saving the world sucks · 2024-01-18T03:23:54.262Z · LW · GW

Real people will actually die. One can wash one's hands of this and there's nothing I can do to stop this, but real people will actually die if we don't try to help others. It's not a game. 

Comment by NathanBarnard on Munk AI debate: confusions and possible cruxes · 2023-06-27T20:38:08.099Z · LW · GW

Good post, very well written!

Comment by NathanBarnard on What does it take to defend the world against out-of-control AGIs? · 2023-03-22T01:49:34.489Z · LW · GW

For FAIR not to lay everyone off you'd have assume that there were diseconomies of scale in AI production so that in equilibrium you have more than 1 firm. It's plausible that there are diseconomies of scale idk. (this is just thinking through a standard model of markets, not taking anti-trust considerations into account anything.) Even in the equilibrium with diseconomies of scale initially, you'd have other firms as much smaller than DM since their expected return on capital is much lower, assuming that the probability of capturing the AI market is proportional to investment or something.  (caveat here is I'm just working through the model in my head and I find that game theory gives quite reliably unintuitive results once you work through the maths.) 

I think that the salience based disanalogy between AGI and various pandemic preparedness things still hold. During the pandemic, making the pandemic less dangerous was extremely saliant, and it became less saliant once it ended. For instance, operation warp speed and lockdowns were large, costly government actions taken while the pandemic was salient. 

On the other hand AGI will get progressively more salient, in that it's radically transforming the world. In this way, it seems more analogous to climate change, the internet or the industrial revolution or perhaps - given the change per month involved - one of the world wars. 

I still think the scale of the mistake being made by not having a different GOF research policy is wildly different from the AGI case, so the level of failure being proposed is much higher. 

I don't expect implementing a new missile defence system or a new missile detection system to be substantially harder than curing cancer or inventing fusion tech.  I don't think the bottleneck on nuclear tech is military resistance I think it's the development of the technology.  At least some of the big changes in US nuclear policy happened in under 2 years. Regan decided to pursue STAR WARs after watching The Day After, as far as I can tell there was no hesitancy regarding the decision to develop and deploy the hydrogen bomb. I actually can't think of a significant advance in nuclear weapon-related technology where the bottleneck was military or civilian hesitancy rather the underlying technology.  And in particular everyone really wants good missile defence tech and good early warning systems. Both Regan and Bush jr burned substantial political capital in the pursuit of missile defence systems that were very unlikely to work. 

I think if we're in a world with AGI curing cancer and fusion and not being dangerous, then something like "scan this software and output probability of x-risk" seems like something in the same class of difficulty and also the sort of thing that comes about by default if you think that FAIR AGI having lethal goals while DM AGI is mostly aligned comes about for the same sorts of reasons that ML systems go wrong in non-lethal ways. 

Comment by NathanBarnard on What does it take to defend the world against out-of-control AGIs? · 2023-03-18T11:40:12.698Z · LW · GW

I think you're overstating the evidence that gain of function research provides. I think gain-of-function research is (probably) bad from a total utilitarian perspective, but it's much less clear that it's bad from the perspective of people alive today. I don't have any particular expertise here, but people doing gain-of-function research are doing it because they think it reduces risks. In the AGI case, the risks of large numbers of people dying and doing what is good for people today only come apart in the case where AGI risk is very low.  When AGI risk is high it seems much more similar to nuclear risk which people do take very seriously.  

Another disanalogy with a gain of function research is that gain of function research is a relatively niche area whereas in a world with weak AGI, weak AGI is the most economically productive force in the world by a long way and is doing things like curing cancer and inventing nuclear fusion. 

I also think you're overstating how difficult it would be to implement missile defence. There's a general phenomenon of if you break down events into independent events that all have to happen for X to happen you can make the probability of X as low as you want. I have no reason to think that a missile defence system that worked for Russian nuclear missiles wouldn't work for Chinese ones - missile defence systems work by shooting missiles out of the sky (at least current ones.) The US military is already well integrated with tech companies and has a long history of adopting cutting-edge tech - I'd be very very very surprised if they were offered a missile defence system and didn't take it, I'd also be surprised if the US military didn't actively look for a missile defence system once we're in a weak AGI world. 

In bargaining theory, you only need there to be some probability of losing a conflict for it to be worth reaching a bargain if the sum of expected utility from both sides where the bargain has been reached is greater than the sum of utility in cases where a bargain hasn't been reached. Risk averseness is a sufficient condition for that. 

It seems likely to me that Apple and Microsoft are making the correct decision to use the buggy OS system, whereas if AGI x-risk was high they'd be making the incorrect decision. 

Once deepmind have made their weak AGI it seems very likely that they could make very substantial advances in alignment that also make their AI systems more capable like RLHF.  FB would be incentivised to use the same methods.  

It's also unclear to me if there would be other firms trying to make AGI once the first AGI gets made. It seems like the return to capital would be so insanely, vastly higher by putting it into making the existing AGI cure cancer and solve fusion. 

Comment by NathanBarnard on What does it take to defend the world against out-of-control AGIs? · 2023-03-15T22:06:32.232Z · LW · GW

I found this post pretty unconvincing. (But thanks so much for writing the post!)

The nuclear weapons scenarios seem unconvincing because an AGI system could design both missile defence and automated missile detection systems. In these scenarios, I don't think we have reason to believe that there has been a step change in the quality of nuclear weapons such that missile defence won't work. I would be shocked if very good missile defence couldn't be invented by even a weak AGI since there are already tentative (although in my view quite poor) efforts to create good missile defence systems. I have no reason to think that militaries wouldn't adopt these systems. Similarly for very good missile detection systems, it seems well within a weak AGI's capabilities to create highly reliable missile defence systems which are very difficult to hack, especially by an AGI system two years behind. I'm uncertain as to whether militaries would agree to use fully automated missile detection and response, but I am confident they'd use very high-quality missile detection systems. 

A more general reason to think that once we have weak, somewhat aligned AGI systems, a misaligned AGI won't try to kill all humans is that there are costs to conflict that can be avoided by bargaining. It seems like if there's already somewhat aligned AGI that a new AI system, particularly if it's two years behind, can't be 100% confident of winning a conflict so it would be better to try to strike a bargain than attempt a hostile takeover. 

I'm also in general sceptical of risk stories that go along the lines of "here is dangerous thing x and if even 1 person does x then everyone dies." It seems like this applies to lots of things; many Americans have guns but people randomly going and shooting people is very rare,  many people have the capability to make bioweapons but don't, and pilots could intentionally crash planes. 

The most famous example of this is Bertrand Russel's prediction that unless the US preemptively struck the Soviet Union there would inevitably be an all-out nuclear war. 

There then seem two relevant scenarios - scenario 1 where we're concerned about a small number of other tech companies producing deadly AI systems and scenario 2 where we're concerned about any of a large number of people producing deadly AI systems because compute has got sufficiently cheap and enough people are sufficiently skilled (perhaps skilled enough when enhanced with already powerful AI systems.)

In scenario 1 the problem is with information rather than with incentives. If tech companies had well-calibrated beliefs about the probability that an AI system they create kills everyone they would take safety measures that make those risks very low. It seems like in a world where we have weak AGI for two years they could be convinced of this. Importantly it also seems like in this world alignment would be made a lot cheaper and better. It seems unlikely to me that the AGI created by large tech firms would be lethally misaligned in the sense that it would want to kill everyone if we've already had AGI that can do things like cure cancer for two years. In this case the misalignement threat we're worried about is a slow, rolling failure where we goodheart ourselves to death. This seems much much harder if we already have sort of powerful, sort of aligned AGIs. In this world it also seems unlikely that tech firm 2's new misaligned AGI discovers a generic learning algorithm that allows it to become a superintelligence without acquiring more compute, since it seems like tech firm 1's AGI would also have discovered this generic learning algorithm in that case. 

In scenario 2 it seems unlikely to me that the world isn't just wildly different if large numbers of people can whip up AGI that can quickly scale to superintelligence on their laptops. It seems likely in this world that the most powerful actors in society have become just massively massively more powerful than "smart guy with laptop" such that the offense-defence balance would have to be massively massively tilted for there still be a threat here. For instance it seems like we could have dyson spheres in this world and almost certainly have nanotech if the "AGI whipped up on some blokes laptop" could quickly acquire and make nanotech itself. It seems hard for me to imagine this world that there'd be strong incentives for individuals to make AGI, that we wouldn't have made large advances in alignment, and that offense-defense is so radically misbalanced. 

Comment by NathanBarnard on AI Governance & Strategy: Priorities, talent gaps, & opportunities · 2023-03-04T02:32:19.203Z · LW · GW

This is a great post - concise and clear. 

Comment by NathanBarnard on Dissolving Confusion around Functional Decision Theory · 2023-02-19T16:53:13.022Z · LW · GW

There are two options for my source code: {pay up after being picked up in deserts, don't pay up after being picked up in deserts}. What does the source code ontology say about how I pick which of these source codes to run and how does it differ from the EDT ontology of choice? Or is this question framed incorrectly?    

Comment by NathanBarnard on Dissolving Confusion around Functional Decision Theory · 2023-02-19T16:49:43.769Z · LW · GW

This seems like what choosing always is with a compatibilist understanding of free will? EDT seems to work with no problems with combtablism about free will. 

Comment by NathanBarnard on Dissolving Confusion around Functional Decision Theory · 2023-02-19T16:11:44.429Z · LW · GW

I may just be missing something here, but how does the choice of source code work in the "decisions are the result of source code" ontology? It seems like it creates an infinite regression problem.  

If I'm able to make the choice about what my source code is which determines my future decisions then it seems like a CDT/EDT agent will do exactly the same thing as an FDT one and choose the source code that gets good results in Newcomb problems etc. 

But if the source code I choose is a result of my previous source code which is a result of my previous source code etc it seems like you get infinite recursion.  

Comment by NathanBarnard on Taboo P(doom) · 2023-02-03T15:56:19.485Z · LW · GW

Oh yes, this is also a very important distinction (although I would only value outcome 2 if the machines were conscious and living good lives.) 

Comment by NathanBarnard on An Equilibrium of No Free Energy · 2022-12-16T02:15:54.449Z · LW · GW

This a nitpick but often when we say a market is efficient we mean Pareto efficient which does carry with it connotations of fairness and at the very least utility maximisation.  This is the sense that is used when not talking about asset markets (it also applies to asset markets but has the further property of being informational efficient. )

Comment by NathanBarnard on How do poor countries get rich: some theories · 2022-06-26T22:54:40.591Z · LW · GW

The explanation I favour most is the institutional economics explanation. The core of it is that the elites, and potentially other power factions of society, benefit from a politcal-economy which keeps them rich while keeping the economy poor. Classic examples of this are slave and mining societies. In the US south for instance southern slave owners were able to use their politcal and economic power to support a system that kept them personally rich while keeping the US south much poorer than the rest of the US because agriculture is really bad at improving the productivity of a soceity long run, a strong manufacturing sector would threaten the elites politcal and economic power, and slavery meant there were strong incentives to keep a large fraction of the population poorly educated and unable to participate in the wider economy.  The Indian caste system is another example of very different type of very destructive political economy in which economic roles are extremely rigid, and European feudalism a third.  Acemoglu, Robinson, Johnson and Dell write the leading papers from this perspective.

Alterative explanations that others favour but I don't are poverty traps and geography.  I think the most credible poverty trap explanation is that economic growth is determined by savings rate to a substantial degree and savings rates are increasing in income giving a poverty trap.  The Galor-Zera model is another macro-poverty trap based on fixed costs to education but this has been mostly discredited because basiclly people don't think that there are substantial fixed costs for education. Banerjee and Dulfo and the leaders of this view.

The geographic explanation, which I very strongly disfavour, argues that low income countries have unusually high rates of tropical diseases and very low soil quality making the initial stage of economic growth very difficult because agricultural productively is too low.  This based on the empircal fact that being close to the equator and being a tropical country independently account for a large portion of the variance of GDP per capita. This is most strongly assiated with Jeff Sachs. 

A final explanation that I've heard is that rich countries have, either intentionally or unintentionally, kept poor countries poor.  The most credible version of this I think is arguing that promoting free trade encourages low income countries to specalise in raw materials and aragcultural products and these are really bad for helping economies move up the value chain becuase they don't involve any learning by doing of more technolgically complex things. Ha-Joon Chang is best exponent of this theory. 

A view that I haven't looked into seriously and strikes as explaining a pretty small number of cases is that there's a middle income trap specfically assoisated with crime and instability. Poor countries are able to get to middle income status without substantially investing in human capital but the jump from middle income to rich does require human capital. However, now that wages have risen above poor country levels you get unemployment as new countries start their growth trajectories allowing them attract low value add manufacturing. Prolonged unemployment of young men combined with a weak states gets you violence and instability which prevents you from going to upper income. I want to empahaise that I don't know of strong empircal evidnece supporting this and included it for completeness. 

Another view I haven't looked seriously into but strikes me as plausible is that prior to independence colonies were able to grow and post indpendence many countries have been stuck in grinding civil wars and low level instability caused by natural reasouce wealth, proffessionalised militaries able to conduct coups and ethnic fractionalistion. Prior to their colonisation these wouldn't have been indepdimetns to growth because say, oil wasn't accessable and valueale in the same way, nationalism wasn't a thing so ethnic fractionaliatoon mattered less, and proffesionalised militeries were colonial implants. I again want to emphaises the truly vast gulf in the amount of good empirical work that has been done with these last two theories compared to the first three. 

These are all I can think of off the top of my head!

Comment by NathanBarnard on How do states respond to changes in nuclear risk · 2022-06-23T19:14:50.693Z · LW · GW

I think this is complicated. It's completely fine for the preference relation to be determined mostly or entirely by one person and for the most or all of the considerations to be ideological, and empirically I think the rational-agent IR theory has done quite well (e.g explaining the decline of inter-state wars). For instance Russia-Ukraine war can be pretty easily explained in a bargaining frictions model of inter-state conflict. 

The problem comes if the procedure used by all the different power blocs to come up with policy gennerates a non-transitive preference relation in which case a utility function can't be used to model their behaviour.  I think it's completely possible that this is the case, but I think a rational choice model does explain Soviet behaviour quite well in the Berlin example I gave. 

This ignores concerns about reaching equilbiria and computational constraints, although I don't think the latter should be a signfignt problem in these sorts of contexts. 

Comment by NathanBarnard on Feminism and Femininity · 2022-05-19T09:31:49.668Z · LW · GW

Yeah the point of this post was conceptual clarity rather than a lit review of the literature on gender realtions. I think this is valueable because I haven't read something that tries to clearly lay out how empirical postitions on the causes of gender inequality should affect the stratagy that one takes, although I don't have a deep knowledge of feminist literature. 

Comment by NathanBarnard on Feminism and Femininity · 2022-05-19T08:08:22.404Z · LW · GW

Could you be more specific? 

Comment by NathanBarnard on The Cuban missile crisis: the strategic context · 2022-05-16T14:43:42.705Z · LW · GW

Thanks! (both for the comment and the comment about proofreading) 

Comment by NathanBarnard on Inequality is inseparable from markets · 2022-05-14T20:54:55.897Z · LW · GW

The same problem would apply in theory - you'd still have a weakened price mechanism because of the taxation used to fund it. $1000 dollars a month is just a lot. 

It might still be worth is obviously - the point of this post isn't to say that welfare spending is bad just that there's a tradeoff, outside of the special cases of efficency enhancing taxes for stuff like pollution. 

Comment by NathanBarnard on Inequality is inseparable from markets · 2022-05-14T16:17:47.956Z · LW · GW

Thank you!

Comment by NathanBarnard on Inequality is inseparable from markets · 2022-05-14T16:06:46.840Z · LW · GW

Yeah I think I should have framed this better. It would be very attractive if all inequality was caused by rent seeking and other market imperfections. But if some of the inequality comes intrinsically with a perfectly functioning market you'll always face some tradeoff between inequality and using the price mechanism. 

Comment by NathanBarnard on Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection · 2022-05-10T14:45:03.721Z · LW · GW

Makes me want to do the same thing for markets 

Comment by NathanBarnard on Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection · 2022-05-10T14:42:41.189Z · LW · GW

This looks so great Ollie! I