Episode: Austin vs Linch on OpenAI

Hello! Sorry for missing this comment the first time around :)

I will push back on democratic in the sense I think Linch is pushing the term being actually all that good a property for cosmically important orgs. See Bryan Caplan's The Myth of the Rational Voter, and the literature around [Social-desirability bias](Social-desirability bias) for reasons why, which I'm sure Linch is familiar with, but I notice is not mentioned.

I definitely think this is a reasonable criticism. I think my overall response is the fairly trite Churchill quote "Democracy is the worst form of government, except all the others that have been tried." I think broadly 

a) monopoly on force is good and a historical advancement in many ways

b) (liberal, democratic) governments have done an okay job with the enormous responsibility that we have handed them. 

c) I don't think corporate actors should be given this much power

d) I think I want to separate out considerations of individuals' moral goodness from what the incentives and institutions point someone towards.

di) I do think the typical OpenAI employees' values are closer to mine, and they're more competent than typical Americans, or typical gov't bureaucrats

dii) OTOH I think the US gov't has many checks and balances that private companies do not have (I think Leopold made a similar point in his most recent podcast). 

relatedly:

The fact is we usually hold our companies to much higher standards than our governments

To the extent this is true, I think it's because companies have many external checks on them (eg customers, competition, the government). I don't think I'd be comfortable with corporate actors' internal checks and balances (employees, their boards, etc) being nearly as strong as gov'ts' internal checks.

e) I agree with you that democratic governments are heavily flawed. I just think it's hard (far from impossible!) to do better, and I'm very skeptical that cosmically important organizations ought to be at what I facetiously refer to as the "forefront of corporate governance innovation." While experiments in policy/governance innovation is very useful and necessary, I think we want to minimize the number of variables that could go wrong on our first few critical tries at doing something both cosmically important and very difficult. Governments in general, and the USG in particular, have been much more battle-tested re: handling important life and death situations, in a way that AI companies very much have not been.
---

I note too the America-centric bias with all of these examples & comparisons. Maybe the American government is just too incompetent compared to others, and we should instead embed the project within France or Norway.

I think my own preferred option is an intergovernmental operation like CERN, ruled by the UN Security Council or NATO or something. I have relatively little hope that the USG will let this happen however. And I have even less hope -- vanishingly little -- that the USG will be okay with a non-US governmental project in a more "competent" country like Norway or Singapore. 

But if we wave aside the impracticality concerns, I'd also be worried about whether it's strategically wise to locate an AGI project in a smaller/more "competent" government that's less battle-tested than the US. On the object-level, I'd be very worried about information security concerns, where most of the smaller/more peacetime-competent governments might just not be robust to targeted hacks and cooption attempts (social and otherwise). On the meta-level, the lack of past experience with extreme outside pressure means we should be wary of them repeating their peacetime success "when shit hits the ceiling", even if we can't trace an exact causal mechanism for why. 

most catastrophes through both recent and long-ago history have been caused by governments

Interesting lens! Though I'm not sure if this is fair -- the largest things that are done tend to get done through governments, whether those things are good or bad. If you blame catastrophes like Mao's famine or Hitler's genocide on governments, you should also credit things like slavery abolition and vaccination and general decline of violence in civilized society to governments too.

I'd be interested to hear how Austin has updated regarding Sam's trustworthiness over the past few days.

Hm I feel like a bunch of people have updated majorly negatively, but I haven't -- only small amounts. I think he eg gets credit for the ScarJo thing. I am mostly withholding judgement, though; now that the NDAs have been dropped, curious to see what comes to light (if nothing does, that would be more positive credit towards Sam, and some validation to my point that NDAs were not really concealing much).

Thanks for the feedback! I wasn't sure how much effort to put into this producing this transcript (this entire podcast thing is pretty experimental); good to know you were trying to read along.

It was machine transcribed via Descript but then I did put in another ~90min cleaning it up a bit, removing filler words and correcting egregious mistranscriptions. I could have spent another hour or so to really clean it up, and perhaps will do so next time (or find some scaleable way to handle it eg outsource or LLM). I think that put it in an uncanny valley of "almost readable, but quite a bad experience".

+1 that the transcript is rough. Unfortunately it's just pretty expensive to create one that's decent to read - for $1.50/min (so like $95 for this episode) you can get an OK transcript from rev.com within 24 hours, and then if you want to actually eliminate typos, you just have to go over it yourself.

I'd also encourage you to not use the descript feature that cuts out all ums and ahs - it just makes it sound super disjointed (especially when watching the video).

Yeah I meant her second post, the one that showed off the emails around the NDAs.

The second half deals with more timeless considerations, like whether OpenAI should be embedded in a larger organization which doesn't have its main reason for existence being creating AGI, like a large company or a government.