Posts
Comments
Do dragon unbelievers accept this stance? My impression is that dragon agnosticism would often be considered almost as bad as dragon belief.
I’m confused as to how the fits in with UK politics. I don’t think the minority party has any kind of veto?
I guess we have the House of Lords but this doesn’t really have a veto (at least not long term) and the House of Commons and House of Lords aren’t always or even usually controlled by different factions.
One extra thing to consider financially is if you have a smart meter then you can get all of your hot water and a chunk of your heating done at off peak rates. Our off peak electricity rates are about equal per kWh to gas rates.
Without this I think our system would be roughly the same cost per year as gas or slightly more, with it we save £200 per year or so I think. (This would be a very long payback time but there was a fully funded scheme we used).
If it helps anyone we are in Scotland and get average COP=2.9
In the UK there is a non-binding but generally observed rule that speed cameras allow you to drive 10% + 2mph above the speed limit(e.g. 35mph in a 30mph zone) before they activate.
This is a bit more of a fudge but better than nothing.
- Someone in your company gets fired by a boss you don't know/particularly like without giving any reason
- You are mad with the boss and want the decision overturned
- You have a credible, attractive BATNA (the Microsoft offer)
These 3 items seem like they would be sufficient to cause something like the Open Letter to happen.
In most cases number 3 is not present which I think is why we don't see things like this happen more often in more organisations.
None of this requires Sam to be hugely likeable or a particularly savvy political operator, just that people generally like him. People seem to suggest he was one or both so this just makes the letter more likely.
I'm sure this doesn't explain it all in OpenAI's case - some/many employees would also have been worried about AI safety which complicates the decision - but I suspect it is the underlying story.
I work in equipment manufacturing for construction so can comment on excavators. Other construction equipment (loaders, dumpers) have a similar story although excavators have more gently duty cycles and require smaller batteries so make sense to electrify first. Diesel-Hydraulic Excavators are also less efficient giving more potential advantage for electric equipment.
- Agree that payback period is relatively low but possibly a bit longer than here - I’ve seen 3-5 years. The ruggedised batteries required for instance can be expensive.
Purchasers of new machines will generally keep them for 5-7 years which is enough to justify the payback but not to make it an obvious easy win.
-
If you have to use a diesel generator you immediately lose a lot of your cost saving. It is surprising how many construction sites lack mains electricity.
-
Many machines go to the rental market. In this case the equipment buyers do not get the benefit of the reduced operating costs. In that case the rental company has to sell the increased rental cost to their customers who are happy with what they are currently using.
-
Total cost of ownership just isn’t the main driver of buyer decisions. This is already a problem with diesel-hydraulic machines - there are many ways to make these more efficient which would have a decent payback period but don’t get implemented because efficiency isn’t a key purchasing driver.
What buyers really need is performance and reliability (plus low up front cost). The advantage of electric is more difficult to sell for reliability because of a lack of track record so going electric is a risk. Users are also rightly concerned that battery range will not be sufficient on high usage days - batteries in current machines often claim a full day but not necessarily with high usage.
- Most likely route for electric in short term is for them to get used in environments where emissions are important (due to regulations or low ventilation such as mines) plus companies wanting to be/look green. This will allow a track record to build up which will give more confidence to buyers.
I suspect the most useful thing a government could do (assuming carbon tax is politically infeasible) would be to legislate for low emissions in cities which would build the track record faster.
Something similar not involving AIs is where chess grandmasters do rating climbs with handicaps. one I know of was Aman Hambleton managing to reach 2100 Elo on chess.com when he deliberately sacrificed his Queen for a pawn on the third/fourth move of every game.
https://youtube.com/playlist?list=PLUjxDD7HNNTj4NpheA5hLAQLvEZYTkuz5
He had to complicate positions, defend strongly, refuse to trade and rely on time pressure to win.
The games weren’t quite the same as Queen odds as he got a pawn for the Queen and usually displaced the opponent’s king to f3/f6 and prevented castling but still gives an idea that probably most amateurs couldn’t beat a grandmaster at Queen odds even if they can beat stockfish. Longer time controls would also help the amateur so maybe in 15 minute games an 1800 could beat Aman up a Queen.
Think you need to update this line too?
This is a bit less than half the rate for the CTA.
Is there a default direction to twist for the butt bump? The pictures all show the greeters facing in the same direction so one must have turned left and the other right! How do I know which way I should twist?
I cannot sign the assurance contract until I understand this fundamental question
Agreed, intended to distinguish between the weak claim “you should stop pushing the bus” and the stronger “there’s no game theoretic angle which encourages you to keep pushing”.
So there's no game theoretic angle, you can just make the decision alone, to stop pushing the frigging bus.
I don’t think this holds if you allow for p(doom) < 1. For a typical AI researcher with p(doom) ~ 0.1 and easy replacement, striking is plausibly an altruistic act and should be applauded as such.
I haven’t tested extensively but first impression is that this is indeed the case. Would be interesting to see if Sydney is similar but I think there’s a limit on number of messages per conversation or something?
When you did this do you let ChatGPT play both sides or were you playing one side? I think it is much better if it gets to play both sides.
I tried this with chatGPT to see just how big the difference was.
ChatGPT is pretty terrible at FEN in both games (Zack and Erik). In Erik’s game it insisted on giving me a position after 13 moves even though 25 moves had happened. When I pointed this out it told me that because there were no captures or pawn moves between moves 13 and 25 the FEN stayed the same…
However it is able to give sensible continuations of >20 ply to checkmate for both positions provided you instruct it not to give commentary and to only provide the moves. The second you allow it to comment on moves it spouts nonsense and starts making illegal moves. I also sometimes had to point out that it was black to play.
In Zack’s game ChatGPT has black set a trap for white (14… Ne5) and has white fall into it (15. Qxd4). After this 15… Nf3+ wins the Queen with a discovered attack from the bishop on g7.
Example continuation:
14... Ne5 15. Qxd4 Nf3+ 16. gxf3 Bxd4 17. Nxd4 Qxd4 18. Be3 Qxb2 19. Rec1 Bh3 20. Rab1 Qf6 21. Bd1 Rfc8 22. Rxc8+ Rxc8 23. Rxb7 Qa1 24. Bxa7 Qxd1#
One other continuation it gave got to checkmate on move 51 without any illegal moves or obvious blunders! (Other than whites initial blunder to fall into the trap)
In Erik’s game ChatGPT manages to play 29 ply of near perfect game for both players:
25... d5 26. g5 d4 27. cxd4 exd4 28. Qd2 Bb3+ 29. Ke1 Nd7 30. Nf5 Ne5 31. Be2 d3 32. Bd1 Bxd1 33. Kxd1 Nxf3 34. Qc3 f6 35. Qc4+ Kh8 36. Qe6 Qa5 37. Kc1 Qc3+ 38. Kb1 Rb8+ 39. Ka2 Qb2#
Stockfish prefers 26… dxc4+ and later on keeps wanting Bc5 for black plus black takes slightly to complete the checkmate than optimal but overall this is very accurate for both players.
Might be worth playing a game against chatGPT while telling it not to give any commentary?
Despite being a GPT-3 instance DALL-E appears to be able to draw an adequate " SolidGoldMagikarp" (if you allow for its usual lack of ability to spell). I tried a couple of alternative prompts without any anomalous results.
FWIW this matches my own experience with one as a company car pretty exactly.
(On mine (UK, right hand drive) the wipers can be activated manually by pressing the button on the end of the left stalk. This also brings up an on-screen menu for selecting constant wiping)
Get out of our houses before we are driven to expend effort killing them, and similarly for all the other places ants conflict with humans (stinging, eating crops, ..)
Ant mafia: "Lovely house you've got there, wouldn't it be a shame if it got all filled up with ants?"
I can't tell you whether this is right or not but this is what ChatGPT thinks:
"Is it possible to put a latex table in a footnote?"
"Yes, it is possible to include a LaTeX table in a footnote. You can do this by using the footnote or footnotetext commands provided by the footmisc package.
Here is an example of how to use these commands to include a table in a footnote:
\usepackage{footmisc}
...
\begin{table}[h]
\centering
\begin{tabular}{c c c}
A & B & C \\
1 & 2 & 3 \\
4 & 5 & 6
\end{tabular}
\caption{Table caption}
\end{table}
...
\footnote{
\begin{table}[h]
\centering
\begin{tabular}{c c c}
D & E & F \\
7 & 8 & 9 \\
10 & 11 & 12
\end{tabular}
\caption{Table caption in footnote}
\end{table}
}
Note that when including a table in a footnote, you should use the table environment as you would normally, but place it inside the footnote or footnotetext command. You should also make sure to use the [h] option for the table environment to prevent the table from floating.
I hope this helps! Let me know if you have any other questions."
I think the article undersells the problems of ChatGPT's hallucinations. One example from the article where ChatGPT is said to win is a recipe for risotto. However, I wouldn't follow a risotto recipe for ChatGPT just because I can't be confident it hasn't hallucinated some portion of the recipe but would happily follow one from Google, even if the format is a bit more annoying. Same issue with calculating load bearing capacity for a beam only more serious!
Having said that, it does seem like there are definitely specific areas where ChatGPT will be more useful. Coding is a good example as verifying the code is usually straightforward and/or would need to be done anyway. In many cases ChatGPT for an overview followed by Google for more detail/verification is probably a good way to go - I think this would be a good idea for the load bearing wall example.
One thing I've found useful is to make sure I identify to the supplier what specifically I need about the product I'm ordering - sometimes they have something similar in stock which meets my requirements.
One thing I think makes a big difference to me is whether I feel like the provider is taking a collaborative or adversarial stance.
- I don't usually skip ads on Youtube content but if the channel is often clickbaity/misrepresenting content then I will
- The printer/ink thing feels very out to get me. The alternative model of printer subscription (e.g. hp) feels alot more collaborative so I don't feel the need to ensure that every page I print is as filled with ink as possible so as to get the "best" deal.
- If the premium charged on foods in an amusement park/movie theatre feels excessive then I will feel less bad about eating my own food.
For the six/man thing my first association was six pack. Obviously the prototypical image would be topless but my guess is topless images aren’t in the training set (or Dall-E is otherwise prevented from producing them)
I realised something a few weeks back which I feel like I should have realised a long time ago.
The size of the human brain isn’t the thing which makes us smart, rather it is an indicator that we are smart.
A trebling of brain size vs a chimp is impressive but trebling a neural network’s size doesn’t give that much of an improvement in performance.
A more sensible story is that humans started using their brains more usefully (evolutionarily speaking) so it made sense for us to devote more of our resources to bigger brains for the marginal gains that would give.
As I said, I feel like I should have known this for ages. I had a cached thought that human’s big brains (and other things) cause us to be smarter and had never re-examined the thought. Now I think that the “and other things” is doing almost all of the heavy lifting and the size is more incidental to the process.
Thanks for publishing this. I’ve been around the rationality community for a few years and heard TAPs mentioned positively a lot without knowing much about them. This roughly matches my best guess as to what they were but the extra detail is super useful, especially in the implementation.
This suggests a different question. For non-participants who are given the program which creates the data, what probability/timeframe to assign to success.
On this one I think that I would have put a high probability to be solved but would have anticipated a longer timeframe.
I think the resulting program has lower length (so whatever string it generates has lower KC)
I don’t think this follows - your code is shorter in python but it includes 3 new built in functions which is hidden complexity.
I do agree with the general point that KC isn’t a great measure of difficulty for humans - we are not exactly arbitrary encoders.
What were the noise levels on the Corsi-Rosenthal?
Humans are very reliable agents for tasks which humans are very reliable for.
For most of these examples (arguably all of them) if humans were not reliable at them then the tasks would not exist or would exist in a less stringent form.
Curious as to what the get under the desks alarm was supposed to help with and how long ago this was? I’m having trouble fitting it into my world model.
I see that the standard Playground Q&A prompt on OpenAI uses a similar technique (although boringly uses "Unknown" instead of "Yo be real").
I think the thing which throws people off is that when GPT-3 goes wrong it goes wrong in ways that are weird to humans.
I wondered if humans sometimes fail at riddles that GPT-3 would think of as weird. I tried a few that I thought would be promising candidates (no prompt other than the questions itself)
Q: If a red house is made with red bricks, a blue house is made with blue bricks, a pink house is made with pink bricks, a black house is made with black bricks, what is a greenhouse made with?
A: Greenhouses are made with glass.
Say silk 4 times
Silk, silk, silk, silk.
What does a cow drink?
A cow drinks water.
You put a roast in a roaster, what do you put in a toaster?
Bread.
These are very cherry picked - I deliberately chose trick riddles that I thought GPT-3 would be good at. I think there's possibly something here about how humans vs GPT-3 rely on context.
I imagine a society of GPT-4s discovering a human and discussing how hollow their intelligence is.
I think the natural/manmade comparison between COVID and Three Mile has alot of merit but there are other differences which might explain the difference. Some of them would imply that there would be a strong response to an AI , others less so.
Local vs global
To prevent nuclear meltdowns you only need to ban them in the US - it doesn't matter what you do elsewhere. This is more complicated for pandemic preparedness.
Active spending vs loss of growth
Its easier to pass a law putting in nuclear regulations which limit growth as this isn't as obvious a loss as spending money from the public purse on measures for pandemics.
Activity of lobbying groups
I get the impression that the anti-nuclear lobby was alot bigger than any pro-pandemic-preparedness lobby. Possibly this is partly caused by the natural vs manmade thing so might be kind of a subpoint.
Tractability of problem
Preventing nuclear disasters seems more tractable than pandemic preparedness
1979 vs 2020
Were our institutions stronger back then?
FWIW I agree that a large AI disaster would cause some strong regulation and international agreements, my concern is more that a small one would not and small ones from weaker AIs seem more likely to happen.
Assuming this is the best an AGI can do, I find this alot less comforting than you appear to. I assume "a very moderate chance" means something like 5-10%?
Having a 5% chance of such a plan working out is insufficient to prevent an AGI from attempting it if the potential reward is large enough and/or they expect they might get turned off anyway.
Given sufficient number of AGIs (something we presumably will have in the world that none have taken over) I would expect multiple attempts so the chance of one of them working becomes high.
There's a theory of humor called benign violation theory.
The BVT claims that humor occurs when three conditions are satisfied: 1) something threatens one's sense of how the world "ought to be", 2) the threatening situation seems benign, and 3) a person sees both interpretations at the same time.
I think your description of pranks etc. fits in nicely with this - you even chose the same words to describe it so maybe you're already aware?
It's worth noting that the while number of courses at Berkeley almost doubled in the period shown, the number of courses per student has increased at a lower rate due to an increase in students.
Eyeballing the graph and looking at Berkeley's enrollment numbers I think the number of courses per student has increased by around 50%. Smaller but still a big effect.
Example:
I have a couple of positions I need to fill at my work. I’ve been off on holiday this week and it occurred to me that I should change one of the roles quite a lot and redistribute work.
I’ve had this issue for a few months and while in work I’ve been a bit overworked to actually take a step back and see this opportunity.
That makes me feel less bad for doing the same...
To a first order approximation I think of bureaucracies as status maximisers. I'll admit that status can be a bit nebulous and could be used to explain almost anything but I think a common sense understanding of status gives a good prediction in most cases.
- Will a bureaucracy usually perform its tasks kinda adequately? Yes
- Will a bureaucracy excel at its tasks? Not unless excellence comes with status, so almost never
- Will a bureaucracy look to expand its remit? Yes
- Will a bureaucracy often look like a blame minimiser? Yes (due to asymmetric justice)
For second order effects I would probably say bureaucracies are effort minimisers.If a bureaucracy's status isn't going to change much between 2 actions, just do whatever is easiest.
From a practical point of view I would expect the pull fan to better ventilate the corners of the room. On the push side the flow is more directional and I think with a push fan you're more likely to end up with turbulent flow in the corners which would noticeably slow air transfer from these regions. From this point of view it's possible that the 2 x pull configuration may actually be better than 2 x push + 2 x pull but I'm no expert.
Of course if the air speed is low then the difference will be minimal.
One rich dude had a whole island and set it up to have lenses on lots of parts of it, and for like a year he’d go around each day and note down the positions of the stars
You can’t just say that without a name or reference! Not that I don’t believe you - I just want to know more!
They're a bit tricky to get the hang of and are petrifying on steep slopes but I highly recommend. Also make getting to the hill more fun.
Something like this is sometimes recommended in marriage courses for dealing with disagreements. The idea is to keep emotions cool and ensure people are understanding what each other are saying.
So there's a technical definition of edge which is your expected gain for every unit that you bet, given your own probability and the bet odds.
I agree that not clumping up the post is probably best but to make the post correct I suggest adding the underlined text into the definition in case people don't click the link.
bet such that you are trying to win a percentage of your bankroll equal to your percent edge.
A short note to start the review that the author isn’t happy with how it is communicated. I agree it could be clearer and this is the reason I’m scoring this 4 instead of 9. The actual content seems very useful to me.
AllAmericanBreakfast has already reviewed this from a theoretical point of view but I wanted to look at it from a practical standpoint.
***
To test whether the conclusions of this post were true in practice I decided to take 5 examples from the Wikipedia page on the Prisoner’s dilemma and see if they were better modeled by Stag Hunt or Schelling Pub:
- Climate negotiations
- Relationships
- Marketing
- Doping in sport
- Cold war nuclear arms race
Detailed analysis of each is at the bottom of the review.
Of these 5, 3 (Climate, Relationships, Arms race) seem to me to be very well modeled by Schelling Pub.
Due to the constraints on communication allowed between rival companies it is difficult to see marketing (where more advertising = defect) as a Schelling Pub game. There probably is an underlying structure which looks a bit like Schelling Pub but it is very hard to move between Nash Equilibria. As a result I would say that Prisoner’s Dilemma is a more natural model for marketing.
The choice of whether to dope in sport is probably best modeled as a Prisoner’s dilemma with an enforcing authority which punishes defection. As a result, I don’t think any of the 3 games are a particularly good model for any individual’s choice. However, negotiations on setting up the enforcing authority and the rules under which it operates are more like Schelling Pub. Originally I thought this should maybe count as half a point for the post but thinking about it further I would say this is actually a very strong example of what the post is talking about – if your individual choice looks like a Prisoner’s Dilemma then look for ways to make it into a Schelling Pub. If this involves setting up a central enforcement agency then negotiate to make that happen.
So I’m scoring it 4 out of 5 Prisoners Dilemmas examined are better modeled as Schelling Pubs, which is in line with the “most” claims of the post title.
The example which was least like Schelling Pub was the one where communication was difficult/impossible due to external rules. I think the value of communication is implicit in the post but it would be helpful to have it spelled out explicitly.
One other thing which might be useful from a practical point is that things which don't initially seem iterated may be able to be iterated if you split them into smaller tasks. You don't have to reduce your nuclear arsenal or decarbonise all at once, you can do a little bit and then check that the others have done the same before continuing. This seems obvious on a national level but maybe not so obvious on a personal level.
***
(Read beyond here only if you're interested in more detail of the examples - it doesn't add to the conclusions)
Below is my analysis of the 5 items chosen from the Prisoner’s dilemmas example list on Wikipedia. In discussing Stag Hunts (SH) I use the post’s list of 4 items which might make something more like a SH than a Prisoner’s Dilemma (PD).
Climate negotiations
- PD shaped
- Each country benefits from stable climate
- Each would prefer that they put in minimum effort to achieve it
- SH shaped?
- Iteration: Yes - it is iterated over years and agreements
- Reputation: It seems like yes, although I’m not sure how this works out in practice
- Enforecable contracts: Not really
- Superrationality: Possibly
- SP shaped?
- It seems yes.
- As an example, recall the coal amendment from the Glasgow talks
- This is an amendment which favours some countries over others and means that those disfavoured will put in relatively more effort for a given amount of CO2 reduction compared to the original wording.
- It seems obvious that the actual agreement probably isn’t on the Pareto frontier
- I think the final figure from the post gives a good mental model of what is going on
- It seems yes.
Relationships
- PD shaped
- In theory, for an individual action, you’re better off if you get your own way over your partner
- SH shaped?
- Iteration: Yes (+ your partner can just leave)
- Reputation: Very important in small communities, less so in large ones
- Enforceable contracts: Sometimes (prenup)
- Superrationality: Potentially yes but not required
- It will be better to hunt stag together just from iteration
- Abusive relationships are not SH shaped as the partner can’t or won’t leave, especially when there’s no reputational effect that the abusive partner cares about
- SP shaped?
Marketing
- PD shaped
- 2 rival companies with equally effective marketing departments are in a roughly PD shaped game (assuming customer pool is fixed size)
- If one spends money on advertising (defect) then the other is disadvantaged if they don’t
- But both would be better off if neither advertised
- Often the customer pool is not of fixed size which would mean that this may not really be a PD in real life
- It is important here to note that collusion between companies is generally forbidden so communication is not allowed
- SH shaped?
- Iteration: Yes.
- Reputation: Not really – there aren’t negative reputation effects to advertising
- Enforceable contracts: No
- Modelling other players: Yes but they are (accurately) modelling each other as playing defect
- In theory the businesses could get into C-C but advertising is so ingrained as the default choice that this would be hard
- Possibly there’s an availability heuristic problem here – I’ll obviously remember the examples of industries which are stuck in D-D as I constantly see their adverts (Pepsi vs Coke, tech companies, supermarkets).
- I tried to think of industries that aren’t advertising much but I’m drawing a blank.
- SP shaped?
- In theory maybe, however the lack of communication makes it extremely hard for companies to change between solutions
Doping in sport
- PD shaped
- Using drugs gives you an edge but has a price potential price with medical dangers
- There is a potential difference from “standard” PD in that doping is disadvantageous to a more able athlete if the less able athlete is not doping (which would make this an alibi game).
- SH shaped?
- Iteration: Yes (although results of previous iterations are not very legible)
- Reputation: Yes (again, legibility problems)
- Enforceable contract: Yes – drugs testing and bans
- Superrationality: No
- The enforceable contract is probably the biggest effect here – just relying on iteration and reputation would be insufficient
- SP shaped?
- Because the solution is an enforceable contract the decision is not very SP shaped on an individual basis
- Negotiating what the contract should be is SP shaped
- What makes a medical exemption ok?
- How much inhaler is an ok amount?
Cold war nuclear arms race
- PD shaped
- Making more nukes is equivalent to defect
- SH shaped?
- Iteration: Yes, treaty followed treaty. Inspections were allowed to verify adherence.
- Reputation: A bit but probably not in a relevant way
- Enforceable contract: Not really – if one country reneged then it’s not like the other country could sue
- Superrationality: Yes. It seems both players realised D-D was terrible and wanted to play C-C. They couldn't rely on just this so iteration was very important.
- SP shaped?
- Deciding the terms of the agreement is SP shaped
- What size arsenal is better for which country?
- Which particular weapons are better for which country?
- Deciding the terms of the agreement is SP shaped
Yes, I agree that some symptoms are likely highly correlated. I didn't intend to rule out that possibility with that sentence - I was just trying to say how I did my math (although I'm not sure how clear I was!). The correct conclusion is in the following sentence:
So having COVID on average gives you ~0.2 persistent symptoms vs not having COVID, with presumably some people having more than one symptom.
Possibly it would be better to add the caveat "0.2 persistent symptoms of those symptoms investigated".
On the whole I agree with Raemon’s review, particularly the first paragraph.
A further thing I would want to add (which would be relatively easy to fix) is that the description and math of the Kelly criterion is misleading / wrong.
The post states that you should:
bet a percentage of your bankroll equivalent to your expected edge
However the correct rule is:
bet such that you are trying to win a percentage of your bankroll equal to your percent edge.
(emphasis added)
The 2 definitions give the same results for 1:1 bets but will give strongly diverging results with other odds.
In addition the post gives a method of calculating one’s edge which gives correct results for 1:1 bets. It is not entirely clear how one would use the formula for non 1:1 bets but it doesn’t seem to indicate a calculation which would give one’s edge correctly. (55%-45%=10% doesn’t seem to readily generalise correctly in the same way that (55%-50%)/50%=10% does).
The post never states that the definition is only for 1:1 bets so the impression given by the post is that these formulae can be used in the cases given in the post. However there are no guarantees that any of the examples in the post are 1:1 bets.
(The post does mention that the example is based on 1:1 bets but it doesn’t imply that the calculation as given only works for such bets.)
As a result the post effectively ends up recommending making incorrectly sized bets. For non 1:1 bets it is possible to calculate recommended bet sizes which have an expected negative impact on the logarithm of ones bankroll and as such actively makes one’s life worse.
(e.g. the bet odds are at 25% but you believe there’s a 50% chance that it will resolve true then your edge is 100% (calculated properly, not using the method in the post as I’m not sure how to use that) and the post would suggest betting your entire bankroll. The correct Kelly calculation gives 1/3rd of your bankroll)
Without this correction I would strongly recommend against including this post in the review.
The post claims:
I have investigated this issue in depth and concluded that even a full scale nuclear exchange is unlikely (<1%) to cause human extinction.
This review aims to assess whether having read the post I can conclude the same.
The review is split into 3 parts:
- Epistemic spot check
- Examining the argument
- Outside the argument
Epistemic spot check
Claim: There are 14,000 nuclear warheads in the world.
Assessment: True
Claim: Average warhead yield <1 Mt, probably closer to 100kt
Assessment: Probably true, possibly misleading. Values I found were:
- US
- W78 warhead: 335-350kt
- W87 warhead: 300 or 475 kt
- Russia
- R-36 missile: 550-750 kt
- R29 missile: 100 or 500kt
The original claim read to me that 100kT was probably pretty close and 1Mt was a big factor of safety (~x10) but whereas the safety factor was actually less than that (~x3). However that’s the advantage of having a safety factor – even if it’s a bit misleading there still is a safety factor in the calculations.
I found the lack of links slightly frustrating here – it would have been nice to see where the OP got the numbers from.
Examining the argument
The argument itself can be summarized as:
- Kinetic destruction can’t be big enough
- Radiation could theoretically be enough but in practice wouldn’t be
- Nuclear winter not sufficient to cause extinction
One assumption in the arguments for 1 & 2 is that the important factor is the average warhead yield and that e.g. a 10Mt warhead doesn’t have an outsized effect. This seems likely and a comment suggests that going over 500kt doesn’t make as much difference as might be thought and that is why warheads are the size that they are.
Arguments 1 & 2 seem very solid. We have done enough tests that our understanding of kinetic destruction is likely to be fairly good so I don’t have much concerns there. Similarly, radiation is well understood and dispersal patterns seem kinda predictable in principle and even if these are wrong the total amount of radiation doesn't change, just the where it is.
Climate change is less easy to model, especially given that the scenario is a long way out of our actual experience. To trust the conclusion we have to trust the models. I haven’t looked into this – different researchers have come to different conclusions and we hope that the true value is between the worst and best case found (or at least not too much worse than the worst case).
One thing that I found frustrating in the post is that the <1% risk of human extinction from full scale nuclear war is given but the post is not explicit which arguments give such high confidence. The post rightly refers to a number of steps which need to happen to make catastrophic climate change happen:
- Given a nuclear exchange how probable are firestorms?
- Given firestorms how probable is black carbon to be lifted into the atmosphere?
- Given lofting, how probable is it to stay aloft for sufficient time?
- Given long term cooling, how probable is human extinction?
But it is not explicit as to which step(s) are most improbable.
The OP points towards an exchange between Robock and Reisner (could a small scale conflict cause a nuclear winter?) where it finds Reisner’s position more compelling. However, finding the position more compelling does not suggest to me that this part of the logic chain can be doing much heavy lifting.
(similarly, I don’t think military planners being aware of the danger of nuclear winter can be doing much heavy lifting to get us up to <1% probability).
So the main part of the logic to conclude high confidence must be that even catastrophic climate change would not cause human extinction.
The OP quotes Luke Oman as saying he believed that it would be ~1 in 10,000 to 1 in 100,000. It also links to him giving his reasoning. This is based mainly on the Toba supervolcano and other similar events failing to wipe out humanity’s ancestors and that the southern hemisphere is likely to be less badly hit than the north in a nuclear exchange. (I would have liked this to have been summarized in the OP as it is probably the strongest argument for non-extinction).
The interview doesn’t go into enough detail for me to assess how strong Luke’s arguments are and this makes it hard for me to get the same level of confidence that Luke has – how close an analogy to full scale nuclear war is Toba likely to be?
The post also suggests food stores and human ingenuity as reasons to think that humanity would avoid extinction.
Given how key this part of the logic chain seems to be I would have liked this section to be more detailed.
Outside the argument
I would have also liked the post to consider uncertainty outside the argument.
A few comments point out other possible mechanisms which might cause additional risk:
Giving <1% probability that full scale nuclear war will cause human extinction implies sufficient confidence that we have thought of all the possible causes of extinction and/or that other possible causes will have similarly low probability of extinction.
This isn’t discussed in the OP which seems like a weakness.
Conclusion
I think that the post (plus the link to the Luke Oman interview) give sufficient evidence – a full scale nuclear exchange is unlikely (<1%) to cause human extinction.
If I wanted to hone in my probability further the post helpfully supplies further reading which is a great feature.
The main subject where I would have liked to see more detail is what I see as the crux of the matter with climate change – given extremely large temperature changes how likely is humanity to go extinct? The arguments in the post itself wouldn’t have been enough for me – it was only reading the Luke Oman interview that convinced me.
Overall I think the post is strong and think it is a good candidate for the 2020 review.
I suppose it depends how general one is aiming to be. If by general intelligence we mean "able to do what a human can do" then no, at this point the method isn't up to that standard.
If instead we mean "able to achieve SOTA on a difficult problem which it wasn't specifically designed to deal with" then PI-MNIST seems like a reasonable starting point.
Also, from a practical standpoint PI-MNIST seems reasonable for a personal research project.
I do think D𝜋's original post felt like it was overstating it's case. From a later comment it seems like they more see it as a starting point to add more steps onto to achieve a more general intelligence (i.e. not just a scaling up of the same thing). So instead of paradigms which are MLP + others or DBM + others we would have S(O)NN + others.
I think there's a mistake which is being repeated in a few comments both here and on D𝜋's post which needs emphasizing. Below is my understanding:
D𝜋 is attempting to create a general intelligence architecture. He is using image classification as a test for this general intelligence but his architecture is not optimized specifically for image identification.
Most attempts on MNIST use what we know about images (especially the importance of location of pixels) and design an architecture based on those facts. Convolutions are an especially obvious example of this. They are very effective at identifying images but the fact that we are inserting some of our knowledge of images into the algorithm precludes it from being a general intelligence methodology (without a lot of modification at least).
The point of using PI-MNIST (where locations of pixels in the dataset are randomized) is that we can't use any of our own understanding of images to help with our model so a model which is good at PI-MNIST is proving a more general intelligence than a model which is good at MNIST.
That is why D𝜋 keeps on emphasizing that this is PI-MNIST.
More “for Covid” vs “with Covid” from England:
https://www.bbc.co.uk/news/health-59862568
Ratio in October was 3:1 (for:with) but this has gone down to 2:1. “For” cases are rising but at a lower fractional rate than “with” cases.
We don’t know which patients are in the hospital because of Covid
BBC reports today (i.e. after post was published) that 3 in 10 people who are in hospital with COVID in England were admitted for something else.