Multitudinous outside views 2020-08-18T06:21:47.566Z · score: 46 (21 votes)
Update more slowly! 2020-07-13T07:10:50.164Z · score: 50 (19 votes)
A Personal (Interim) COVID-19 Postmortem 2020-06-25T18:10:40.885Z · score: 171 (66 votes)
Market-shaping approaches to accelerate COVID-19 response: a role for option-based guarantees? 2020-04-27T22:43:26.034Z · score: 39 (10 votes)
Potential High-Leverage and Inexpensive Mitigations (which are still feasible) for Pandemics 2020-03-09T06:59:19.610Z · score: 35 (14 votes)
Ineffective Response to COVID-19 and Risk Compensation 2020-03-08T09:21:55.888Z · score: 29 (15 votes)
Link: Does the following seem like a reasonable brief summary of the key disagreements regarding AI risk? 2019-12-26T20:14:52.509Z · score: 11 (5 votes)
Updating a Complex Mental Model - An Applied Election Odds Example 2019-11-28T09:29:56.753Z · score: 10 (4 votes)
Theater Tickets, Sleeping Pills, and the Idiosyncrasies of Delegated Risk Management 2019-10-30T10:33:16.240Z · score: 26 (14 votes)
Divergence on Evidence Due to Differing Priors - A Political Case Study 2019-09-16T11:01:11.341Z · score: 27 (11 votes)
Hackable Rewards as a Safety Valve? 2019-09-10T10:33:40.238Z · score: 18 (5 votes)
What Programming Language Characteristics Would Allow Provably Safe AI? 2019-08-28T10:46:32.643Z · score: 5 (5 votes)
Mesa-Optimizers and Over-optimization Failure (Optimizing and Goodhart Effects, Clarifying Thoughts - Part 4) 2019-08-12T08:07:01.769Z · score: 17 (9 votes)
Applying Overoptimization to Selection vs. Control (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 3) 2019-07-28T09:32:25.878Z · score: 19 (6 votes)
What does Optimization Mean, Again? (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 2) 2019-07-28T09:30:29.792Z · score: 29 (6 votes)
Re-introducing Selection vs Control for Optimization (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 1) 2019-07-02T15:36:51.071Z · score: 31 (7 votes)
Schelling Fences versus Marginal Thinking 2019-05-22T10:22:32.213Z · score: 23 (14 votes)
Values Weren't Complex, Once. 2018-11-25T09:17:02.207Z · score: 34 (15 votes)
Oversight of Unsafe Systems via Dynamic Safety Envelopes 2018-11-23T08:37:30.401Z · score: 11 (5 votes)
Collaboration-by-Design versus Emergent Collaboration 2018-11-18T07:22:16.340Z · score: 12 (3 votes)
Multi-Agent Overoptimization, and Embedded Agent World Models 2018-11-08T20:33:00.499Z · score: 9 (4 votes)
Policy Beats Morality 2018-10-17T06:39:40.398Z · score: 15 (15 votes)
(Some?) Possible Multi-Agent Goodhart Interactions 2018-09-22T17:48:22.356Z · score: 21 (5 votes)
Lotuses and Loot Boxes 2018-05-17T00:21:12.583Z · score: 29 (6 votes)
Non-Adversarial Goodhart and AI Risks 2018-03-27T01:39:30.539Z · score: 65 (15 votes)
Evidence as Rhetoric — Normative or Positive? 2017-12-06T17:38:05.033Z · score: 1 (1 votes)
A Short Explanation of Blame and Causation 2017-09-18T17:43:34.571Z · score: 1 (1 votes)
Prescientific Organizational Theory (Ribbonfarm) 2017-02-22T23:00:41.273Z · score: 3 (4 votes)
A Quick Confidence Heuristic; Implicitly Leveraging "The Wisdom of Crowds" 2017-02-10T00:54:41.394Z · score: 1 (2 votes)
Most empirical questions are unresolveable; The good, the bad, and the appropriately under-powered 2017-01-23T20:35:29.054Z · score: 7 (5 votes)
A Cruciverbalist’s Introduction to Bayesian reasoning 2017-01-12T20:43:48.928Z · score: 1 (2 votes)
Map:Territory::Uncertainty::Randomness – but that doesn’t matter, value of information does. 2016-01-22T19:12:17.946Z · score: 6 (11 votes)
Meetup : Finding Effective Altruism with Biased Inputs on Options - LA Rationality Weekly Meetup 2016-01-14T05:31:20.472Z · score: 1 (2 votes)
Perceptual Entropy and Frozen Estimates 2015-06-03T19:27:31.074Z · score: 17 (12 votes)
Meetup : Complex problems, limited information, and rationality; How should we make decisions in real life? 2013-10-09T21:44:19.773Z · score: 3 (4 votes)
Meetup : Group Decision Making (the good, the bad, and the confusion of welfare economics) 2013-04-30T16:18:04.955Z · score: 4 (5 votes)


Comment by davidmanheim on Industrial literacy · 2020-10-06T20:45:55.572Z · score: 4 (2 votes) · LW · GW

That all makes sense - I'm less certain that there is a reachable global maximum that is a Pareto improvement in terms of inputs over the current system. That is, I expect any improvement to require more of some critical resource - human time, capital investment, or land.

Comment by davidmanheim on Industrial literacy · 2020-10-05T09:49:46.464Z · score: 8 (7 votes) · LW · GW

No, the claim as written is true - agriculture will ruin soil over time, which has happened in recent scientific memory in certain places in Africa. And if you look at the biblical description of parts of the middle east, it's clear that desertification had taken a tremendous toll over the past couple thousand years. That's not because of fertilizer usage, it's because agriculture is about extracting food and moving it elsewhere, usually interrupting the cycle of nutrients, which happens organically otherwise. Obviously, natural habitats don't do this in the same way, because the varieties of plants shift over time, fauna is involved, etc. 

Comment by davidmanheim on Industrial literacy · 2020-10-05T09:42:20.746Z · score: 10 (5 votes) · LW · GW

Yes, in the modern world, where babies are seen as precious, that is true. It clearly wasn't as big a deal when infant mortality was very high.

Comment by davidmanheim on Industrial literacy · 2020-10-05T09:40:12.636Z · score: 7 (5 votes) · LW · GW

This is disingenuous, I think. Of course they don't exist at the necessary scale yet, because the market is small. If the market grew, and was profitable, scaling would be possible. Rare earths aren't rare enough to be a real constraint, we'd just need to mine more of them.  The only thing needed would be to make more of things we know how to make. (And no, that wouldn't happen, because the new tech being developed would get developed far faster, and used instead.) 

Comment by davidmanheim on Industrial literacy · 2020-10-05T09:35:34.758Z · score: 14 (9 votes) · LW · GW

This isn't critiquing the claim, though. Yes, there are alternatives that are available, but those alternatives - multi-cropping, integrating livestock, etc. are more labor intensive, and will produce less over the short term. And I'm very skeptical that the maximum is only local - have you found evidence that you can use land more efficiently, while keeping labor minimal, and produce more? Because if you did, there's a multi-billion dollar market for doing that. Does that make the alternatives useless, or bad ideas? Not at all, and I agree that changes are likely necessary for long-term stability - unless other technological advances obviate the need for them. But we can't pretend that staying at the maximum isn't effectively necessary.

Comment by davidmanheim on Expansive translations: considerations and possibilities · 2020-10-05T04:51:19.140Z · score: 4 (2 votes) · LW · GW

Agreed - and this reminds me of the observation that all of physics is contained in a single pebble; with enough undesrstnding, you could infer all of physics from close observation of quantum effects, find gravity at a very small scale if you had sensitive enough instruments, know much of natural history, liked the fact that earth has running water that made the stone smooth, that it must be in a universe more than a certain age given its composition, etc. With enough detail, any facet of a story requires effectively unlimited detail to fully understand.

And that makes it clear that we don't intend for every translation to be of unlimited depth - but the depth of the translation matters, and we trade off between depth of translation and accuracy. Translating Sherbert Lemon as Lemon Sorbet is probably a lack of understanding and an overly direct literal-but-incorrect meaning, while translating it as Crembo might be a reasonable choice because of the context, but is not at all a literal translation.

Comment by davidmanheim on Expansive translations: considerations and possibilities · 2020-09-30T11:22:31.534Z · score: 12 (2 votes) · LW · GW

As the post notes, inferential distance relates to differing worldviews and life experiences. This was written to an audience that mostly understands what inferential distance has to do with different worldviews - how would you explain it to a different audience?

Well, a typical translation doesn't try to bridge the gap between languages, it just picks something on the far side of the gap that seems similar to the one on the near side. But that leaves something out.

An example of this is in translations of Harry Potter, where Dumbledore's password is translated into a local sweet. The UK versions has "Sherbet Lemon" while the US version has "Lemon drop." Are these the same? I assumed so, but actually it seems the UK version has a "fizzy sweet powder" on the inside. In Danish and Swedish, it's (mis?) translated as lemon ice cream - which isn't the same at all. And in Hebrew, it's translated as Krembo, which doesn't even get close to translating the meaning correctly - it's supposed to be an "equivalent children’s dessert" - but the translation simply doesn't work, because you can't carry a Krembo around in your pocket, since it would melt. Does this matter? Well, the difference between a kindly old wizard who carries around a sucking candy, and one who carries around a kind-of-big marshmallow dessert. But that's beside the point - you don't translate the life experience that growing up eating sherbert lemons gives you, you find an analogue. 

The only way to translate a specific word or term accurately could be to provide so much background that the original point is buried, and the only way to translate an idea is to find an analogue that the reader already understands. And that's why translation is impossible - but we do it anyways, and just accept that the results are fuzzy equivalents, and accept that worldviews are different enough that bridging the gap is impossible.

Comment by davidmanheim on Puzzle Games · 2020-09-29T13:16:28.349Z · score: 2 (1 votes) · LW · GW

Tier 3, I think: Hoplite, on Android. 

The free game is basically a roguelike, but it's full information on each level, with only a little bit of strategy for which abilities to pick, and the Challenge mode available in the paid version, for $3, has a lot more straight puzzles.

Comment by davidmanheim on Stripping Away the Protections · 2020-09-23T21:17:17.157Z · score: 6 (1 votes) · LW · GW

To what extent are these dynamics the inevitable result of large organizations?


I want to note that I've previously argued that much of the dynamics are significantly forced by structure - but not in this context, and I'm thinking about how much or little of that argument applies here. (I'll need to see what yo say in later posts in the series.)

Comment by davidmanheim on Needed: AI infohazard policy · 2020-09-22T15:54:52.723Z · score: 2 (1 votes) · LW · GW

I think there needs to be individual decisionmaking (on the part of both organizations and individual researchers, especially in light of the unilateralists' curse,) alongside a much broader discussion about how the world should handle unsafe machine learning, and more advanced AI.

I very much don't think that the AI safety community debating and coming up with shared, semi-public guidelines for, essentially, what to withhold from the broader public, done without input from the wider ML / AI and research community who are impacted and whose work is a big part of what we are discussing, would be wise. That community needs to be engaged in any such discussions.

Comment by davidmanheim on Needed: AI infohazard policy · 2020-09-22T04:40:09.201Z · score: 1 (2 votes) · LW · GW
There's some intermediate options available instead of just "full secret" or "full publish"... and I haven't seen anyone mention that...

OpenAI's phased release of GPT2 seems like a clear example of exactly this. And there is a forthcoming paper looking at the internal deliberations around this from Toby Shevlane, in addition to his extant work on the question of how disclosure potentially affects misuse.

Comment by davidmanheim on Needed: AI infohazard policy · 2020-09-22T04:34:24.546Z · score: 2 (1 votes) · LW · GW

The first thing I would note is that stakeholders need to be involved in making any guidelines, and that pushing for guidelines from the outside is unhelpful, if not harmful, since it pushes participants to be defensive about their work. There are also an extensive literature discussing the general issue of information dissemination hazards and the issues of regulation in other domains, such as nuclear weapons technology, biological and chemical weapons, and similar.

There is also a fair amount of ongoing work on synthesizing this literature and the implications for AI. Some of it is even on this site. For example, see: and

So there is tons of discussion about this already, and there is plenty you should read on the topic - I suspect you can start with the paper that provided the name for your post, and continuing with sections of GovAI's research agenda.

Comment by davidmanheim on Are aircraft carriers super vulnerable in a modern war? · 2020-09-21T10:58:02.220Z · score: 4 (2 votes) · LW · GW

Noting that this is correct, but incomplete. They are very important for force projection even in near-peer engagements, since otherwise you likely can't get your planes to where you need them. The question that matters for this is who wins the area-denial / anti-aircraft battle, i.e. can drones and similar get close enough to sink anything, and this is the critical question anyways, since your carriers and planes are useless if you can't get close enough. And this isn't my area, but my very preliminary impression is that AA/AD makes aerial combat fairly defense-dominant.

Comment by davidmanheim on What's the best overview of common Micromorts? · 2020-09-04T08:57:32.990Z · score: 4 (2 votes) · LW · GW
"Could someone write a LW style book review of the Norm Chronicles?"


Comment by davidmanheim on Reflections on AI Timelines Forecasting Thread · 2020-09-03T06:04:37.643Z · score: 3 (2 votes) · LW · GW
"A good next step would be to create more consensus on the most productive interpretation for AGI timeline predictions. "

Strongly agree with this. I don't think the numbers are meaningful, since AGI could mean anything from "a CAIS system-of-systems that can be used to replace most menial jobs with greater than 50% success," to "a system that can do any one of the majority of current jobs given an economically viable (<$10m) amount of direct training and supervision" to "A system that can do everything any human is able to do at least as well as that human, based only on available data and observation, without any direct training or feedback, for no marginal cost."

Comment by davidmanheim on What's the best overview of common Micromorts? · 2020-09-03T05:49:34.223Z · score: 12 (7 votes) · LW · GW

Scott's answer is a good one - you should read "The Norm Chronicles." But I think the question has a problem. Micromorts are a time-agnostic measure of dying, and the problem is that most risks you take don't actually translate well into micromorts.

Smoking a cigarette, which reduces your life expectancy by around 30 seconds, translates into either zero micromorts, or one, depending on how you set up the question. Increasing your relative risk of dying from cancer in 30 years isn't really the same as playing Russian roulette with a 1-million-chamber gun. Similarly, a healthy 25 year old getting COVID has about a 70-micromort risk based on direct mortality from COVID. But that number ignores the risks of chronic fatigue, later complications, or reduced life expectancy (all of which we simply don't know enough to quantify well.)

The answer that health economists have to this question is the QALY, which has its own drawbacks. For example, QALYs can't uniformly measure the risks of Russian roulette, since the risk depends on the age and quality of life of the player.

What we're left with is that the actual question we want answered has a couple more dimensions than a single metric can capture - and as I have mentioned once or twice elsewhere, reductive. metrics. have various. problems.

Comment by davidmanheim on nostalgebraist: Recursive Goodhart's Law · 2020-08-28T07:46:51.037Z · score: 2 (1 votes) · LW · GW

I'd agree with the epistemic warning ;)

I don't think the model is useful, since it's non-predictive. And we have good reasons to think that human brains are actually incoherent. Which means I'm skeptical that there is something useful to find by fitting a complex model to find a coherent fit for an incoherent system.

Comment by davidmanheim on Multitudinous outside views · 2020-08-28T07:44:23.185Z · score: 2 (1 votes) · LW · GW

I don't think that weights are the right answer - not that they aren't better than nothing, but as the Tesla case shows, the actual answer is having a useful model with which to apply reference classes. For example, once you have a model of stock prices as random walks, the useful priors are over the volatility rather than price, or rather, the difference between implied options volatility and post-hoc realized volatility for the stock, and other similar stocks. (And if your model is stochastic volatility with jumps, you want priors over the inputs to that.) At that point, you can usefully use the reference classes, and which one to use isn't nearly as critical.

In general, I strongly expect that in "difficult" domains, causal understanding combined with outside view and reference classes will outperform simply using "better" reference classes naively.

Comment by davidmanheim on nostalgebraist: Recursive Goodhart's Law · 2020-08-27T10:44:06.941Z · score: 2 (1 votes) · LW · GW

I talked about this in terms of "underspecified goals" - often, the true goal doesn't usually exist clearly, and may not be coherent. Until that's fixed, the problem isn't really Goodhart, it's just sucking at deciding what you want.

I'm thinking of a young kid in a candy store who has $1, and wants everything, and can't get it. What metric for choosing what to purchase will make them happy? Answer: There isn't one. What they want is too unclear for them to be happy. So I can tell you in advance that they're going to have a tantrum later about wanting to have done something else no matter what happens now. That's not because they picked the wrong goal, it's because their desires aren't coherent.

Comment by davidmanheim on nostalgebraist: Recursive Goodhart's Law · 2020-08-27T10:36:49.366Z · score: 6 (3 votes) · LW · GW

Strongly agree - and Goodhart's law is at least 4 things. Though I'd note that anti-inductive behavior / metric gaming is hard to separate from goal mis-specification, for exactly the reasons outlined in the post.

But saying there is a goal too complex to be understandable and legible implies that it's really complex, but coherent. I don't think that's the case of individuals, and I'm certain it isn't true of groups. (Arrow's theorem, etc.)

Comment by davidmanheim on Multitudinous outside views · 2020-08-19T07:20:48.140Z · score: 2 (1 votes) · LW · GW
I suspect final forecasts that are "good enough" are often shockingly simple, and the hard part of a forecast is building/extracting a "correct enough" simplified model of reality and getting a small amount of the appropriate data that you actually need.

I think that it's often true that good forecasts can be simple, but I also think that the gulf between "good enough" and "very good" usually contains a perverse effect, where slightly more complexity makes the model perhaps slightly better in expectation, and far worse in properly estimating variance or accounting for uncertainties outside the model. That means that for the purpose of forecasting, you get much worse (brier scores) before you get better.

As a concrete example, this is seen when people forecast COVID deaths. They start with a simple linear trend, then say they don't really think it's linear, it's actually exponential, so they roughly adjust their confidence and have appropriate uncertainties around a bad model. Then they get fancier, and try using a SIR model that gives "the" answer, and the forecaster simulates 100 runs to create a distribution by varying R_0 withing a reasonable range. That gives an uncertainty range, and a very narrow resulting distribution - which the forecaster is more narrowly willing to adjust, because their model accounts for the obvious sources of variance. Then schools are reopened, or treatment methods improve, or contact rates drop as people see case counts rise, and the model's assumptions are invalidated in a different way than was expected.

I think while consulting many models is a good reminder, the hard part is choosing which model(s) to use in the end. I think your ensemble of models can often do much better than an unweighted average of all the models you've considered, since some models are a) much less applicable, b) much more brittle, c) much less intuitively plausible, or d) much too strongly correlated than other models you have.

As I said to Luke in a comment to his link to an excellent earlier post that discusses this, I think there is far more to be said about how to do model fusion, and agreed with his point in his paper that ensembles which simply average models are better than single models, but still worse than actually figuring out what each model tells you.

Comment by davidmanheim on Model Combination and Adjustment · 2020-08-18T20:23:50.455Z · score: 8 (4 votes) · LW · GW

Link rot notice: (Technical note: I say "model combination" rather than "model averaging" on purpose.) <- that link should now point HERE instead.

Comment by davidmanheim on Multitudinous outside views · 2020-08-18T20:20:06.588Z · score: 15 (7 votes) · LW · GW

I don't think I had seen that, and wow, it definitely covers basically all of what I was thinking about trying to say in this post, and a bit more.

I do think there is something useful to say about how reference class combinations work, and using causal models versus correlational ones for model combination given heterogeneous data - but that will require formulating it more clearly than I have in my head right now. (I'm working on two different projects where I'm getting it straighter in my head, which led to this post, as a quick explanation why people need to stop using "reference classes" that don't work well because they can't find a better one, as if "reference class" is an argument about correctness of a prediction.)

Comment by davidmanheim on Multitudinous outside views · 2020-08-18T09:28:34.897Z · score: 5 (3 votes) · LW · GW

I tried to kind-of do this with the examples here. Unfortunately, I don't actually have a clear recollection of what I did for forecasting many questions, as it's been a long time since the original tournament. And for more recent questions, I often comment of metaculus - but if longer analyses seem worthwhile, maybe that should be a post to accompany, say, my 2021 yearly predictions. (I'm naively expecting to have free time by then.)

But for most questions, the outside view is relatively easy. That doesn't mean it's the standard "use a reference class," since as I said, models dictate that. My favorite example of that is asset price forcasts, where I remember that some super-forecasters were building their own models of asset prices and probability of movement by a certain amount in a given time period from historical data, and I was just forecasting the implicit price distribution given by options prices, and absolutely dominating brier scores for those questions. (Occasionally I had very slight modifications to reflect my inside view of surprises outside that model, like stock splits and dividends, where actually modelling it correctly was annoying and not worthwhile.)

For other questions, like forecasting life-spans of dictators, the answer is fundamentally hard, and I don't think reference classes are nearly as valuable. And for COVID, I've written about my very early expectations - but maybe you think that a follow-up on why superforecasters mostly disagreed with my forecasts / I modeled things differently than them over the past 3-4 months would be interesting and useful. (I would need to check what amount of that type of discussion I can discuss publicly.)

Edit to add: There are also some interesting things to discuss around epistemic superiority, and how to deal with a relative lack of expertise in choosing between expert views or in deciding how and when it makes sense to disagree as a "general expert" with forecasting expertise. That's a bit more philosophical, but I'm hoping to discuss related issues in a paper on elicitation I am writing.

Comment by davidmanheim on By what metric do you judge a reference class? · 2020-08-18T06:24:02.871Z · score: 5 (3 votes) · LW · GW

I just wrote a different post which discusses this issue in slightly different terms, with a few links which might be helpful:

Comment by davidmanheim on Will OpenAI's work unintentionally increase existential risks related to AI? · 2020-08-17T17:06:13.688Z · score: 2 (1 votes) · LW · GW

Oh. Right. I should have gotten the reference, but wasn't thinking about it.

Comment by davidmanheim on Will OpenAI's work unintentionally increase existential risks related to AI? · 2020-08-12T05:58:13.711Z · score: 5 (3 votes) · LW · GW

I'd focus even more, (per my comment to Vanniver's response,) and ask "What parts of OpenAI are most and least valuable, and how do these relate to their strategy - and what strategy is best?"

Comment by davidmanheim on Will OpenAI's work unintentionally increase existential risks related to AI? · 2020-08-12T05:56:35.271Z · score: 4 (2 votes) · LW · GW

I would reemphasize that the "does OpenAI increase risks" is a counterfactual question. That means we need to be clearer about what we are asking as a matter of predicting what the counterfactuals are, and consider strategy options for going forward. This is a major set of questions, and increasing or decreasing risks as a single metric isn't enough to capture much of interest.

For a taste of what we'd want to consider, what about the following:

Are we asking OpenAI to pick a different, "safer" strategy?

Perhaps they should focus more on hiring people to work on safety and strategy, and hire fewer capabilities researchers. That brings us to the Dr. Wily/Dr. Light question - Perhaps Dr. Capabilities B. Wily shouldn't be hired, and Dr. Safety R. Light should be, instead. That means Wily does capabilities research elsewhere, perhaps with more resources, and Light does safety research at OpenAI. But the counterfactual is that Light would do (perhaps slightly less well funded) research on safety anyways, and Wily would work on (approximately as useful) capabilities research at OpenAI - advantaging OpenAI in any capabilities races in the future.

Are we asking OpenAI to be larger, and (if needed,) we should find them funding?

Perhaps the should hire both, along with all of Dr. Light and Dr. Wily's research teams. Fast growth will dilute OpenAI's culture, but give them an additional marginal advantage over other groups. Perhaps bringing them in would help OpenAI in race dynamics, but make it more likely that they'd engage in such races.

How much funding would this need? Perhaps none - they have cash, they just need to do this. Or perhaps tons, and we need them to be profitable, and focus on that strategy, with all of the implications of that. Or perhaps a moderate amount, and we just need OpenPhil to give them another billion dollars, and then we need to ask about the counterfactual impact of that money.

Or OpenAI should focus on redirecting their capabilities staff to work on safety, and have a harder time hiring the best people who want to work on capabilities? Or OpenAI should be smaller and more focused, and reserve cash?

These are all important questions, but need much more time than I, or I suspect, most of the readers here have available - and are probably already being discussed more usefully by both OpenAI, and their advisors.

Comment by davidmanheim on Forecasting AI Progress: A Research Agenda · 2020-08-11T14:43:21.942Z · score: 5 (3 votes) · LW · GW
Now the perhaps harder step is trying to get traction on them

Yes, very much so. We're working on a few parts of this now, as part of a different project, but I agree that it's tricky. And there are a number of other things that seem like potentially very useful projects if others are interested in collaborations, or just some ideas / suggestions about how they could be approached.

(On the tables, unfortunately the tables were pasted in as images from another program. We should definitely see if we can get higher-resolution, even if we can't convert to text easily.)

Comment by davidmanheim on Understanding information cascades · 2020-08-04T06:13:05.276Z · score: 4 (2 votes) · LW · GW

Or perhaps less unreasonably, we need clear epistemic superiority hierarchies, likely per subject area. And it occurs to me that this could be a super-interesting agent-based/graph theoretic modeling study of information flow and updating. As a nice bonus, this can easily show how ignoring epistemic hierarchies will cause conspiracy cascades - and perhaps show that it will lead to the divergence of rational agent beliefs which Jaynes talks about in PT:LoS.

Comment by davidmanheim on What a 20-year-lead in military tech might look like · 2020-08-02T09:33:02.341Z · score: 3 (2 votes) · LW · GW

Great - I was just surprised that nothing in that vein was cited, and you missed a few issues that are heavily discussed, including area denial tech, multipolarity, or the reduced value of land warfare compared to economic impacts and destruction or compromise of matériel.

Comment by davidmanheim on Understanding information cascades · 2020-08-02T08:07:42.873Z · score: 4 (2 votes) · LW · GW

Good find - I need to look into this more. The paper is on scihub, and it says it needs to be non-cyclical, so yes.

"All the examples in which communicating values of a union-consistent function fails to bring about consensus... must contain a cycle; if there are no cycles in the communication graph, consensus on the value of any union consistent function must be reached."

Comment by davidmanheim on What a 20-year-lead in military tech might look like · 2020-07-31T12:02:13.273Z · score: 5 (4 votes) · LW · GW

It probably would have been good to do a basic search for what has been done on this topic by experts.

Comment by davidmanheim on Are we in an AI overhang? · 2020-07-29T14:24:13.804Z · score: 2 (1 votes) · LW · GW

Agreed, but I suspect that replacing those hard-coded elements will get easier over time as well.

Comment by davidmanheim on Are we in an AI overhang? · 2020-07-29T10:56:59.199Z · score: 4 (2 votes) · LW · GW

Presumably, because with a big-enough X, we can generate text descriptions of scenes from cameras and feed them in to get driving output more easily than the seemingly fairly slow process to directly train a self-driving system that is safe. And if GPT-X is effectively magic, that's enough.

I'm not sure I buy it, though. I think that once people agree that scaling just works, we'll end up scaling the NNs used for self driving instead, and just feed them much more training data.

Comment by davidmanheim on Can you get AGI from a Transformer? · 2020-07-24T09:10:07.989Z · score: 4 (2 votes) · LW · GW

I'm unsure that GPT3 can output, say, a ipython notebook to get the values it wants.

That would be really interesting to try...

Comment by davidmanheim on Six economics misconceptions of mine which I've resolved over the last few years · 2020-07-22T17:45:05.837Z · score: 4 (2 votes) · LW · GW

re: Index funds, if it's just about risk tolerance, you're better off with options on indexes + index funds than you are picking stocks.

Comment by davidmanheim on Math. proof of the superiority of independent guesses? · 2020-07-19T13:59:07.256Z · score: 2 (1 votes) · LW · GW

If the guesses are unbiased, the law of large numbers can be used to show this:

(If you look at the proof, you can see where the independence assumption comes in.)

Comment by davidmanheim on What are some good public contribution opportunities? (100$ bounty) · 2020-07-15T14:03:32.225Z · score: 2 (3 votes) · LW · GW

I'd be really happy if someone were to figure out how to clearly characterize which Goodhart failure mode is occurring in a toy world with simple optimizers. (Bonus: and also look at what types of agents do or do not display the different failure modes.)

For example, imagine you have a blockworld, where the agent is supposed to push blocks to a goal, and is scored based on distance from the goal. It would be good to have a clear way to delineate which failures can / do occur, and provide the failure category.

A change in regime failure might happen if the agent finds a strategy that works in the training world, where, say, you are only supposed to push the blocks right, and the goal is against the right wall, but in the test set the goal is elsewhere.

An extremal Goodhart failure might be that the training world is 10x10, and in the test set there is a 20x20 world, and the agent stops pushing after moving it 10 blocks.

A causal Goodhart failure might be if the goal is movable, and the agent accidentally pushes it away from where it moves the blocks towards.

Comment by davidmanheim on Update more slowly! · 2020-07-13T17:04:59.584Z · score: 4 (2 votes) · LW · GW

I should clarify that what I mean by "slow" was supposed to be in the cognitive / Kahneman sense. In most cases, as I said, "if you don't need to make any decisions, at least file everything away and decide that it's unclear." Instead, what I see people do is jump to updating / grabbing a hypothesis, acting "fast." The failure modes that this can cause will include over or under-updating, ignoring "missing" hypotheses and not thinking of alternative explanations, narrowing the hypotheses too early, not updating marginally, etc.

Given that, I think I agree on all of the substantive points. However, I will note that including an explicit "other" category is tricky, but critical. The problem is that given such a category, over time it's more plausible than anything else. It turns into the equivalent of "the witch down the road did it," which is super-parsimonious and can explain anything.

And slowness in the sense you thought I was talking about is equivalent to lowering the weight on evidence, or to having a higher weight on priors. Strength of priors and how much to weight evidence are good questions to discuss, and can be tricky, but weren't my point in this post.

Comment by davidmanheim on Your Prioritization is Underspecified · 2020-07-12T17:24:35.048Z · score: 2 (1 votes) · LW · GW


Comment by davidmanheim on Your Prioritization is Underspecified · 2020-07-12T07:17:52.846Z · score: 4 (2 votes) · LW · GW

I think there is a critical piece which could be added about Satisficing over strategies. "Satisficing" would be giving a bit of thought to each priority structure and the tasks, finding the thing that is highest value based on the first pass, and just doing that without spending lots of time on meta-level concerns.

The above post also ignores heuristic approaches to VoI (Value of Information). I wrote about that here, but the short answer is that you should spend 5 seconds thinking about whether there is something that would inform your decision, and if there is and it's very easy to get, you should get it, and if it's hard to get but very valuable then you should read the post.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-07T12:59:49.117Z · score: 4 (2 votes) · LW · GW

I don't understand the hypothetical.

If every country in the world had closed their borders well enough to stop all movement before it left China, yes, spread would have been prevented. But that's unfeasible even if there was political will, since border closures are never complete, and there was already spread outside of China by mid-January.

Once there is spread somewhere, you can't reopen borders. And even if you keep them closed, no border closure is 100% effective - unless you have magical borders, spread will inevitably end up in your country. And at that point, countries are either ready to suppress domestic spread without closures, or they aren't, and end up closing later instead of earlier.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-07T08:18:44.472Z · score: 2 (1 votes) · LW · GW

In general, I think that earlier closures would potentially have delayed spread enough to save lives due to getting vaccines and testing further along than they were.

I'm also claiming that now, with a fully in place and adequate test-and-trace program, including screening for passengers and isolation for positives, border closures have low marginal benefit. Without such a test and trace program, travel modifies the spread dynamics by little enough that it won't matter for places that don't have spread essentially controlled. The key case where it would matter is if the border closures delayed spread by long enough to put in place such systems, in which case they would have been very valuable. And yes, border closures in place have allowed this in some places, but certainly not the US or UK.

So, conditional on the policy failures, I think border closures were effectively only a way to signal, and if they distracted from putting in place testing and other systems by even a small amount, they were net negative.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-07T08:10:02.137Z · score: 2 (1 votes) · LW · GW

See the back-and-forth with John Wentsworth in the comments earlier -

Comment by davidmanheim on When a status symbol loses its plausible deniability, how much power does it lose? · 2020-07-07T08:07:44.659Z · score: 4 (3 votes) · LW · GW

The claim that Harvard is just a status symbol is that the entire variance in success from attending Harvard is explained by the two factors of 1) the characteristics of individual people entering the program, and 2) the prestige from being able to claim they graduated.

This seems implausible - so to extend this, I'd say all of the variance can be explained by those two plus a third factor, 3) the value of networking with Harvard students, faculty and staff.

In either case, the central point is that benefit from the services provided by Harvard are unrelated to the education they claim to provide.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-05T18:23:47.784Z · score: 2 (1 votes) · LW · GW

Again, it didn't actually stop spread - it slowed it slightly. Borders haven't been actually closed. Flights have continued, you just need connections to get a visa. But people have been able to return home - and dual citizens have been able to travel both ways - the entire time.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-04T19:43:58.658Z · score: 5 (3 votes) · LW · GW

Assuming away the political problem of making it stick, it seems clear that without universal border closures by countries, it would have made only a minor difference in spread - most cases that came to Europe, the US, and elsewhere didn't come from China.

If some set of countries were willing to completely shut down all borders, those countries might have avoided infections - might, but I'm skeptical. Even now, the countries that shut down international travel still have a fair amount of international travel, from diplomatic travel to repatriation of citizens to shipping and trucking. So it could plausibly have delayed spread by a month. In places that mounted a really effective response, a month might have made the difference between slow control and faster control. In most places, I think it would have shifted spread a couple weeks later.

Comment by davidmanheim on Conditions for Mesa-Optimization · 2020-07-02T20:56:37.799Z · score: 4 (2 votes) · LW · GW

The better link is the final version -

The link in the original post broke because it included the trailing period by accident.

Comment by davidmanheim on A Personal (Interim) COVID-19 Postmortem · 2020-07-02T20:55:09.425Z · score: 3 (2 votes) · LW · GW

I agree that it could be a death spiral, and think the caution is in general warranted. My personal situation was one where I had fairly little personal interaction with members of the community - though this is likely less true not - but that was why I decided that explicitly considering the consensus opinions was reasonable.