FHI paper published in Science: interventions against COVID-19 2020-12-16T21:19:00.441Z
How to do remote co-working 2020-05-08T19:38:11.623Z
How important are model sizes to your timeline predictions? 2019-09-05T17:34:14.742Z
What are some good examples of gaming that is hard to detect? 2019-05-16T16:10:38.333Z
Any rebuttals of Christiano and AI Impacts on takeoff speeds? 2019-04-21T20:39:51.076Z
Some intuition on why consciousness seems subjective 2018-07-27T22:37:44.587Z
Updating towards the simulation hypothesis because you think about AI 2016-03-05T22:23:49.424Z
Working at MIRI: An interview with Malo Bourgon 2015-11-01T12:54:58.841Z
Meetup : 'The Most Good Good You Can Do' (Effective Altruism meetup) 2015-05-14T18:32:18.446Z
Meetup : Utrecht- Brainstorm and ethics discussion at the Film Café 2014-05-19T20:49:07.529Z
Meetup : Utrecht - Social discussion at the Film Café 2014-05-12T13:10:07.746Z
Meetup : Utrecht 2014-04-20T10:14:21.859Z
Meetup : Utrecht: Behavioural economics, game theory... 2014-04-07T13:54:49.079Z
Meetup : Utrecht: More on effective altruism 2014-03-27T00:40:37.720Z
Meetup : Utrecht: Famine, Affluence and Morality 2014-03-16T19:56:44.267Z
Meetup : Utrecht: Effective Altruism 2014-03-03T19:55:11.665Z


Comment by soerenmind on Will OpenAI's work unintentionally increase existential risks related to AI? · 2021-01-04T16:59:59.372Z · LW · GW

No. Amodei led the GPT-3 project, he's clearly not opposed to scaling things. Idk why they're leaving but since they're all starting a new thing together, I presume that's the reason.

Comment by soerenmind on New SARS-CoV-2 variant · 2020-12-21T19:12:54.793Z · LW · GW

Some expert commentary here:


  • We previously thought a strain from Spain was spreading faster than the rest but it was just because og people returning from holiday in Spain.
  • Chance events can help a strain spread faster.
  • The UK (and Denmark) do more gene sequencing than other countries - that may explain why they picked up the new variant first.
  • The strain has acquired 17 mutations at once which is very high. Not clear what that means.
Comment by soerenmind on Continuing the takeoffs debate · 2020-11-24T11:00:08.501Z · LW · GW

For example, moving from a 90% chance to a 95% chance of copying a skill correctly doubles the expected length of any given transmission chain, allowing much faster cultural accumulation. This suggests that there’s a naturally abrupt increase in the usefulness of culture

This makes sense when there's only one type of thing to teach / imitate. But some things are easier to teach and imitate than others (e. g. catching a fish vs. building a house). And while there may be an abrupt jump in the ability to teach or imitate each particular skill, this argument doesn't show that there will be a jump in the number of skills that can be taught /imitated. (Which is what matters)

Comment by soerenmind on Covid Covid Covid Covid Covid 10/29: All We Ever Talk About · 2020-11-01T11:25:40.762Z · LW · GW

Right, to be clear that's the sort of number I have in mind and wouldn't call far far lower.

Comment by soerenmind on Covid Covid Covid Covid Covid 10/29: All We Ever Talk About · 2020-10-31T11:38:19.679Z · LW · GW

the infection fatality rate is far, far lower [now]


Just registering that, based on my reading of people who study the IFR over time, this is a highly contentious claim especially in the US.

Comment by soerenmind on interpreting GPT: the logit lens · 2020-08-31T22:56:19.199Z · LW · GW

Are these known facts? If not, I think there's a paper in here.

Comment by soerenmind on Will OpenAI's work unintentionally increase existential risks related to AI? · 2020-08-21T14:32:08.715Z · LW · GW
But what if they reach AGI during their speed up?

I agree, but I think it's unlikely OpenAI will be the first to build AGI.

(Except maybe if it turns out AGI isn't economically viable).

Comment by soerenmind on Will OpenAI's work unintentionally increase existential risks related to AI? · 2020-08-17T19:13:39.771Z · LW · GW

OpenAI's work speeds up progress, but in a way that's likely smooth progress later on. If you spend as much compute as possible now, you reduce potential surprises in the future.

Comment by soerenmind on Are we in an AI overhang? · 2020-08-02T22:02:09.752Z · LW · GW

Last year it only took Google Brain half a year to make a Transformer 8x larger than GPT-2 (the T5). And they concluded that model size is a key component of progress. So I won't be surprised if they release something with a trillion parameters this year.

Comment by soerenmind on Delegate a Forecast · 2020-07-30T19:49:55.138Z · LW · GW

I'm not sure if a probability counts as continuous?

If so, what's the probability that this paper would get into Nature (main journal) if submitted? Or even better, how much more likely is it to get into The Lancet Public Health vs Nature? I can give context by PM.

Comment by soerenmind on The Puzzling Linearity of COVID-19 · 2020-06-30T14:03:59.371Z · LW · GW

"Why are most COVID-19 infection curves linear?

Many countries have passed their first COVID-19 epidemic peak. Traditional epidemiological models describe this as a result of non-pharmaceutical interventions that pushed the growth rate below the recovery rate. In this new phase of the pandemic many countries show an almost linear growth of confirmed cases for extended time-periods. This new containment regime is hard to explain by traditional models where infection numbers either grow explosively until herd immunity is reached, or the epidemic is completely suppressed (zero new cases). Here we offer an explanation of this puzzling observation based on the structure of contact networks. We show that for any given transmission rate there exists a critical number of social contacts, Dc, below which linear growth and low infection prevalence must occur. Above Dc traditional epidemiological dynamics takes place, as e.g. in SIR-type models. When calibrating our corresponding model to empirical estimates of the transmission rate and the number of days being contagious, we find Dc ~ 7.2. Assuming realistic contact networks with a degree of about 5, and assuming that lockdown measures would reduce that to household-size (about 2.5), we reproduce actual infection curves with a remarkable precision, without fitting or fine-tuning of parameters. In particular we compare the US and Austria, as examples for one country that initially did not impose measures and one that responded with a severe lockdown early on. Our findings question the applicability of standard compartmental models to describe the COVID-19 containment phase. The probability to observe linear growth in these is practically zero."

Comment by soerenmind on The ground of optimization · 2020-06-22T23:42:14.710Z · LW · GW

Seconded that the academic style really helped, particularly discussing the problem and prior work early on. One classic introduction paragraph that I was missing is "what have prior works left unaddressed?".

Comment by soerenmind on FHI paper on COVID-19 government countermeasures · 2020-06-08T23:26:03.789Z · LW · GW

Think of it like one-sided vs two-sided. You can have a 95% CI that overlaps with zero, like [-2, 30], because 2.5% of the probability mass is on >30 and 2.5% on <-2, but still the probability of >0 effect can be >95%. This can also happen with Frequentist CIs.

A credible interval is the Bayesian analog to a confidence interval.

Comment by soerenmind on FHI paper on COVID-19 government countermeasures · 2020-06-08T23:14:13.747Z · LW · GW

We have no info on that, sorry. That's because we have a single feature which is switched on when most schools are closed. Universities were closed 75% of the time when that happened IIRC.

Comment by soerenmind on How to do remote co-working · 2020-05-09T12:46:02.445Z · LW · GW

Yes these are also great options. I used them in the past but somehow didn't keep it up.

Co-working with a friend is good option for people like myself who benefit from having someone who expects me to be there (and who I'm socially comfortable with).

Comment by soerenmind on The Puzzling Linearity of COVID-19 · 2020-04-25T02:53:53.433Z · LW · GW

Maybe this is not the type of explanation you're looking for but logistic curves (and other S-curves) look linear for surprisingly long.

Comment by soerenmind on Will COVID-19 survivors suffer lasting disability at a high rate? · 2020-04-21T23:27:28.613Z · LW · GW

The second study has a classic 'adjusting for observed confounders' methodology which comes with classic limitations such as that you don't observe all confounders. For example, they control for alcohol, drug abuse, but not smoking (!)

The first study also acknowledges possible confounding but I haven't checked it in detail.

Comment by soerenmind on Any rebuttals of Christiano and AI Impacts on takeoff speeds? · 2020-03-30T22:16:46.847Z · LW · GW

Looking forward to it :)

Comment by soerenmind on AGI in a vulnerable world · 2020-03-30T22:02:19.827Z · LW · GW

I'm using the colloquial meaning of 'marginal' = 'not large'.

Comment by soerenmind on AGI in a vulnerable world · 2020-03-27T13:40:32.672Z · LW · GW

Hmm, in my model most of the x-risk is gone if there is no incentive to deploy. But I expect actors will deploy systems because their system is aligned with a proxy. At least this leads to short-term gains. Maybe the crux is that you expect these actors to suffer a large private harm (death) and I expect a small private harm (for each system, a marginal distributed harm to all of society)?

Comment by soerenmind on AGI in a vulnerable world · 2020-03-27T13:08:31.714Z · LW · GW

I agree that coordination between mutually aligned AIs is plausible.

I think such coordination is less likely in our example because we can probably anticipate and avoid it for human-level AGI.

I also think there are strong commercial incentives to avoid building mutually aligned AGIs. You can't sell (access to) a system if there is no reason to believe the system will help your customer. Rather, I expect systems to be fine-tuned for each task, as in the current paradigm. (The systems may successfully resist fine-tuning once they become sufficiently advanced.)

I'll also add that two copies of the same system are not necessarily mutually aligned. See for example debate and other self-play algorithms.

Comment by soerenmind on AGI in a vulnerable world · 2020-03-26T18:10:58.245Z · LW · GW

This reasoning can break if deployment turns out to be very cheap (i.e. low marginal cost compared to fixed cost); then there will be lots of copies of the most impressive system. Then it matters a lot who uses the copies. Are they kept secret and only deployed for internal use? Or are they sold in some form? (E.g. the supplier sells access to its system so customers can fine-tune e.g. to do financial trading.)

Comment by soerenmind on AGI in a vulnerable world · 2020-03-26T18:05:18.591Z · LW · GW
And once there is at least one AGI running around, things will either get a lot worse or a lot better very quickly.

I don't expect the first AGI to have that much influence (assuming gradual progress). Here's an example of what fits my model: there is one giant-research-project AGI that costs $10b to deploy (and maybe $100b to R&D), 100 slightly worse pre-AGIs that cost perhaps $100m each to deploy, and 1m again slightly worse pre-AGIs that cost $10k to each copy. So at any point in time we have a lot of AI systems that, together, are more powerful than the small number of most impressive systems.

Comment by soerenmind on AGI in a vulnerable world · 2020-03-26T14:16:15.388Z · LW · GW

Small teams can also get cheap access to impressive results by buying it from large teams. The large team should set a low price if it has competitors who also sell to many customers.

Comment by soerenmind on What would be the consequences of commoditizing AI? · 2020-03-21T17:28:15.129Z · LW · GW

Would be pretty interested in your ideas about how to commoditize AI.

Comment by soerenmind on March Coronavirus Open Thread · 2020-03-18T02:42:52.688Z · LW · GW

Right now I expect they just used hospital admission forms. If I was self-reporting 5 pages of medical history while I'm critically ill I'd probably skip some fields. Interesting that they did find high rates of diabetes etc though.

Comment by soerenmind on March Coronavirus Open Thread · 2020-03-17T17:28:24.174Z · LW · GW

Data point: There were no asthma patients among a group of 140 hospitalized COVID-19 cases in Wuhan.

But nobody had other allergic diseases either. No hay fever? Seems curious.

Comment by soerenmind on March Coronavirus Open Thread · 2020-03-15T15:18:06.181Z · LW · GW

1/13 people have Asthma. How much worse off are we?

Comment by soerenmind on Credibility of the CDC on SARS-CoV-2 · 2020-03-09T14:28:39.654Z · LW · GW

Is this also wrong?

It may be possible that a person can get COVID-19 by touching a surface or object that has the virus on it and then touching their own mouth, nose, or possibly their eyes, but this is not thought to be the main way the virus spreads.

It's certainly contrary to most sources I've seen. Instead CDC claim it spreads "between people who are in close contact with one another (within about 6 feet)" (i. e. through droplets in the air).

Comment by soerenmind on What "Saving throws" does the world have against coronavirus? (And how plausible are they?) · 2020-03-05T10:19:06.851Z · LW · GW

1. We can slow down the spread through hand-washing, social distancing etc for long enough to develop a vaccine (or other measures) on time.

2. A vaccine is brought to market without the usual safety testing. Apparently we already have one that works in mice (from personal communication).

3. >10% get infected but the death rate has been greatly overestimated due to sampling bias. That one seems probable to me.

4. Antivirals

Comment by soerenmind on Draft: Models of Risks of Delivery Under Coronavirus · 2020-02-28T13:23:30.909Z · LW · GW

Coronaviruses may survive a lot longer, depending on specifics.

Comment by soerenmind on [AN #78] Formalizing power and instrumental convergence, and the end-of-year AI safety charity comparison · 2019-12-26T18:26:32.644Z · LW · GW

Sounds like we agree :)

Comment by soerenmind on [AN #78] Formalizing power and instrumental convergence, and the end-of-year AI safety charity comparison · 2019-12-26T14:09:24.866Z · LW · GW


On 2): Being overparameterized doesn't mean you fit all your training data. It just means that you could fit it with enough optimization. Perhaps the existence of some Savant people shows that the brain could memorize way more than it does.

On 3): The number of our synaptic weights is stupendous too - about 30000 for every second in our life.

On 4): You can underfit at the evolution level and still overparameterize at the individual level.

Overall you convinced me that underparameterization is less likely though. Especially on your definition of overparameterization, which is relevant for double descent.

Comment by soerenmind on [AN #78] Formalizing power and instrumental convergence, and the end-of-year AI safety charity comparison · 2019-12-26T01:31:27.108Z · LW · GW

Why do you think that humans are, and powerful AI systems will be, severely underparameterized?

Comment by soerenmind on [AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations · 2019-12-07T21:45:11.706Z · LW · GW

Potential paper from DM/Stanford for a future newsletter:

It addresses the problem that an RL agent will delude itself by finding loopholes in a learned reward function.

Comment by soerenmind on Strategic implications of AIs' ability to coordinate at low cost, for example by merging · 2019-12-07T15:47:56.420Z · LW · GW

Also interesting to see that all of these groups were able to coordinate to the disadvantage of less coordinates groups, but not able to reach peace among themselves.

One explanation might be that the more coordinated groups also have harder coordination problems to solve because their world is bigger and more complicated. Might be the same with AI?

Comment by soerenmind on Seeking Power is Often Robustly Instrumental in MDPs · 2019-12-07T15:35:22.290Z · LW · GW

If X is "number of paperclips" and Y is something arbitrary that nobody optimizes, such as the ratio of number of bicycles on the moon to flying horses, optimizing X should be equally likely to increase or decrease Y in expectation. Otherwise "1-Y" would go in the opposite direction which can't be true by symmetry. But if Y is something like "number of happy people", Y will probably decrease because the world is already set up to keep Y up and a misaligned agent could disturb that state.

Comment by soerenmind on Seeking Power is Often Robustly Instrumental in MDPs · 2019-12-06T12:13:23.133Z · LW · GW

I should've specified that the strong version is "Y decreases relative to a world where neither of X nor Y are being optimized". Am I right that this version is not true?

Comment by soerenmind on Seeking Power is Often Robustly Instrumental in MDPs · 2019-12-05T21:40:02.180Z · LW · GW

Thanks for writing this! It always felt like a blind spot to me that we only have Goodhart's law that says "if X is a proxy for Y and you optimize X, the correlation breaks" but we really mean a stronger version: "if you optimize X, Y will actively decrease". Your paper clarifies that what we actually mean is an intermediate version: "if you optimize X, it becomes a harder to optimize Y". My conclusion would be that the intermediate version is true but the strong version false then. Would you say that's an accurate summary?

Comment by soerenmind on [1911.08265] Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model | Arxiv · 2019-11-28T19:15:09.468Z · LW · GW

Posted a little reaction to this paper here.

Comment by soerenmind on [AN #75]: Solving Atari and Go with learned game models, and thoughts from a MIRI employee · 2019-11-28T18:56:18.011Z · LW · GW

My tentative view on MuZero:

  • Cool for board games and related tasks.
  • The Atari demo seems sketchy.
  • Not a big step towards making model-based RL work - instead, a step making it more like model-free RL.


  • A textbook benefit for model-based RL is that world models (i.e. prediction of observations) generalize to new reward functions and environments. They've removed this benefit.
  • The other textbook benefit of model-based RL is data efficiency. But on Atari, MuZero is just as inefficient as model-free RL. In fact, MuZero moves a lot closer to model-free methods by removing the need to predict observations. And it's roughly equally inefficient. Plus it trains with 40 TPUs per game where other algorithms use a single GPU and similar training time. What if they spent that extra compute to get more data?
  • In the low-data setting they outperform model-free methods. But they suspiciously didn't compare to any model-based method. They'd probably lose there because they'd need a world model for data efficiency.
  • MuZero only plans for K=5 steps ahead - far less than AlphaZero. Two takeaways: 1) This again looks more similar to model-free RL which has essentially K=1. 2) This makes me more optimistic that model-free RL can learn Go with just a moderate efficiency (and stability?) loss (Paul has speculated this. Also, the trained AlphaZero policy net is apparently still better than Lee Sedol without MCTS).
Comment by soerenmind on AlphaStar: Impressive for RL progress, not for AGI progress · 2019-11-25T20:54:44.808Z · LW · GW
the big questions are just how large a policy you would need to train using existing methods in order to be competitive with a human (my best guess would be a ~trillion to a ~quadrillion)

Curious where this estimate comes from?

Comment by soerenmind on AlphaStar: Impressive for RL progress, not for AGI progress · 2019-11-10T08:28:51.629Z · LW · GW

Why just a 10x speedup over model free RL? I would've expected much more.

Comment by soerenmind on [AN #72]: Alignment, robustness, methodology, and system building as research priorities for AI safety · 2019-11-07T16:01:19.229Z · LW · GW

Should I share the Alignment Research Overview in its current Google Doc form or is it about to be published somewhere more official?

Comment by soerenmind on Misconceptions about continuous takeoff · 2019-10-13T01:55:36.060Z · LW · GW

Yes. To the extent that the system in question is an agent, I'd roughly think of many copies of it as a single distributed agent.

Comment by soerenmind on Towards an empirical investigation of inner alignment · 2019-10-12T10:33:20.282Z · LW · GW

Hmmm my worry isn't so nuch that we have an unusual definition of inner alignment. It's more the opposite: that outsiders associate this line of research with quackery (which only gets worse if our definition is close to the standard one).

Comment by soerenmind on Misconceptions about continuous takeoff · 2019-10-12T10:25:46.635Z · LW · GW

Re whether ML is easy to deploy: most compute these days goes into deployment. And there are a lot of other deployment challenges that you don't have during training where you train a single model under lab conditions.

Comment by soerenmind on Misconceptions about continuous takeoff · 2019-10-12T10:24:52.181Z · LW · GW

Fair - I'd probably count "making lots of copies of a trained system" as a single system here.

Comment by soerenmind on Misconceptions about continuous takeoff · 2019-10-12T10:17:31.278Z · LW · GW

Yes - the part that I was doubting is that it provides evidence for relatively quick takeoff.

Comment by soerenmind on How important are model sizes to your timeline predictions? · 2019-10-11T01:25:43.349Z · LW · GW

For the record, two people who I consider authorities on this told me some version of "model sizes matter a lot".