Applications of Economic Models to Physiology? 2019-12-10T18:09:43.494Z · score: 27 (5 votes)
What is Abstraction? 2019-12-06T20:30:03.849Z · score: 20 (4 votes)
Paper-Reading for Gears 2019-12-04T21:02:56.316Z · score: 68 (20 votes)
Gears-Level Models are Capital Investments 2019-11-22T22:41:52.943Z · score: 81 (30 votes)
Wrinkles 2019-11-19T22:59:30.989Z · score: 61 (22 votes)
Evolution of Modularity 2019-11-14T06:49:04.112Z · score: 81 (28 votes)
Book Review: Design Principles of Biological Circuits 2019-11-05T06:49:58.329Z · score: 115 (46 votes)
Characterizing Real-World Agents as a Research Meta-Strategy 2019-10-08T15:32:27.896Z · score: 26 (9 votes)
What funding sources exist for technical AI safety research? 2019-10-01T15:30:08.149Z · score: 21 (8 votes)
Gears vs Behavior 2019-09-19T06:50:42.379Z · score: 49 (15 votes)
Theory of Ideal Agents, or of Existing Agents? 2019-09-13T17:38:27.187Z · score: 16 (8 votes)
How to Throw Away Information 2019-09-05T21:10:06.609Z · score: 20 (7 votes)
Probability as Minimal Map 2019-09-01T19:19:56.696Z · score: 40 (12 votes)
The Missing Math of Map-Making 2019-08-28T21:18:25.298Z · score: 33 (16 votes)
Don't Pull a Broken Chain 2019-08-28T01:21:37.622Z · score: 27 (13 votes)
Cartographic Processes 2019-08-27T20:02:45.263Z · score: 23 (8 votes)
Embedded Agency via Abstraction 2019-08-26T23:03:49.989Z · score: 33 (12 votes)
Time Travel, AI and Transparent Newcomb 2019-08-22T22:04:55.908Z · score: 12 (7 votes)
Embedded Naive Bayes 2019-08-22T21:40:05.972Z · score: 15 (6 votes)
Computational Model: Causal Diagrams with Symmetry 2019-08-22T17:54:11.274Z · score: 40 (15 votes)
Markets are Universal for Logical Induction 2019-08-22T06:44:56.532Z · score: 64 (26 votes)
Why Subagents? 2019-08-01T22:17:26.415Z · score: 108 (35 votes)
Compilers/PLs book recommendation? 2019-07-28T15:49:17.570Z · score: 10 (4 votes)
Results of LW Technical Background Survey 2019-07-26T17:33:01.999Z · score: 43 (15 votes)
Cross-Validation vs Bayesian Model Comparison 2019-07-21T18:14:34.207Z · score: 21 (7 votes)
Bayesian Model Testing Comparisons 2019-07-20T16:40:50.879Z · score: 13 (3 votes)
From Laplace to BIC 2019-07-19T16:52:58.087Z · score: 13 (3 votes)
Laplace Approximation 2019-07-18T15:23:28.140Z · score: 27 (8 votes)
Wolf's Dice II: What Asymmetry? 2019-07-17T15:22:55.674Z · score: 30 (8 votes)
Wolf's Dice 2019-07-16T19:50:03.106Z · score: 33 (12 votes)
Very Short Introduction to Bayesian Model Comparison 2019-07-16T19:48:40.400Z · score: 27 (8 votes)
How much background technical knowledge do LW readers have? 2019-07-11T17:38:37.839Z · score: 31 (10 votes)
Embedded Agency: Not Just an AI Problem 2019-06-27T00:35:31.857Z · score: 13 (8 votes)
Being the (Pareto) Best in the World 2019-06-24T18:36:45.929Z · score: 175 (83 votes)
ISO: Automated P-Hacking Detection 2019-06-16T21:15:52.837Z · score: 6 (1 votes)
Real-World Coordination Problems are Usually Information Problems 2019-06-13T18:21:55.586Z · score: 29 (12 votes)
The Fundamental Theorem of Asset Pricing: Missing Link of the Dutch Book Arguments 2019-06-01T20:34:06.924Z · score: 43 (13 votes)
When Observation Beats Experiment 2019-05-31T22:58:57.986Z · score: 15 (6 votes)
Constraints & Slackness Reasoning Exercises 2019-05-21T22:53:11.048Z · score: 47 (16 votes)
The Simple Solow Model of Software Engineering 2019-04-08T23:06:41.327Z · score: 26 (10 votes)
Declarative Mathematics 2019-03-21T19:05:08.688Z · score: 60 (25 votes)
Constructing Goodhart 2019-02-03T21:59:53.785Z · score: 31 (12 votes)
From Personal to Prison Gangs: Enforcing Prosocial Behavior 2019-01-24T18:07:33.262Z · score: 81 (28 votes)
The E-Coli Test for AI Alignment 2018-12-16T08:10:50.502Z · score: 60 (23 votes)
Competitive Markets as Distributed Backprop 2018-11-10T16:47:37.622Z · score: 44 (16 votes)
Two Kinds of Technology Change 2018-10-11T04:54:50.121Z · score: 61 (22 votes)
The Valley of Bad Theory 2018-10-06T03:06:03.532Z · score: 64 (30 votes)
Don't Get Distracted by the Boilerplate 2018-07-26T02:15:46.951Z · score: 44 (22 votes)
ISO: Name of Problem 2018-07-24T17:15:06.676Z · score: 32 (13 votes)
Letting Go III: Unilateral or GTFO 2018-07-10T06:26:34.411Z · score: 22 (7 votes)


Comment by johnswentworth on What determines the balance between intelligence signaling and virtue signaling? · 2019-12-11T04:17:46.753Z · score: 2 (1 votes) · LW · GW

It seems like a lot of examples of virtue signalling require sacrificing intelligence, but sacrificing virtue seems like a less common requirement to signal intelligence. So one possible model would be that, rather than a pareto frontier on which the two trade off symmetrically, intelligent decisions are an input which are destructively consumed to produce virtue signals - like trees are consumed to produce paper.

Comment by johnswentworth on What determines the balance between intelligence signaling and virtue signaling? · 2019-12-10T19:45:11.811Z · score: 4 (2 votes) · LW · GW

Could you expand a bit on why you expect a trade-off between intelligence/virtue signalling, as opposed to two independent axes? I can sort of see a case where intelligence is the "cost" part of "costly virtue signalling", and virtue is the "cost" part of "costly intelligence signalling", like the examples in toxoplasma of rage. On the other hand, looking at those examples of the dangers of runaway IQ signalling, they generally don't seem to trade-off against virtue.

Comment by johnswentworth on Multiple conditions must be met to gain causal effect · 2019-12-09T20:31:32.954Z · score: 2 (1 votes) · LW · GW

I think of "gears-level model" and "causal DAG" as usually synonymous. There are some arguable exceptions - e.g. some non-DAG markov models are arguably gears-level - but DAGs are the typical use case.

The obvious objection to this idea is "what about feedback loops?", and the answer is "it's still a causal DAG when you expand over time" - and that's exactly what gears-level understanding of a feedback loop requires. Same with undirected markov models: they typically arise from DAG models with some of the nodes unobserved; a gears-level model hypothesizes what those hidden factors are. The hospital example includes both of these: a feedback loop, with some nodes unobserved. But if you expand out the actual gears-level model, distinguishing between different people with different diseases at different times, then it all looks DAG-shaped; the observed data just doesn't include most of those nodes.

This generalizes: the physical world is always DAG-shaped, on a fundamental level. Everything else is an abstraction on top of that, and it can always be grounded in DAGs if needed.

Instead of stopping at: "It has to do with gears." keep going to get more specific, find subsets of things with gears: "gear AND oval-shape AND a sprocket is missing AND there is a cardan shaft AND ..." But if indeed only things with gears are affected do not expand with "gears AND needs oil" because that already follows from gears.

The advantage of using causal DAGs for our model, even when most of the nodes are not observed, is that it tells us which things need to be included in the AND-clauses and which do not. For instance, "gear AND oval-shaped" vs "gear AND needs oil" - the idea that the second can be ignored "because that already follows from gears" is a fact which derives from DAG structure. For a large model, there's an exponential number of logical clauses which we could form; a DAG gives formal rules for which clauses are relevant to our analysis.

Comment by johnswentworth on Understanding “Deep Double Descent” · 2019-12-09T18:48:39.584Z · score: 6 (3 votes) · LW · GW
the peak of a flat minimum is a slightly better approximation for the posterior predictive distribution over the entire hypothesis class. Sometimes I even wonder if something like this explains why Occam's Razor works...

That's exactly correct. You can prove it via the Laplace approximation: the "width" of the peak in each principal direction is the inverse of an eigenvalue of the Hessian, and each eigenvalue contributes to the marginal log likelihood . So, if a peak is twice as wide in one direction, its marginal log likelihood is higher by , or half a bit. For models in which the number of free parameters is large relative to the number of data points (i.e. the interesting part of the double-descent curve), this is the main term of interest in the marginal log likelihood.

In Jaynes' Logic of Science book, in the chapter on model comparison, I believe he directly claims that this is how/why Occam's Razor works.

That said, it's not clear that this particular proof would generalize properly to systems which perfectly fit the data. In that case, there's an entire surface on which is constant, so Laplace needs some tweaking.

Comment by johnswentworth on Computational Model: Causal Diagrams with Symmetry · 2019-12-09T17:50:18.558Z · score: 3 (2 votes) · LW · GW

My guess here is that there are some instrumentally convergent abstractions/algorithms which both a brain and a hypothetical AGI needs to use. But a brain will have implemented some of those as hacks on top of methods which evolved earlier, whereas an AI could implement those methods directly. So for instance, one could imagine the brain implementing simple causal reasoning as a hack on top of pre-existing temporal sequence capabilities. When designing an AI, it would probably make more sense to use causal DAGs as the fundamental, and then implement temporal sequences as abstract stick-dags which don't support many (if any) counterfactuals.

Possibly better example: tree search and logic. Humans seem to handle these mostly as hacks on top of pattern-matchers and trigger-action pairs, but for an AI it makes more sense to implement tree search as a fundamental.

Comment by johnswentworth on Computational Model: Causal Diagrams with Symmetry · 2019-12-08T19:20:31.980Z · score: 3 (2 votes) · LW · GW

I generally agree with this thinking, although I'll highlight that the brain and a hypothetical AI might not use the same primitives - they're on very different hardware, after all. Certainly the general strategy of "start with a few primitives, and see if they can represent all these other things" is the sort of strategy I'm after. I currently consider causal DAGs with symmetry the most promising primitive. It directly handles causality and recursion, and alongside a suitable theory of abstraction, I expect it will allow us to represent things like spatial/temporal relations, hierarchies, analogies, composition, and many others, all in a unified framework.

Comment by johnswentworth on What do the Charter Cities Institute likely mean when they refer to long term problems with the use of eminent domain? · 2019-12-08T19:01:38.117Z · score: 2 (1 votes) · LW · GW

My guess is that they're giving a nod to Kelo v. New London. It was a big supreme court case at the time (2005), when the city of New London tried to use eminent domain to buy up a big chunk of land and then sell it to a private developer. I don't know the details of the case well, but my understanding is that it was a pretty typical case of "urban renewal": government uses eminent domain to buy out a neighborhood full of poor people, kick the poor folks out, and then sells the land over to a private developer to build "higher-class" real estate.

A lot of people are politically opposed to this (for obvious reasons), so presumably the Charter Cities Institute decided to avoid getting entangled in that kind of thing.

Comment by johnswentworth on What is Abstraction? · 2019-12-07T19:38:32.870Z · score: 2 (1 votes) · LW · GW
How you do lossy compression depends on what you want.

I think this is technically true, but less important than it seems at first glance. Natural abstractions are a thing, which means there's instrumental convergence in abstractions - some compressed information is relevant to a far wider variety of objectives than other compressed information. Representing DNA sequences as strings of four different symbols is a natural abstraction, and it's useful for a very wide variety of goals; MD5 hashes of those strings are useful only for a relatively narrow set of goals.

Somewhat more formally... any given territory has some Kolmogorov complexity, a maximally-compressed lossless map. That's a property of the territory alone, independent of any goal. But it's still relevant to goal-specific lossy compression - it will very often be useful for lossy models to re-use the compression methods relevant to lossless compression.

For instance, maybe we have an ASCII text file which contains only alphanumeric and punctuation characters. We can losslessly compress that file using e.g. Huffman coding, which uses fewer bits for the characters which appear more often. Now we decide to move on to lossy encoding - but we can still use the compressed character representation found by Huffman, assuming the lossy method doesn't change the distribution of characters too much.

Comment by johnswentworth on The Epsilon Fallacy · 2019-12-07T18:11:40.132Z · score: 4 (5 votes) · LW · GW
All of those appear to be wrong.

Let's go through them one by one:

  • "it ain’t a bunch of small things adding together" -> Still 100% true. Eyeballing the EIA's data, wind + natgas account for ~80% of the decrease in carbon emissions. That's 2 things added together.
  • "Practically all of the reduction in US carbon emissions over the past 10 years has come from that shift" -> False, based on EIA 2005-2017 data. More than half of the reduction came from the natgas shift (majority, not just plurality), but not practically all.
  • "all these well-meaning, hard-working people were basically useless" -> False for the wind people.
  • "PV has been an active research field for thousands of academics for several decades. They’ve had barely any effect on carbon emissions to date" -> Still true. Eyeballing the numbers, solar is maybe 10% of the reduction to date. That's pretty small to start with, and on top of that, little of the academic research has actually translated to the market, much less addressed the major bottlenecks of solar PV (e.g. installation).
  • "one wedge will end up a lot more effective than all others combined. Carbon emission reductions will not come from a little bit of natgas, a little bit of PV, a little bit of many other things" -> Originally intended as a prediction further into the future, and I still expect this to be the case. That said, as of today, "one wedge will end up a lot more effective" looks false, but "Carbon emission reductions will not come from a little bit of natgas, a little bit of PV, a little bit of many other things" looks true.

... So a couple of them are wrong, though none without at least some kernel of truth in there. And a couple of them are still completely true.

And it needed to be that, rather than something weaker-and-truer like "Substantially the biggest single element in the carbon reduction has been the natgas transition", because the more general thesis is that here, and in many other places, one approach so dominates the others that working on anything else is a waste of time.

No, as the next section makes clear, it does not need to be one approach dominating everything else; that just makes for memorable examples. 80/20 is the rule, and 20% of causes can still be more than one cause. 80/20 is still plenty strong for working on the other 80% of causes to be a waste of time.

Comment by johnswentworth on Recent Progress in the Theory of Neural Networks · 2019-12-06T22:29:41.271Z · score: 3 (2 votes) · LW · GW
a naive estimate of the expected test loss would be the average training loss using samples of the posterior.

That's exactly the problem - that is generally not a good estimate of the expected test loss. It isn't even an unbiased estimate. It's just completely wrong.

The right way to do this is to just calculate the expected test loss.

Comment by johnswentworth on Recent Progress in the Theory of Neural Networks · 2019-12-06T20:21:22.351Z · score: 2 (1 votes) · LW · GW

That piece you link uses a definition of overfitting which doesn't really make sense from a Bayesian perspective. "The difference between the performance on the training set and the performance on the test set" is not what we care about; we care about the difference between the expected performance on the test set and the actual performance on the test set.

Indeed, it's entirely possible that the training data and the test data are of qualitatively different types, drawn from entirely different distributions. A Bayesian method with a well-informed model can often work well in such circumstances. In that case, the performance on the training and test sets aren't even comparable-in-principle.

For instance, we could have some experiment trying to measure the gravitational constant, and use a Bayesian model to estimate the constant from whatever data we've collected. Our "test data" is then the "true" value of G, as measured by better experiments than ours. Here, we can compare our expected performance to actual performance, but there's no notion of performance comparison between train and test.

Comment by johnswentworth on The Epsilon Fallacy · 2019-12-06T18:09:24.827Z · score: 5 (3 votes) · LW · GW

Those numbers say:

  • The counterfactual decrease in emissions from low demand growth was larger than all other factors combined.
  • Just looking at actual decreases (not counterfactual), the decrease in emissions from switching coal to natgas was larger than the decrease in emissions from everything else combined (even with subsidies on everything else and no subsidies on natgas).

I agree that, based on these numbers, the largest factor is not "a lot more effective than all others combined" as predicted in the post. But I wouldn't say it "demolishes the thesis" - however you slice it, the largest factor is still larger than everything else combined (and for actual decreases, that largest factor is still natgas).

Have there been substantial reductions in emissions from solar and wind? Yes. But remember the key point of the post: we need to consider opportunity costs. Would emissions be lower today if all the subsidies supporting solar/wind had gone to natgas instead? If all those people campaigning for solar/wind subsidies had instead campaigned for natgas subsidies? And that wouldn't have taken magical predictive powers ten years ago - at that time, the shift was already beginning.

Comment by johnswentworth on Understanding “Deep Double Descent” · 2019-12-06T00:39:37.060Z · score: 10 (5 votes) · LW · GW

Off the top of your head, do you know anything about/have any hypotheses about how double descent interacts with the gaussian processes interpretation of deep nets? It seems like the sort of theory which could potentially quantify the inductive bias of SGD.

Comment by johnswentworth on Multiple conditions must be met to gain causal effect · 2019-12-05T19:54:34.795Z · score: 5 (3 votes) · LW · GW

Two relevant things.

First, the Epsilon Fallacy: the idea that effects are the result of many tiny causes adding up. In practice, 80/20 is a thing, and most things most of the time do have a small number of "main" root causes which account for most of the effect. So it's not necessarily wrong to look for "exactly one cause" - as in e.g. optimizing runtime of a program, there's often one cause which accounts for most of the effect. In the "logical-and" case you mention, I'd usually expect to see either

  • most of the things in the and-clause don't actually vary much in the population (i.e. most of them are almost always true or almost always false), and just one or two account for most of the variance, OR
  • a bunch of the things in the and-clause are highly correlated due to some underlying cause.

Of course there are exceptions to this, in particular for traits under heavy selection pressure - if we always hammer down the nail that sticks out, then all the nails end up at around the same height. If we repeatedly address bottlenecks/limiting factors in a system, then all limiting factors will end up roughly equally limiting, and 80/20 doesn't happen.

Second: the right "language" in which to think about this sort of thing is not flat boolean logic (i.e. "effect = (A or B) and C and D") but rather causal diagrams. The sort of medical studies you mention - i.e. "saliva is a risk factor for cancer but only if taken orally in small doses over a long period of time" - are indeed pretty dumb, but the fix is not to look for a giant and-clause of conditions which result in the effect. The fix is to build a gears-level model of the system, figure out the whole internal cause-and-effect graph.

Comment by johnswentworth on Recent Progress in the Theory of Neural Networks · 2019-12-05T01:37:27.205Z · score: 7 (5 votes) · LW · GW

Thank you for writing this. I only heard about the gaussian process results a month or so ago, and hadn't gotten around to slogging through the papers yet. Reading this helped situate things and give a broad-strokes overview.

Comment by johnswentworth on What are the requirements for being "citable?" · 2019-11-29T06:16:06.600Z · score: 2 (1 votes) · LW · GW

Could auto-export posts to pdf, and just have a pile of those somewhere?

Comment by johnswentworth on What are the requirements for being "citable?" · 2019-11-28T21:37:33.864Z · score: 7 (3 votes) · LW · GW

Google scholar inclusion criteria:

Comment by johnswentworth on Unknown Knowns · 2019-11-28T17:15:52.708Z · score: 4 (2 votes) · LW · GW

Second Bena's nomination

Comment by johnswentworth on Could someone please start a bright home lighting company? · 2019-11-27T03:32:47.960Z · score: 3 (2 votes) · LW · GW

That is SO COOL, thanks for the link.

Comment by johnswentworth on Could someone please start a bright home lighting company? · 2019-11-26T22:46:00.756Z · score: 6 (4 votes) · LW · GW

How would very bright indoor lights handle the problem?

With the sun, it's already very far away (relative to the size of a room, or even the size of the whole earth), so light intensity doesn't vary much as we move around - stays basically constant. But in a house, you'll sometimes be 10X further from the light source than other times - so the light will sometimes be 100X brighter than other times. (I don't know if other people have this problem, but those kinds of sharp light gradients tend to give me headaches - so mostly I either use dim lighting inside or go outside.)

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-25T23:58:36.569Z · score: 3 (2 votes) · LW · GW

I definitely agree that combining models - especially by averaging them in some way - is very blackboxy. The individual models being averaged can each be gears-level models, though.

Circling back to my main definition: it's the top-level division which makes a model gearsy/non-gearsy. If the top-level is averaging a bunch of stuff, then that's a black-box model, even if it's using some gears-level models internally. If the top-level division contains gears, then that's a gears-level model, even if the gears themselves are black boxes. (Alternatively, we could say that "gears" vs "black box" is a characterization of each level/component of the model, rather than a characterization of the model as a whole.)

I'm curious if you agree with the conception of gears being capital investments towards specific expertise, and black boxes being capital investments towards generalizable advantage.

I don't think black boxes are capital investments towards generalizable advantage. Black box methods are generalizable, in the sense that they work on basically any system. But individual black-box models are not generalizable - a black-box method needs to build a new model whenever the system changes. That's why black-box methods don't involve an investment - when a black-box method encounters a new problem/system, it starts from scratch. Something like "learn how to do A/B tests" is an investment in learning how to apply a black-box method, but the A/B tests themselves are not an investment (or to the extent they are, they're an investment which depreciates very quickly) - they won't pay off over a very long time horizon.

So learning how to apply a black-box method, in general, is a capital investment towards generalizable advantage. But actually using a black-box method - i.e. producing a black-box model - is usually not a capital investment.

(BTW, learning how to produce gears-level models is a capital investment which makes it cheaper to produce future capital investments.)

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-25T23:41:17.187Z · score: 2 (1 votes) · LW · GW

Ok, so for the primes examples, I'd say that the gears-level model is using prior information in the form of the universal prior. I'd think of the universal prior as a black-box method for learning gears-level models; it's a magical thing which lets us cross the bridge from one to the other (sometimes). In general, "black-box methods for finding gears-level models" is one way I'd characterize the core problems of AGI.

One "box" in the primes example is just the integers from 0-20; the gears-level model gives us insight into what happens outside that range, while the black-box model does not.

Similarly for the being from another dimension: they're presumably using a universal prior. And they may not bother thinking outside the box - they may only want to make accurate predictions about whatever questions are in front of them - but F = ma is a model which definitely can be used for all sorts of things in our universe, not just whatever specific physical outcomes the being wants to predict.

But I still don't think I've properly explained what I mean by "outside the box".

For the primes problem, a better example of "outside the box" would be suddenly introducing some other kind of "number", like integer matrices or quadratic integers or something. A divisibility-based model would generalize (assuming you kept using a divisibility-based criterion) - not in the sense that the same program would work, but in the sense that we don't need to restart from scratch when figuring out the pattern. The black-box model, on the other hand, would need to start more-or-less from scratch.

For the being from another dimension, a good example of "outside the box" would be a sudden change in fundamental constants - not so drastic as to break all the approximations, but enough that e.g. energies of chemical reactions all change. In that case, F = ma would probably still hold despite the distribution shift.

So I guess the best summary of what I mean by "outside the box" is something like "counterfactual changes which don't correspond to anything in the design space/data".

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-25T23:08:17.862Z · score: 3 (2 votes) · LW · GW

I would characterize game theory, as applied to human interactions, as a gearsy model. It's not a very high-fidelity model - a good analogy would be "spherical cow in a vacuum" or "sled on a perfectly smooth frictionless incline" in physics. And the components in game-theory models - the agents - are themselves black boxes which are really resistant to being broken open. But a model with multiple agents in it is not a single monolithic black box, therefore it's a gears-level model.

This is similar to my response to Kaj above: there's a qualitative change in going from a model which treats the entire system as a single monolithic black box, to a model which contains any internal structure at all. As soon as we have any internal structure, the model will no longer apply to any random system in the wild - it will only apply to systems which share the relevant gears. In the case of game theory, our game-theoretic models are only relevant to systems with interacting agenty things; it won't help us to e.g. design a calculator or find a short path through a maze. Those agenty things are the gears.

As in any gears-level model, the gears themselves can be black boxes, and that's definitely the case for agents in game theory.

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-25T01:15:41.301Z · score: 3 (2 votes) · LW · GW

Gears-level models which don't use prior knowledge or offer outside-the-box insights.

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-24T18:21:45.728Z · score: 5 (4 votes) · LW · GW

I agree with the principle here, but I think the two are competitive in practice far more often than one would naively expect. For instance, people do use black-box optimizers for designing arithmetic logic units (ALUs), the core component of a calculator. Indeed, circuit optimizers are a core tool for digital hardware design these days (see e.g. espresso for a relatively simple one) - and of course there's a whole academic subfield devoted to the topic.

Competitiveness of the two methods comes from hybrid approaches. If evolution can solve a problem, then we can study the evolved solution to come up with a competitive gears-level model. If a gears-level approach can solve a problem, then we can initialize an iterative optimizer with the gears-level solution and let it run (which is what circuit designers do).

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-24T18:10:45.688Z · score: 3 (2 votes) · LW · GW

I think you're pointing to a true and useful thing, but "sliding scale" isn't quite the right way to characterize it. Rather, I'd say that we're always operating at some level(s) of abstraction, and there's always a lowest abstraction level in our model - a ground-level abstraction, in which the pieces are atomic. For a black-box method, the ground-level abstraction just has the one monolithic black box in it.

A gearsy method has more than just one object in its ground-level abstraction. There's some freedom in how deep the abstraction goes - we could say a gear is atomic, or we could go all the way down to atoms - and the objects at the bottom will always be treated as black boxes. But I'd say it's not quite right to think of the model as "partially black-box" just because the bottom-level objects are atomic; it's usually the top-level breakdown that matters. E.g., in the maze example from the post, the top and bottom halves of the maze are still atomic black boxes, but our gearsy insight is still 100% gearsy - it is an insight which will not ever apply to some random black box in the wild.

Gears/no gears is a binary distinction; there's a big qualitative jump between a black-box method which uses no information about internal system structure, and a gearsy model which uses any information about internal structure (even just very simple information). We can add more gears, reduce the black-box components in a gears level model. But as soon as we make the very first jump from one monolithic black box to two atomic gears, we've gone from a black-box method which applies to any random system, to a gears-level investment which will pay out on our particular system and systems related to it.

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-24T17:53:18.835Z · score: 3 (2 votes) · LW · GW

Can you give 2-3 examples?

Comment by johnswentworth on Gears-Level Models are Capital Investments · 2019-11-24T06:15:36.075Z · score: 2 (1 votes) · LW · GW

I'll clarify what I mean a bit.

We have some black box with a bunch of dials on the outside. A black-box optimizer is one which fiddles the dials to make the box perform well, without any knowledge of the internals of the box. If the dials themselves are sufficiently expressive, it may find creative and interesting solutions - as in the genetic algorithm example you mention. But it cannot find dials which we had not noticed before. A black-box method can explore previously-unexplored areas of the design space, but it cannot expand our concept of what the "design space" is beyond the space we've told it to search.

A black-box algorithm doesn't think outside the box it's given.

Comment by johnswentworth on Market Rate Food Is Luxury Food · 2019-11-23T17:58:34.237Z · score: 0 (2 votes) · LW · GW

How does the price cap suggestion avoid the usual econ-101 rule that a price cap either does nothing or causes a shortage?

Comment by johnswentworth on Wrinkles · 2019-11-23T17:02:57.329Z · score: 2 (1 votes) · LW · GW

One of the main questions I haven't found a satisfying answer to yet is whether denervation/renervation is causal for sarcopenia. Apparently a huge amount of resources went into imaging neuromuscular junctions for a while - the physiological reviews article I linked spends half the article on the topic - but that seems to be driven by historical accident more than anything. After wading through a ton of it I still haven't seen any decisive evidence on whether denervation is the main cause of muscle atrophy, or muscle atrophy causes nerve atrophy. It sounds like either is sufficient to cause the other experimentally, but it's not clear which actually comes first in aging. (And of course research is made difficult by authors sometimes making statements about causality which their data/experimental procedure doesn't actually establish.)

Comment by johnswentworth on Wrinkles · 2019-11-22T17:48:23.056Z · score: 5 (3 votes) · LW · GW

The best source I've seen on the topic is this Physiological Reviews article (I've read other sources as well, but didn't keep around links for most of them).

Reversibility is specifically addressed - age-related muscle loss (aka sarcopenia) is not really reversible. There are things people can do at any age to add muscle (e.g. exercise), but muscle is lost if exercise/diet/etc is held constant. Masters athletes are a visible example of this.

Also, it's not just skeletal muscle. For example, the pupil muscle squeezes the lens of the eye to adjust focus. In old age, that muscle loses mass, resulting in slower focusing speed. (Source: Physiological Basis of Aging and Geriatrics; the chapter on the eye is one of the best in the book). And of course there's loss of muscle mass in various sphincters, resulting in e.g. incontinence and other digestive problems. None of those muscles are suffering from lack of use.

Comment by johnswentworth on Historical forecasting: Are there ways I can get lots of data, but only up to a certain date? · 2019-11-22T02:31:10.437Z · score: 13 (5 votes) · LW · GW

Ray Dalio mentions in his Big Debt Crises book that he did this by reading through newspaper archives. Obviously this has some shortcomings - not a lot of consistent quantitative data (other than asset prices), comes with a lot of interpretation from the writers, the writers are journalists, only works for relatively recent history, etc. But it seems like a great way to learn what sounded reasonable to laymen at the time.

Comment by johnswentworth on Embedded Agents · 2019-11-21T18:58:09.791Z · score: 12 (3 votes) · LW · GW

This post (and the rest of the sequence) was the first time I had ever read something about AI alignment and thought that it was actually asking the right questions. It is not about a sub-problem, it is not about marginal improvements. Its goal is a gears-level understanding of agents, and it directly explains why that's hard. It's a list of everything which needs to be figured out in order to remove all the black boxes and Cartesian boundaries, and understand agents as well as we understand refrigerators.

Comment by johnswentworth on Babble · 2019-11-21T18:42:35.391Z · score: 7 (3 votes) · LW · GW

This post opens with the claim that most human thinking amounts to babble-and-prune. My reaction was (1) that's basically right, (2) babble-and-prune is a pretty lame algorithm, (3) it is possible for humans to do better, even though we usually don't. More than anything else, "Babble" convinced me to pay attention to my own reasoning algorithms and strive to do better. I wrote a couple posts which are basically "how to think better than babble" - Mazes and Crayon and Slackness and Constraints Exercises - and will probably write more on the topic in the future.

"Babble" is the baseline for all that. It's a key background concept; the reason the techniques in "Mazes and Crayon" or "Slackness and Constraints" are important is because without them, we have to fall back on babble-and-prune. That's the mark to beat.

Comment by johnswentworth on Wrinkles · 2019-11-21T00:54:55.802Z · score: 2 (1 votes) · LW · GW

I would not predict that in general - loss of muscle is a major hallmark of aging, and I would expect that to have some impact. But I doubt that any age-related difference in muscle activity matters much for wrinkles.

Also, this isn't just about facial wrinkles. The same model applies to wrinkles on the hands, elbows, etc.

Comment by johnswentworth on Wrinkles · 2019-11-20T16:52:53.291Z · score: 5 (3 votes) · LW · GW

The wrinkle model requires compression - there has to be some squeezing from the sides. Botox kills the facial muscles which do the squeezing. (I haven't actually researched this, just an educated guess, but one reason gears-level models are important is that they let us make pretty darn good educated guesses.)

Comment by johnswentworth on Hard to find factors messing up experiments: Examples? · 2019-11-16T18:28:39.357Z · score: 8 (5 votes) · LW · GW

A few stories from my undergrad (some firsthand, some secondhand):

  • One of the standard experiments in the undergrad intro physics lab involved a pendulum. One professor would loosen the fasteners on the top before the lab began. When students got confusing results, he would ask them what could be the cause, and they would stand there scratching their heads as the bar to which the pendulum was attached went clanging back and forth.
  • During one professor's time at grad school, a voltage probe was showing bizarre oscillations of tens of volts. After some investigation, it turned out that the "ground" ran down through a pipe into the local groundwater, and a long wire elsewhere in the system ran over the ground for a ways, and the whole thing formed a giant antenna. They were picking up radio waves.
  • No undergrad lab involving an oscilloscope would be complete without someone noticing mysterious oscillations in their circuit. Usually these turn out to be oscillations at 60 Hz - anywhere urban or indoors is flooded with 60 Hz waves from the power lines & outlets.
  • While calibrating a pH sensor, I noticed that the supposedly-deionized water was off from where it should be. Our lab manager thought it was the local air quality (we were in LA). Apparently there's a fair bit of variance in the pH of DI water exposed to air depending on where you are.

Also, as a source of more of these stories, you might check out the classic experiments measuring various physical constants - gravitational constant, electron mass & charge, etc. Usually they involve a whole series of tricks to control for various error sources.

Comment by johnswentworth on Evolution of Modularity · 2019-11-14T18:13:42.363Z · score: 7 (3 votes) · LW · GW

Yeah, Alon briefly mentions that line of study as well, although he doesn't discuss it much. Personally, I think connection costs are less likely to be the main driver of biological modularity in general, for two main reasons:

  • If connection costs were a taut constraint, then we'd expect to see connection costs taking up a large fraction of the organism's resources. I don't think that's true for most organisms most of the time (though the human brain is arguably an exception). And qualitatively, if we look at the cost of e.g. signalling molecules in a bacteria, they're just not that expensive - mainly because they don't need very high copy number.
  • Connection costs are not a robust way to produce modularity - we need a delicate balance between cost and benefit, so that neither overwhelms the other. Given how universal modularity is in biology, across so many levels of organization and basically all known organisms, it seems like a less delicate mechanism is needed to explain it.

I do find it plausible that connection cost is a major driver in some specific systems - in particular, the sanity checks pass for the human brain. But I doubt that it's the main cause of modularity across so many different systems in biology.

Comment by johnswentworth on [Team Update] Why we spent Q3 optimizing for karma · 2019-11-11T17:54:40.458Z · score: 5 (3 votes) · LW · GW

This line of thinking makes a major assumption which has, in my experience, been completely wrong: the assumption that a "big thing" in terms of impact is also a "big thing" in terms of engineering effort. I have seen many changes which are only small tweaks from an engineering standpoint, but produce 25% or 50% increase in a metric all on their own - things like making a button bigger, clarifying/shortening some text, changing something from red to green, etc. Design matters, it's relatively easy to change, but we don't know how to change it usefully without tests.

Comment by johnswentworth on [Team Update] Why we spent Q3 optimizing for karma · 2019-11-09T03:16:13.467Z · score: 14 (4 votes) · LW · GW

I actually agree with the overall judgement there - optimizing simple metrics really hard is mainly useful for things like e.g. landing pages, where the goals really are pretty simple and there's not too much danger of Goodharting. Lesswrong mostly isn't like that, and most of the value in micro-optimizing would be in the knowledge gained, rather than the concrete result of increasing a metric. I do think there's a lot of knowledge there to gain, and I think our design-level decisions are currently far away from the pareto frontier in ways that won't be obvious until the micro-optimization loop starts up.

I will also say that the majority of people I've worked with have dramatically underestimated the magnitude of impact this sort of thing has until they saw it happen first-hand, for whatever that's worth. (I first saw it in action at a company which achieved supercritical virality for a short time, and A/B-test-driven micro-optimization was the main tool responsible for that.) If this were a start-up, and we needed strong new user and engagement metrics to get our next round of funding, then I'd say it should be the highest priority. But this isn't a startup, and I totally agree that A/B tests won't solve the most crucial uncertainties.

Comment by johnswentworth on LW Team Updates - November 2019 (Subscriptions & More) · 2019-11-09T00:53:12.093Z · score: 11 (5 votes) · LW · GW

I love the pingbacks. I enabled them literally five minutes ago, opened a random post to check that it was working, and immediately saw an interesting post from early this year that I'd missed.

Comment by johnswentworth on [Team Update] Why we spent Q3 optimizing for karma · 2019-11-08T05:50:52.504Z · score: 15 (4 votes) · LW · GW

Responding to both of you with one comment again: I sort of alluded to it in the A/B testing comment, but it's less about any particular feature that's missing and more about the general mindset. If you want to drive up metrics fast, then the magic formula is a tight iteration loop: testing large numbers of small changes to figure out which little things have disproportionate impact. Any not-yet-optimized UI is going to have lots of little trivial inconveniences and micro-confusions; identifying and fixing those can move the needle a lot with relatively little effort. Think about how facebook or amazon A/B tests every single button, every item in every sidebar, on their main pages. That sort of thing is very easy, once a testing framework is in place, and it has high yields.

As far as bigger projects go... until we know what the key factors are which drive engagement on LW, we really don't have the tools to prioritize big projects. For purposes of driving up metrics, the biggest project right now is "figure out which things matter that we didn't realize matter". A/B tests are one of the main tools for that - looking at which little tweaks have big impact will give hints toward the bigger issues. Recorded user sessions (a la FullStory) are another really helpful tool. Interviews and talking to authors can be a substitute for that, although users usually don't understand their own wants/needs very well. Analytics in general is obviously useful, although it's tough to know which questions to ask without watching user sessions directly.

Comment by johnswentworth on [Team Update] Why we spent Q3 optimizing for karma · 2019-11-08T05:35:33.061Z · score: 16 (7 votes) · LW · GW

Responding to both of you here: A/B tests are a mental habit which takes time to acquire. Right now, you guys are thinking in terms of big meaty projects, which aren't the sort of thing A/B tests are for. I wouldn't typically make a single A/B test for a big, complicated feature like shortform - I'd run lots of little A/B tests for different parts of it, like details of how it's accessed and how it's visible. It's the little things: size/location/wording of buttons, sorting on the homepage, tweaking affordances, that sort of thing. Think nudges, not huge features. Those are the kinds of things which let you really drive up the metrics with relatively little effort, once you have the tests in place. Usually, it turns out that one or two seemingly-innocuous details are actually surprisingly important.

It's true that you don't necessarily need A/B tests to attribute growth to particular changes, especially if the changes are big things or one-off events, but that has some serious drawbacks even aside from the statistical uncertainty. Without A/B tests, we can't distinguish between the effects of multiple changes made in the same time window, especially small changes, which means we can't run lots of small tests. More fundamentally, an A/B test isn't just about attribution, it's about having a control group - with all the benefits that a control group brings, like fine-grained analysis of changes in behavior between test buckets.

Comment by johnswentworth on [Team Update] Why we spent Q3 optimizing for karma · 2019-11-08T00:58:02.990Z · score: 25 (16 votes) · LW · GW

I'm gonna heckle a bit from the peanut gallery...

First, trying to optimize a metric without an A/B testing framework in place is kinda pointless. Maybe the growth achieved in Q3 was due to the changes made, but looking at the charts, it looks like a pretty typical quarter. It's entirely plausible that growth would have been basically the same even without all this stuff. How much extra karma was actually generated due to removing login expiry? That's exactly the sort of thing an A/B test is great for, and without A/B tests, the best we can do is guess in the dark.

Second (and I apologize if I'm wrong here), that list of projects does not sound like the sort of thing someone would come up with if they sat down for an hour with a blank slate and asked "how can the LW team get more karma generated?" They sound like the sort of projects which were probably on the docket anyway, and then you guys just checked afterward to see if they raised karma (except maybe some of the one-shot projects, but those won't help long-term anyway).

Third, I do not think 7% was a mistaken target. I think Paul Graham was right on this one: only hitting 2% is a sign that you have not yet figured out what you're doing. Trying to optimize a metric without even having a test framework in place adds a lot of evidence to that story - certainly in my own start-up experience, we never had any idea what we were doing until well after the test framework was in place (at any of the companies I've worked at). Analytics more generally were also always crucial for figuring out where the low-hanging fruit was and which projects to prioritize, and it sounds like you guys are currently still flying blind in that department.

So, maybe re-try targeting one metric for a full quarter after the groundwork is in place for it to work?

Comment by johnswentworth on Building Intuitions On Non-Empirical Arguments In Science · 2019-11-07T21:06:13.212Z · score: 3 (2 votes) · LW · GW

Ah, I see. Thanks.

Comment by johnswentworth on Building Intuitions On Non-Empirical Arguments In Science · 2019-11-07T18:38:34.887Z · score: 6 (3 votes) · LW · GW
Again, Bayesians would start with a very low prior for Atlantis, and assess the evidence as very low, and end up with a probability distribution something like Khafre 80%, Khufu 19.999999%, Atlantis 0.000001%

This isn't quite how a pure Bayesian analysis would work. We should end up with higher probability for Khafre/Khufu, even if the prior starts with comparable weight on all three.

We want to calculate the probability that the sphinx was built by Atlanteans, given the evidence: P[atlantis | evidence]. By Bayes' rule, that's proportional to P[evidence | atlantis] times the prior P[atlantis]. Let's just go ahead and fix the prior at 1/3 for the sake of exposition, so that the heavy lifting will be done by P[evidence | atlantis].

The key piece: what does P[evidence | atlantis] mean? If the new-agers say "ah, the Atlantis theory predicts all of this evidence perfectly", does that mean that P[evidence | atlantis] is very high? No, because we expect that the new-agers would have said that regardless of what evidence was found. A theory cannot assign high probability to all possible evidence, because the theory's evidence-distribution must sum to one. To properly compute P[evidence | atlantis], we have to step back and ask "before seeing this evidence, what probability would I assign it, assuming the sphinx was actually built by Atlanteans?"

What matters most for computing P[evidence | atlantis] is that the Atlantis theory puts nonzero probability on all sorts of unusual hypothetical evidence-scenarios. For instance, if somebody ran an ultrasound on the sphinx, and found that it contained pure aluminum, or a compact nuclear reactor, or a cavity containing tablets with linear A script on them, or anything else that Egyptians would definitely not have put in there... the Atlantis theory would put nonzero probability on all those crazy possibilities. But there's a lot of crazy possibilities, and allocating probability to all of them means that there can't be very much left for the boring possibilities - remember, it all has to add up to one, so we're on a limited probability budget here. On the other hand, Khafre/Khufu both assign basically-zero probability to all the crazy possibilities, which leaves basically their entire probability budget on the boring stuff.

So when the evidence actually ends up being pretty boring, P[evidence | atlantis] has to be a lot lower than P[evidence | khafre] or P[evidence | khufu].

Comment by johnswentworth on Building Intuitions On Non-Empirical Arguments In Science · 2019-11-07T18:05:06.761Z · score: 5 (3 votes) · LW · GW

Regarding the particles with mass-ratios following sphere integrals: this isn't quite an analogy for many worlds. Particle masses following that sort of pattern is empirically testable: as particle mass measurements gain precision over time, the theory would predict that their mass ratios continue to match the pattern. Many worlds is a different beast: it is mathematically equivalent to other interpretations of quantum mechanics. The different interpretations provably make the same predictions for everything, always. They use the same exact math. The only difference is interpretation.

Comment by johnswentworth on Book Review: Design Principles of Biological Circuits · 2019-11-06T18:03:05.731Z · score: 10 (7 votes) · LW · GW

The second claim was actually my main goal with this post. It is a claim I have heard honest arguments against, and even argued against myself, back in the day. A simple but not-particularly-useful version of the argument would be something like "the shortest program which describes biological behavior may be very long", i.e. high Kolmogorov complexity. If that program were too long to fit in a human brain, then it would be impossible for humans to "understand" the system, in some sense. We could fit the program in a long book, maybe, but since the program itself would be incompressible it would just look like thousands of pages of random noise - indeed, it would be random noise, in a very rigorous sense.

That said, while I don't think either Alon or I were making claims about what particular structure we're likely to find here, I do think there is a particular kind of structure here. I do not currently know what that structure is, but I think answering that question (or any of several equivalent questions, e.g. formalizing abstraction) is the main problem required to solve embedded agency and AGI in general.

Also see my response to Ofer, which discusses the same issues from a different starting point.

Comment by johnswentworth on But exactly how complex and fragile? · 2019-11-05T18:47:20.788Z · score: 0 (2 votes) · LW · GW
Surely "the whole point of AI safety research" is just to save the world, no?

Suppose you're an engineer working on a project to construct the world's largest bridge (by a wide margin). You've been tasked with safety: designing the bridge so that it does not fall down.

One assistant comes along and says "I have reviewed the data on millions of previously-built bridges as well as record-breaking bridges specifically. Extrapolating the data forward, it is unlikely that our bridge will fall down if we just scale-up a standard, traditional design."

Now, that may be comforting, but I'm still not going to move forward with that bridge design until we've actually run some simulations. Indeed, I'd consider the simulations the core part of the bridge-safety-engineer's job; trying to extrapolate from existing bridges would be at most an interesting side-project.

But if the bridge ends up standing, does it matter whether we were able to guarantee/verify the design or not?

The problem is model uncertainty. Simulations of a bridge have very little model uncertainty - if the simulation stands, then we can be pretty darn confident the bridge will stand. Extrapolating from existing data to a record-breaking new system has a lot of model uncertainty. There's just no way one can ever achieve sufficient levels of confidence with that kind of outside-view reasoning - we need the levels of certainty which come with a detailed, inside-view understanding of the system.

If the world ends up being saved, does it matter whether we were able to "verify" that or not?

Go find an engineer who designs bridges, or buildings, or something. Ask them: if they were designing the world's largest bridge, would it matter whether they had verified the design was safe, so long as the bridge stood up?

Comment by johnswentworth on Book Review: Design Principles of Biological Circuits · 2019-11-05T18:19:15.642Z · score: 24 (9 votes) · LW · GW

Yeah, that's a natural argument. The counterargument which immediately springs to mind is that, until we've completely and totally solved biology, there's always going to be some systems we don't understand yet - just because they haven't been understood yet does not mean they're opaque. It boils down to priors: do we have reasons to expect large variance in opaqueness? Do we have reason to expect low variance in opaqueness?

My own thoughts can be summarized by three main lines of argument:

  • If we look at the entire space of possible programs, it's not hard to find things which are pretty darn opaque to humans. Crypto and computational complexity theory provide some degree of foundation to that idea. So human-opaque systems do exist.
  • We can know that something is non-opaque (by understanding it), but we can't know for sure that something is opaque. Lack of understanding is Bayesian evidence in favor of opaqueness, but the strength of that evidence depends a lot on who's tried to understand it, how much effort was put in, what the incentives look like, etc.
  • I personally have made arguments of the form "X is intractably opaque to humans" about many different systems in multiple different fields in the past (not just biology). In most cases, I later turned out to be wrong. So at this point I have a pretty significant prior against opacity.

So I'd say the book provides evidence in favor of a generally-low prior on opaqueness, but people trying and failing to understand a system is the main type of evidence regarding opacity of particular systems.

Unfortunately, these are all outside-view arguments. I do think an inside view is possible here - it intuitively feels like the kinds of systems which turned out not to be opaque (e.g. biological circuits) have visible, qualitative differences from the kinds of systems which we have some theoretical reasons to consider opaque (e.g. pseudorandom number generators). They're the sort of systems people call "emergent". Problem is, we don't have a useful formalization of that idea, and I expect that figuring it out requires solving a large chunk of the problems in the embedded agency cluster.

When I get around to writing a separate post on Alon's last chapter (evolution of modularity), that will include some additional relevant insight to the question.