Models of Value of Learning 2020-07-07T19:08:31.785Z · score: 20 (6 votes)
High Stock Prices Make Sense Right Now 2020-07-03T20:16:53.852Z · score: 78 (31 votes)
Mediators of History 2020-06-27T19:55:48.485Z · score: 25 (9 votes)
Abstraction, Evolution and Gears 2020-06-24T17:39:42.563Z · score: 22 (6 votes)
The Indexing Problem 2020-06-22T19:11:53.626Z · score: 38 (7 votes)
High-School Algebra for Data Structures 2020-06-17T18:09:24.550Z · score: 20 (9 votes)
Causality Adds Up to Normality 2020-06-15T17:19:58.333Z · score: 12 (3 votes)
Cartesian Boundary as Abstraction Boundary 2020-06-11T17:38:18.307Z · score: 24 (6 votes)
Public Static: What is Abstraction? 2020-06-09T18:36:49.838Z · score: 64 (15 votes)
Everyday Lessons from High-Dimensional Optimization 2020-06-06T20:57:05.155Z · score: 112 (45 votes)
Speculations on the Future of Fiction Writing 2020-05-28T16:34:45.599Z · score: 34 (22 votes)
Highlights of Comparative and Evolutionary Aging 2020-05-22T17:01:30.158Z · score: 49 (18 votes)
Pointing to a Flower 2020-05-18T18:54:53.711Z · score: 51 (18 votes)
Conjecture Workshop 2020-05-15T22:41:31.984Z · score: 34 (10 votes)
Project Proposal: Gears of Aging 2020-05-09T18:47:26.468Z · score: 62 (24 votes)
Writing Causal Models Like We Write Programs 2020-05-05T18:05:38.339Z · score: 52 (18 votes)
Generalized Efficient Markets and Academia 2020-04-30T21:59:56.285Z · score: 45 (18 votes)
Motivating Abstraction-First Decision Theory 2020-04-29T17:47:31.896Z · score: 38 (11 votes)
What's Hard About Long Tails? 2020-04-23T16:32:36.266Z · score: 24 (10 votes)
Intuitions on Universal Behavior of Information at a Distance 2020-04-20T21:44:42.260Z · score: 26 (8 votes)
Integrating Hidden Variables Improves Approximation 2020-04-16T21:43:04.639Z · score: 15 (3 votes)
Noise Simplifies 2020-04-15T19:48:39.452Z · score: 24 (13 votes)
Probabilistic Blue-Eyed Islanders 2020-04-15T17:55:00.429Z · score: 26 (7 votes)
Transportation as a Constraint 2020-04-06T04:58:28.862Z · score: 142 (49 votes)
Mediation From a Distance 2020-03-20T22:02:46.545Z · score: 15 (4 votes)
Alignment as Translation 2020-03-19T21:40:01.266Z · score: 44 (15 votes)
Abstraction = Information at a Distance 2020-03-19T00:19:49.189Z · score: 23 (5 votes)
Positive Feedback -> Optimization? 2020-03-16T18:48:52.297Z · score: 19 (7 votes)
Adaptive Immune System Aging 2020-03-13T03:47:22.056Z · score: 64 (23 votes)
Please Press "Record" 2020-03-11T23:56:27.699Z · score: 49 (14 votes)
Trace README 2020-03-11T21:08:20.669Z · score: 33 (10 votes)
Name of Problem? 2020-03-09T20:15:11.760Z · score: 9 (2 votes)
The Lens, Progerias and Polycausality 2020-03-08T17:53:30.924Z · score: 62 (22 votes)
Interfaces as a Scarce Resource 2020-03-05T18:20:26.733Z · score: 129 (40 votes)
Trace: Goals and Principles 2020-02-28T23:50:12.900Z · score: 20 (5 votes)
johnswentworth's Shortform 2020-02-27T19:04:55.108Z · score: 8 (1 votes)
Value of the Long Tail 2020-02-26T17:24:28.707Z · score: 47 (18 votes)
Theory and Data as Constraints 2020-02-21T22:00:00.783Z · score: 42 (13 votes)
Exercises in Comprehensive Information Gathering 2020-02-15T17:27:19.753Z · score: 102 (46 votes)
Demons in Imperfect Search 2020-02-11T20:25:19.655Z · score: 69 (22 votes)
Category Theory Without The Baggage 2020-02-03T20:03:13.586Z · score: 98 (34 votes)
What Money Cannot Buy 2020-02-01T20:11:05.090Z · score: 200 (93 votes)
Algorithms vs Compute 2020-01-28T17:34:31.795Z · score: 28 (6 votes)
Coordination as a Scarce Resource 2020-01-25T23:32:36.309Z · score: 111 (36 votes)
Material Goods as an Abundant Resource 2020-01-25T23:23:14.489Z · score: 65 (27 votes)
Constraints & Slackness as a Worldview Generator 2020-01-25T23:18:54.562Z · score: 39 (16 votes)
Technology Changes Constraints 2020-01-25T23:13:17.428Z · score: 81 (30 votes)
Theory of Causal Models with Dynamic Structure? 2020-01-23T19:47:22.825Z · score: 25 (5 votes)
Formulating Reductive Agency in Causal Models 2020-01-23T17:03:44.758Z · score: 30 (7 votes)
(A -> B) -> A in Causal DAGs 2020-01-22T18:22:28.791Z · score: 35 (9 votes)


Comment by johnswentworth on DARPA Digital Tutor: Four Months to Total Technical Expertise? · 2020-07-08T02:42:05.376Z · score: 15 (7 votes) · LW · GW

Was there any attempt to check generalizability of the acquired skills? One concern I have is that the digital tutor may have just taught-to-the-test, since the people who built the system presumably knew what the test questions were and how students would be tested.

Comment by johnswentworth on Breaking Questions Down · 2020-07-07T20:34:39.926Z · score: 5 (3 votes) · LW · GW

One recommendation: the goal is not just to break a question into smaller questions sufficient to answer the original. The goal is to carve reality at the joints; to structure the search for answers in a way which reflects the structure of reality. This is useful mainly because it allows re-use and generalization of intermediate answers.

In a lot of simple cases, it's fairly obvious how to structure questions - e.g. query "probability of catching covid when doing X", rather than "probability that covid came from X given covid", even though either one might technically be sufficient information for a super-query.

The hard/interesting cases are those where the right break-down is nonobvious. For these cases, remember that it's useful to carve reality at the joints because it allows re-use/generalization of intermediate answers. Reversing that idea suggests a strategy for generating good intermediate questions: rather than starting from one goal-question, start from several related goal-questions, and then look for intermediate-questions which shed light on multiple goals. Rather than a tree, the question-breakdown should look like a DAG. By forcing intermediate questions to support multiple use-cases, we naturally push ourselves to look for questions which will have general relevance.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-06T16:52:18.708Z · score: 3 (2 votes) · LW · GW

I pretty much always expect positive total returns from stocks. Nothing in the OP contradicts the EMH - predicting crashes in such a way that we could profit from it is still Hard. The correct discount rate to use in EMH pricing depends on capital supply and demand, but that discount rate is still generally positive. Just like high bond prices mean that the yield is close to zero, not negative, high stock prices mean that expected returns are close to zero, not negative. (This isn't always the case - e.g. negative interest rates in the European banks' overnight markets during the previous decade - but that requires some fairly unusual conditions to maintain.)

So, yes, I do expect positive total returns from stocks over the next few months, though not very high on average and with quite a bit of variance.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-05T18:17:37.188Z · score: 5 (3 votes) · LW · GW

The money markets are the main direct source of cheap (short-term) liquidity for banks; the relevant interest rates are LIBOR, fed funds, and repo. My current understanding is that retail deposits are ultimately the main source of funds in these markets - some banks (think Bank of America) specialize in retail and net-lend into the overnight money markets, while others net-borrow.

That money goes into Treasuries by default - i.e. whenever banks don't have anything higher-margin to put it into. That is a profitable activity, to my understanding, at least in the long term. It's borrowing short-term (overnight) and lending long-term (Treasury term), thereby getting paid to assume interest rate risk, which is exactly the main business of a bank.

In case people want to know more about this stuff, most of my understanding comes from Perry Mehrling's coursera course (which I recommend), as well as the first third of Stigum's Money Markets.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-04T04:14:17.109Z · score: 2 (1 votes) · LW · GW

This is also something I have yet to study in depth, but the Fed model makes a lot of sense (specifically the capital structure substitution version). Under that model, companies generally use stock issues/buybacks paired with bond buybacks/issues to adjust how much of their funding is from stocks vs bonds, in order to maximize expected earnings per share. That creates a transmission mechanism between bonds and stocks, and is the main thing I'd think about for the sorts of stuff you're talking about.

More generally, I'm on board with Cochrane's money as stock theory, although I think he doesn't implement it quite right - the assets backing dollar value on a day-to-day basis are the SOMA portfolio, not the whole government's assets and cash flows.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-04T03:04:32.917Z · score: 4 (6 votes) · LW · GW

That would make sense if the increase in savings was expected to be mostly permanent. But in this case, the savings rate seems likely to revert to normal in a year or so. Why aren't investors who see that selling enough now for stock prices to anticipate that return to normal?

When one stock is overvalued relative to another stock, investors can buy the undervalued and sell the overvalued. When the entire market (including bonds) is "overvalued", that doesn't work - there's nothing to substitute into, and even an "overvalued" stock/bond is usually better than plain cash.

The relative price of two financial assets is mainly about expectations. The overall price level of the market is mainly about supply/demand of capital right now.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-04T00:30:50.335Z · score: 3 (2 votes) · LW · GW

True, though I would guess that bank holdings dwarf those, even looking at marginal investors. Banks not holding stocks is the more important side of this model, because that's the part which allows Treasury yields to be systematically lower than they "should" be compared to stocks.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-03T23:35:14.192Z · score: 2 (1 votes) · LW · GW

One comment on the low interest rates issue: I generally think about the equity premium puzzle via a Volker fence model. The institutions which own Treasuries (e.g. banks) do so with massive amounts of cheap leverage, and those are the only assets they're allowed to hold with that much leverage. For individuals without access to cheap leverage, it rarely makes sense to hold large amounts in Treasuries - stock returns are so much higher, because the cheap leverage of banks pushes down the returns on Treasuries. So we get this split model, where there's one class of investors which determines Treasury rates and a mostly-separate class which determines equity prices, with a regulatory barrier between the two (namely, bank leverage regulations).

Epistemic status: properly checking the numbers on this is still on my todo list, but the qualitative predictions look realistic and it matches how financial institutions actually work in practice.

Comment by johnswentworth on High Stock Prices Make Sense Right Now · 2020-07-03T23:18:38.978Z · score: 7 (4 votes) · LW · GW

I mentioned that I usually use a model with exogenous cash-flows, and this gets at the reason.

When I look at how most people around me manage their money, it does not look like "adjust allocations based on the business cycle". It looks like "keep savings in fixed-allocation retirement portfolio, or index fund, or just buy and hold for years". Most people don't actually trade much in response to market fluctuations; they trade when they have spare cash to invest or need to take out cash in order to pay bills. That doesn't mean their allocations don't change - e.g. if equity prices fall, then everyone who's simply bought-and-held will have a smaller proportion of their wealth in stocks without trading at all.

Of course, there's a whole industry full of professionals who do adjust more actively, but as a general rule traders on any timescale faster than years make money by forecasting the behavior of people trading larger amounts of money on slower timescales than them. The large-scale cash flows from everyday people are the "ground truth" which others make money by forecasting, whether explicitly or (more often) implicitly.

In short: it's mainly fund flows that drive overall prices, not vice-versa.

Note that the EMH is still compatible with this: the small fraction of active traders will still eat up any opportunities to beat the market. Those fund flows determine the behavior of the market which those traders are trying to beat - they determine the baseline.

Epistemic status on this: I don't have conclusive data, except in a few relatively-isolated markets (which are simpler to analyze). I trust the model because its qualitative predictions look much more realistic than a prices-drive-fund-flows model. For instance, one of my original big motivators of using a fund-flows-drive-prices model is that it directly predicts that price motion should usually look Brownian (to first order), whereas the usual models don't actually predict that - they just predict expectations, without actually saying anything about the shape of the distribution of moves.

Comment by johnswentworth on The ground of optimization · 2020-06-30T17:04:55.220Z · score: 2 (1 votes) · LW · GW

Fascinating - but why is this an objection? Is it just the inelegance of not being able to look at a single time slice and answer the question of whether optimization is happening?

No, the issue is that the usual definition of an optimization problem (e.g. ) has no built-in notion of time, and the intuitive notion of optimization (e.g. "the system makes Y big") has no built-in notion of time (or at least linear time). It's this really fundamental thing that isn't present in the "original problem", so to speak; it would be very surprising and interesting if time had to be involved when it's not present from the start. 

If I specifically try to brainstorm things-which-look-like-optimization-but-don't-involve-objective-improvement-over-time, then it's not hard to come up with examples:

  • Rather than a function-value "improving" along linear time, I could think about a function value improving along some tree or DAG - e.g. in a heap data structure, we have a tree where the "function value" always "improves" as we move from any leaf toward the root. There, any path from a leaf to the root could be considered "time" (but the whole set of nodes at the "same level" can't be considered a time-slice, because we don't have a meaningful way to compare whole sets of values; we could invent one, but it wouldn't actually reflect the tree structure).
  • The example from the earlier comment: a one-shot non-iterative optimizer
  • A distributed optimizer: the system fans out, tests a whole bunch of possible choices in parallel, then selects the best of those.
  • Various flavors of constraint propagation, e.g. the simplex algorithm (and markets more generally)
Comment by johnswentworth on Optimized Propaganda with Bayesian Networks: Comment on "Articulating Lay Theories Through Graphical Models" · 2020-06-29T15:22:45.862Z · score: 6 (4 votes) · LW · GW

(and are conditionally independent given the value of )

I believe this should say "conditionally dependent".

Comment by johnswentworth on The ground of optimization · 2020-06-29T04:50:20.382Z · score: 4 (2 votes) · LW · GW

Another big thing to note in examples like e.g. iteratively computing a square root for the quadratic formula or iteratively computing eigenvalues to solve a matrix: the optimization problems we're solving are subproblems, not the original full problem. These crucially differ from most of the examples in the OP in that the system's objective function (in your sense) does not match the objective function (in the usual intuitive sense). They're iteratively optimizing a subproblem's objective, not the "full" problem's objective.

That's potentially an issue for thinking about e.g. AI as an optimizer: if it's using iterative optimization on subproblems, but using those results to perform some higher-level optimization in a non-iterative manner, then aligning the sobproblem-optimizers may not be synonymous with aligning the full AI. Indeed, I think a lot of reasoning works very much like this: we decompose a high-dimensional problem into coupled low-dimensional subproblems (i.e. "gears"), then apply iterative optimizers to the subproblems. That's exactly how eigenvalue algorithms work, for instance: we decompose the full problem into a series of optimization subproblems in narrower and narrower subspaces, while the "high-level" part of the algorithm (i.e. outside the subproblems) doesn't look like iterative optimization.

Comment by johnswentworth on Abstraction, Evolution and Gears · 2020-06-26T21:01:48.311Z · score: 2 (1 votes) · LW · GW

Awesome. Thanks for the spot-check, I'll probably use this as a dramatic example going forward.

Comment by johnswentworth on A Personal (Interim) COVID-19 Postmortem · 2020-06-26T15:04:50.895Z · score: 25 (13 votes) · LW · GW

I also had too-strong priors and "expert" ideas to be properly fox-like in my predictions, and not quick enough to update about how things were actually going based on the data. Because I was slow to move from the base-rate, I underestimated the severity of COVID-19 for too long. I'm unsure how to fix that, since most of the time it's the right move, and paying attention to every new event is very expensive in terms of mental energy. (Suggestions welcome!)

Bottom line: in order to outperform base-rate, somebody somewhere has to do the expensive updates. No way around that. So the options are (a) do the expensive updates yourself, (b) give up and go with base rate, (c) find someone else who's doing the updates. (c) is the obvious answer, but bear in mind that recognizing real expertise is Hard, and you've already noticed just how questionable the "expertise" of many supposed experts actually is.

In the easy case, the nominal experts are in a position where we get reasonably-frequent feedback on their performance (which obviously is usually not the case for rare events). In principle, one might get around that by an expert specializing in noticing rare events across domains.

Comment by johnswentworth on Abstraction, Evolution and Gears · 2020-06-26T14:42:13.415Z · score: 2 (1 votes) · LW · GW

Do you have a source on that? My back-of-the-envelope says 7 is not enough. For 1 kg located 1 m away from 60 kg, gravitional force is on the order of 10^-9 N. IIRC, angular uncertainty in a billiards system roughly doubles with each head-on collision, so the exponential growth would only kick in around a factor of 128, which still wouldn't be large enough to notice most of the time. (Though I believe the uncertainty grows faster for off-center hits, so that might be what I'm missing.)

Comment by johnswentworth on Abstraction, Evolution and Gears · 2020-06-25T14:59:33.036Z · score: 2 (1 votes) · LW · GW

I think that there are some things that are sensitively dependant on other parts of the system, and we usually just call those bits random.

One key piece missing here: the parts we call "random" are not just sensitively dependent on other parts of the system, they're sensitively dependent on many other parts of the system. E.g. predicting the long-run trajectories of billiard balls bouncing off each other requires very precise knowledge of the initial conditions of every billiard ball in the system. If we have no knowledge of even just one ball, then we have to treat all the long-run trajectories at random.

That's why sensitive dependence on many variables matters: lack of knowledge of just one of them wipes out all of our signal. If there's a large number of such variables, then we'll always be missing knowledge of at least one, so we call the whole system random.

Comment by johnswentworth on Abstraction, Evolution and Gears · 2020-06-24T19:47:10.934Z · score: 2 (1 votes) · LW · GW

The low level details may be important as a means to an end.

The end in question being f(X), specifically. The agent will be indifferent between any X-values with the same f(X) - or, to put it differently, f(X) mediates the impact of X on the agent's goals.

Similarly with the resources thing - the impact of current choices on far-future action space is mediated by resource quantities.

Comment by johnswentworth on The ground of optimization · 2020-06-24T19:42:25.230Z · score: 2 (1 votes) · LW · GW

Ok, well, it seems like the one-shot non-iterative optimizer is an optimizer in a MUCH stronger sense than a random program, and I'd still expect a definition of optimization to say something about the sense in which that holds.

Comment by johnswentworth on Abstraction, Evolution and Gears · 2020-06-24T19:39:35.003Z · score: 2 (1 votes) · LW · GW

The image this is a caption for isn't showing up.

Thanks, this should be fixed now.

Comment by johnswentworth on The ground of optimization · 2020-06-24T17:44:38.294Z · score: 2 (1 votes) · LW · GW

This argument applies to literally any deterministic program with nonempty output. Are you saying that every program is an optimizer?

Comment by johnswentworth on The ground of optimization · 2020-06-24T15:34:26.038Z · score: 2 (1 votes) · LW · GW

Ok, two problems with this:

  • There's no reason why that target set would be smaller than the basin of attraction. Given one such optimization problem, there are no obvious perturbations we could make which would leave the result in the target region.
  • The target region is not a subset of the basin of attraction. The system doesn't evolve from a larger region to a smaller subset (as in the Venn-diagram visuals in the OP), it just evolves from one set to another.

The first problem explicitly violates the OP's definition of an optimizer, and the second problem violates one of the unspoken assumptions present in all of the OP's examples.

Comment by johnswentworth on The ground of optimization · 2020-06-23T22:30:34.700Z · score: 2 (1 votes) · LW · GW

Not sure.

Comment by johnswentworth on The ground of optimization · 2020-06-23T18:12:00.796Z · score: 2 (1 votes) · LW · GW

I'm not sure what you're trying to say here. What's the state space (in which both the start and end state of the optimizer live), what's the basin of attraction (i.e. set of allowed initial conditions), and what's the target region within the state space? And remember, the target region needs to be a subset of the allowed initial conditions.

Comment by johnswentworth on Homeostasis and “Root Causes” in Aging · 2020-06-23T18:03:14.144Z · score: 4 (2 votes) · LW · GW

That's a tough question to answer; there's an awful lot of stem-cell types and telomerase activity is not necessarily binary. I wouldn't be shocked if some of the more-differentiated and/or slower-dividing types don't express it.

We can do a back-of-the-envelope estimate: wikipedia quotes typical telomere length of 11k bp (base pairs) at birth, and one replication eats up 20 bp. That's a limit of ~500 divisions, so any human stem cell which divides much faster than every ~(80 yrs)/500 = 60 days needs to express telomerase just to keep dividing throughout our lives. In practice, I'd expect that to be an extreme underestimate, since my understanding is that oxidative damage eats up telomeres considerably faster than replication does, especially for infrequently-dividing cells.

Comment by johnswentworth on Locality of goals · 2020-06-23T15:22:59.018Z · score: 10 (4 votes) · LW · GW

Nice post!

One related thing I was thinking about last week: part of the idea of abstraction is that we can pick a Markov blanket around some variable X, and anything outside that Markov blanket can only "see" abstract summary information f(X). So, if we have a goal which only cares about things outside that Markov blanket, then that goal will only care about f(X) rather than all of X. This holds for any goal which only cares about things outside the blanket. That sounds like instrumental convergence: any goal which does not explicitly care about things near X itself, will care only about controlling f(X), not all of X.

This isn't quite the same notion of goal-locality that the OP is using (it's not about how close the goal-variables are to the agent), but it feels like there's some overlapping ideas there.

Comment by johnswentworth on The ground of optimization · 2020-06-22T22:05:55.105Z · score: 13 (7 votes) · LW · GW

My biggest objection to this definition is that it inherently requires time. At a bare minimum, there needs to be an "initial state" and a "final state" within the same state space, so we can talk about the system going from outside the target set to inside the target set.

One class of cases which definitely seem like optimization but do not satisfy this property at all: one-shot non-iterative optimization. For instance, I could write a convex function optimizer which works by symbolically differentiating the objective function and then algebraically solving for a point at which the gradient is zero.

Is there an argument that I should not consider this to be an optimizer?

Comment by johnswentworth on The ground of optimization · 2020-06-22T21:54:37.500Z · score: 6 (3 votes) · LW · GW

The set of optimizing systems is smaller than the set of all AI services, but larger than the set of goal-directed agentic systems.


A tree is an optimizing system but not a goal-directed agent system.

I'm not sure this is true, at least not in the sense that we usually think about "goal-directed agent systems".

You make a case that there's no distinct subsystem of the tree which is "doing the optimizing", but this isn't obviously relevant to whether the tree is agenty. For instance, the tree presumably still needs to model its environment to some extent, and "make decisions" to optimize its growth within the environment - e.g. new branches/leaves growing toward sunlight and roots growing toward water, or the tree "predicting" when the seasons are turning and growing/dropping leaves accordingly.

One to think about whether "the set of optimizing systems is smaller than the set of all AI services, but larger than the set of goal-directed agentic systems" is that it's equivalent to Scott's (open) question does agent-like behavior imply agent-like architecture?

Comment by johnswentworth on The ground of optimization · 2020-06-22T21:46:10.162Z · score: 6 (3 votes) · LW · GW

At first I particularly liked the idea of identifying systems with "an optimizer" as those which are robust to changes in the object of optimization, but brittle with respect to changes in the engine of optimization.

On reflection, it seems like a useful heuristic but not a reliable definition. A counterexample: suppose we do manage to build a robust AI which maximizes some utility function. One desirable property of such an AI is that it's robust to e.g. one of its servers going down or corrupted data on a hard drive; the AI itself should be robust to as many interventions as possible. Ideally it would even be robust to minor bugs in its own source code. Yet it still seems like the AI is the "engine", and it optimizes the rest of the world.

Comment by johnswentworth on The ground of optimization · 2020-06-22T21:39:40.485Z · score: 6 (3 votes) · LW · GW

This is excellent! Very well done, I would love to see more work like this.

I have a whole bunch of things to say along separate directions so I'll break them into separate comments. This first one is just a couple minor notes:

  • For the universe section, the universe doesn't push "toward" maxent, it just wanders around and usually ends up in maxent states because that's most of the states. The basin of attraction includes all states.
  • Regarding "whether dynamical systems theory explicitly studies attractors that operate along a subset of the system’s dimensions", I believe there's an old theorem that the long-term behavior of dynamical systems on a compact space is always ergodic on some manifold within the space. That manifold has a name which I don't remember, which is probably what you want to look for.
Comment by johnswentworth on Plausible cases for HRAD work, and locating the crux in the "realism about rationality" debate · 2020-06-22T20:28:42.573Z · score: 8 (4 votes) · LW · GW

... we don't merely want a precise theory that lets us build an agent; we want our theory to act like a box that takes in an arbitrary agent (such as one built using ML and other black boxes) and allows us to analyze its behavior.

FWIW, this is what I consider myself to be mainly working towards, and I do expect that the problem is directly solvable. I don't think that's a necessary case to make in order for HRAD-style research to be far and away the highest priority for AI safety (so it's not necessarily a crux), but I do think it's both sufficient and true.

Comment by johnswentworth on The Indexing Problem · 2020-06-22T20:00:36.675Z · score: 3 (2 votes) · LW · GW

For a few months, yes. Planning to focus on aging for 2-3 months, then probably back to AI-adjacent theory.

Comment by johnswentworth on Public Static: What is Abstraction? · 2020-06-22T14:25:29.250Z · score: 4 (2 votes) · LW · GW


Comment by johnswentworth on What's Your Cognitive Algorithm? · 2020-06-19T14:37:27.525Z · score: 3 (2 votes) · LW · GW

I've read some of his stuff on strategy. It seemed like there were a lot of interesting insights in there, but it was all presented in the sort of way that sounds sciency to non-science-people but didn't really communicate a proper model. If someone knows of or could write a good explanation of the models underlying his ideas, I'd be very interested to read that.

Comment by johnswentworth on What's Your Cognitive Algorithm? · 2020-06-19T01:43:06.492Z · score: 5 (3 votes) · LW · GW

I do think it's all ultimately implemented on top of something babbly, yeah. The babbly part seems like the machine code of the brain - ultimately everything has to be implemented in that.

I think what I mean by "gearsy reasoning" is something different than how you're using the phrase. It sounds like you're using it as a synonym for systematic or system-2 reasoning, whereas I see gears as more specifically about decomposing systems into their parts. Gearsy reasoning doesn't need to look very systematic, and systematic reasoning doesn't need to be gearsy - e.g. simply breaking things into steps is not gearsy reasoning in itself. So the specific "tight associations" you list do not sound like the things I associate with gearsy thinking specifically.

As an example, let's say I'm playing a complicated board game and figuring out how to get maximum value out of my resources. The thought process would be something like:

  • Ok, main things I want are X, Y, Z -> what resources do I need for all that?
  • (add it up)
  • I have excess A and B but not enough C -> can I get more C?
  • I have like half a dozen ways of getting more C, it's basically interchangeable with B at a rate of 2-to-1 -> do I have enough B for that?
  • ...

So that does look like associative babbling; the "associations" it's following are mainly the relationships between objects given by the game actions, plus the general habit of checking what's needed (i.e. the constraints) and what's available.

I guess one insight from this: when engaged in gears-thinking, it feels like the associations are more a feature of the territory than of my brain. It's not about taste, it's about following the structure of reality (or at least that's how it feels).

Comment by johnswentworth on What's Your Cognitive Algorithm? · 2020-06-18T23:27:23.328Z · score: 20 (7 votes) · LW · GW

I definitely agree that most people most of the time are using an algorithm like babble-with-associations, with feedback refining the associations over time. That said, I do think it is possible to do better, and at least some people (especially STEM people working on hard problems) learn to use other methods at least sometimes. In the rationality-movement-I'd-like-to-see, systematizing and training such more-efficient-thinking-algorithms is a central piece of the curriculum, and a major goal of the movement.

I've been thinking about explicit models of more-efficient-thinking-algorithms for a while now - Mazes and Crayon was an early piece along those lines, and of course a lot of the gears pieces are tied in. I have at least one thinking-algorithm which is more efficient on novel problems than vanilla babble-with-associations and which I think is trainable. I've been working towards a write-up (Everyday Lessons from High-Dimensional Optimization was partly intended as background for that write-up).

That write-up isn't ready yet, but here are some sparknotes:

  • We use gears-level models because black-box problem solving in high dimensions is inefficient. The idea of "gears" is to decompose the system into coupled low-dimensional subsystems.
  • My current best guess is that equations/constraints are a universal representation for gears. E.g. in a causal model (specifically structural equations), the equations/constraints are the structural equations themselves. Likewise in physics/chemistry/engineering/etc: the constraints are the system's governing equations. When we want to problem-solve on the system, those governing equations act as constraints on our problem.
    • I expect that humans' native representation of gears-level models amounts to something equivalent to constraints as well.
  • This immediately suggests a whole class of problem-solving strategies, in particular:
    • constraint relaxation to generate heuristics (e.g. "could I solve this problem ignoring this constraint?")
    • dual methods (e.g. thinking about tautness/slackness of constraints - which constraints are "easy", and which are limiting our ability to solve the problem)
  • There's also the question of how to figure out the equations/constraints in the first place. Presumably we could look for constraints-on-the-constraints, but then we go up a whole ladder of meta-levels... except we actually don't. Here again, we can leverage duality: constraints-on-constraints are partial solutions to the original problem. So as we go "up" the meta-ladder, we just switch back-and-forth between two problems. One specific example of this is simultaneously looking for a proof and a counterexample in math. A more visual example is in this comment.
    • Concretely, this process looks like trying out some solution-approaches while looking for any common barrier they run into, then thinking about methods to specifically circumvent that barrier (ignoring any other constraints), then looking for any common barrier those methods run into, etc...

The key feature of this class of thinking-algorithms is that they can get around the inefficiency issues in high-dimensional problems, as long as the problem-space decomposes - which real-world problem spaces usually do, even in cases where you might not expect it. This is basically just taking some standard tricks from AI (constraint relaxation, dual methods), and applying them directly to human thinking.

Note that this can still involve babbly thinking for the lower-level steps; the whole thing can still be implemented on top of babble-with-associations. The key is that we only want to babble at low-dimensional subproblems, while using a more systematic approach for the full high-dimensional problem.

Comment by johnswentworth on Growing Independence · 2020-06-16T00:24:51.616Z · score: 7 (4 votes) · LW · GW

I think there is actually something fairly important about the topic of raising kids, which is relevant to more common LessWrong themes. [...] Many LW folk don't seem to have kids, and part of me is worried that all our philosophy and strategy is sort of missing something important if we don't have a background sense of "what raising kids is like" subtly informing our judgments.

+1 to this specifically. Taking an unformed brain and guiding it along a path to become stronger seems like the equivalent of a lab class in rationality.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-16T00:17:11.105Z · score: 4 (2 votes) · LW · GW

I think your description here is basically right, except for the very last sentence. If we just had one meta-level, then yeah, it would end up basically the same as explore/exploit but at a more abstract level. But by recursively applying the trick, it ends up working well on a much wider class of problems than explore/exploit by itself.

The really neat thing is that recursively applying this particular trick does not require moving up an infinite ladder of meta-levels. There are only two levels, and moving "up" alternates between them.

To see why, consider what it looks like on a math problem where we're trying to prove some theorem:

  • A "constraint" would be something like a conditional counterexample - i.e. "for any proof which uses strategy X, here's a counterexample"
  • When searching for such counterexamples, our "constraints" are partial proofs - i.e. if we have a proof assuming X, then any counterexample must leverage/assume the falsehood of X.

So when we use constraints to guide our search for proofs, we look for counterexamples. When we use constraints to guide our search for counterexamples, we look for proofs.

More visually, consider solving a maze. We can find constraints like this:

When we're searching for that constraint, what are the meta-constraints to that search? Well, they're paths. A constraint is a path of connected walls; a meta-constraint is a path of connected spaces.

In general, we're switching between solving an optimization problem and solving the dual problem.

Of course, we may also use some random search at any given step before switching.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-15T05:35:56.550Z · score: 5 (3 votes) · LW · GW

I had a long conversation with my girlfriend at one point about picking hobbies. She used to just try things at random, and I was encouraging her to be more strategic. We started by making a list of things she liked/didn't like about previous hobbies she'd tried. Then, we turned those into criteria for potential hobbies - things like "should involve exercise", "should involve producing something", "should be good to show off to friends", or "should grow in skill over time". Then we considered broad classes of activities which addressed specific criteria, like "sports" or "art". We tried to be systematic about generating the activity-classes - e.g. starting from "what possible kinds of things can we produce?" rather than just generating random ideas and checking them against the constraints.

This sort of reasoning doesn't really look like uniformly-random exploration followed by exploitation. In particular, the pattern I want to highlight in this case is:

  • identify constraints (i.e. the criteria)
  • search by backtracking from the constraints, rather than trying things uniformly-randomly

That's not incompatible with exploration+exploitation (blackbox search can be used in sub-steps), but it is an additional component which is not implied by exploitation/exploration alone. It avoids uniformly-random exploration, and instead steers toward more-likely-useful possibilities. The constraints tell us which-way-to-go in our high-dimensional search space.

This part in particular is interesting:

But where did these decision procedures come from?

Well, we've been trying each of them for millennia. Each depend on exploitation (of friends, restaurants, cookbooks), which in turn are the result of exploration (going to parties, searching on Yelp, looking up reviews), and so on ad infinitum.

One of the hobbies my girlfriend eventually picked up was baking, and she recently wrote a post about how baking is not about following opaque decision procedures handed down through the millennia. That post has some great examples of gearsy reasoning; I recommend checking it out.

Comment by johnswentworth on What are some Civilizational Sanity Interventions? · 2020-06-15T01:03:31.233Z · score: 4 (2 votes) · LW · GW

I agree this is a likely outcome, though I also think there's at least a 30% chance that the blackhats could find ways around it. Journals can't just lock it down to people the editors know personally without losing the large majority of their contributors.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-14T16:37:20.687Z · score: 2 (1 votes) · LW · GW

Gradient descent requires somehow computing the gradient. There are both blackbox and non-blackbox ways of doing that, and the blackbox methods are much more expensive.

Backpropagation is the main non-blackbox method for computing the gradient; it looks at the steps used to compute the function and propagates information backward through those steps. If the function takes m steps to compute, backprop takes O(m) to compute the gradient.

The main blackbox method for computing gradients is to evaluate the function itself, then evaluate it with a small change in one coordinate, then evaluate it with a small change in the next coordinate, and so forth - one evaluation for each coordinate. If the function takes m steps to compute and has n inputs, then this takes O(mn).

Comment by johnswentworth on What are some Civilizational Sanity Interventions? · 2020-06-14T02:35:16.523Z · score: 14 (11 votes) · LW · GW

One idea I was thinking about over the last few days: academic hoaxes have been used many times over the past few decades to reveal shoddy standards in journals/subfields. The Sokal affair is probably the most famous, but there's a whole list of others linked on its wikipedia page. Thing is, that sort of hoax always took a fair bit of effort - writing bullshit which sounds good isn't trivial! So, as a method for policing scientific rigor, it was hard to scale up without a lot of resources.

But now we have GPT2/3, which potentially changes the math dramatically.

I'd guess that a single small team - possibly even a single person - could generate and submit hundreds or even thousands of bullshit papers, in parallel. That sort of sustained pressure would potentially change journals' incentives in a way which the occasional sting doesn't. There'd probably be an arms race for a little while - journals/reviewers coming up with cheap ways to avoid proper checks, bullshit-generators coming up with ways around those defenses - but I think there's a decent chance that the end result would be proper rigor in reviews.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-14T00:49:11.793Z · score: 7 (4 votes) · LW · GW

It sounds like you're using "simulated annealing" as synonymous with something like "mix exploration and exploitation"? I'm not entirely sure what you're using it to point to, but it definitely sounds like you're using it as a metaphor for some particular aspect of search algorithms, rather than talking about the actual math of simulated annealing. What are you trying to emphasize?

Anyway, "mix exploration and exploitation" is not a search algorithm by itself. It's a general trick which can be used in conjunction with a wide variety of search algorithms. This also applies to simulated annealing specifically: it's a technique used in conjunction with other search algorithms. (E.g. scipy's implementation takes an argument for which optimizer to use in the minimizer_kwargs.)

It sounds like you might be arguing that there is something fundamentally different going on when we employ "gearsy reasoning." If so, that is my point of disagreement. 

I am definitely arguing that gearsy reasoning does something fundamentally different. It does use black-box optimizers, but it uses them locally, operating on individual gears. Black-box optimization works badly on high-dimensional systems, so we decompose such systems into coupled low-dimensional systems, and then use black-box optimizers on the low-dimensional components. The interesting part of gearsy reasoning is how to account for the coupling, which is where things like causality and propagation come in.

The account I've just given seems to me like an intuitive and accurate description of how I've solved every problem I've ever worked on, and how it appears to me that other people work as well.

The account you've given may well be accurate, but it's an inherently incomplete account. Simulated annealing - or whatever you're trying to point to with that phrase - is not itself a search algorithm. Even if all problem solving involves some mix of exploration and exploitation, that still doesn't tell us which direction to go in a thousand-dimensional space when "exploiting". Likewise, exploration clearly isn't completely random - so how do we decide which directions to explore? That's what gears are for.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-13T19:47:31.130Z · score: 4 (2 votes) · LW · GW

Ah, I see what you're saying. This definitely gets at some interesting points.

First point is efficiency. As you mentioned in the top-level comment, SA "seems to work well if you have the ability to create and evaluate an enormous number of designs, at extremely low cost relative to the value of success, and high accuracy". So SA would be a bad way for a human (or group of humans) to design a bridge, but if you have a cheap and accurate bridge-simulator, then it's not unreasonable. There are probably much more efficient algorithms which leverage the problem structure more, but that requires writing complicated code.

Note that there's nothing particularly special about SA in this regard - most black-box optimization methods work in largely the same cases.

(Ultimately, one of the main areas I want to apply the ideas in the OP to is better methods for human thinking - i.e. things humans can do to learn/problem-solve more quickly/successfully. So that's largely the use-case I have in mind. Obviously computational optimization is a whole different game with different constraints, and we do use black-box optimizers in many computational problems with considerable success.)

Second point is problem specification. Once we move from talking about humans designing things to computers designing things (via black-box optimization), problem specification becomes the big bottleneck. We need both (a) a specification of the environment/problem-space, e.g. a physics simulator for bridge-design, and (b) specification of the objective, e.g. a function to compute whether a bridge has "failed" in a simulation and how much the bridge "costs". The challenge is that both of these pieces need to work correctly over the entire (exponentially large) space of parameters which the optimizer could test. Obviously we can't test correctness of the simulator and objective by checking it on every possible input, so we need some other way to check that it's correct - which brings us back to gears.

In (certain parts of) chip design, this problem has turned out to be tractable. In that case, the "objective" is specified by humans using gearsy reasoning to design a chip, then black-boxing that chip design and requiring the optimizer to produce something functionally identical to the design (but more efficient by various metrics).

(Here again, gears matter mainly for the part of the thinking-work which the humans need to do.)

As to a one-sentence summary of my main claim: in complex high-dimensional problems, you (a human) need to break the system up into sparsely-coupled low-dimensional subsystems, and leverage that representation, in order to achieve a workable solution.

Comment by johnswentworth on Everyday Lessons from High-Dimensional Optimization · 2020-06-13T17:41:40.120Z · score: 7 (4 votes) · LW · GW

Simulated annealing is subject to exactly the same sorts of problems as the algorithms discussed in the post. It's effectively an e-coli-style algorithm which varies its step size.

I think there's a lot of popular misunderstanding of SA. People talk about it as though it's going to find the global optimum, which it isn't. If you need an actual global optimum, then SA isn't any better than brute force (in a big-O sense). People also talk about it approximating a global optimum, which it does marginally better than naive gradient descent, but it still requires qualitatively similar conditions to gradient descent in order to actually work well. It can handle a few bumps and hills, but at the large scale, the objective function still needs to be shaped roughly like a large bowl in order for SA to work well; there has to be a broad basin of attraction for the optimum. If the best points are narrow, deep wells with big hills around them, then SA isn't going to help any.

Basically, it's gradient descent with less noise-sensitivity.

None of this is to say it's a bad method; gradient descent with less noise-sensitivity is great! But it's not going to find global optima in high-dimensional non-convex problems without exponential amounts of compute.

Comment by johnswentworth on Why do all out attacks actually work? · 2020-06-13T17:10:44.706Z · score: 2 (1 votes) · LW · GW

But why doesn't the all out attack work against status?

I think it often does. All out attacks do actually work quite often.

Comment by johnswentworth on Why do all out attacks actually work? · 2020-06-13T17:08:41.698Z · score: 6 (3 votes) · LW · GW

In principle I agree with Villiam, though often these situations are sufficiently unlike markets that thinking of it in EMH terms will lead intuitions astray. So I'll emphasize some other aspects (though it's still useful to consider how the aspects below generalize to other EMH arguments).

Situations where all out attacks work are usually situations where people nominally trying to do the thing are not actually trying to do the thing. This is often for typical Inadequate Equilibria reasons - i.e. people are rewarded for looking like they're making effort, rather than for success, because it's often a lot easier to verify that people look-like-they're-making-effort than that they're actually making progress.

I think this happens a lot more in everyday life than people realize/care to admit: employers in many areas will continue to employ employees without complaint as long as it looks like they're trying to do The Thing, even if The Thing doesn't get done very quickly/very well - there just needs to be a plausible-sounding argument that The Thing is more difficult than it looks. (I've worked in several tech startups, and this incentive structure applied to basically everyone.) Whether consciously or unconsciously, a natural result is that employees don't really put forth their full effort to finish things as quickly and perfectly as possible; there's no way for the employer to know that The Thing could have been done faster/better.

(Thought experiment: would you do your job differently if you were going to capture the value from the product for yourself, and wouldn't get paid anything besides that?)

The whole status-attack problem slots neatly into this sort of scenario: if I come along and say that I can do The Thing in half the time and do a better job of it too, then obviously that's going to come across as an attack on whoever 's busy looking-like-they're-doing The Thing.

Comment by johnswentworth on Why do all out attacks actually work? · 2020-06-12T22:21:15.103Z · score: 12 (9 votes) · LW · GW

My current model of this centers on status, similar to your last paragraph. I'll flesh it out a bit more.

Suppose I build a net-power-generating fusion reactor in my garage. In terms of status, this reflects very badly on an awful lot of high-status physicists and engineers who've sunk massive amounts of effort and resources into the same goal and completely failed to achieve it. This applies even without actually building the thing: if I claim that I can build a net-power-generating fusion reactor in my garage, then that's a status grab (whether I intend it that way or not); I'm claiming that I can run circles around all those people who've tried and failed. People react to that the way people usually react to status grabs: they slap it down. After all, if they don't slap it down, then the status-grab "succeeds" - it becomes more plausible in the eyes of third parties that I actually could build the thing, which in turn lowers the status of all the people who failed to do so.

Now, flip this back around: if I want to avoid being perceived as making a status grab (and therefore being slapped down), then I need to avoid being perceived as claiming to be able to do anything really big-sounding. And, as you mention, the easiest way to avoid the perception of a grand claim is to honestly believe that I can't do the grand thing.

From the inside, this means that we try to predict what we'll be able to do via an algorithm like:

  • How much social status would I have if I did X?
  • How much social status do I have?
  • If the first number is much larger than the second, then I probably can't do the thing.

Presumably this is not an especially accurate algorithm, but it is a great algorithm for avoiding conflict. It avoids making claims (even unintentionally) for which we will be slapped down.

I'm pretty sure Yudkowsky sketched a model like this in Inadequate Equilibria, which is probably where I got it from.

Comment by johnswentworth on Goal-directedness is behavioral, not structural · 2020-06-12T21:19:16.560Z · score: 6 (2 votes) · LW · GW

Well, the questions I care about (and the ones Rohin asked) are actually about goal-directed AI. It's about whether it must be goal-directed, and whether making it not/less goal-directed improves its safety. So I'm clearly not considering "what humans want" first, even if it would be a nice consequence.

Yeah, I definitely see that you're trying to do a useful thing here, and the fact that you're not doing some other useful thing doesn't make the current efforts any less useful.

That said, I would suggest that, if you're thinking about a notion of "goal-directedness" which isn't even intended to capture many of the things people often call "goal-directedness", then maybe finding a better name for the thing you want to formalize would be a useful step. It feels like the thing you're trying to formalize is not actually goal-directedness per se, and figuring out what it is would likely be a big step forward in terms of figuring out the best ways to formalize it and what properties it's likely to have.

(Alternatively, if you really do want a general theory of goal-directedness, then I strongly recommend brainstorming many use-cases/examples and figuring out what unifies all of them.)

Drawing an analogy to my current work: if I want to formulate a general notion of abstraction, then that project is about making it work on as many abstraction-use-cases as possible. On the other hand, if I just want a formulation of abstraction to solve one or two particular problems, then a solution to that might not need to be a general formulation of abstraction - and figuring out what it does need to be would probably help me avoid the hard work of building a fully general theory.

Comment by johnswentworth on Self-Predicting Markets · 2020-06-12T16:32:08.067Z · score: 16 (6 votes) · LW · GW

I just got around to actually looking at Hertz' most recent financials. Short version:

  • $19B of debt
  • $26B of assets, mostly highly illiquid (e.g. cars, property)

So this bankruptcy is not a situation where the company has negative net value, it's a situation where the company doesn't have the revenue to cover its debt-servicing costs in the short term, because their assets aren't liquid. But the company's assets are still worth substantially more than their debt, so it seems reasonably likely that stockholders will still end up with a fair bit of value. Indeed, assuming Hertz isn't forced into a fire-sale, their assets are (supposedly) worth $7B more than their debt, and their market cap is only $900M as of the OP, so there's an awful lot of space for good things to happen there just based on fundamentals.

Comment by johnswentworth on Ideology/narrative stabilizes path-dependent equilibria · 2020-06-11T20:52:26.712Z · score: 3 (2 votes) · LW · GW

De Mesquita and Smith, as well as CGP Grey, discuss some of the structural reasons for this: in technological advanced liberal democracies, wealth is produced primarily by educated knowledge workers. Therefore, one can’t neglect the needs of the population at large like you can in a dictatorship, or you will cut off the flow of revenue that funds your state-apparatus.

But that structural consideration doesn’t seem to be most of the story to me. It seems like the main factor is ideology.


I can barely imagine a cabal of the majority of high ranking military officials agreeing to back a candidate that lost an election, even if that a cabal assessed that backing that candidate would be more profitable for them. My impression of military people in general is that they are extremely proud Americans, for whom the ideals of freedom and democracy are nigh-spiritual in their import.

Sounds like you're trying to build a model shaped like "(educated knowledge workers) -> democracy <- ideology". I suggest that a better model is "(educated knowledge workers) -> ideology -> democracy" - i.e. the dominant ideology is itself downstream of economic "power", and is determined by realpolitik-style dynamics.

Looking at the example of American military people, I'd say they are selected for being proud Americans with ideals of freedom and democracy. Why that particular selection pressure? There's presumably more than one gear in the middle, and I don't know what they all are, but that's the sort of selection pressure I'd expect to spontaneously show up in a country economically dominated by educated knowledge workers.

That points toward a class of answers for this question:

In so far as we do live in a world where we have the ideology of Democracy, right in exactly the places where it needs to be to protect our republic, how did that happen?

Again, I don't know all the gears, but I'd expect that an economically-dominant middle class somehow selected for a governmental structure which favors them, and that selection pressure propagates through the system (i.e. if I select X for some outcome, and Y is causally between X and the outcome, then I'll end up exerting selection pressure on Y via X).