Posts

Testing The Natural Abstraction Hypothesis: Project Update 2021-09-20T03:44:43.061Z
Writing On The Pareto Frontier 2021-09-17T00:05:32.310Z
Optimizing Multiple Imperfect Filters 2021-09-15T22:57:16.961Z
Framing Practicum: Comparative Advantage 2021-09-09T23:59:09.468Z
Information At A Distance Is Mediated By Deterministic Constraints 2021-08-31T16:50:13.483Z
How To Write Quickly While Maintaining Epistemic Rigor 2021-08-28T17:52:21.692Z
Framing Practicum: Turnover Time 2021-08-24T16:29:04.701Z
What fraction of breakthrough COVID cases are attributable to low antibody count? 2021-08-22T04:07:46.495Z
Framing Practicum: Timescale Separation 2021-08-19T18:27:55.891Z
Framing Practicum: Dynamic Equilibrium 2021-08-16T18:52:00.632Z
Staying Grounded 2021-08-14T17:43:53.003Z
Framing Practicum: Bistability 2021-08-12T04:51:53.287Z
Framing Practicum: Stable Equilibrium 2021-08-09T17:28:48.338Z
Slack Has Positive Externalities For Groups 2021-07-29T15:03:25.929Z
Working With Monsters 2021-07-20T15:23:20.762Z
Generalizing Koopman-Pitman-Darmois 2021-07-15T22:33:03.772Z
The Additive Summary Equation 2021-07-13T18:23:06.016Z
Potential Bottlenecks to Taking Over The World 2021-07-06T19:34:53.016Z
The Language of Bird 2021-06-27T04:44:44.474Z
Notes on War: Grand Strategy 2021-06-18T22:55:30.174Z
Variables Don't Represent The Physical World (And That's OK) 2021-06-16T19:05:08.512Z
The Apprentice Experiment 2021-06-10T03:29:27.257Z
Search-in-Territory vs Search-in-Map 2021-06-05T23:22:35.773Z
Selection Has A Quality Ceiling 2021-06-02T18:25:54.432Z
Abstraction Talk 2021-05-25T16:45:15.996Z
SGD's Bias 2021-05-18T23:19:51.450Z
How to Play a Support Role in Research Conversations 2021-04-23T20:57:50.075Z
Updating the Lottery Ticket Hypothesis 2021-04-18T21:45:05.898Z
Computing Natural Abstractions: Linear Approximation 2021-04-15T17:47:10.422Z
Specializing in Problems We Don't Understand 2021-04-10T22:40:40.690Z
Testing The Natural Abstraction Hypothesis: Project Intro 2021-04-06T21:24:43.135Z
Core Pathways of Aging 2021-03-28T00:31:49.698Z
Another RadVac Testing Update 2021-03-23T17:29:10.741Z
Chaos Induces Abstractions 2021-03-18T20:08:21.739Z
What's So Bad About Ad-Hoc Mathematical Definitions? 2021-03-15T21:51:53.242Z
How To Think About Overparameterized Models 2021-03-03T22:29:13.126Z
RadVac Commercial Antibody Test Results 2021-02-26T18:04:09.171Z
The Prototypical Negotiation Game 2021-02-20T21:33:34.195Z
Utility Maximization = Description Length Minimization 2021-02-18T18:04:23.365Z
Fixing The Good Regulator Theorem 2021-02-09T20:30:16.888Z
Making Vaccine 2021-02-03T20:24:18.756Z
Simulacrum 3 As Stag-Hunt Strategy 2021-01-26T19:40:42.727Z
Exercise: Taboo "Should" 2021-01-22T21:02:46.649Z
Recognizing Numbers 2021-01-20T19:50:51.908Z
Science in a High-Dimensional World 2021-01-08T17:52:02.261Z
How Hard Would It Be To Make A COVID Vaccine For Oneself? 2020-12-21T16:19:10.415Z
What confusions do people have about simulacrum levels? 2020-12-14T20:20:35.626Z
Parable of the Dammed 2020-12-10T00:08:44.493Z
Non-Book Review: Patterns of Conflict 2020-11-30T21:05:24.389Z
The Pointers Problem: Human Values Are A Function Of Humans' Latent Variables 2020-11-18T17:47:40.929Z

Comments

Comment by johnswentworth on Writing On The Pareto Frontier · 2021-09-17T22:42:48.480Z · LW · GW

I mean, the argument does kinda rely on someone else having written it better, which does not often happen when "better" is comparing to Scott.

Comment by johnswentworth on Writing On The Pareto Frontier · 2021-09-17T16:08:46.376Z · LW · GW

Good question! Rough argument: if someone else has already written it better, then do your readers a favor and promote that to them instead.

Obviously this is an imperfect argument - for instance, writing is a costly signal that you consider a topic important, and it's also a way to clarify your own thoughts or promote your own brand. So Pareto optimality isn't necessarily relevant to things I'm writing for my own benefit (as opposed to readers'), and it's not relevant when the writing is mostly a costly signal of importance aimed at my social circle. Also, even if we accept the argument, then Pareto optimality is only a necessary condition for net value, not a sufficient condition; plenty of things are on some Pareto frontier but still not worth reading for anyone.

Comment by johnswentworth on Dating Minefield vs. Dating Playground · 2021-09-16T16:30:08.498Z · LW · GW

I've seen that graph (of what percentage of couples met in various ways) a few times now, and what I really want to know is: why do several different channels all plateau at the same levels? E.g. bar/restaurant, coworkers, and online all seem to plateau just below 20% for a while. Church, neighbors, and college all seem to hang out around 8% for a while. What's up with that?

Comment by johnswentworth on Optimizing Multiple Imperfect Filters · 2021-09-16T16:21:55.952Z · LW · GW

Yeah, I didn't want to spend a paragraph on definitions which nobody would be able to keep straight anyway. "False positive" and "false negative" are just very easy-to-confuse terms in general. That's why I switched to "duds" and "missed opportunities" in the sales funnel section.

Comment by johnswentworth on Optimizing Multiple Imperfect Filters · 2021-09-16T01:24:46.959Z · LW · GW

Fixed, thank you.

Comment by johnswentworth on Information At A Distance Is Mediated By Deterministic Constraints · 2021-09-12T01:07:19.844Z · LW · GW

More like: exponential family distributions are a universal property of information-at-a-distance in large complex systems. So, we can use exponential models without any loss of generality when working with information-at-a-distance in large complex systems.

That's what I hope to show, anyway.

Comment by johnswentworth on Information At A Distance Is Mediated By Deterministic Constraints · 2021-09-11T22:38:04.656Z · LW · GW

Yup, that's the direction I want. If the distributions are exponential family, then that dramatically narrows down the space of distributions which need to be represented in order to represent abstractions in general. That means much simpler data structures - e.g. feature functions and Lagrange multipliers, rather than whole distributions.

Comment by johnswentworth on Information At A Distance Is Mediated By Deterministic Constraints · 2021-09-11T01:00:14.663Z · LW · GW

Roughly speaking, the generalized KPD says that if the long-range correlations are low dimensional, then the whole distribution is exponential family (modulo a few "exceptional" variables). The theorem doesn't rule out the possibility of high-dimensional correlations, but it narrows down the possible forms a lot if we can rule out high-dimensional correlations some other way. That's what I'm hoping for: some simple/common conditions which limit the dimension of the long-range correlations, so that gKPD can apply.

This post says that those long range correlations have to be mediated by deterministic constraints, so if the dimension of the deterministic constraints is low, then that's one potential route. Another potential route is some kind of information network flow approach - i.e. if lots of information is conserved along one "direction", then that should limit information flow along "orthogonal directions", which would mean that long-range correlations are limited between "most" local chunks of the graph.

Comment by johnswentworth on Framing Practicum: Comparative Advantage · 2021-09-10T17:07:01.737Z · LW · GW

I like the insights on research specialization.

On cellular signalling: "control by intercellular signalling" is not necessarily analogous to a command economy. After all, even in a market economy, we have lots of interagent signalling in the form of e.g. prices. Indeed, many hormones function quite similarly to prices (i.e. they signal abundance or scarcity of an associated resource), and biological signalling is largely decentralized - different organs specialize in different signals and their associated functions. The "rewards" need not be energy or oxygen or even growth of a cell population; indeed, we don't necessarily need a "reward" signal at the cellular level at all in order for the economic analogy to apply. That's part of the idea of this post: we can apply comparative advantage (and opportunity cost, markets, etc) even when the "subsystems" are not themselves optimizers which "want" anything. There can be just one "central" set of pareto-optimization objectives, but the optimization is implemented in a decentralized way by "trading" until the opportunity costs of different subsystems equilibrate.

Comment by johnswentworth on Framing Practicum: Comparative Advantage · 2021-09-10T16:50:39.157Z · LW · GW

I have at least two practicum posts planned for markets, looking at them from different angles.

One is a direct follow-up to this post: we say two "subsystems" (of the sort used in this post) are in "equilibrium" when they can't make any "trade" which would yield a pareto gain on the objectives. In this post, we saw that that means the two subsystems have the same trade off ratios (aka opportunity costs). Those ratios are prices - specifically, the prices at which any of the equilibrated subsystems can "trade" with any other subsystems or the external world. The equilibrated subsystems are a "market", and their shared prices are the defining feature of that market.

Under that angle, market size isn't particularly relevant. Markets are about pareto optimality and trade-equilibrium.

The other angle is markets as a limit in games with many identical players. As the number of players grows, identical players compete to make deals, and only the best bids/offers win. So, we end up with a "shared price" for whatever deals are made.

Under that angle, market size is a central question.

Comment by johnswentworth on The alignment problem in different capability regimes · 2021-09-09T22:01:36.193Z · LW · GW

Claim: the core of the alignment problem is conserved across capability levels. If a particular issue only occurs at a particular capability level, then the issue is usually "not really about alignment" in some sense.

Roughly speaking, if I ask a system for something, and then the result is not really what I wanted, but the system "could have" given the result I wanted in some sense, then that's an alignment problem regardless of whether the system is a superintelligent AI or google maps. Whether it's a simple system with a bad user interface, or a giant ML system with an unfriendly mesa-optimizer embedded in it, the conceptual core of the problem isn't all that different.

The difference is mainly in how-bad-it-is for the system to be misaligned (for a given degree-of-misalignment). That does have important implications for how we think about AI safety - e.g. we can try to create systems which are reasonably safe without really solving the alignment problem. But I think it's useful to distinguish safety vs alignment here - e.g. a proposal to make an AI safe by making sure it doesn't do anything very far out of the training distribution might be a reasonable safety proposal without really saying much about the alignment problem.

Similarly, proposals along the lines of "simulate a human working on the alignment problem for a thousand years" are mostly safety proposals, and pass the buck on the alignment parts of the problem. (Which is not necessarily bad!)

The distinction matters because, roughly speaking, alignment advances should allow us to leverage more-capable systems while maintaining any given safety level. On the other hand, safety-without-alignment mostly chooses a point on the safety-vs-capabilities pareto surface without moving that surface. (Obviously this is a severe oversimplification of a problem with a lot more than two dimensions, but I still think it's useful.)

Comment by johnswentworth on Core Pathways of Aging · 2021-09-09T18:43:26.404Z · LW · GW

Not especially. Sequencing transposons in particular is tricky, and requires special techniques. It sounds like the authors of that paper used pretty standard techniques, which mostly ignore transposons; they mainly looked at point-mutations.

It does still provide some very interesting tangentially-related data, especially about the phylogeny (i.e. the "loss of clonal diversity" with age). Pretty cool paper overall; this exact methodology plus a transposon-specific sequencing pipeline is exactly the sort of study I'd really like to see.

Comment by johnswentworth on Gradient descent is not just more efficient genetic algorithms · 2021-09-08T18:20:44.538Z · LW · GW

Yup, that's roughly what I was picturing. (Really I was picturing a smooth approximation of that, but the conclusion is the same regardless.)

In general, "shouldn't stray from the center of the surface by a symmetry argument" definitely should not work for GD in practice - either because numerical noise knocks us off the line-where-both-are-equal, or because the line-where-both-are-equal itself curves.

So, unless the line-where-both-are-equal is perfectly linear and the numerics are perfectly symmetric all the way to the lowest bits, GD will need to take steps of size ~epsilon to stay near the center of the surface.

Comment by johnswentworth on Gradient descent is not just more efficient genetic algorithms · 2021-09-08T17:02:43.132Z · LW · GW

First, I love this example. Second, I think it's wrong.

It's true in idealized mathematical terms - if you're right at the point where the submodules agree, then the gradient will be along the continue-to-agree direction. But that's not what matters numerically - i.e. it's not what matters in actual practice. Numerically, the speed of (non-stochastic) gradient descent is controlled by the local condition number, and the condition number for this two-module example would be enormous - meaning that gradient descent will move along the submodules' parameter space extremely slowly. Unless the surface along which the submodules match is perfectly linear (and the parameters are already exactly on that surface), every step will take it just a little bit off the surface, so it will end up taking extremely small steps (so that it's always within numerical precision of the surface).

Comment by johnswentworth on How To Write Quickly While Maintaining Epistemic Rigor · 2021-09-08T03:00:56.797Z · LW · GW

Secret secondary goal of this post: get people to pay attention to the process-which-generates their ideas/beliefs/etc.

Comment by johnswentworth on Framing Practicum: Selection Incentive · 2021-09-06T16:46:29.291Z · LW · GW
  1. Sales jobs tend to bring in a lot of young people, and a high proportion of them aren't able to sell much and leave - i.e. there's high selection pressure on salespeople. There's a selection incentive to engage in sketchy sales practices, e.g. lying about the product. Individual salespeople may intend to be honest, and even believe what they're saying, but they'll still be selected for saying the product is better than it is. So, the salespeople who stick around will be those who mislead customers about the product, even if they do so accidentally.
  2. Competing products are selected by customers; they face different selection incentives depending on whether customers are mostly one-time or repeat users. If customers are mostly one time, then there's a selection incentive for the product to look good before purchase, but less incentive to actually be good. If customers are mostly repeat users, then there's more incentive to actually be good.
  3. I own a few pairs of flip-flops, so I sometimes have to choose which pair to wear; the criteria by which I make that choice create selection incentives for the worn flip-flops. (I.e. if I look at which flip-flops I actually end up wearing, I'll find that they're selected according to those incentives.) In particular, I usually choose whichever flip-flops are most easily available at the time, which is usually whatever pair I wore most recently. (So the selection incentives change sometimes when I wear different flip-flops; they're bistable.)
Comment by johnswentworth on When Money Is Abundant, Knowledge Is The Real Wealth · 2021-09-06T02:05:56.703Z · LW · GW

I do indeed agree with that.

Comment by johnswentworth on The Coordination Frontier: Sequence Intro · 2021-09-05T04:48:14.450Z · LW · GW

coordination is basically the most important thing [citation needed]

Citation. Well, sort of. That version of the post was a little shy about calling it the most important thing; the original was more direct about that, but wasn't as good a post.

I'm looking forward to this sequence, it sounds excellent.

Comment by johnswentworth on Information At A Distance Is Mediated By Deterministic Constraints · 2021-09-04T15:42:03.616Z · LW · GW

We can still view these as travelling through many layers - the light waves have to propagate through many lightyears of mostly-empty space (and it could attenuate or hit things along the way). The photo has to last many years (and could randomly degrade a little or be destroyed at any moment along the way).

What makes it feel like "one hop" intuitively is that the information is basically-perfectly conserved at each "step" through spacetime, and there's in a symmetry in how the information is represented.

Comment by johnswentworth on johnswentworth's Shortform · 2021-09-02T18:15:47.064Z · LW · GW

Takeaways From "The Idea Factory: Bell Labs And The Great Age Of American Innovation"

Main takeaway: to the extent that Bell Labs did basic research, it actually wasn’t all that far ahead of others. Their major breakthroughs would almost certainly have happened not-much-later, even in a world without Bell Labs.

There were really two transistor inventions, back to back: Bardain and Brattain’s point-contact transistor, and then Schockley’s transistor. Throughout, the group was worried about some outside group beating them to the punch (i.e. the patent). There were semiconductor research labs at universities (e.g. at Purdue; see pg 97), and the prospect of one of these labs figuring out a similar device was close enough that the inventors were concerned about being scooped.

Most inventions which were central to Bell Labs actually started elsewhere. The travelling-wave tube started in an academic lab. The idea for fiber optic cable went way back, but it got its big kick at Corning. The maser and laser both started in universities. The ideas were only later picked up by Bell.

In other cases, the ideas were “easy enough to find” that they popped up more than once, independently, and were mostly-ignored long before deployment - communication satellites and cell communications, for instance.

The only fundamental breakthrough which does not seem like it would have soon appeared in a counterfactual world was Shannon’s information theory.

So where was Bell’s big achievement? Mostly in development, and the research division was actually an important component of that. Without in-house researchers chewing on the same problems as the academic labs, keeping up-to-date with all the latest findings and running into the same barriers themselves, the development handoff would have been much harder. Many of Bell Labs’ key people were quite explicitly there to be consulted - i.e. “ask the guy who wrote the book”. I think it makes most sense to view most of the Labs’ research that way. It was only slightly ahead of the rest of the world at best (Shannon excepted), and often behind, but having those researchers around probably made it a lot easier to get new inventions into production.

Major reason this matters: a lot of people say that Bell was able to make big investments in fundamental research because they had unusually-long time horizons, protected by a monopoly and a cozy government arrangement (essentially a Schumpeterian view). This is contrasted to today's silicon valley, where horizons are usually short. But if Bell's researchers generally weren't significantly ahead of others, and mostly just helped get things to market faster, then this doesn't seem to matter as much. The important question is not whether something silicon-valley-like induces more/less fundamental research in industrial labs, but whether academics heeding the siren call of startup profits can get innovations to market as quickly as Bell Labs' in-house team could. And by that metric, silicon valley looks pretty good: Bell Labs could get some impressive things through the pipe very quickly when rushed, but they usually had no reason to hurry, and they acted accordingly.

Comment by johnswentworth on How To Write Quickly While Maintaining Epistemic Rigor · 2021-09-02T17:52:52.152Z · LW · GW

Drawing from my own posts:

Comment by johnswentworth on How To Write Quickly While Maintaining Epistemic Rigor · 2021-09-02T17:00:06.153Z · LW · GW

Addendum (also added to the post): I'm worried that people will read this post think "ah, so that's the magic bullet for a LW post", then try it, and be heartbroken when their post gets like one upvote. Accurately conveying one's thought process and uncertainty is not a sufficient condition for a great post; clear explanation and novelty and interesting ideas all still matter (though you certainly don't need all of those in every post). Especially clear explanation - if you find something interesting, and can clearly explain why you find it interesting, then (at least some) other people will probably find it interesting too.

Comment by johnswentworth on A lost 80s/90s metaphor: playing the demo · 2021-09-02T15:43:49.474Z · LW · GW

This is a great name for something I've been wanting a name for.

I've wanted it as a useful frame for thinking about politics, especially the power of elected officials (both legislators and the executive). It seems like most of the time, most of the decisions about how some regulation or project or whatever will actually go happen at a much lower level in the bureaucracy; leadership has neither the knowledge nor the processing bandwidth to exert any meaningful control. At best they can appoint people who they think are aligned with their goals, but this runs into the problem that recognizing real expertise itself requires some expertise in the area. Ultimately, the decrees they hand down end up doing entirely different things than what they advertised; the nominal leaders don't really know how to make decrees which have the effects they say they do. (Not that this is necessarily a problem from the standpoint of elected officials - they're largely selected for symbolism these days anyway. "Vote for me if you hate the outgroup" is not a platform which hinges on actually-effective policies.)

The previous analogy I had used was that elected officials are mostly "pretending to lead the parade" - i.e. they walk in front of the parade and pretend that it's following them, rather than following a predetermined route. In some ways, I like the "playing the demo" analogy better - it doesn't capture the symbolic aspects as much, but it better captures the idea that some complicated non-human logic is actually running the show.

The same frame applies to many other kinds of large organizations too, like big companies. To a large extent, leadership is symbolic, and has limited power to either observe or control what lower-level people are doing on a minute-to-minute basis. (In principle, incentive design is the main way one can actually exert control on a reasonably-granular level, but the sort of people who end up in most leadership positions usually don't do that sort of thing. Mostly, to the extent that they do anything useful, they solve coordination problems between departments/teams/subunits.)

Also, the opening section was hysterical.

Comment by johnswentworth on How To Write Quickly While Maintaining Epistemic Rigor · 2021-09-01T19:40:28.755Z · LW · GW

This is definitely the use-case where "explain how you came to think Y" is hardest; there's a vague ball of intuitions playing a major role in the causal pathway. On the other hand, making those intuitions more legible (e.g. by using analogies between psych and ML) tends to have unusually high value.

I suspect that, from Eliezer's perspective, a lot of sequences came from roughly this process. He was trying to work back through his own pile of intuitions and where they came from, then serialize and explain as much of it as possible. It's been a generator for a lot of my own writing as well - for instance, the Constraints/Scarcity posts came from figuring out how to make a broad class of intuitions legible, and the review of Design Principles of Biological Circuits came from realizing that the book had been upstream of a bunch of my intuitions about AI. It's not coincidence that those were relatively popular posts - figuring out the logic which drives some intuitions, and making that logic legible, is valuable. It allows us to more directly examine and discuss the previously-implicit/intuitive arguments.

I wouldn't quite liken it to persuasion. I think the thing you're trying to point to is that the author does most of the work of crossing the inductive gap. In general, when two people communicate, either one can do the work of translating into terms the other person understands (or they can split that work, or a third party can help, etc... the point is that someone has to do it.). When trying to persuade someone, that burden is definitely on the persuader. But that's not exclusively a feature of persuasion - it's a useful habit to have in general, to try to cross most of the inductive gap oneself, and it's important for clear writing in general. The goal is still to accurately convey some idea/intuition/information, not to persuade the reader that the idea/intuition/information is right.

Comment by johnswentworth on [Crosspost] On Hreha On Behavioral Economics · 2021-08-31T19:52:18.920Z · LW · GW

G&R are happy to admit that in many, many cases, people behave in loss-averse ways, including most of the classic examples given by Kahneman and Tversky. They just think that this is because of other cognitive biases, not a specific cognitive bias called “loss aversion”. They especially emphasize Status Quo Bias and the Endowment Effect.

Interesting tangent: if we start with the kind of inexploitability arguments typically used to justify utility functions, and modify them to account for agents having internal state, then we get subagents. Rather than inexploitable decision-makers always being equivalent to utility-maximizers, we find that inexploitable decision-makers are equivalent to committees of utility-maximizers where each "committee member" has a veto. (In particular, this model handles markets, which are the ur-example of inexploitability yet turn out not to be equivalent to a single utility maximizer.)

What sort of "biases" would someone expecting a utility-maximizer would find when studying such a subagent-based decision-maker? In other words, how does a subagent-based decision-maker's decisions differ from a utility-maximizer's decisions? Mainly, there are cases where the subagent will choose A over B if it already has A, and B over A if it already has B. (Interpretation: there are two subagents, one of which wants A, and one of which wants B. If we already have A, then the A-preferring subagent will veto any offer to switch; if we already have B, then the B-preferring subagent will veto any offer to switch.) In other words: there's a tendency toward inaction, and toward assigning "more value" to whatever the agent currently has. Status Quo Bias, and Endowment Effect.

And yet, neither of these supposed "biases" is actually exploitable. Such decision-makers can't be money-pumped.

Comment by johnswentworth on Framing Practicum: Incentive · 2021-08-29T23:24:39.276Z · LW · GW
  1. Aqueducts. Water "wants" to flow downhill. Humans want the water to flow from remote mountain springs into our cities and homes. So, we provide an incentive gradient: the water can go (locally) downhill fastest by following our aqueduct.
    Could the water go down faster by some other route? Well, it could spray through a leak in the pipe/channel, for instance. 
  2. Yudkowsky claims that every cause wants to become a cult - i.e. there is a positive feedback loop which amplifies cult-like aspects of causes. Leaders are incentivized to play along with this - i.e. to "give the cause what it wants" in exchange for themselves being in charge. Note that this incentive pressure applies regardless of whether the leaders actually want their cause to become a cult.
    What would it look like for a cause's leaders satisfy this incentive via some other strategy? Basically, they could take the "extreme" members who want to push that positive feedback loop, and give them some position/outlet which satisfies the relevant group-status needs without actually pushing marginal people out of the group.
  3. Filters (the physical kind, like coffee filters). Filters select for very small particles, so if we look at what makes it through, it's "incentivized" to be small.
    But things could satisfy the incentive (i.e. sneak through the filter) in other ways - e.g. a microorganism could literally eat its way through, or weakly-soluble salts could dissolve and re-precipitate on the other side.
Comment by johnswentworth on How to turn money into AI safety? · 2021-08-26T15:23:55.368Z · LW · GW

I agree with the bit in the post about how it makes sense to invest in a lot of different approaches by different small teams. Similarly with hiring people to work on various smaller/specific questions. This makes sense at small scale, and there's probably still room to scale it up more at current margins. The problem comes when one tries to pour a lot of money into that sort of approach: spending a lot of money on something is applying optimization pressure, whether we intend to or not, and if we don't know what we're optimizing for then the default thing which happens is that we Goodhart on people trying to look good to whoever's making the funding decisions.

So, yes at small scale and probably at current margins, but this is a strategy which can only scale so far before breaking down.

Comment by johnswentworth on How to turn money into AI safety? · 2021-08-26T02:51:16.370Z · LW · GW

There's apparently a lot of funding looking for useful ways to reduce AI X-risk right now.

Comment by johnswentworth on Framing Practicum: Turnover Time · 2021-08-25T16:44:16.265Z · LW · GW

Solid examples, and good variety. They all felt like natural fits for the frame, and I could easily imagine setting up a rough model for any of them.

Comment by johnswentworth on Framing Practicum: Timescale Separation · 2021-08-25T16:41:15.566Z · LW · GW

One theme in these: they're all conclusions which seem pretty intuitive. One of the nice things about timescale separation is that it gives us a formal justification for a lot of intuitively-sensible conclusions.

Comment by johnswentworth on Framing Practicum: Timescale Separation · 2021-08-25T16:39:52.966Z · LW · GW

Good examples. One theme these highlight: we intuitively use timescale separation all the time in our day-to-day lives.

Comment by johnswentworth on Framing Practicum: Timescale Separation · 2021-08-25T16:38:40.973Z · LW · GW

I love the "examples where the frame doesn't apply" idea.

What is a "winner of the market"?

Comment by johnswentworth on Framing Practicum: Timescale Separation · 2021-08-25T16:32:09.431Z · LW · GW

I love the idea of defining one's relation to fashion by focus on short-term vs long-term equilibrium.

Comment by johnswentworth on Framing Practicum: Stable Equilibrium · 2021-08-25T16:25:07.301Z · LW · GW

I love the first one. Explicitly trying to avoid time is a brilliant spin on the exercise.

Comment by johnswentworth on How to turn money into AI safety? · 2021-08-25T15:38:06.426Z · LW · GW

A lot of the difficulty comes from the fact that AI safety is a problem we don't understand; the field is pre-paradigmatic. We don't know how best to frame the problem. We don't know what questions to ask, what approximations to make, how to break the problem into good subproblems, what to pay attention to or what to ignore. All of these issues are themselves major open problems.

That makes a lot of the usual scaling-up approaches hard.

  • We can't write a textbook, because we don't know what needs to go in it. The one thing we know for sure is that the things we might currently think to write down are not sufficient; we do not yet have all the pieces.
  • We can't scale up existing training programs, because we don't quite know what skills/knowledge are crucial for AI safety research. We do know that no current program trains quite the right mix of skills/knowledge; otherwise AI safety would already fit neatly into that paradigm.
  • Existing organizations have limited ability to absorb more people, because they don't understand the problem well enough to effectively break it into pieces which can be pursued in parallel. Figuring that out is part of what existing orgs are trying to do.
  • Previous bullet also applies to people who have some legible achievements and could found a new org.
  • Finally, nobody currently knows how to formulate the core problems of the field in terms of highly legible objectives. Again, that's a major open problem.

I learned about the abundance of available resources this past spring. My own approach to leveraging more resources is to try to scale up the meta-level skills of specializing in problems we don't understand. That's largely what the framing practicum material is for - this is what a "textbook" looks like for fields where we don't yet know what the textbook should contain, because figuring out the right framing tools is itself part of the problem.

Comment by johnswentworth on Welcome & FAQ! · 2021-08-25T00:07:41.201Z · LW · GW

I recommend that the title make it clearer that non-members can now submit alignment forum content for review, since this post is cross-posted on LW.

Comment by johnswentworth on Generator Systems: Coincident Constraints · 2021-08-24T21:14:09.923Z · LW · GW

This a great and under-appreciated point.

Comment by johnswentworth on Framing Practicum: Dynamic Equilibrium · 2021-08-24T21:06:12.820Z · LW · GW

Aha! This makes more sense now. Thanks for chasing that down, I feel much less confused.

Comment by johnswentworth on Framing Practicum: Dynamic Equilibrium · 2021-08-24T17:58:06.008Z · LW · GW

Huh. Now I am confused. Why is a cell which turns over on a timescale of months so over-represented in turnover? Skin cells, for instance, turn over at least that fast and should be at least as numerous.

Comment by johnswentworth on Framing Practicum: Timescale Separation · 2021-08-24T17:01:58.219Z · LW · GW

Thanks, got it.

Comment by johnswentworth on Framing Practicum: Dynamic Equilibrium · 2021-08-24T17:01:16.122Z · LW · GW

#1 in particular is definitely a useful frame to use in practice.

Comment by johnswentworth on Framing Practicum: Dynamic Equilibrium · 2021-08-24T16:56:41.039Z · LW · GW

I'd guess blood cells and neutrophils dominate turnover largely because there's so many of them; IIRC blood cells turn over on a timescale of months, which isn't especially fast. The stomach lining presumably turns over very quickly because it's exposed to extreme chemical stress (mitigated by a mucus layer, but that can only do so much), so I'd guess that's the dominant "gut cell" term.

That's an interesting thing to know because it tells us what processes are likely to eat up bodily resources, aside from obvious things like moving muscles or firing neurons.

Comment by johnswentworth on Framing Practicum: Bistability · 2021-08-24T16:44:04.079Z · LW · GW

I find 2 particularly interesting, because it matches my experience, but I have no idea what mechanism drives the system into discrete-ish states. Now I think about it, clouds seem related: we often see a "partly cloudy" sky with lots of discrete clouds scattered around and empty space between them, rather than a uniform less-concentrated cloudiness throughout the sky. That suggests bistability in cloud formation. What's up with that?

Comment by johnswentworth on Framing Practicum: Bistability · 2021-08-24T16:40:13.958Z · LW · GW

Good economic examples in 2 & 3. I find 1 particularly novel - it works surprisingly well as a bistable equilibrium, with the "in recycling bin" equilibrium more stable. If a container is near that state - e.g. it's mostly empty and sitting around in some random spot - then usually someone will empty it and throw it in the bin.

Comment by johnswentworth on Framing Practicum: Stable Equilibrium · 2021-08-24T16:33:47.980Z · LW · GW

I particularly like 2 & 3 - they evoke great visualizations in my head. I imagine a fast-forwarded video showing things appearing and disappearing, but density staying at roughly the same level over time.

Comment by johnswentworth on What fraction of breakthrough COVID cases are attributable to low antibody count? · 2021-08-23T03:31:20.924Z · LW · GW

Yeah, I'm not really imagining a lab test on the matter. It seems like the sort of thing where someone with the right dataset could do some clever math and back out a reasonable estimate.

Comment by johnswentworth on What fraction of breakthrough COVID cases are attributable to low antibody count? · 2021-08-23T03:29:49.734Z · LW · GW

Great analysis, though it's narrower than what I originally had in mind. The question didn't really nail down one use-case, so here's a few other possibilities:

  • One advantage of the testing approach is that, if antibody counts are high, it potentially offers very high confidence in immunity. Whereas when taking vaccines blindly, one needs potentially quite a few to attain high certainty (especially since people who have low antibody counts after the first couple are more likely to have something going on which messes up later doses too). So, if someone is really paranoid and wants high certainty, the test approach is potentially cheaper.
  • Similarly, if one wants to orchestrate a large group of people being together, then very high confidence in immunity is potentially valuable.
  • On a personal level, if government restrictions prevent multiple boosters, a test could be useful for deciding what level of precautions to take.

Mostly I'm thinking about this as an individual/private group strategy, not as a whole-population thing. Though even at the whole-population level, I do think there would be a lot of value in being able to say "do X, and once the test passes you can completely stop all these annoying precautions without having any significant chance of catching COVID". (As opposed to what we have now, where a lot of vaccine hesitancy comes from "but I can still catch COVID even with the vaccine".)

Comment by johnswentworth on Framing Practicum: Stable Equilibrium · 2021-08-20T21:18:08.795Z · LW · GW

Good insights. The inside/outside assignment becomes especially important when we have have multiple processes which equilibrate at different timescales - e.g. a commodity price may have both a short-term equilibrium (which just balances near-term supply and demand) and a long-term equilibrium (in which new buyers/sellers start businesses/shut down businesses in response to prices). In that situation, we explicitly declare the long-term changes to be "outside" (aka "exogenous") when analyzing the short-term equilibrium.

Comment by johnswentworth on Framing Practicum: Dynamic Equilibrium · 2021-08-20T00:54:42.677Z · LW · GW

Lots of great economic examples here. #2 in particular makes some great points about incentives inducing an equilibrium, in ways that a lot of overly-simple economic models wouldn't capture very well.

Comment by johnswentworth on A Modest Proposal: Logging for Environmentalists · 2021-08-18T22:24:57.790Z · LW · GW

I was thinking something similar recently. I'd add that you could go beyond wood - crop offal more generally should be very very cheap, and we basically just need to bury it someplace where the carbon won't leak out quickly.

We already use biomass to generate energy (expensive/inefficient), and energy to capture carbon (also expensive/inefficient); it seems like using biomass for carbon capture should be way cheaper/more efficient.