Comment by johnswentworth on Declarative Mathematics · 2019-04-19T16:44:53.775Z · score: 2 (1 votes) · LW · GW

Perhaps the difference is what you're imagining as "under the hood". Nobody wants to think about the axiom of choice when solving a differential equation.

Comment by johnswentworth on The Simple Solow Model of Software Engineering · 2019-04-11T16:45:45.694Z · score: 8 (4 votes) · LW · GW

Possible point of confusion: equilibrium does not imply static equilibrium.

If a firm can't find someone to maintain their COBOL accounting software, and decides to scrap the old mainframe and have someone write a new piece of software with similar functionality but on modern infra, then that's functionally the same as replacement due to depreciation.

If that sort of thing happens regularly, then we have a dynamic equilibrium. As an analogy, consider the human body: all of our red blood cells are replaced every couple of months, yet the total number of red blood cells is in equilibrium. Replacement balances removal. Most cell types in the human body are regularly replaced this way, at varying timescales.

That's the sort of equilibrium we're talking about here. It's not that the same software sticks around needing maintenance forever; it's that software is constantly repaired or replaced, but mostly provides the same functionality.

Comment by johnswentworth on The Simple Solow Model of Software Engineering · 2019-04-10T23:02:40.833Z · score: 6 (3 votes) · LW · GW

Yeah, in retrospect I should have been more clear about that. Thanks for drawing attention to it, other people probably interpreted it the same way you did.

Comment by johnswentworth on The Simple Solow Model of Software Engineering · 2019-04-10T21:17:39.438Z · score: 4 (2 votes) · LW · GW

Sounds like you're mostly talking about ops, which is a different beast.

An example from my previous job, to illustrate the sort of things I'm talking about: we had a mortgage app, so we called a credit report api, an api to get house data from an address, and an api to pull current pricing from the mortgage securities market (there were others, but those three were the most important). Within a six-month span, the first two apis made various small breaking changes to the format returned, and the third was shut down altogether and had to be switched to a new service.

(We also had the whole backend setup on Kubernetes, and breaking changes there were pretty rare. But as long as the infrastructure is working, it's tangential to most engineers' day-to-day work; the bulk of the code is not infrastructure/process related. Though I suppose we did have a slew of new bugs every time anything in the stack "upgraded" to a new version.)

Comment by johnswentworth on The Simple Solow Model of Software Engineering · 2019-04-10T17:49:35.094Z · score: 6 (3 votes) · LW · GW

Good points, this gets more into the details of the relevant models. The short answer is that capital equilibrates on a faster timescale than growth happens.

About a year ago I did some research into where most capital investments in the US end up - i.e. what the major capital sinks are. The major items are:

  • infrastructure: power grid, roads, railroads, data transmission, pipelines, etc.
  • oil wells
  • buildings (both residential and commercial)
  • vehicles

Most of the things on that list need constant repair/replacement, and aren't expanding much over time. The power grid, roads and other infrastructure (excluding data transmission) currently grow at a similar rate to the population, whereas they need repair/replacement at a much faster rate - so most of the invested capital goes to repair/replacement. Same for oil wells: shale deposits (which absorbed massive capital investments over the past decade) see well production drop off sharply after about two years. After that, they get replaced by new wells nearby. Vehicles follow a similar story to infrastructure: total number of vehicles grows at a similar rate to the population, but they wear out much faster than a human lifetime, so most vehicle purchases are replacements of old vehicles.

Now, this doesn't mean that people "prioritize maintenance above new stuff"; replacement of an old capital asset serves the same economic role as repairing the old one. But it does mean that capital mostly goes to repair/replace rather than growth.

Since capital equilibrates on a faster timescale than growth, growth must be driven by other factors - notably innovation and population growth. In the context of a software company, population growth (i.e. growing engineer headcount) is the big one. Few companies can constantly add new features without either adding new engineers or abandoning old products/features. (To the extent that companies abandon old products/features in order to develop new ones, that would be economic innovation, at least if there's net gain.)

Comment by johnswentworth on The Simple Solow Model of Software Engineering · 2019-04-10T05:32:28.567Z · score: 2 (1 votes) · LW · GW

Yeah, I used to have conversations like this all the time with my boss. Practically anything in an economics textbook can be applied to management in general, and software engineering in particular, with a little thought. Price theory and coordination games were most relevant to my previous job (at a mortgage-tech startup).

The Simple Solow Model of Software Engineering

2019-04-08T23:06:41.327Z · score: 26 (10 votes)
Comment by johnswentworth on How good is a human's gut judgement at guessing someone's IQ? · 2019-04-07T16:09:50.997Z · score: 2 (1 votes) · LW · GW

Yeah, sorry, I should have been more clear there. The mechanisms which trade off immune strength against other things are the underlying cause. Testosterone levels in males are a good example - higher testosterone increases attractiveness, physical strength, and spatial-visual reasoning, but it's an immune suppressor.

Comment by johnswentworth on Asymptotically Benign AGI · 2019-04-02T03:27:21.396Z · score: 3 (2 votes) · LW · GW

We cannot "prove" that something is physically impossible, only that it is impossible under some model of physics. Normally that distinction would be entirely irrelevant, but when dealing with a superintelligent AI, it's quite likely to understand the physics better than we do. For all we know, it may turn out that Alcubierre drives are possible, and if so then the AI could definitely break out that way and would have an incentive to do so.

I agree that the AI is not really boxed here; it's the "myopia" that makes the difference. But one of two things should generally be true:

  • The AI doesn't want to get out of the box, in which case the box doesn't need to be secure in the first place.
  • The AI cannot get out of the box, in which case the AI doesn't need to be safe (but also won't be very useful).

This case seems like the former, so long as hacking the human is easier than getting out of the box. But that means we don't need to make the box perfect anyway.

Comment by johnswentworth on Asymptotically Benign AGI · 2019-04-01T16:24:38.322Z · score: 2 (1 votes) · LW · GW

Can you expand a bit on why a commitment to give a high reward won't save us? Is it a matter of the AI seeking more certainty, or is there some other issue?

Comment by johnswentworth on Asymptotically Benign AGI · 2019-04-01T16:23:02.467Z · score: 2 (1 votes) · LW · GW

This is pretty thorough from an engineering standpoint. I still would not at all trust such a box to hold a superintelligent AI. The unknown unknowns remain too big a factor. Just because an unknown information channel does not have any humans listening to it directly, does not mean that it can't be used to push around matter, or couple to some other channel, or otherwise leave an opening.

Also, the human and their clothes are definitely not the only potentially-useful tools in the box: there's also the computer, the computer's power supply, whatever's cooling the room, whatever's keeping the air fresh, whatever's used for input, and whatever's used for output. If any of those things were outside the room, then they would offer a side channel for information to escape. They can be buried in concrete, but they'll still need to be within the vacuums and whatnot.

In general, trying to outsmart a superintelligent AI is a Bad Idea, and building an AI-proof box definitely falls in that category. If an AI is not safe with arbitrarily large amounts of power, then it is not safe, full stop.

Comment by johnswentworth on Asymptotically Benign AGI · 2019-04-01T07:16:18.578Z · score: 2 (1 votes) · LW · GW

So I think this proposal is a specific instance of a general principle: wireheading acts as a safety valve. As long as the AI is prone to wireheading, and you leave it some path to wireheading, the AI will never bother with anything more difficult than that path. In this particular case, it will never do anything harder than hacking its human operator, and that's where most of the safety benefit comes from.

This kind of setup creates a direct trade-off between safety and capability: the harder it is to hack the human operator, the more likely that some other escape channel is easier - manipulating electrical circuitry to push atoms around, or some weird thing like that. The AI will become an existential threat if you send in an operator who really doesn't want to give it 100% reward, and make that operator hard to hack - then the AI has a strong incentive to gain whatever power it needs. (I definitely do not trust a door with a kill switch to keep a superintelligent AI in a box.)

My intuition says that nothing else in the proposal actually matters for safety, except the wireheading safety valve aspect. In particular, I think just giving the AI direct exploration abilities won't mess up safety, as long as the wireheading path is present and there's not "99.99% certainty is better than 99.98%"-type failure mode.

Comment by johnswentworth on Asymptotically Benign AGI · 2019-04-01T02:03:05.802Z · score: 2 (1 votes) · LW · GW
The reward belongs to a finite set of rationals between 0 and 1.

Once the AI becomes sufficiently powerful, it is definitely going to hack the operator any time it expects the operator to give a reward less than 1. So the operator's input is really binary, at least once the AI has learned an accurate model. Given that, why allow non-binary rewards at all? Is it just supposed to provide faster learning early on?

Along similar lines: once the AI has learned an accurate model, why would we expect it to ever provide anything useful at all, rather than just hacking the operator all day? Do we think that hacking the human is likely to be harder than obtaining perfect rewards every time without hacking the human? Seems like that would depend very heavily on the problem at hand, and on the operator's feedback strategy.

To put it differently: this setup will not provide a solution to any problem which is more difficult than hacking the human operator.

Comment by johnswentworth on Review of Q&A [LW2.0 internal document] · 2019-03-30T00:46:22.918Z · score: 17 (7 votes) · LW · GW

Two minor comments:

  • It would be nice for other people to be able to throw more into the bounty pool somehow.
  • A shorter-term, smaller-scale thing-to-try on the bounty front might be to make karma transferable, and let people create karma bounties. That would avoid having to deal with money.
Comment by johnswentworth on Review of Q&A [LW2.0 internal document] · 2019-03-30T00:39:53.466Z · score: 6 (3 votes) · LW · GW

Totally on board with that. The important point is eliminating risk-management overhead by (usually) only having to reward someone who contributes value, in hindsight.

Comment by johnswentworth on Review of Q&A [LW2.0 internal document] · 2019-03-30T00:12:56.166Z · score: 6 (3 votes) · LW · GW

I posted a handful of example questions I have been/would be interested in on Raemon's bounty question. I think these examples address several of the challenges in section 4:

  • All of them are questions which I'd expect many people on lesswrong to be independently interested in, and which would make great blog-post-material. So the bounties wouldn't be the sole incentive; even those who don't get the bounty are likely to derive some value from the exercise.
  • There's not really a trust/credentials problem; it would be easy for me to tell whether a given response had been competently executed. If I needed to hire someone in advance of seeing their answer, then there would be a trust problem, but the best-answer-gets-paid format mostly solves that. Even if there are zero competent responses, I've bought useful information: I've learned that the question is tougher than I thought. Also, they're all the sort of project that I expect a smart, generally-educated non-expert to be able to execute.
  • They are all motivated by deeper/vaguer questions, but answers to the questions as stated have enough value to justify themselves. Directly answering the deeper questions would not be the objective of any of them.
  • They're all sufficiently clean-cut that I wouldn't expect much feedback to be necessary mid-effort.

I see that second point as the biggest advantage of bounties over a marketplace: just paying for the best answer means I don't need to go to the effort of finding someone who's competent and motivated and so forth. I don't need to babysit someone while they work on the problem, to make sure we're always on the same page. I don't need to establish careful criteria for what counts as "finished". I can just throw out my question, declare a bounty, and move on. That's a much lower-effort investment on the asker's side than a marketplace.

In short, with a bounty system, competition between answerers solves most of the trust problems which would otherwise require lots of pre-screening and detailed contract specifications.

Bounties will also likely need to be higher to compensate for answer-side risk, but that's a very worthwhile tradeoff for those of us who have some money and don't want to deal with hiring and contracts and other forms of baby-sitting.

Comment by johnswentworth on What would you need to be motivated to answer "hard" LW questions? · 2019-03-29T21:56:32.404Z · score: 23 (5 votes) · LW · GW

I would like more concrete examples of nontrivial questions people might be interested in. Too much of this conversation is too abstract, and I worry people are imagining different things.

Toward that end, here are a few research projects I've either taken on or considered, which I would have been happy to outsource and which seem like a good fit for the format:

  • Go through the data on spending by US colleges. Look at how much is actually charged per student (including a comparison of sticker price to actual tuition), how much is spent per student, and where all the money is spent. Graph how these have changed over time, to figure out exactly which expenditures account for the rapid growth of college cost. Where is all the extra money going? (I've done this one; results here.)
  • Go through the data on aggregate financial assets held, and on real capital assets held by private citizens/public companies/the state (i.e. patents, equipment, property, buildings, etc) to find out where money invested ultimately ends up. What are the main capital sinks in the US economy? Where do marginal capital investments go? (I've also done this one, but haven't gotten around to writing it up.)
  • Go through the genes of JCVI's minimal cell, and write up an accessible explanation of the (known) functionality of all of its genes (grouping them into pathways/systems as needed). The idea is to give someone with minimal bio background a comprehensive knowledge of everything needed for bare-minimum life. Some of this will have to be speculative, since not all gene functions are known, but a closed list of known-unknowns sure beats unknown-unknowns.
  • Something like Laura Deming's longevity FAQ, but focused on the macro rather than micro side of what's known - i.e. (macroscopic) physiology of vascular calcification and heart disease, alzheimers, cancer, and maybe a bit on statistical models of old-age survival rates. In general, there seems to be lots of research on the micro side, lots known on the macro side, but few-if-any well-understood mechanistic links from tone to the other; so understanding both sides in depth is likely to have value.
  • An accessible explanation of Cox' Theorem, especially what each piece means. The tough part: include a few examples in which a non-obvious interpretation of a system as a probabilistic model is directly derived via Cox' Theorem. I have tried to write this at least four separate times, and the examples part in particular seems like a great exercise for people interested in embedded agency.
Comment by johnswentworth on Parable of the flooding mountain range · 2019-03-29T21:17:07.119Z · score: 5 (3 votes) · LW · GW

Reading this, I figured you were talking about local-descent-type optimization algorithms, i.e. gradient descent and variants.

From that perspective, there's two really important pieces missing from these analogies:

  • The mountaineers can presumably backtrack, at least as long as the water remains low enough
  • The mountaineers can presumably communicate

With backtracking, even a lone mountaineer can do better sometimes (and never any worse) by continuing to explore after reaching the top of a hill - as long as he keeps an eye on the water, and makes sure he has time to get back. In an algorithmic context, this just means keeping track of the best point seen, while continuing to explore.

With backtracking and communication, the mountaineers can each go explore independently, then all come back and compare notes (again keeping track of water etc), all go to the highest point found, and maybe even repeat that process. In an algorithmic context, this just means spinning off some extra threads, then taking the best result found by any of them.

In an evolutionary context, those pieces are probably not so relevant.

Comment by johnswentworth on Declarative Mathematics · 2019-03-23T19:30:53.365Z · score: 4 (2 votes) · LW · GW

Those are great examples! That's exactly the sort of thing I see the tools currently associated with neural nets being most useful for long term - applications which aren't really neural nets at all. Automated differentiation and optimization aren't specific to neural nets, they're generic mathematical tools. The neural network community just happens to be the main group developing them.

I really look forward to the day when I can bust out a standard DE solver, use it to estimate the frequency of some stable nonlinear oscillator, and then compute the sensitivity of that frequency to each of the DE's parameters with an extra two lines of code.

Comment by johnswentworth on Declarative Mathematics · 2019-03-23T07:27:38.177Z · score: 5 (3 votes) · LW · GW

Geometric algebra is really neat, thanks for the links. I've been looking for something like that since I first encountered Pauli matrices back in quantum. I would describe it as an improved language for talking about lots of physical phenomena; that makes it a component of a potentially-better interface layer for many different mathematical frameworks. That's really the key to a successful declarative framework: having an interface layer/language which makes it easy to recognize and precisely formulate the kinds of problems the framework can handle.

I'm generally suspicious of anything combining neural nets and nonlinear DEs. As James Mickens would say, putting a numerical solver for a chaotic system in the middle of another already-notoriously-finicky-and-often-unstable-system is like asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. This does not lead to rising property values in Tokyo! That said, it does seem like something along those lines will have to work inside learning frameworks sooner or later, so it's cool to see a nice implementation that puts everything under one roof.

Declarative Mathematics

2019-03-21T19:05:08.688Z · score: 57 (24 votes)
Comment by johnswentworth on Asking for help teaching a critical thinking class. · 2019-03-07T04:10:31.954Z · score: 9 (4 votes) · LW · GW

There used to be a set of Walter Lewin's physics 101 lectures on MIT opencourseware; they're probably still floating around on youtube somewhere. The very first lecture, he explained that his grandmother used to argue people were taller lying down that standing up - y'know, because there's less weight compacting them when lying down. And of course this is completely ridiculous, but he does the experiment anyway: carefully measures the height of a student lying down, then standing up. On the surface, he's using this to illustrate the importance of tracking measurement uncertainty, but the ultimate message is epistemic: turns out people are a bit shorter standing up.

He talks about this as an example of why we need to carefully quantify uncertainty, but it's a great example for epistemological hygiene more generally. It sounds like something ridiculous and low-status to believe, a "crazy old people" sort of thing, but it's not really that implausible on its own merits - and it turns out to be true.

Anyway, besides that one example, I'd say it's generally easy to make people believe something quickly just by insinuating that the alternative hypothesis is somehow low-status, something which only weird people believe. Heck, whole scientific fields have fallen for that sort of trick for decades at a time - behaviorism, frequentism, Copenhagen interpretation... Students will likely be even more prone to it, since they're trained to tie epistemics to status: "truth" in school is whatever gets you a gold star when you repeat it back to the teacher.

Comment by johnswentworth on Unconscious Economies · 2019-02-27T17:05:54.511Z · score: 14 (6 votes) · LW · GW

Thanks for writing this. Multiple times I've looked for a compact, self-contained explanation of this idea, thinking "surely it's common knowledge within econ?".

Comment by johnswentworth on How good is a human's gut judgement at guessing someone's IQ? · 2019-02-27T01:12:31.494Z · score: 4 (2 votes) · LW · GW

It's been a few years, so I don't have sources to cite, but I remember looking into this at one point and finding that immune health during developmental years is a common major underlying cause for intelligence, attractiveness, physical fitness, and so forth. This makes a lot of sense from an evolutionary standpoint: infectious disease used to be the #1 killer, so immune health would be the thing which everything else traded off against.

One consequence is that things like e.g. attractiveness and intelligence actually do positively correlate, so peoples' halo-effect estimates actually do work, to some extent.

Comment by johnswentworth on Constructing Goodhart · 2019-02-14T18:37:52.423Z · score: 2 (1 votes) · LW · GW

The problem is that you invoke the idea that it's starting from something close to pareto-optimal. But pareto optimal with respect to what? Pareto optimality implies a multi-objective problem, and it's not clear what those objectives are. That's why we need the whole causality framework: the multiple objectives are internal nodes of the DAG.

The standard description of overfitting does fit into the DAG model, but most of the usual solutions to that problem are specific to overfitting; they don't generalize to Goodhart problems in e.g. management.

Comment by johnswentworth on When should we expect the education bubble to pop? How can we short it? · 2019-02-10T01:29:52.651Z · score: 11 (6 votes) · LW · GW

That model works, but it requires irrational agents to make it work. The bubble isn't really "stable" in a game-theoretic equilibrium sense; it's made stable by assuming that some of the actors aren't rational game-theoretic agents. So it isn't a true Nash equilibrium unless you omit all those irrational agents.

The fundamental difference with a signalling arms race is that the model holds up even without any agent behaving irrationally.

That distinction cashes out in expectations about whether we should be able to find ways to profit. In a market bubble, even if it's propped up by irrational investors, we expect to be able to find ways around that liquidity problem - like shorting options or taking opposite positions on near-substitute assets. If there's irrational agents in the mix, it shouldn't be surprising to find clever ways to relieve them of their money. But if everyone is behaving rationally, if the equilibrium is a true Nash equilibrium, then we should not expect to find some clever way to do better. That's the point of equilibria, after all.

Comment by johnswentworth on When should we expect the education bubble to pop? How can we short it? · 2019-02-09T22:10:50.980Z · score: 7 (4 votes) · LW · GW

It could pop under political pressure to allow student loan forgiveness, and indeed I've heard plenty of people who want exactly that.

Comment by johnswentworth on When should we expect the education bubble to pop? How can we short it? · 2019-02-09T22:08:33.272Z · score: 10 (7 votes) · LW · GW

If we buy into Bryan Caplan's model, then it's not really a bubble so much as zero-sum arms race. It's less like tulips, and more like keeping up with the Joneses. Keeping up with the Joneses doesn't pop; it's a stable phenomenon.

In the case of education, people who are diligent/smart/conformist get a degree, employers mostly want to hire those people, so then everyone else tries to get a degree in order to keep up, and the diligent/smart/conformist people then have to get more degrees to stand out. That's a signalling arms race, but it's stable: nobody gains by doing something else.

We shouldn't expect to find ways to "short the bubble" for exactly that reason: it's stable. If there were ways to gain by shorting, then it wouldn't be stable. Sure, we'd all be better off if we all agreed to less education, but the Nash equilibrium is everyone defecting. Policy position for 2020: ban higher education!

Comment by johnswentworth on Constructing Goodhart · 2019-02-04T00:21:00.082Z · score: 3 (2 votes) · LW · GW

The problem with is that it's not clear why would seem like a good proxy in the first place. With an inequality constraint, has positive correlation with the objective everywhere except the boundary. You get at this idea with knowing only , but I think it's more a property of dimensionality than of objective complexity - even with a complicated objective, it's usually easy to tell how to change a single variable to improve the objective if everything else is held constant.

It's the "held constant" part that really matters - changing one variable while holding all else constant only makes sense in the interior of the set, so it runs into Goodhart-type tradeoffs once you hit the boundary. But you still need the interior in order for the proxy to look good in the first place.

Comment by johnswentworth on Constructing Goodhart · 2019-02-03T22:57:53.003Z · score: 4 (3 votes) · LW · GW

Assuming you mean , optimizing for , and using as the proxy, this is a pretty nice formulation. Then, increasing will improve the objective over most of the space, until we run into the boundary (a.k.a the pareto frontier), and then Goodhart kicks in. That's actually a really clean, simple formulation.

Constructing Goodhart

2019-02-03T21:59:53.785Z · score: 27 (10 votes)
Comment by johnswentworth on How does Gradient Descent Interact with Goodhart? · 2019-02-03T20:32:01.897Z · score: 7 (4 votes) · LW · GW

Another piece I'd guess is relevant here is generalized efficient markets. If you generate a DAG and start out with random parameters, then start optimizing for a proxy node right away, then you're not going to be near any sort of pareto frontier, so trade-offs won't be an issue. You won't see a Goodhart effect.

In practice, most of the systems we deal with already have some optimization pressure. They may not be optimal for our main objective, but they'll at least be pareto-optimal for any cross-section of nodes. Physically, that's because people do just fine locally optimizing whatever node they're in charge of - it's the nonlocal tradeoffs between distant nodes that are tough to deal with (at least without competitive price mechanisms).

So if you want to see Goodhart effects, first you have to push up to that pareto frontier. Otherwise, changes applied to optimize the proxy are not going to have systematically negative impact on other nodes in parallel to the proxy; the impacts will just be random.

Comment by johnswentworth on How does Gradient Descent Interact with Goodhart? · 2019-02-02T08:06:11.527Z · score: 8 (5 votes) · LW · GW

If we want to think about reasonably realistic Goodhart issues, random functions on seem like the wrong setting. John Maxwell put it nicely in his answer:

If your proxy consists of something you're trying to maximize plus unrelated noise that's roughly constant in magnitude, you're still best off maximizing the heck out of that proxy, because the very highest value of the proxy will tend to be a point where the noise is high and the thing you're trying to maximize is also high.

That intuition is easy to formalize: we have our "true" objective that we want to maximize, but we can only observe plus some (differentiable) systematic error . Assuming we don't have any useful knowledge about that error, the expected value given our information will still be maximized when is maximized. There is no Goodhart.

I'd think about it on a causal DAG instead. In practice, the way Goodhart usually pops up is that we have some deep, complicated causal DAG which determines some output we really want to optimize. We notice that some node in the middle of that DAG is highly predictive of happy outputs, so we optimize for that thing as a proxy. If our proxy were a bottleneck in the DAG - i.e. it's on every possible path from inputs to output - then that would work just fine. But in practice, there are other nodes in parallel to the proxy which also matter for the output. By optimizing for the proxy, we accept trade-offs which harm nodes in parallel to it, which potentially adds up to net-harmful effect on the output.

For example, there's the old story about soviet nail factories evaluated on number of nails made, and producing huge numbers of tiny useless nails. We really want to optimize something like the total economic value of nails produced. There's some complicated causal network leading from the factory's inputs to the economic value of its outputs. If we pick a specific cross-section of that network, we might find that economic value is mediated by number of nails, size, strength, and so forth. If we then choose number of nails as a proxy, then the factories trade off number of nails against any other nodes in that cross-section. But we'll also see optimization pressure in the right direction for any nodes which effect number of nails without effecting any of those other variables.

So that at least gives us a workable formalization, but we haven't really answered the question yet. I'm gonna chew on it some more; hopefully this formulation will be helpful to others.

Comment by johnswentworth on What kind of information would serve as the best evidence for resolving the debate of whether a centrist or leftist Democratic nominee is likelier to take the White House in 2020? · 2019-02-01T19:35:20.076Z · score: 8 (6 votes) · LW · GW

What you want from a prediction market is not the chance of a given candidate winning the presidency, but the chance of a given candidate winning the presidency if they win the nomination. So, for each of the listed democratic candidates, take Predictit's probability that they win the presidency and divide that by Predictit's probability that they win the nomination: P[presidency | nomination] = P[presidency & nomination] / P[nomination] = P[presidency]/P[nomination], ignoring the chance that someone gets elected president without winning the nomination.

Just looking at the most recently traded prices, I see:

  • Harris: .19/.24 = .79
  • Biden: .12/.16 = .75
  • Warren: .07/.10 = .70
  • Sanders: .10/.15 = .67
  • Brown: .07/.11 = .64
  • O'Rourke: .08/.13 = .62
  • Booker: .06/.10 = .60

That said, the price spreads are ridiculously wide and the trade volume is a trickle, so the error bars on all of those implied probabilities are huge. We'll probably get tighter estimates this time next year.

Comment by johnswentworth on From Personal to Prison Gangs: Enforcing Prosocial Behavior · 2019-01-26T19:10:01.197Z · score: 2 (1 votes) · LW · GW

Iterated prisoners' dilemma is used to model the breakdown of reputation. Roughly speaking, when the interaction count is high, there's plenty of time to realize you're playing against a defector and to punish them, so defectors don't do very well - that's a reputation system in action. But as the interaction count gets lower, defectors can "hit-and-run", so they flourish, and the reputation system breaks down. The link goes into all of this in much more depth.

Dunbar just comes in as a (very) rough estimate for where the transition point occurs.

Comment by johnswentworth on From Personal to Prison Gangs: Enforcing Prosocial Behavior · 2019-01-26T06:39:58.004Z · score: 2 (1 votes) · LW · GW

Yes! Click the link that says "you should click that link, it's really cool".

From Personal to Prison Gangs: Enforcing Prosocial Behavior

2019-01-24T18:07:33.262Z · score: 78 (27 votes)
Comment by johnswentworth on In what way has the generation after us "gone too far"? · 2019-01-24T16:13:58.625Z · score: 9 (5 votes) · LW · GW

Deep learning and tensorflow. Dear god. These days, every freshman with a semester of python under their belt thinks they can "do machine learning" while barely understanding calculus, much less probability. When I was their age, we had to code our downhill gradients by hand, in both directions. You wanted it on a GPU? You wrote shader code!

Comment by johnswentworth on The Relationship Between Hierarchy and Wealth · 2019-01-23T19:34:32.472Z · score: 9 (4 votes) · LW · GW

This was a great post, interesting topic and tons of relevant facts.

One criticism: it seems to slice the world along socially salient lines rather than causal mediators. For instance, a bunch of stuff gets glommed into "freedom", much of which doesn't seem very related - "freedom" seems like an unnatural category for purposes of this discussion. That makes claims like "freedom causes poverty" kinda tough to interpret.

If we're asking "what causes hierarchy?", then I'd expect the root answer to be "large-scale coordination problems with low communication requirements", followed by various conditions which tend to induce those kinds of problems. For instance:

  • large demand for capital-intensive goods (e.g. irrigation, roads, other infrastructure)
  • natural monopolies (including military)
  • large heavily-mixed populations, which tend to induce low trust/high defection, messing up market coordination mechanisms
  • increasing social connectedness
  • increasing economic specialization

The various case-studies mentioned in the post sound like they offer a lot of evidence about which conditions are more/less relevant to hierarchy formation. But the discussion doesn't really slice it like that, so we're left without even knowing which way the causal arrows point.

Comment by johnswentworth on The E-Coli Test for AI Alignment · 2019-01-18T01:26:32.702Z · score: 2 (1 votes) · LW · GW

I agree that an e-coli's lack of reflective capability makes it useless for reasoning directly about iterated amplification or anything like it.

On the other hand, if we lack the tools to think about the values of a simple single-celled organism, then presumably we also lack the tools to think about whether amplification-style processes actually converge to something in line with human values.

Comment by johnswentworth on What makes people intellectually active? · 2019-01-03T00:13:17.618Z · score: 6 (4 votes) · LW · GW

As Kaj pointed out, most of the answers so far focus on feedback and reward. As an answer, that feels correct, but incomplete. I know so many people who are clearly very smart, surrounded by friends who give them positive feedback on whatever they're doing, but it doesn't end up channeling into intellectual development. If every intellectually-active person were linked to an idea-focused community, then the feedback answer would make sense, but I doubt that's the case. So what's missing?

I don't have a complete answer, but I remember a quote (maybe from Feynman?) about keeping a stock of unsolved problems in your head. Whenever you learn some new trick or method, you try applying it to one of those unsolved problems. At least for me, that's mostly how my "sprawling intellectual framework" develops. Some of them are open technical problems, others are deficits in my current social or economic models of the world. This feels connected to what Martin talks about - some people notice holes in their understanding and then keep an eye out for solutions. You hear something that doesn't sound right, doesn't quite make sense, and you reflexively start digging. Maybe you find an answer quickly, otherwise you carry the problem around in the back of your head.

I don't know why some people do this and others don't, but as a causal factor, it feels orthogonal to social feedback. It still feels like I don't have all the puzzle pieces, though. This question will continue to sit in the back of my head.

Comment by johnswentworth on What makes people intellectually active? · 2018-12-30T15:37:30.303Z · score: 5 (3 votes) · LW · GW

I think Martin's describing something more like "curiosity" than OCD. It's not obsessing over the problem so much as finding the problem interesting, wondering whether there's more to it, digging deeper.

Comment by johnswentworth on In Defense of Finance · 2018-12-18T04:37:50.517Z · score: 12 (5 votes) · LW · GW

One good reason why risk preference would be bimodal: the Volker rule. Banks are generally prohibited and/or penalized for holding riskier asset classes. Both regulations and intrabank risk rules stipulate maximum leverage ratios for each asset class. Meanwhile, non-banks usually just can't get leverage ratios anywhere near what banks get, at all.

So, you get one class of investors (banks) who use high leverage to buy safe assets, pushing their return down very low. The returns on those assets are then too low for non-banks to hold them in large quantities, so non-banks hold the riskier stuff with bimodal returns.

I don't know how well this represents reality, but that's how I've thought about it for a while now.

Comment by johnswentworth on The E-Coli Test for AI Alignment · 2018-12-16T19:16:18.718Z · score: 4 (3 votes) · LW · GW

First two yes, last one no. There is a communication gap in any case, and crossing that communication gap is ultimately the AI's job. Answering questions will look different in the two cases: maybe typing yes/no at a prompt vs swimming up one of two channels on a microfluidic chip. But the point is, communication is itself a difficult problem, and an AI alignment method should account for that.

The E-Coli Test for AI Alignment

2018-12-16T08:10:50.502Z · score: 55 (20 votes)
Comment by johnswentworth on What is abstraction? · 2018-12-16T00:10:23.710Z · score: 28 (12 votes) · LW · GW

I've thought a lot about this exact problem, because it's central to a bunch of hard problems in biology, ML/AI, economics, and psychology/neuroscience (including embedded agents). I don't have a full answer yet, but I can at least give part of an answer.

First, the sort of abstraction used in pure math and CS is sort of an unusual corner case, because it's exact: it doesn't deal with the sort of fuzzy boundaries we see in the real world. "Tennis" is a fuzzy abstraction of many real-world activities, and there are edge cases which are sort-of-tennis-but-maybe-not. Most of the interesting problems involve non-exact abstraction, so I'll mostly talk about that, with the understanding that math/CS-style abstraction is just the case with zero fuzz.

I only know of one existing field which explicitly quantifies abstraction without needing hard edges: statistical mechanics. The heart of the field is things like "I have a huge number of tiny particles in a box, and I want to treat them as one abstract object which I'll call "gas". What properties will the gas have?" Jaynes puts the tools of statistical mechanics on foundations which can, in principle, be used for quantifying abstraction more generally. (I don't think Jaynes had all the puzzle pieces, but he had a lot more than anyone else I've read.) It's rather difficult to find good sources for learning stat mech the Jaynes way; Walter Grandy has a few great books, but they're not exactly intro-level.

Anyway, (my reading of) Jaynes' answer to the main question: abstraction is mainly about throwing away or ignoring information, in such away that we can still make strong predictions about some aspects of the underlying concrete system. Examples:

  • We have a gas consisting of some huge number of particles. We throw away information about the particles themselves, instead keeping just a few summary statistics: average energy, number of particles, etc. We can then make highly precise predictions about things like e.g. pressure just based on the reduced information we've kept, without having to think about each individual particle. That reduced information is the "abstract layer" - the gas and its properties.
  • We have a bunch of transistors and wires on a chip. We arrange them to perform some logical operation, like maybe a NAND gate. Then, we throw away information about the underlying details, and just treat it as an abstract logical NAND gate. Using just the abstract layer, we can make predictions about what outputs will result from what inputs. Even in this case, there's fuzz - 0.01 V and 0.02 V are both treated as logical zero, and in rare cases there will be enough noise in the wires to get an incorrect output.
  • I tell my friend that I'm going to play tennis. I have ignored a huge amount of information about the details of the activity - where, when, what racket, what ball, with whom, all the distributions of every microscopic particle involved - yet my friend can still make some strong predictions based on the abstract information I've provided.
  • When we abstract formulas like "1+1=2" or "2+2=4" into "n+n=2n", we're obviously throwing out information about the value of n, while still making whatever predictions we can given the information we kept. This is what generalization is all about in math and programming: throw out as much information as you can, while still maintaining the core "prediction".
  • Finally, abstract art. IMO, the quintessential abstract art pieces convey the idea of some thing without showing the thing itself - I remember one piece at SF's MOMA which looks like writing on a blackboard but contains no actual letters/numbers. In some sense, that piece is anti-abstract: it's throwing away information about our usual abstraction - the letters/numbers - but retaining the rest of the visual info of writing on a chalkboard. By doing so, it forces us to notice the abstraction process itself.
Comment by johnswentworth on Introducing the Longevity Research Institute · 2018-12-14T23:01:12.623Z · score: 4 (3 votes) · LW · GW

Is vium's data on long-lived mice something they're willing to share more generally? What kind of data do they have?

Comment by johnswentworth on On Rigorous Error Handling · 2018-11-17T23:34:28.277Z · score: 6 (4 votes) · LW · GW

I generally agree with the problem described, and I agree that "small amount of well-defined failure modes" is a necessary condition for the error codes to be useful. But that doesn't really tell us how to come up with a good set of errors. I'll suggest a more constructive error ontology.

When an error occurs, the programmer using the library mostly needs to know:

  • Is it my mistake, a bug in the library, or a hardware-level problem (e.g. connection issue)?
  • If it's my mistake, what did I do wrong?

Why these questions? Because these are the questions which determine what the programmer needs to do next. If you really want to keep the list of errors absolutely minimal, then three errors is not a bad starting point: bad input, internal bug, hardware issue. Many libraries won't even need all of these - e.g. non-network libraries probably don't need to worry about hardware issues at all.

Which of the three categories can benefit from more info, and what kind of additional info?

First, it is almost never a good idea to give more info on internal bugs, other than logging it somewhere for the library's maintainers to look at. Users of the library will very rarely care about why the library is broken; simply establish that it is indeed a bug and then move on.

For hardware problems, bad connection is probably the most ubiquitous. The user mostly just needs to know whether it's really a bad connection (e.g. comcast having a bad day) or really the user's mistake (e.g. input the wrong credentials). Most libraries probably only need at most one actual hardware error, but user mistakes masquerading as hardware problems are worth looking out for separately.

That just leaves user mistakes, a.k.a. bad inputs. This is the one category where it makes sense to give plenty of detail, because the user needs to know what to fix. Of course, communication is a central problem here: the whole point of this class of errors is to communicate to the programmer exactly how their input is flawed. So, undocumented numerical codes aren't really going to help.

(Amusingly, when I hit "submit" for this comment, I got "Network error: Failed to fetch". This error did its job: I immediately knew what the problem was, and what I needed to do to fix it.)

Comment by johnswentworth on Competitive Markets as Distributed Backprop · 2018-11-12T18:14:12.582Z · score: 6 (3 votes) · LW · GW

The original piece continues where this post leaves off to discuss how this logic applies inside the firm. The main takeaway there is that most firms do not have competitive internal resource markets, so each part of the company usually optimizes for some imperfect metric. The better those metrics approximate profit in competitive markets, the closer the company comes to maximizing overall profit. This model is harder to quantify, but we can predict that e.g. deep production pipelines will be less efficient than broad pipelines.

I'm still writing the piece on non-equilibrium markets. The information we get on how the market is out of equilibrium is rather odd, and doesn't neatly map to any other algorithm I know. The closest analogue would be message-passing algorithms for updating a Bayes net when new data comes in, but that analogy is more aesthetic than formal.

Comment by johnswentworth on Competitive Markets as Distributed Backprop · 2018-11-12T17:45:19.395Z · score: 4 (2 votes) · LW · GW

"Price = derivative" is certainly well-known. I haven't seen anyone else extend the connection to backprop before, but there's no way I'm first person to think of it.

Competitive Markets as Distributed Backprop

2018-11-10T16:47:37.622Z · score: 44 (16 votes)
Comment by johnswentworth on Bayes Questions · 2018-11-09T22:49:21.641Z · score: 2 (1 votes) · LW · GW

Ok, that sounds right.

Comment by johnswentworth on Bayes Questions · 2018-11-09T22:30:07.787Z · score: 2 (1 votes) · LW · GW

At what point is the data used?

Comment by johnswentworth on Real-time hiring with prediction markets · 2018-11-09T22:28:18.084Z · score: 10 (6 votes) · LW · GW

One hypothesis for why current hiring practices seem not-very-good: there's usually no feedback mechanism. There are sometimes obvious cases, where a hire ended up being really good or really bad, but there's no fine-grained way to measure how someone is doing - let alone how much value they add to the organization.

Any prediction market proposal to fix hiring first needs to solve that problem. You need a metric for performance, so you have a ground truth to use for determining bet pay-offs. And to work in practice, that metric also needs to get around Goodhart's Law somehow. (See here for a mathy explanation of roughly this problem.)

Now for the flip side: if we had an accurate, Goodhart-proof metric for employee performance, then we probably wouldn't need a fancy prediction market to utilize it. Don't get me wrong, a prediction market would be a very fast and efficient way to incorporate all the relevant info. But even a traditional HR department can probably figure out what they need to do in order to improve their metric, once they have a metric to improve.

Comment by johnswentworth on Bayes Questions · 2018-11-09T19:12:09.424Z · score: 2 (1 votes) · LW · GW

That sampling method sounds like it should work, assuming it's all implemented correctly (not sure what method you're using to sample from the posterior distribution of , ).

Worst case in a million being dominated by parameter uncertainty definitely makes sense, given the small sample size and the rate at which those distributions fall off.

Comment by johnswentworth on Bayes Questions · 2018-11-08T00:22:08.891Z · score: 3 (2 votes) · LW · GW

Having ~ten data points makes this way more interesting. That's exactly the kind of problem that I specialize in.

For the log-normal distribution, it should be possible to do the integral for explicitly. The integral is tractable for the normal distribution - it comes out proportional to a power of the sample variance - so just log-transform the data and use that. If you write down the integral for normal-distributed data explicitly and plug it into wolframalpha or something, it should be able to handle it. That would circumvent needing to sample and .

I don't know if there's a corresponding closed form for Birnbaum-Saunders; I had never even heard of it before this. The problem is still sufficiently low-dimensional that it would definitely be tractable computationally, but it would probably be a fair bit of work to code.

Two Kinds of Technology Change

2018-10-11T04:54:50.121Z · score: 61 (22 votes)

The Valley of Bad Theory

2018-10-06T03:06:03.532Z · score: 54 (26 votes)

Don't Get Distracted by the Boilerplate

2018-07-26T02:15:46.951Z · score: 44 (22 votes)

ISO: Name of Problem

2018-07-24T17:15:06.676Z · score: 32 (13 votes)

Letting Go III: Unilateral or GTFO

2018-07-10T06:26:34.411Z · score: 22 (7 votes)

Letting Go II: Understanding is Key

2018-07-03T04:08:44.638Z · score: 12 (3 votes)

The Power of Letting Go Part I: Examples

2018-06-29T01:19:03.474Z · score: 38 (15 votes)

Problem Solving with Mazes and Crayon

2018-06-19T06:15:13.081Z · score: 121 (55 votes)

Fun With DAGs

2018-05-13T19:35:49.014Z · score: 38 (15 votes)

The Epsilon Fallacy

2018-03-17T00:08:01.203Z · score: 68 (18 votes)

The Cause of Time

2013-10-05T02:56:46.150Z · score: 0 (19 votes)

Recent MIRI workshop results?

2013-07-16T01:25:02.704Z · score: 2 (7 votes)