Posts

Don't Make Your Problems Hide 2020-06-27T20:24:53.408Z · score: 51 (21 votes)
Map Errors: The Good, The Bad, and The Territory 2020-06-27T05:22:58.674Z · score: 24 (9 votes)
Negotiating With Yourself [Transcript] 2020-06-26T23:55:15.638Z · score: 22 (4 votes)
[Link] COVID-19 causing deadly blood clots in younger people 2020-04-27T21:02:32.945Z · score: 45 (16 votes)
Choosing the Zero Point 2020-04-06T23:44:02.083Z · score: 132 (55 votes)
The Real Standard 2020-03-30T03:09:02.607Z · score: 16 (10 votes)
Adding Up To Normality 2020-03-24T21:53:03.339Z · score: 68 (28 votes)
Does the 14-month vaccine safety test make sense for COVID-19? 2020-03-18T18:41:24.582Z · score: 59 (22 votes)
Rationalists, Post-Rationalists, And Rationalist-Adjacents 2020-03-13T20:25:52.670Z · score: 73 (26 votes)
AlphaStar: Impressive for RL progress, not for AGI progress 2019-11-02T01:50:27.208Z · score: 100 (47 votes)
orthonormal's Shortform 2019-10-31T05:24:47.692Z · score: 9 (1 votes)
Fuzzy Boundaries, Real Concepts 2018-05-07T03:39:33.033Z · score: 62 (16 votes)
Roleplaying As Yourself 2018-01-06T06:48:03.510Z · score: 92 (35 votes)
The Loudest Alarm Is Probably False 2018-01-02T16:38:05.748Z · score: 182 (74 votes)
Value Learning for Irrational Toy Models 2017-05-15T20:55:05.000Z · score: 0 (0 votes)
HCH as a measure of manipulation 2017-03-11T03:02:53.000Z · score: 1 (1 votes)
Censoring out-of-domain representations 2017-02-01T04:09:51.000Z · score: 2 (2 votes)
Vector-Valued Reinforcement Learning 2016-11-01T00:21:55.000Z · score: 2 (2 votes)
Cooperative Inverse Reinforcement Learning vs. Irrational Human Preferences 2016-06-18T00:55:10.000Z · score: 2 (2 votes)
Proof Length and Logical Counterfactuals Revisited 2016-02-10T18:56:38.000Z · score: 3 (3 votes)
Obstacle to modal optimality when you're being modalized 2015-08-29T20:41:59.000Z · score: 3 (3 votes)
A simple model of the Löbstacle 2015-06-11T16:23:22.000Z · score: 2 (2 votes)
Agent Simulates Predictor using Second-Level Oracles 2015-06-06T22:08:37.000Z · score: 2 (2 votes)
Agents that can predict their Newcomb predictor 2015-05-19T10:17:08.000Z · score: 1 (1 votes)
Modal Bargaining Agents 2015-04-16T22:19:03.000Z · score: 3 (3 votes)
[Clearing out my Drafts folder] Rationality and Decision Theory Curriculum Idea 2015-03-23T22:54:51.241Z · score: 6 (7 votes)
An Introduction to Löb's Theorem in MIRI Research 2015-03-23T22:22:26.908Z · score: 16 (17 votes)
Welcome, new contributors! 2015-03-23T21:53:20.000Z · score: 4 (4 votes)
A toy model of a corrigibility problem 2015-03-22T19:33:02.000Z · score: 4 (4 votes)
New forum for MIRI research: Intelligent Agent Foundations Forum 2015-03-20T00:35:07.071Z · score: 36 (37 votes)
Forum Digest: Updateless Decision Theory 2015-03-20T00:22:06.000Z · score: 5 (5 votes)
Meta- the goals of this forum 2015-03-10T20:16:47.000Z · score: 3 (3 votes)
Proposal: Modeling goal stability in machine learning 2015-03-03T01:31:36.000Z · score: 1 (1 votes)
An Introduction to Löb's Theorem in MIRI Research 2015-01-22T20:35:50.000Z · score: 2 (2 votes)
Robust Cooperation in the Prisoner's Dilemma 2013-06-07T08:30:25.557Z · score: 73 (71 votes)
Compromise: Send Meta Discussions to the Unofficial LessWrong Subreddit 2013-04-23T01:37:31.762Z · score: -2 (18 votes)
Welcome to Less Wrong! (5th thread, March 2013) 2013-04-01T16:19:17.933Z · score: 27 (28 votes)
Robin Hanson's Cryonics Hour 2013-03-29T17:20:23.897Z · score: 29 (34 votes)
Does My Vote Matter? 2012-11-05T01:23:52.009Z · score: 19 (37 votes)
Decision Theories, Part 3.75: Hang On, I Think This Works After All 2012-09-06T16:23:37.670Z · score: 23 (24 votes)
Decision Theories, Part 3.5: Halt, Melt and Catch Fire 2012-08-26T22:40:20.388Z · score: 31 (32 votes)
Posts I'd Like To Write (Includes Poll) 2012-05-26T21:25:31.019Z · score: 14 (15 votes)
Timeless physics breaks T-Rex's mind [LINK] 2012-04-23T19:16:07.064Z · score: 22 (29 votes)
Decision Theories: A Semi-Formal Analysis, Part III 2012-04-14T19:34:38.716Z · score: 23 (28 votes)
Decision Theories: A Semi-Formal Analysis, Part II 2012-04-06T18:59:35.787Z · score: 16 (19 votes)
Decision Theories: A Semi-Formal Analysis, Part I 2012-03-24T16:01:33.295Z · score: 24 (26 votes)
Suggestions for naming a class of decision theories 2012-03-17T17:22:54.160Z · score: 5 (8 votes)
Decision Theories: A Less Wrong Primer 2012-03-13T23:31:51.795Z · score: 73 (77 votes)
Baconmas: The holiday for the sciences 2012-01-05T18:51:10.606Z · score: 5 (5 votes)
Advice Request: Baconmas Website 2012-01-01T19:25:40.308Z · score: 11 (11 votes)

Comments

Comment by orthonormal on Map Errors: The Good, The Bad, and The Territory · 2020-06-29T03:27:58.976Z · score: 2 (1 votes) · LW · GW

#1 is just inevitable in all but a few perfectly specified domains. The map can't contain the entire territory.

#2 is what I'm discussing in this post; it's the one we rationalists try most to notice and combat. (Beliefs paying rent and all.)

#3 is fine; I'm not as worried about [maps that admit they don't know what's beyond the mountain] as I'm worried about [maps that fabricate the territory beyond the mountain].

#4. For sufficiently perfect predictive power, the difference between map and territory becomes an epiphenomenon, so I don't worry about this either.

Comment by orthonormal on Self-sacrifice is a scarce resource · 2020-06-28T17:29:19.364Z · score: 10 (5 votes) · LW · GW

This is super important, and I'm curious what your process of change was like.

(I'm working on an analogous change- I've been terrified of letting people down for my whole adult life.)

Comment by orthonormal on A reply to Agnes Callard · 2020-06-28T17:21:14.546Z · score: 14 (9 votes) · LW · GW

I forget who responded with the kernel of this argument, but it wasn't mine:

Saying the incentives should be different doesn't mean pretending they are different. In an ideal world, news organizations would have good standards and would not give in to external pressure on those standards. In our world, news organizations have some bad standards and give in to external pressure from time to time.

Reaching a better world has to come from making de-escalation treaties or changing the overall incentives. Unilaterally disarming (by refusing to even sign petitions) has the completely predictable consequence that the NYT will compromise their standards in directions that we dislike, because the pressure would be high in each direction but ours.

Comment by orthonormal on DontDoxScottAlexander.com - A Petition · 2020-06-28T17:12:10.300Z · score: 10 (2 votes) · LW · GW

What they say is that they don't respect pseudonyms in stories unless there's a compelling reason to do so in that particular case. There appears to be a political bias to the exceptions, but good luck getting an editor to admit that even to themself, let alone to others.

Comment by orthonormal on Map Errors: The Good, The Bad, and The Territory · 2020-06-27T19:03:14.761Z · score: 2 (1 votes) · LW · GW

That more or less covers the advice at the end, but the rest of my post feels very valuable to my model of rationality.

Comment by orthonormal on Covid 6/25: The Dam Breaks · 2020-06-26T00:55:40.640Z · score: 27 (14 votes) · LW · GW

When you say that "our civilization was inadequate [to the task of suppressing COVID-19]", I just want to emphasize that "our civilization" means only the USA, not Western civilization in general. The EU got hit harder at first and has since then performed well; you can blame them for not taking it seriously early enough, but you certainly can't accuse them of the level of dysfunction you see here.

In general, I like the framing that the United States is running on the worst legacy code of any Western democracy; the UK's is older but was more amenable to modern patches. Never underestimate the degree to which the US government is just the least efficient government of any developed nation.

Comment by orthonormal on The EMH Aten't Dead · 2020-05-19T16:23:58.206Z · score: 2 (1 votes) · LW · GW
The catch though, from a couple of times I've tried placing big bets on unlikely events, is that (most) bookmakers don't seem to accept them. They might accept a $100 bet but not a $1000 one on such odds. They suspect you have inside information. (The same happens I've heard if you repeatedly win at roulette in some casinos. Goons appear and instruct you firmly that you may only bet on the low-stakes tables from now on.)

Right, the EMH doesn't fully apply when sharks can't swoop in with bets large enough to overwhelm the confederacy of Georges. The odds bookies offer are a hybrid between a market and a democracy.

Comment by orthonormal on The EMH Aten't Dead · 2020-05-19T16:18:45.412Z · score: 7 (4 votes) · LW · GW

Right, April's rally wasn't due to "actually, everything is great now", it was due to "whew, it looks like the most apocalyptic scenarios we were seeing in March aren't likely, and there's a limit to how bad it's going to get".

Comment by orthonormal on Does the 14-month vaccine safety test make sense for COVID-19? · 2020-05-03T17:06:41.639Z · score: 8 (5 votes) · LW · GW

You're right; the current plan condenses and overlaps the three phases in order to save a lot of time.

Comment by orthonormal on On “COVID-19 Superspreader Events in 28 Countries: Critical Patterns and Lessons” · 2020-05-01T21:00:17.130Z · score: 2 (1 votes) · LW · GW

[EDIT: Probably not a valid counterexample; see steve2152's comment below]

Subways and other public transit aren’t present [in the list of superspreader events]

There was that bus in China, which also suggests that recirculated air might transfer aerosols (since many people sat in between the spreader and those who became infected).

Comment by orthonormal on What will be the big-picture implications of the coronavirus, assuming it eventually infects >10% of the world? · 2020-05-01T20:24:30.229Z · score: 12 (5 votes) · LW · GW

Update on Diamond Princess: as of now, Wikipedia says that the death toll is 14, or 2% of the passengers who tested positive within the first month. However, the dead all seem to have been elderly (there were many elderly passengers, as expected for a cruise liner). More specifically, 11 of them were over 70, another was over 60, and two others were of undisclosed age due to family wishes.

I don't know how to adjust those results for demographics, and of course you can't use them to predict what would happen without hospital care. But it's a promising sign (relative to Wei'd predictions) that we've made it this far without anything obviously worse than what happened in Italy and Spain, and even those have seen far less than 0.1% of their population die. NYC is estimated to have a 20% rate of infection, and it too has had less than 0.1% of its population die (though this may rise somewhat, as their wave of cases crested fairly recently).

Comment by orthonormal on Covid-19 4/30: Stuck in Limbo · 2020-05-01T20:05:46.772Z · score: 2 (1 votes) · LW · GW

My point was that because of the reporting backlog, the spike and decrease in your table appeared sharper than they were; the actual curve was in line with other places.

Comment by orthonormal on Covid-19 4/30: Stuck in Limbo · 2020-05-01T05:47:58.794Z · score: 2 (1 votes) · LW · GW
Still, [Louisiana] seems like a clear explosion and fast peak, followed by a clear negative trend.

The date of test and the date reported give two very different pictures.

Comment by orthonormal on [Link] COVID-19 causing deadly blood clots in younger people · 2020-04-29T04:53:16.676Z · score: 3 (2 votes) · LW · GW
Not a doctor - just wild speculation - I'm not even going to do this myself without real medical advice - would aspirin possibly make a difference as it helps for 'normal' blood clots?

I forget, what's the current epistemic status of "don't use NSAIDs if you might have COVID-19"? I think they were recommended against for a while, then said to be fine. And I haven't seen them mentioned on LW recently.

Comment by orthonormal on Does the 14-month vaccine safety test make sense for COVID-19? · 2020-04-25T03:20:41.447Z · score: 3 (2 votes) · LW · GW

Thank you, this is the sort of answer I was looking for- I'd naively had the prior that "no effect" was the only non-negligible possibility besides "positive effect".

Comment by orthonormal on Does the 14-month vaccine safety test make sense for COVID-19? · 2020-04-25T03:19:31.871Z · score: 2 (1 votes) · LW · GW

Informed volunteers would be heroes, but I think there are enough heroes to make vaccines available months sooner and to save millions of lives. At least it should be in the Overton window to ask for a voluntary trial with the understanding that there's substantial risk.

Comment by orthonormal on April Coronavirus Open Thread · 2020-04-24T04:22:06.755Z · score: 2 (1 votes) · LW · GW

This article and the linked study propose that smokers are getting infected less often due to the nicotine rather than the smoking. Is it worth getting nicotine gum and patches?

Comment by orthonormal on Conflict vs. mistake in non-zero-sum games · 2020-04-23T22:42:59.222Z · score: 2 (1 votes) · LW · GW

Ah, I'm just bad at recognizing sarcasm. In fact, I'm going to reword my comment above to remove the sarcasm.

Comment by orthonormal on Conflict vs. mistake in non-zero-sum games · 2020-04-23T19:50:01.963Z · score: 3 (2 votes) · LW · GW

[Edited to remove sarcasm.]

It's more negative for the rich than for the poor, and as such reduces inequality.

Wouldn't that predict that San Francisco, which has built almost nothing since the 1970s in most neighborhoods, should have low inequality?

Comment by orthonormal on What Happens During a Recession · 2020-04-20T21:18:54.291Z · score: 4 (3 votes) · LW · GW

The life expectancy result really, really surprises me. I'd expect that quality of life in nursing facilities also improves with additional staff.

Perhaps in a sane world, we'd subsidize costly end-of-life medical interventions less, but subsidize wages of staff at nursing homes more.

Comment by orthonormal on What Happens During a Recession · 2020-04-20T21:12:54.665Z · score: 4 (3 votes) · LW · GW

I'm having a tough time seeing any consistent effect of recessions on the labor force participation rate chart, for any of the demographics. It all looks like long-term trends plus very-short-term noise. Is there something I ought to be looking for?

Comment by orthonormal on April Coronavirus Open Thread · 2020-04-19T16:16:50.436Z · score: 4 (2 votes) · LW · GW

Great point. And South Korea is one of the few places I trust to have counted almost all of their cases, so that calculation has to be basically right. I think that completely settles it.

Comment by orthonormal on April Coronavirus Open Thread · 2020-04-18T06:51:13.569Z · score: 17 (7 votes) · LW · GW

[EDIT: Bucky points out that these cases make up too high a proportion of new cases for novel reinfection to be the primary mechanism, which means that there's negligible evidence to move the basic prior that general immunity should persist for a while (once the virus is well and truly defeated by the immune system).]

In South Korea, 2% of previously recovered patients have again tested positive and are again in isolation. There are several other explanations besides a general lack of acquired immunity (which would be the worst possible case, from a public health standpoint). But it seems critical that someone look at the evidence for the most dangerous possibility.

Comment by orthonormal on April Coronavirus Open Thread · 2020-04-15T05:53:09.360Z · score: 5 (3 votes) · LW · GW

So wow, something like this (just the basic version like leggi was discussing, putting patients on their belly instead of their back) is proving to be strikingly effective:

The biggest change: Instead of quickly sedating people who had shockingly low levels of oxygen and then putting them on mechanical ventilators, many doctors are now keeping patients conscious, having them roll over in bed, recline in chairs and continue to breathe on their own — with additional oxygen — for as long as possible.
The idea is to get them off their backs and thereby make more lung available.
[...]
Some patients, by taking oxygen and rolling onto their sides or on their bellies, have quickly returned to normal levels [of blood oxygen level]. The tactic is called proning.
Comment by orthonormal on April Coronavirus Open Thread · 2020-04-15T05:51:16.970Z · score: 2 (1 votes) · LW · GW

So wow, something like this (just the basic version, putting patients on their belly instead of their back) is proving to be strikingly effective:

The biggest change: Instead of quickly sedating people who had shockingly low levels of oxygen and then putting them on mechanical ventilators, many doctors are now keeping patients conscious, having them roll over in bed, recline in chairs and continue to breathe on their own — with additional oxygen — for as long as possible.
The idea is to get them off their backs and thereby make more lung available.
[...]
Some patients, by taking oxygen and rolling onto their sides or on their bellies, have quickly returned to normal levels [of blood oxygen level]. The tactic is called proning.
Comment by orthonormal on Where should LessWrong go on COVID? · 2020-04-14T17:40:27.520Z · score: 7 (4 votes) · LW · GW

Nobody's forcing you to help with this! And if you just want to point out why particular proposed models are bad, that's a good way to help as well.

Academics are indeed very smart, but under time pressure they have many additional constraints, most particularly the need to have everything pass peer review (now or later), which entails some unfortunate requirements like

  • "tighter confidence intervals look better"
  • "a single model is more justifiable than an ensemble"
  • "you can justify a handpicked parameter more easily than a handpicked distribution over that parameter"
  • "if your model looks at all like someone else's you'd better cite them, so either keep things in spherical cow territory or do a long literature search while people are dying"

We're not constrained by the same factors, and so it's perhaps possible to do better.

Comment by orthonormal on Ends Don't Justify Means (Among Humans) · 2020-04-14T03:03:04.141Z · score: 11 (3 votes) · LW · GW
The notion of untrusted hardware seems like something wholly outside the realm of classical decision theory.  (What it does to reflective decision theory I can't yet say, but that would seem to be the appropriate level to handle it.)

It's nice to see the genesis of corrigibility before Eliezer had unconfused himself enough to take that first step.

Comment by orthonormal on Where should LessWrong go on COVID? · 2020-04-14T02:46:01.831Z · score: 10 (5 votes) · LW · GW

I'd like a better model of possible infection and death trajectories than the models driving discussion at present. I think we might be capable of that.

Metaculus only represents the outputs of people's models. There's been a lot of talk in the rationalsphere criticizing overly simplistic/rigid/overconfident models, but little explicit discussion of what a better one might look like. It would be great if such models could be done through Guesstimate, but still valuable in other formats as long as they're easily readable.

Comment by orthonormal on Seemingly Popular Covid-19 Model is Obvious Nonsense · 2020-04-13T21:20:22.663Z · score: 5 (3 votes) · LW · GW

With actual numbers very, very large, this isn't remotely a concern; the domain of a correct continuous model might be "so long as there are at least 100 positive tests per week" or the like. Once we're there, we obviously need to treat things more discretely.

It's just not a sufficient reason for the modelers to make this egregious an optimistic error in setting R as a function of social distancing measures.

those have other drawbacks, like needing far more data than we have to calibrate and build, or needing you to make up inputs

Those are exactly the drawbacks Zvi is pointing to! And they're not even putting distributions on the parameter values the pulled from their asses!

Comment by orthonormal on Review of "Lifecycle Investing" · 2020-04-13T21:08:55.307Z · score: 4 (2 votes) · LW · GW

Is this strategy just counting on bankruptcy as a personal bailout in case you're leveraged 3:1 and the stock market doesn't go back up at least a bit after the pandemic?

Comment by orthonormal on The One Mistake Rule · 2020-04-10T23:55:51.792Z · score: 7 (4 votes) · LW · GW

Not betting, in the sense of keeping the money in index funds or somewhere else on the risk/reward Pareto frontier of easy strategies, at least limits your expected downside compared to entering shark-infested waters in an imperfect cage.

Comment by orthonormal on The One Mistake Rule · 2020-04-10T19:05:10.811Z · score: 22 (12 votes) · LW · GW

I think the need for this is relatively higher in an anti-inductive environment like trying to beat the market than in an inductive environment like materials science or disease modeling. In the former, you don't dare deploy until you're confident it's missing nothing; in the latter, what you have mid-iteration is still very useful, especially if you're making big time-sensitive decisions.

Comment by orthonormal on The One Mistake Rule · 2020-04-10T18:58:34.597Z · score: 4 (2 votes) · LW · GW

Similarly, a linear regression to some data will often behave well for interpolation but less well for distant extrapolation.

Comment by orthonormal on An Orthodox Case Against Utility Functions · 2020-04-10T05:01:40.480Z · score: 2 (1 votes) · LW · GW

I've been using computable to mean a total function (each instance is computable in finite time).

I'm thinking of an agent outside a universe about to take an action, and each action will cause that universe to run a particular TM. (You could maybe frame this as "the agent chooses the tape for the TM to run on".) For me, this is analogous to acting in the world and causing the world to shift toward some outcomes over others.

By asserting that U should be the computable one, I'm asserting that "how much do I like this outcome" is a more tractable question than "which actions result in this outcome".

An intuition pump in a human setting:

I can check whether given states of a Go board are victories for one player or the other, or if the game is not yet finished (this is analogous to U being a total computable function). But it's much more difficult to choose, for an unfinished game where I'm told I have a winning strategy, a move such that I still have a winning strategy. The best I can really do as a human is calculate a bit and then guess at how the leaves will probably resolve if we go down them (this is analogous to eval being an enumerable but not necessarily computable function).

In general, individual humans are much better at figuring out what outcomes we want than we are at figuring out exactly how to achieve those outcomes. (It would be quite weird if the opposite were the case.) We're not good at either in an absolute sense, of course.

Comment by orthonormal on Why I'm Not Vegan · 2020-04-10T04:36:37.277Z · score: 2 (1 votes) · LW · GW

The total carbon footprint of the typical American diet is... one set of numbers says 5-15 tons per household, another says 6 tons per person, of CO2-equivalent emissions. And carbon offsets are sold from 10 cents per ton to $40 per ton, with an average of $3.30. (I'm betting the cheapest ones are likely fake, but I don't think all of the offsets are fake.)

So your entire diet would cost somewhere between 50 cents and $600 per year (with a point estimate of $20, but dominated in expectation by the right tail), and going vegan would maybe save a majority of that. I don't expect the other environmental damage to be as costly as the carbon emissions, but of course I could be wrong.

However, if (as I do) you have more uncertainty over animal suffering- and thus a higher average expected value- cutting out meat entirely seems like it's worth a significant but not overwhelming dollar equivalent, measured in hundreds or in thousands of dollars each year.

And that's enough for me to at least 80-20 it, cutting out any meat that I was just eating by habit (instead of because my body feels it needs it or because it's a particularly delicious meal).

Comment by orthonormal on [Announcement] LessWrong will be down for ~1 hour on the evening of April 10th around 10PM PDT (5:00AM GMT) · 2020-04-09T21:34:07.207Z · score: 9 (3 votes) · LW · GW

Multiple tech companies I've worked at have had "get off MongoDB" projects that took more than a year, so that colors my interpretation of it. It may be less bad if your team never grows beyond 10 people, I guess?

(That, and I've lost and mangled training data because of MongoDB.)

Comment by orthonormal on [Announcement] LessWrong will be down for ~1 hour on the evening of April 10th around 10PM PDT (5:00AM GMT) · 2020-04-09T19:22:36.231Z · score: 11 (6 votes) · LW · GW

This is probably not the place to state Opinions about the database running on MongoDB, is it?

Comment by orthonormal on Choosing the Zero Point · 2020-04-09T19:21:04.964Z · score: 2 (1 votes) · LW · GW

To respond to the Godwin example, if your reference class is "Germans in the 1930s", I assert that there are far more altruistically effective actions one can take than "be a sincere reformist Nazi", to a much greater extent than "become entirely vegan" is a more altruistic option than "reduce meat/egg consumption by 2/3".

I agree that choosing the right reference class is difficult and subjective. The alternative of "imagine if you never existed" is interesting, but has the problem of the valley of bad rationality: people realize "I've already caused a carbon footprint and animal suffering" long before they realize "the amount of work it takes to offset more than I've caused is actually not that much". That leaves them feeling like they're deep in the Negative Zone for too long, with the risks I've mentioned.

Comment by orthonormal on Assessing Kurzweil's 1999 predictions for 2019 · 2020-04-09T16:53:30.997Z · score: 10 (6 votes) · LW · GW

Boy, lots of the remote learning stuff does suffer from predicting 2019 instead of 2020...

I suggest running inter-rater sanity checks and sometimes asking the rater if their answers should be inverted; I bet that at least one reader will misremember the instructions as "how true on a scale of 1 to 5" instead of your intended "how false on a scale of 1 to 5".

Comment by orthonormal on An Orthodox Case Against Utility Functions · 2020-04-09T00:36:50.237Z · score: 2 (1 votes) · LW · GW

Let's talk first about non-embedded agents.

Say that I'm given the specification of a Turing machine, and I have a computable utility mapping from output states (including "does not halt") to [0,1]. We presumably agree that is possible.

I agree that it's impossible to make a computable mapping from Turing machines to outcomes, so therefore I cannot have a computable utility function from TMs to the reals which assigns the same value to any two TMs with identical output.

But I can have a logical inductor which, for each TM, produces a sequence of predictions about that TM's output's utility. Every TM that halts will eventually get the correct utility, and every TM that doesn't will converge to some utility in [0,1], with the usual properties for logical inductors guaranteeing that TMs easily proven to have the same output will converge to the same number, etc.

That's a computable sequence of utility functions over TMs with asymptotic good properties. At any stage, I could stop and tell you that I choose some particular TM as the best one as it seems to me now.

I haven't really thought in a long while about questions like "do logical inductors' good properties of self-prediction mean that they could avoid the procrastination paradox", so I could be talking nonsense there.

Comment by orthonormal on Choosing the Zero Point · 2020-04-09T00:05:25.343Z · score: 6 (4 votes) · LW · GW

That's a good point. On the other hand, many people make their reference class the most impressive one they belong to rather than the least impressive one. (At least I did, when I was in academia; I may have been excellent in mathematics within many sets of people, but among the reference class "math faculty at a good institution" I was struggling to feel okay.) Impostor syndrome makes this doubly bad, if the people in one's reference class who are struggling don't make that fact visible.

There are two opposite pieces of advice here, and I don't know how to tell people which is true for them- if anything, I think they might gravitate to the wrong piece of advice, since they're already biased in that direction.

Comment by orthonormal on Choosing the Zero Point · 2020-04-09T00:00:50.137Z · score: 3 (2 votes) · LW · GW

I think the self is not especially unified in practice for most people- the elephant and the rider, as it were. (Even the elephant can have something like subagents.) That's not quite true, but it's more true than the idea of a human as a unitary agent.

I'm mostly selfish and partly altruistic, and the altruistic part is working hard to make sure that its negotiated portion of the attention/energy/resource budget doesn't go to waste. Part of that is strategizing about how to make the other parts come along for the ride more willingly.

Reframing things to myself, in ways that don't change the truth value but do change the emphasis, is very useful. Other parts of me don't necessarily speak logic, but they do speak metaphor.

I agree that you and I experience the world very differently, and I assert that my experience is the more common one, even among rationalists.

Comment by orthonormal on Choosing the Zero Point · 2020-04-08T23:55:01.964Z · score: 3 (2 votes) · LW · GW

Of course you shouldn't plan to reset the zero point after actions! That's very different.

I use this sparingly, for observing big new facts that I didn't cause to be true. That doesn't change the relative expected utilities of various actions, so long as my expected change in utility from future observations is zero.

Comment by orthonormal on An Orthodox Case Against Utility Functions · 2020-04-08T06:02:34.808Z · score: 2 (1 votes) · LW · GW

I mean the sort of "eventually approximately consistent over computable patterns" thing exhibited by logical inductors, which is stronger than limit-computability.

Comment by orthonormal on An Orthodox Case Against Utility Functions · 2020-04-08T02:43:56.351Z · score: 5 (3 votes) · LW · GW

I think that computable is obviously too strong a condition for classical utility; enumerable is better.

Imagine you're about to see the source code of a machine that's running, and if the machine eventually halts then 2 utilons will be generated. That's a simpler problem to reason about than the procrastination paradox, and your utility function is enumerable but not computable. (Likewise, logical inductors obviously don't make PA approximately computable, but their properties are what you'd want the definition of approximately enumerable to be, if any such definition were standard.)

I suspect that the procrastination paradox leans heavily on the computability requirement as well.

Comment by orthonormal on Choosing the Zero Point · 2020-04-08T00:48:29.669Z · score: 16 (8 votes) · LW · GW

Thank you for confirming. I wanted to be sure I wasn't putting words in your mouth.

I think I just have a very different model than you of what most people tend to do when they're constantly horrified by their own actions.

I'm sorry about the animal welfare relevance of this analogy, but it's the best one I have:

The difference between positive reinforcement and punishment is staggering; you can train a circus animal to do complex tricks using either method, but only under the positive reinforcement method will the animal voluntarily engage further with the trainer. Train an animal with punishment and it will tend to avoid further training, will escape the circus if at all possible.

This is why I think your psychology is unusual. I expect a typical person filled with horror about a behavior to change that behavior for a while (do the trained trick), but eventually find a way to not think about it (avoid the trainer) or change their beliefs in order to not find it horrible any longer (escape the circus). I can believe that your personal history makes the horror an extremely motivating force for you. I just don't think that's the default way for people to respond to those sort of experiences and feelings.

It's also the reason why I want people to reset their zero point such that helpful actions do in fact feel like they push the world into the positive. That gives a positive reinforcement to helpful actions, rather than punishing oneself from any departure from helpful actions. And I expect that to help most people go farther.

Comment by orthonormal on On R0 · 2020-04-08T00:28:51.338Z · score: 5 (3 votes) · LW · GW

Possibly more important than even the grocery store vector are the people whose exposure level is high enough that their reference class hasn't been mostly infected yet, but will be soon. I think of suburban parents who think it's fine to go down to the neighbors, and let their kids play with other kids, etc. The ones who think they're "doing enough" but really aren't.

Interventions on their behavior, from [public health messaging in the right places] to [literally ticketing them], could do quite a bit.

Comment by orthonormal on Choosing the Zero Point · 2020-04-07T17:47:45.910Z · score: 13 (6 votes) · LW · GW

(Splitting replies on different parts into different subthreads.)

The real problem that I have (and I suspect others have) with framing a significant sacrifice as the "bare standard of human decency" is that it pattern-matches purity ethics far more than utilitarianism. (A purity ethic derived from utilitarianism is still a purity ethic.)

For me, the key difference (keeping the vegetarian/vegan example) is whether it is a better outcome for one person to become a vegan and another to keep eating meat as usual, or for two people to each reduce their meat/egg consumption by two-thirds. The "insufficiently horrified" framing makes it sound like neither of the two people in the latter case really count, while at least one person in the former does count.

Do you agree (without getting into which outcome is easier for activism to achieve) that the latter outcome is preferable to the former? And separately, does it aesthetically feel better or worse?

Comment by orthonormal on Choosing the Zero Point · 2020-04-07T17:41:23.948Z · score: 6 (4 votes) · LW · GW

(Splitting replies on different parts into different subthreads.)

Correct me if I'm wrong, but I hear you say that your sense of horror is load-bearing, that you would take worse actions if you did not feel a constant anguish over the suffering that is happening.

That could be true for you, but it seems counter to the way most people work. Constant anguish tends not to motivate, it instead leads to psychological collapse, or to frantic measures when patience would achieve more, or to protected beliefs that resist challenge in any small part.

Comment by orthonormal on Choosing the Zero Point · 2020-04-07T17:33:25.344Z · score: 3 (2 votes) · LW · GW

(Splitting replies on different parts into different subthreads.)

One part of this helped me recognize an important emendation: if many bad things are continuing to happen, then a zero point of "how things are right now" will still lead you inexorably into the negatives. I was intuitively thinking of "the expected trajectory of the world if I were instead a random person from my reference class" as my reference point, but I didn't crystallize that and put it in my post. Thank you, I'll add that in.