Comment by nshepperd on Let Values Drift · 2019-06-21T03:35:25.551Z · score: 10 (3 votes) · LW · GW

I don't see the usual commonsense understanding of "values" (or the understanding used in economics or ethics) as relying on values being ontologically fundamental in any way, though. But you've the fact that they're not to make a seemingly unjustified rhetorical leap to "values are just habituations or patterns of action", which just doesn't seem to be true.

Most importantly, because the "values" that people are concerned with then they talk about "value drift" are idealized values (ala. extrapolated volition), not instantaneous values or opinions or habituations.

For instance, philosophers such as EY consider that changing one's mind in response to a new moral argument is not value drift because it preserves one's idealized values, and that it is generally instrumentally positive because (if it brings one's instantaneous opinions closer to their idealized values) it makes one better at accomplishing their idealized values. So indeed, we should let the EAs "drift" in that sense.

On the other hand, getting hit with a cosmic ray which alters your brain, or getting hacked by a remote code execution exploit is value drift because it does not preserve one's idealized values (and is therefore bad, according to the usual decision theoretic argument, because it makes you worse at accomplishing them). And those are the kind of problems we worry about with AI.

Comment by nshepperd on Let Values Drift · 2019-06-21T01:37:35.928Z · score: 12 (6 votes) · LW · GW

When we talk of values as nouns, we are talking about the values that people have, express, find, embrace, and so on. For example, a person might say that altruism is one of their values. But what would it mean to “have” altruism as a value or for it to be one of one’s values? What is the thing possessed or of one in this case? Can you grab altruism and hold onto it, or find it in the mind cleanly separated from other thoughts?

Since this appears to be a crux of your whole (fallacious, in my opinion) argument, I'm going to start by just criticizing this point. This argument proves far too much. It proves that:

  • People don't have beliefs, memories or skills
  • Books don't have concepts
  • Objects don't have colors
  • Shapes don't have total internal angles

It seems as if you've rhetorically denied the existence of any abstract properties whatsoever, for the purpose of minimizing values as being "merely" habituations or patterns of action. But I don't see why anyone should actually accept that claim.

Comment by nshepperd on Interpretations of "probability" · 2019-05-10T05:10:25.290Z · score: 4 (2 votes) · LW · GW

Doesn't it mean the same thing in either case? Either way, I don't know which way the coin will land or has landed, and I have some odds at which I'll be willing to make a bet. I don't see the problem.

(Though my willingness to bet at all will generally go down over time in the "already flipped" case, due to the increasing possibility that whoever is offering the bet somehow looked at the coin in the intervening time.)

Comment by nshepperd on Interpretations of "probability" · 2019-05-10T04:47:17.331Z · score: 17 (9 votes) · LW · GW

The idea that "probability" is some preexisting thing that needs to be "interpreted" as something always seemed a little bit backwards to me. Isn't it more straightforward to say:

  1. Beliefs exist, and obey the Kolmogorov axioms (at least, "correct" beliefs do, as formalized by generalizations of logic (Cox's theorem), or by possible-world-counting). This is what we refer to as "bayesian probabilities", and code into AIs when we want to them to represent beliefs.
  2. Measures over imaginary event classes / ensembles also obey the Kolmogorov axioms. "Frequentist probabilities" fall into this category.

Personally I mostly think about #1 because I'm interested in figuring out what I should believe, not about frequencies in arbitrary ensembles. But the fact is that both of these obey the same "probability" axioms, the Kolmogorov axioms. Denying one or the other because "probability" must be "interpreted" as exclusively either #1 or #2 is simply wrong (but that's what frequentists effectively do when they loudly shout that you "can't" apply probability to beliefs).

Now, sometimes you do need to interpret "probability" as something -- in the specific case where someone else makes an utterance containing the word "probability" and you want to figure out what they meant. But the answer there is probably that in many cases people don't even distinguish between #1 and #2, because they'll only commit to a specific number when there's a convenient instance of #2 that make #1 easy to calculate. For instance, saying 1/6 for a roll of a "fair" die.

People often act as though their utterances about probability refer to #1 though. For instance when they misinterpret p-values as the post-data probability of the null hypothesis and go around believing that the effect is real...

Comment by nshepperd on Functional Decision Theory vs Causal Decision Theory: Expanding on Newcomb's Problem · 2019-05-03T06:07:02.923Z · score: 9 (5 votes) · LW · GW

No, that doesn't work. It seems to me you've confused yourself by constructing a fake symmetry between these problems. It wouldn't make any sense for Omega to "predict" whether you choose both boxes in Newcomb's if Newcomb's were equivalent to something that doesn't involve choosing boxes.

More explicitly:

Newcomb's Problem is "You sit in front of a pair of boxes, which are either- both filled with money if Omega predicted you would take one box in this case, otherwise only one is filled". Note: describing the problem does not require mentioning "Newcomb's Problem"; it can be expressed as a simple game tree (see here for some explanation of the tree format):


In comparison, your "Inverse Newcomb" is "Omega gives you some money iff it predicts that you take both boxes in Newcomb's Problem, an entirely different scenario (ie. not this case)."

The latter is more of the form "Omega arbitrarily rewards agents for taking certain hypothetical actions in a different problem" (of which a nearly limitless variety can be invented to justify any chosen decision theory¹), rather than being an actual self-contained problem which can be "solved".

The latter also can't be expressed as any kind of game tree without "cheating" and naming "Newcomb's Problem" verbally --- or rather, you can express a similar thing by embedding the Newcomb game tree and referring to the embedded tree, but that converts it into a legitimate decision problem, which FDT of course gives the correct answer to (TODO: draw an example ;).

(¹): Consider Inverse^2 Newcomb, which I consider the proper symmetric inverse of "Inverse Newcomb": Omega puts you in front of two boxes and says "this is not Newcomb's Problem, but I have filled both boxes with money iff I predicted that you take one box in standard Newcomb". Obviously here FDT takes both boxes and a tidy $1,000,1000 profit (plus the $1,000,000 from Standard Newcomb). Whereas CDT gets... $1000 (plus $1000 from Standard Newcomb).

Comment by nshepperd on The Cacophony Hypothesis: Simulation (If It is Possible At All) Cannot Call New Consciousnesses Into Existence · 2019-04-25T17:19:39.715Z · score: 3 (-1 votes) · LW · GW

Yes, you need to have a theory of physics to write down a transition rule for a physical system. That is a problem, but it's not at all the same problem as the "target format" problem. The only role the transition rule plays here is it allows one to apply induction to efficiently prove some generalization about the system over all time steps.

In principle a different more distinguished concise description of the system's behaviour could play the a similar role (perhaps, the recording of the states of the system + the shortest program that outputs the recording?). Or perhaps there's some way of choosing a distinguished "best" formalization of physics. But that's rather out of scope of what I wanted to suggest here.

But then you are measuring proof shortness relative to that system. And you could be using one of countless other formal systems which always make the same predictions, but relative to which different proofs are short and long.

It would be a O(1) cost to start the proof by translating the axioms into a more convenient format. Much as Kolmogorov complexity is "language dependent" but not asymptotically because any particular universal turing machine can be simulated in any other for a constant cost.

The assumption (including that it takes in and puts out in arabic numerals, and uses “*” as the multuplication command, and that buttons must be pressed,… and all the other things you need to actually use it) includes that.

These are all things that can be derived from a physical description of the calculator (maybe not in fewer steps than it takes to do long multiplication, but certainly in fewer steps than less trivial computations one might do with a calculator). There's no observer dependency here.

Comment by nshepperd on The Cacophony Hypothesis: Simulation (If It is Possible At All) Cannot Call New Consciousnesses Into Existence · 2019-04-21T22:33:33.034Z · score: 2 (1 votes) · LW · GW

That's not an issue in my formalization. The "logical facts" I speak of in the formalized version would be fully specified mathematical statements, such as "if the simulation starts in state X at t=0, the state of the simulation at t=T is Y" or "given that Alice starts in state X, then <some formalized way of categorising states according to favourite ice cream flavour> returns Vanilla". The "target format" is mathematical proofs. Languages (as in English vs Chinese) don't and can't come in to it, because proof systems are language-ignorant.

Note, the formalized criterion is broader than the informal "could you do something useful with this simulation IRL" criterion, even though the latter is the 'inspiration' for it. For instance, it doesn't matter whether you understand the programming language the simulation is written in. If someone who did understand the language could write the appropriate proofs, then the proofs exist.

Similarly, if a simulation is run under Homomorphic_encryption, it is nevertheless a valid simulation, despite the fact that you can't read it if you don't have the decryption key. Because a proof exists which starts by "magically" writing down the key, proving that it's the correct decryption key, then proceeding from there.

An informal criterion which maybe captures this better would be: If you and your friend both have (view) access to a genuine computation of some logical facts X, it should be possible to convince your friend of X in fewer words by referring to the alleged computation (but you are permitted unlimited time to think first, so you can reverse engineer the simulation, bruteforce some encryption keys, learn Chinese, whatever you like, before talking). A bit like how it's more efficient to convince your friend that 637265729567*37265974 = 23748328109134853258 by punching the numbers into a calculator and saying "see?" than by handing over a paper with a complete long multiplication derivation (assuming you are familiar with the calculator and can convince your friend that it calculates correctly).

Comment by nshepperd on The Cacophony Hypothesis: Simulation (If It is Possible At All) Cannot Call New Consciousnesses Into Existence · 2019-04-16T07:41:05.793Z · score: 25 (8 votes) · LW · GW

This idea is, as others have commented, pretty much Dust theory.

The solution, in my opinion, is the same as the answer to Dust theory: namely, it is not actually the case that anything is a simulation of anything. Yes, you can claim that (for instance) the motion of the atoms in a pebble can be interpreted as a simulation of Alice, in the sense that anything can be mapped to anything... but in a certain more real sense, you can't.

And that sense is this: an actual simulation of Alice running on a computer grants you certain powers - you can step through the simulation, examine what Alice does, and determine certain facts such as Alice's favourite ice cream flavour (these are logical facts, given the simulation's initial state). If the simulation is an upload of your friend Alice, then by doing so you learn meaningful new facts about your friend.

In comparison, a pebble "interpreted" as a simulation of Alice affords you no such powers, because the interpretation (mapping from pebble states to simulation data) is entirely post-hoc. The only way to pin down the mapping---such that you could, for instance, explicitly write it down, or take the pebble's state and map it to an answer about Alice's favourite ice cream---is to already have carried out the actual simulation, separately, and already know these things about Alice.

In general, "legitimate" computations of certain logical facts (such as the answers one might ask about simulations of people) should, in a certain sense, make it easier to calculate those logical facts then doing so from scratch.

A specific formalization of this idea would be that a proof system equipped with an oracle (axiom schema) describing the states of the physical system which allegedly computed these facts, as well as its transition rule, should be able to find proofs for those logical facts in less steps than one without such axioms.

Such proofs will involve first coming up with a mapping (such as interpreting certain electrical junctions as nand gates), proving them valid using the transition rules, then using induction to jump to "the physical state at timestep t is X therefore Alice's favourite ice cream colour is Y". Note that the requirement that these proofs be short naturally results in these "interpretations" being simple.

As far as I know, this specific formalization of the anti-Dust idea is original to me, though the idea that "interpretations" of things as computations ought to be "simple" is not particularly new.

Comment by nshepperd on Highlights from "Integral Spirituality" · 2019-04-16T06:31:58.499Z · score: 2 (1 votes) · LW · GW

We can (and should) have that discussion, we should just have it on a separate post

Can you point to the specific location that discussion "should" happen at?

Comment by nshepperd on Highlights from "Integral Spirituality" · 2019-04-16T06:01:46.539Z · score: 13 (5 votes) · LW · GW

Since you invited it, I'm going to say it.

How dare Gordon make me read this bullshit.

This has nothing to do with rationality and in my opinion doesn't belong on LessWrong, let alone on the front page.

(choosing a section at random to criticise):

And science itself was always completely honest about its limitations: science cannot say whether God exists or does not exist; whether there is an Absolute or not; why we are here, what our ultimate nature is, and so on.

This is just wrong. Science can and does say these things. Non-overlapping magisteria is a fantasy, tremendously useful for the political objective of preventing conflict between religious authorities and scientists, but no less a fantasy for that.

Comment by nshepperd on The Hard Work of Translation (Buddhism) · 2019-04-15T17:59:21.246Z · score: 5 (4 votes) · LW · GW

The two parts I mentioned are simply the most obviously speculative and unjustified examples. I also don't have any real reason to believe the vaguer pop psychology claims about building stories, backlogs, etc.

The post would prob­a­bly have been a bit cleaner to not men­tion the few wild spec­u­la­tions he men­tions, but get­ting caught up on the tiny de­tails seems to miss the for­est from the trees.

It seems to me LW has a big epistemic hygiene problem, of late. We need to collectively stop make excuses for posting wild speculations as if they were fact, just because the same post also contains some interesting 'insight'. We should be downvoting such posts and saying to the author "go away and write this again, with the parts that you can't justify removed".

Doing so may reveal that the 'insight' either does or does not successfully stand alone when the supporting 'speculation' is removed. Either way, we learn something valuable about the alleged insight; and we benefit directly by not spreading claims that lack evidence.

Comment by nshepperd on The Hard Work of Translation (Buddhism) · 2019-04-12T21:09:13.500Z · score: 17 (10 votes) · LW · GW

For a post that claims to be a "translation" of Buddhism, this seems to contain:

  • No Pali text;
  • No specific references to Pali text, or any sources at all;
  • No actual translation work of any kind.

On the other hand, it does contain quite a bit of unjustified speculation. "Literal electrical resistance in the CNS", really? "Rewiring your CNS"? Why should I believe any of this?

Why are people upvoting this?

Comment by nshepperd on Boundaries - A map and territory experiment. [post-rationality] · 2019-02-02T04:47:04.439Z · score: 8 (3 votes) · LW · GW

"Above the map"? "Outside the territory"? This is utter nonsense. Rationality insists no such thing. Explicitly the opposite, in fact.

Given things like this too:

Existing map-less is very hard. The human brain really likes to put maps around things.

At this point I have to wonder if you're just rounding off rationality to the nearest thing to which you can apply new-age platitudes. Frankly, this is insulting.

Comment by nshepperd on Masculine Virtues · 2019-01-31T17:54:16.311Z · score: 8 (4 votes) · LW · GW

You don't need to estimate this.

A McGill University study found that more than 60 percent of college-level soccer players reported symptoms of concussion during a single season. Although the percentage at other levels of play may be different, these data indicate that head injuries in soccer are more frequent than most presume.

A 60% chance of concussion is more than enough for me to stay far away.

Comment by nshepperd on No option to report spam · 2019-01-08T17:40:48.609Z · score: 6 (3 votes) · LW · GW

Prevention over removal. Old LW required a certain amount of karma in order to create posts, and we correspondingly didn't have a post spam problem that I remember. I strongly believe that this requirement should be re-introduced (with or without a moderator approval option for users without sufficient karma).

Comment by nshepperd on Topological Fixed Point Exercises · 2018-11-21T01:16:05.388Z · score: 24 (6 votes) · LW · GW

Proof of #4, but with unnecessary calculus:

Not only is there an odd number of tricolor triangles, but they come in pairs according to their orientation (RGB clockwise/anticlockwise). Proof: define a continuously differentiable vector field on the plane, by letting the field at each vertex be 0, and the field in the center of each edge be a vector of magnitude 1 pointing in the direction R->G->B->R (or 0 if the two adjacent vertices are the same color). Extend the field to the complete edges, then the interiors of the triangles by some interpolation method with continuous derivative (eg. cosine interpolation).

Assume the line integral along one unit edge in the direction R->G or G->B or B->R to be 1/3. (Without loss of generality since we can rescale the graph/vectors to make this true). Then a similar parity argument to Sperner's 1d lemma (or the FTC) shows that the clockwise line integral along each large edge is 1/3, hence the line integral around the large triangle is 1/3+1/3+1/3=1.

By green's theorem, this is equal to the integrated curl of the field in the interior of the large triangle, and hence equal (by another invocation of green's theorem) to the summed clockwise line integrals around each small triangle. The integrals around a unicolor or bicolor triangle are 0 and -1/3 + 1/3 + 0 = 0 respectively, leaving only tricolor triangles, whose integral is again 1 depending on orientation. Thus: (tricolor clockwise) - (tricolor anticlockwise) = 1. QED.

Comment by nshepperd on Strong Votes [Update: Deployed] · 2018-11-08T03:02:27.143Z · score: 2 (1 votes) · LW · GW

Your interpretation of the bolded part is correct.

Comment by nshepperd on Strong Votes [Update: Deployed] · 2018-11-08T02:32:57.826Z · score: 24 (6 votes) · LW · GW

We got to discussing this on #lesswrong recently. I don't see anyone here pointing this out yet directly, so:

Can you technically Strong Upvote everything? Well, we can’t stop you. But we’re hoping a combination of mostly-good-faith + trivial inconveniences will result in people using Strong Upvotes when they feel it’s actually important.

This approach, hoping that good faith will prevent people from using Strong votes "too much", is a good example of an Asshole Filter (linkposted on LW last year). You've set some (unclear) boundaries, then due to not enforcing them, reward those who violate them with increased control over the site conversation. Chris_Leong gestures towards this without directly naming it in a sibling comment.

In my opinion “maybe put limits on strong upvotes if this seems to be a problem” is not the correct response to this problem, nor would be banning or otherwise 'disciplining' users who use strong votes "too much". The correct response is to remove the asshole filter by altering the incentives to match what you want to happen. Options include:

  1. Making votes normal by default but encouraging users to use strong votes freely, up to 100% of the time, so that good faith users are not disadvantaged. (Note: still disenfranchises users who don't notice that this feature exists, but maybe that's ok.)
  2. Making votes strong by default so that it's making a "weak" vote that takes extra effort. (Note: this gives users who carefully make weak votes when they have weak opinions less weight, but at least they do this with eyes open and in the absence of perverse incentives.)
  3. #2 but with some algorithmic adjustment to give careful users more weight instead of less. This seems extremely difficult to get right (cf. slashdot metamoderation). Probably the correct answer there is some form of collaborative filtering.

Personally I favour solution #1.

I'll add that this is not just a hypothetical troll-control issue. This is also a UX issue. Forcing users to navigate an unclear ethical question and prisoner's dilemma—how much strong voting is "too much"—in order to use the site is unpleasant and a bad user experience. There should not be a "wrong" action available in the user interface.

PS. I'll concede that making strong votes an actually limited resource that is enforced by the site economically (eg. with Token Bucket quota) would in a way also work, due to eliminating the perceived need for strong votes to be limited by "good faith". But IMO the need is only perceived, and not real. Voting is for expressing preferences, and preferences are unlimited.

Comment by nshepperd on Kalman Filter for Bayesians · 2018-10-23T04:21:49.306Z · score: 3 (2 votes) · LW · GW

Good post!

Is it common to use Kalman filters for things that have nonlinear transformations, by approximating the posterior with a Gaussian (eg. calculating the closest Gaussian distribution to the true posterior by JS-divergence or the like)? How well would that work?

Grammar comment--you seem to have accidentally a few words at

Measuring multiple quantities: what if we want to measure two or more quantities, such as temperature and humidity? Furthermore, we might know that these are [missing words?] Then we now have multivariate normal distributions.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-21T22:57:58.293Z · score: 4 (2 votes) · LW · GW

How big was your mirror, and how much of your face did you see in it?

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-20T02:43:01.580Z · score: 10 (2 votes) · LW · GW

C is basically a statement that, if included in a valid argument about the truth of P, causes the argument to tell us either P or ~P. That’s definitionally what it means to be able to know the criterion of truth.

That's not how algorithms work and seems... incoherent.

That you want to deny C is great,

I did not say that either.

because I think (as I’m finding with Said), that we already agree, and any disagreement is the consequence of misunderstanding, probably because it comes too close to sounding to you like a position that I would also reject, and the rest of the fundamental disagreement is one of sentiment, perspective, having worked out the details, and emphasis.

No, I don't think we do agree. It seems to me you're deeply confused about all of this stuff.

Here's an exercise: Say that we replace "C" by a specific concrete algorithm. For instance the elementary long multiplication algorithm used by primary school children to multiply numbers.

Does anything whatsoever about your argument change with this substitution? Have we proved that we can explain multiplication to a rock? Or perhaps we've proved that this algorithm doesn't exist, and neither do schools?

Another exercise: suppose, as a counterfactual, that Laplace's demon exists, and furthermore likes answering questions. Now we can take a specific algorithm C: "ask the demon your question, and await the answer, which will be received within the minute". By construction this algorithm always returns the correct answer. Now, your task is to give the algorithm, given only these premises, that I can follow to convince a rock that Euclid's theorem is true.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-19T23:29:01.944Z · score: 4 (2 votes) · LW · GW

It seems that you don't get it. Said just demonstrated that even if C exists it wouldn't imply a universally compelling argument.

In other words, this:

Suppose we know the criterion of truth, C; that is, there exists (not counterfactually but actually as in anyone can observe this thing) a procedure/​algorithm to assess if any given statement is true. Let P be a statement. Then there exists some argument, A, contingent on C such that A implies P or ~P. Thus for all P we can know if P or ~P. This would make A universally compelling, i.e. A is a mind-independent argument for the truth value of all statements that would convince even rocks.

appears to be a total non sequitur. How does the existence of an algorithm enable you to convince a rock of anything? At a minimum, an algorithm needs to be implemented on a computer... Your statement, and therefore your conclusion that C doesn't exist, doesn't follow at all.

(Note: In this comment, I am not claiming that C (as you've defined it) exists, or agreeing that it needs to exist for any of my criticisms to hold.)

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-19T18:34:30.674Z · score: 10 (2 votes) · LW · GW

It doesn't seem to be a strawman of what eg. gworley and TAG have been saying, judging by the repeated demands for me to supply some universally compelling "criterion of truth" before any of the standard criticisms can be applied. Maybe you actually disagree with them on this point?

It doesn't seem like applying full force in criticism is a priority for the 'postrationality' envisioned by the OP, either, or else they would not have given examples (compellingness-of-story, willingness-to-life) so trivial to show as bad ideas using standard arguments.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-19T00:54:22.820Z · score: 4 (5 votes) · LW · GW

As for my story about how the brain works: yes, it is obviously a vast simplification. That does not make it false, especially given that “the brain learns to use what has worked before and what it thinks is likely to make it win in the future” is exactly what Eliezer is advocating in the above post.

Even if true, this is different from "epistemic rationality is just instrumental rationality"; as different as adaptation executors are from fitness maximisers.

Separately, it's interesting that you quote this part:

The important thing is to hold nothing back in your criticisms of how to criticize; nor should you regard the unavoidability of loopy justifications as a warrant of immunity from questioning.

Because it seems to me that this is exactly what advocates of "postrationality" here are not doing, when they take the absence of universally compelling arguments as license to dismiss rationality and truth-based arguments against their positions.¹

Eliezer also says this:

Always apply full force, whether it loops or not—do the best you can possibly do, whether it loops or not—and play, ultimately, to win.

It seems to me that applying full force in criticism of postrationality amounts to something like the below:

"Indeed, compellingness-of-story, willingness-to-life, mythic mode, and many other non-evidence-based criteria are alternative criteria which could be used to select beliefs. However we have huge amounts of evidence (catalogued in the Sequences, and in the heuristics and biases literature) that these criteria are not strongly correlated to truth, and therefore will lead you to holding wrong beliefs, and furthermore that holding wrong beliefs is instrumentally harmful, and, and [the rest of the sequences, Ethical Injunctions, etc]..."

"Meanwhile, we also have vast tracts of evidence that science works, that results derived with valid statistical methods replicate far more often than any others, that beliefs approaching truth requires accumulating evidence by observation. I would put the probability that rational methods are the best criteria I have for selecting beliefs at . Hence, it seems decisively not worth it to adopt some almost certainly harmful 'postrational' anti-epistomology just because of that probability. In any case, per Ethical Injunctions, even if my probabilities were otherwise, it would be far more likely that I've made a mistake in reasoning than that adopting non-rational beliefs by such methods would be a good idea."

Indeed, much of the Sequences could be seen as Eliezer considering alternative ways of selecting beliefs or "viewing the world", analyzing these alternative ways, and showing that they are contrary to and inferior to rationality. Once this has been demonstrated, we call them "biases". We don't cling to them on the basis that "we can't know the criterion of truth".

Advocates of postrationality seem to be hoping that the fact that P(Occam's razor) < makes these arguments go away. It doesn't work like that. P(Occam's razor) = at most makes of these arguments go away. And we have a lot of evidence for Occam's razor.

¹ As gworley seems to do here and here seemingly expecting me to provide a universally compelling argument in response.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-18T17:31:12.692Z · score: 5 (3 votes) · LW · GW

I'll have more to say later but:

The way that I’d phrase it is that there’s a difference between considering a claim to be true, and considering its justification universally compelling.

Both of these are different from the claim actually being true. The fact that Occam's razor is true is what causes the physical process of (occamian) observation and experiment to yield correct results. So you see, you've already managed to rephrase what I've been saying into something different by conflating map and territory.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-18T02:50:55.632Z · score: 16 (5 votes) · LW · GW

This stuff about rain dancing seems like just the most banal epistemological trivialities, which have already been dealt with thoroughly in the Sequences. The reasons why such "tests" of rain dancing don't work are well known and don't need to be recapitulated here.

But to do that, you need to use a meta-model. When I say that we don’t have direct access to the truth, this is what I mean;

This has nothing to do with causal pathways, magic or otherwise, direct or otherwise. Magic would not turn a rock into a philosopher even if it should exist.

Yes, carrying out experiments to determine reality relies on Occam's razor. It relies on Occam's razor being true. It does not in any way rely on me possessing some magical universally compelling argument for Occam's razor. Because Occam's razor is in fact true in our universe, experiment does in fact work, and thus the causal pathway for evaluating our models does in fact exist: experiment and observation (and bayesian statistics).

I'm going to stress this point because I noticed others in this thread make this seemingly elementary map-territory confusion before (though I didn't comment on it there). In fact it seems to me now that conflating these things is maybe actually the entire source of this debate: "Occam's razor is true" is an entirely different thing from "I have access to universally compelling arguments for Occam's razor", as different as a raven and the abstract concept of corporate debt. The former is true and useful and relevant to epistemology. The latter is false, impossible and useless.

Because the former is true, when I say "in fact, there is a causal pathway to evaluate our models: looking at reality and doing experiments", what I say is, in fact, true. The process in fact works. It can even be carried out by a suitably programmed robot with no awareness of what Occam's razor or "truth" even is. No appeals or arguments about whether universally compelling arguments for Occam's razor exist can change that fact.

(Why am I so lucky as to be a mind whose thinking relies on Occam's razor in a world where Occam's razor is true? Well, animals evolved via natural selection in an Occamian world, and those whose minds were more fit for that world survived...)

But honestly, I'm just regurgitating Where Recursive Justification Hits Bottom at this point.

This is a reinforcement learning system which responds to rewards: if particular thoughts or assumptions (...) have led to actions which brought the organism (internally or externally generated rewards), then those kinds of thoughts and assumptions will be reinforced.

This seems like a gross oversimplification to me. The mind is a complex dynamical system made of locally reinforcement-learning components, which doesn't do any one thing all the time.

In other words, we end up having the kinds of beliefs that seem useful, as evaluated by whether they succeed in giving us rewards. Epistemic and instrumental rationality were the same all along.

And this seems simply wrong. You might as well say "epistemic rationality and chemical action-potentials were the same all along". Or "jumbo jets and sheets of aluminium were the same all along". A jumbo jet might even be made out of sheets of aluminium, but a randomly chosen pile of the latter sure isn't going to fly.

As for your examples, I don't have anything to add to Said's observations.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T15:15:55.398Z · score: 12 (3 votes) · LW · GW

Indeed, the scientific history of how observation and experiment led to a correct understanding of the phenomenon of rainbows is long and fascinating.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T14:28:04.061Z · score: 2 (3 votes) · LW · GW

I'm sorry, what? In this discussion? That seems like an egregious conflict of interest. You don't get to unilaterally decide that my comments are made in bad faith based on your own interpretation of them. I saw which comment of mine you deleted and honestly I'm baffled by that decision.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T04:39:01.399Z · score: 4 (2 votes) · LW · GW

If I may summarize what I think the key disagreement is, you think we can know truth well enough to avoid the problem of the criterion and gain nothing from addressing it.

and to be pointed about it I think believing you can identify the criterion of truth is a “comforting” belief that is either contradictory or demands adopting non-transcendental idealism

Actually... I was going to edit my comment to add that I'm not sure that I would agree that I "think we can know truth well enough to avoid the problem of the criterion" either, since your conception of this notion seems to intrinsically require some kind of magic, leading me to believe that you somehow mean something different by this than I would. But I didn't get around to it in time! No matter.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T02:36:20.453Z · score: 12 (3 votes) · LW · GW

If I may summarize what I think the key disagreement is, you think we can know truth well enough to avoid the problem of the criterion and gain nothing from addressing it.

That's not my only disagreement. I also think that your specific proposed solution does nothing to "address" the problem (in particular because it just seems like a bad idea, in general because "addressing" it to your satisfaction is impossible), and only serves as an excuse to rationalize holding comforting but wrong beliefs under the guise of doing "advanced philosophy". This is why the “powerful but dangerous tool” rhetoric is wrongheaded. It's not a powerful tool. It doesn't grant any ability to step outside your own head that you didn't have before. It's just a trap.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T00:49:56.100Z · score: 10 (2 votes) · LW · GW

If you don't think Eliezer's perspective (cf. The Simple Truth) is sufficient to constitute a criterion of truth, I don't really know what to say to you, to be honest.

As to...

(not that you necessarily would, but you’ve been linking him heavily and I want to cut short the need for a round of conversation where we get over this hurdle)

I'll stop linking the Sequences when people stop needing to read them. :)

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T00:44:05.858Z · score: 7 (5 votes) · LW · GW

I don't have to solve the problem of induction to look out my window and see whether it is raining. I don't need 100% certainty, a four-nines probability estimate is just fine for me.

Where's the "just go to the window and look" in judging beliefs according to "compellingness-of-story"?

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T00:33:59.698Z · score: 12 (3 votes) · LW · GW

Of course not, and that’s the point.

The point... is that judging beliefs according to whether they achieve some goal or anything-- is no more reliable than judging beliefs according to whether they are true, is in no way a solution to the problem of induction or even a sensible response to it, and most likely only makes your epistemology worse?

Indeed, which is why metarationality must not forget to also include all of rationality within it!

Can you explain this in a way that doesn't make it sound like an empty applause light? How can I take compellingness-of-story into account in my probability estimates without violating the Kolmogorov axioms?

To say a little more on danger, I mean dangerous to the purpose of fulfilling your own desires.

Yes, that's exactly the danger.

Unlike politics, which is an object-level danger you are pointing to, postrationality is a metalevel danger, but specifically because it’s a more powerful set of tools rather than a shiny thing people like to fight over. This is like the difference between being weary of generally unsafe conditions that cannot be used and dangerous tools that are only dangerous if used by the unskilled.

Thinking you're skilled enough to use some "powerful but dangerous" tool is exactly the problem. You will never be skilled enough to deliberately adopt false beliefs without suffering the consequences.

Ethical Injunctions:

But surely… if one is aware of these reasons… then one can simply redo the calculation, taking them into account. So we can rob banks if it seems like the right thing to do after taking into account the problem of corrupted hardware and black swan blowups. That’s the rational course, right?

There’s a number of replies I could give to that.

I’ll start by saying that this is a prime example of the sort of thinking I have in mind, when I warn aspiring rationalists to beware of cleverness.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-16T00:11:45.206Z · score: 15 (5 votes) · LW · GW

Because there’s no causal pathway through which we could directly evaluate whether or not our brains are actually tracking reality.

I don't know what "directly" means, but there certainly is a causal pathway, and we can certainly evaluate whether our brains are tracking reality. Just make a prediction, then go outside and look with your eyes to see if it comes true.

Schizophrenics also think that they have causal access to the truth as granted by their senses, and might maintain that belief until their death.

So much the worse for schizophrenics. And so?

“Well we can’t go below 20%, but we can influence what that 20% consists of, so let’s swap that desire to believe ourselves to be better than anyone else into some desire that makes us happier and is less likely to cause needless conflict. Also, by learning to manipulate the contents of that 20%, we become better capable at noticing when a belief comes from the 20% rather than the 80%, and adjusting accordingly”.

I have a hard time believing that this sort of clever reasoning will lead to anything other than making your beliefs less accurate and merely increasing the number of non-truth-based beliefs above 20%.

The only sensible response to the problem of induction is to do our best to track the truth anyway. Everybody who comes up with some clever reason to avoid doing this thinks they've found some magical shortcut, some powerful yet-undiscovered tool (dangerous in the wrong hands, of course, but a rational person can surely use it safely...). Then they cut themselves on it.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-15T17:12:14.866Z · score: 14 (4 votes) · LW · GW

Two points:

  1. Advancing the conversation is not the only reason I would write such a thing, but actually it serves a different purpose: protecting other readers of this site from forming a false belief that there's some kind of consensus here that this philosophy is not poisonous and harmful. Now the reader is aware that there is at least debate on the topic.

  2. It doesn't prove the OP's point at all. The OP was about beliefs (and "making sense of the world"). But I can have the belief "postrationality is poisonous and harmful" without having to post a comment saying so, therefore whether such a comment would advance the conversation need not enter into forming that belief, and is in fact entirely irrelevant.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-15T16:37:08.949Z · score: 24 (8 votes) · LW · GW

Well, this is a long comment, but this seems to be the most important bit:

The general point here is that the human brain does not have magic access to the criteria of truth; it only has access to its own models.

Why would you think "magic access" is required? It seems to me the ordinary non-magic causal access granted by our senses works just fine.

All that you say about beliefs often being critically mistaken due to eg. emotional attachment, is of course true, and that is why we must be ruthless in rejecting any reasons for believing things other than truth -- and if we find that a belief is without reasons after that, we should discard it. The problem is this seems to be exactly the opposite of what "postrationality" advocates: using the lack of "magic access" to the truth as an excuse to embrace non-truth-based reasons for believing things.

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-15T01:49:13.406Z · score: 30 (11 votes) · LW · GW

At the risk of putting words in your mouth, it sounds instead as if you think we can assess the criterion of truth, which we cannot and have known we cannot for over 2000 years.

But of course we can, as evidenced by the fact that people make predictions that turn out to be correct, and carry out plans and achieve goals based on those predictions all the time.

We can't assess whether things are true with 100% reliability, of course. The dark lords of the matrix could always manipulate your mind directly and make you see something false. They could be doing this right now. But so what? Are you going to tell me that we can assess 'telos' with 100% reliability? That we can somehow assess whether it is true that believing something will help fulfill some purpose, with 100% reliability, without knowing what is true?

The problem with assessing beliefs or judgements with anything other than their truth is exactly that the further your beliefs are from the truth, the less accurate any such assessments will be. Worse, this is a vicious positive feedback loop if you use these erroneous 'telos' assessments to adopt further beliefs, which will most likely also be false, and make your subsequent assessments even more inaccurate.

As usual, Eliezer put it best in Ethical Injunctions.

I will at least agree with you that it’s dangerous to folks who are not already rationalists

Being a rationalist isn't a badge that protects you from wrong thinking. Being a rationalist is the discipline and art of correct thought. When you stop practising correct thought, when you stop seeking truth, being a rationalist won't save you, because at that moment you aren't one.

People used to come to this site all the time complaining about the warning about politics: Politics is the Mind-Killer. They would say "for ordinary people, sure it might be dangerous, but we rationalists should be able to discuss these things safely if we're so rational", heedless of the fact that the warning was meant not for ordinary people, but for rationalists. The message was not "if you are weak, you should avoid this dangerous thing; you may demonstrate strength by engaging the dangerous thing and surviving" but "you are weak; avoid this dangerous thing in order to become strong".

Comment by nshepperd on Outline of Metarationality, or much less than you wanted to know about postrationality · 2018-10-14T23:14:17.402Z · score: 10 (10 votes) · LW · GW

Thank you for providing this information.

However, if this is really what 'postrationality' is about, then I think it remains safe to say that it is a poisonous and harmful philosophy that has no place on LW or in the rationality project.

Further, since the criterion for knowing what is true is unreliably known, we must be choosing that criterion on some other basis than truth, and so instead view that prior criterion as coming from usefulness to some purpose we have.

You appear to be saying that since it's impossible to be absolutely certain that any particular thing is the truth, that makes it ok to instead substitute any other easier to solve criteria. This is an incredibly weak justification for anything.

Thus, for example, rationality is important to the purpose of predicting and understanding the world often because we, through experience, come to know it to be correlated with making predictions that later happen, but other criteria, like compellingness-of-story and willingness-to-life, may be better drivers in terms of creating the world we would like to later find ourselves in.

This talk of alternative criteria having equal value sounds very good and cosmopolitan, but actually we know exactly what happens when you stop using truth as your criteria for "truth". Nothing good.

Comment by nshepperd on Moderation Reference · 2018-09-14T02:18:18.691Z · score: 14 (4 votes) · LW · GW

This is an important point, one which was stressed in EY's description of "Crocker's Rules" (and apparently missed by many, as mentioned later on): imposing something on oneself—a discipline—may be very useful, but is a very different thing to expecting that thing of others, and the justifications often do not carry over.

Comment by nshepperd on Zetetic explanation · 2018-09-09T01:26:56.633Z · score: 9 (4 votes) · LW · GW

But if you’re putting Benquo in that category, I really don’t see how we’re going to get more than, say, a post a month on LW, at which point why have LW instead of a collection of personal blogs?

I'd just like to comment that in my opinion, if we only had one post a month on LW, but it was guaranteed to be good and insightful and useful and relevant to the practice of rationality and not wrong in any way, that would be awesome.

The world is full of content. Attention is what is scarce.

Comment by nshepperd on Subsidizing Prediction Markets · 2018-08-20T03:14:00.984Z · score: 16 (7 votes) · LW · GW

Robin has published at least two papers that seem like necessary background reading on this topic:

Note that the logarithmic market scoring rule comes with a built in method of subsidy (in fact, requires such subsidy), and guarantees that there will always be nonzero liquidity.

Removing fees is of course a valuable and necessary first step. There's never any reason to do something so self-defeating as deliberately destroying liquidity by imposing fees.

Comment by nshepperd on Aligned AI May Depend on Moral Facts · 2018-06-15T03:16:37.727Z · score: 9 (3 votes) · LW · GW

What exactly is "moral facts exist" supposed to mean? This whole approach smells off to me -- it looks like you're trying to manipulate your confusion as if it were a known quantity. What metaethical baggage is brought to the table by supposing that "do moral facts exist" is a coherent question at all (assuming you do mean something specific by it)?

Comment by nshepperd on Why Universal Comparability of Utility? · 2018-05-13T16:59:56.883Z · score: 12 (4 votes) · LW · GW

The VNM theorem is best understood as an operator that applies to a function that obeys the axioms and rewrites that function in the form where U is the resulting "utility function" producing a real number. So it rewrites your function into one that compares "expected utilities".

To apply this to something in the real world, a human or an AI, one must decide exactly what refers to and how are interpreted.

  • We can interpret as the actual revealed choices of the agent. Ie. when put in a position to take action to cause either or to happen, what do they do? If the agent's thinking doesn't terminate (within the allotted time), or it chooses randomly, we can interpret that as . The possibilities are fully enumerated, so completeness holds. However, you will find that any real agent fails to obey some of the other axioms.
  • We can interpret as the expressed preferences of the agent. That is to say, present the hypothetical and ask what the agent prefers. Then we say that if the agent says they prefer ; we say that if the agent says they prefer , and we say that if the agent says they are equal or can't decide (within the allotted time). Again completeness holds, but you will again always find that some of the other axioms will fail.
  • In the case of humans, we can interpret as some extrapolated volition of a particular human. In which case we say that if the person would choose if only they thought faster, knew more, were smarter, were more the person they wished they would be, etc. One might fancifully describe this as defining as the person's "true preferences". This is not a practical interpretation, since we don't know how to compute extrapolated volition in the general case. But it's perfectly mathematically valid, and it's not hard to see how it could be defined so that completeness holds. It's plausible that the other axioms could hold too -- most people consider the rationality axioms generally desirable to conform to, so "more the person they wished they would be" plausibly points in a direction that results in such rationality.
  • For some AIs whose source code we have access to, we might be able to just read the source code and define using the actual code that computes preferences.

There are a lot of variables here. One could interpret the domain of as being a restricted set of lotteries. This is the likely interpretation in something like a psychology experiment where we are constrained to only asking about different flavours of ice cream or something. In that case the resulting utility function will only be valid in this particular restricted domain.

Comment by nshepperd on [deleted post] 2017-10-07T22:41:25.179Z

> I'd probably still have the same "speak up and share advice" module, but I'd add a function to the front that injects some gentleness and some status-dynamic-defusing words and phrases.

In this case, I have to object to this advice. You can tie yourself in knots trying to figure out what the most gentle way to say something is, and end up being perceived as condescending etc. anyway for believing that X is "obvious" or that someone else "should have already thought of it" (as again, what is obvious to one person may not be obvious or salient to another). Better to just state the obvious.

Comment by nshepperd on On the importance of Less Wrong, or another single conversational locus · 2016-11-27T19:06:01.168Z · score: 14 (16 votes) · LW · GW

I think you're right that wherever we go next needs to be a clear schelling point. But I disagree on some details.

  1. I do think it's important to have someone clearly "running the place". A BDFL, if you like.

  2. Please no. The comments on SSC are for me a case study in exactly why we don't want to discuss politics.

  3. Something like reddit/hn involving humans posting links seems ok. Such a thing would still be subject to moderation. "Auto-aggregation" would be bad however.

  4. Sure. But if you want to replace the karma system, be sure to replace it with something better, not worse. SatvikBeri's suggestions below seem reasonable. The focus should be on maintaining high standards and certainly not encouraging growth in new users at any cost.

  5. I don't believe that the basilisk is the primary reason for LW's brand rust. As I see it, we squandered our "capital outlay" of readers interested in actually learning rationality (which we obtained due to the site initially being nothing but the sequences) by doing essentially nothing about a large influx of new users interested only in "debating philosophy" who do not even read the sequences (Eternal November). I, personally, have almost completely stopped commenting since quite a while, because doing so is no longer rewarding.

Comment by nshepperd on Conservation of Expected Evidence · 2016-10-03T04:50:52.201Z · score: 0 (0 votes) · LW · GW

"For a true Bayesian, it is impossible to seek evidence that confirms a theory"

The important part of the sentence here is seek. The isn't about falsificationism, but the fact that no experiment you can do can confirm a theory without having some chance of falsifying it too. So any observation can only provide evidence for a hypothesis if a different outcome could have provided the opposite evidence.

For instance, suppose that you flip a coin. You can seek to test the theory that the result was HEADS, by simply looking at the coin with your eyes. There's a 50% chance that the outcome of this test would be "you see the HEADS side", confirming your theory (p(HEADS | you see HEADS) ~ 1). But this only works because there's also a 50% chance that the outcome of the test would have shown the result to be TAILS, falsifying your theory (P(HEADS | you see TAILS) ~ 0). And in fact there's no way to measure the coin so that one outcome would be evidence in favour of HEADS (P(HEADS | measurement) > 0.5), without the opposite result being evidence against HEADS (P(HEADS | ¬measurement) < 0.5).

Comment by nshepperd on Open Thread, Apr. 20 - Apr. 26, 2015 · 2016-01-15T06:41:43.555Z · score: 0 (0 votes) · LW · GW

Under UDT: pay iff you need human contact so much that you'd spend $1000 to be visited by a weird oracle who goes around posing strange decision theory dilemmas.

Comment by nshepperd on The Brain as a Universal Learning Machine · 2015-09-07T04:32:22.378Z · score: 0 (0 votes) · LW · GW

The ferret rewiring experiments, the tongue based vision stuff, the visual regions learning to perform echolocation computations in the blind, this evidence together is decisive against the evolved modularity hypothesis as I've defined that hypothesis, at least for the cortex. The EMH posits that the specific cortical regions rely on complex innate circuitry specialized for specific tasks. The evidence disproves that hypothesis.

It seems a little strange to treat this as a triumphant victory for the ULH. At the most, you've shown that the "fundamentalist" evolved modularity hypothesis is false. You didn't really address how the ULH explains this same evidence.

And there are other mysteries in this model, such as the apparent universality of specific cognitive heuristics and biases, or of various behaviours like altruism, deception, sexuality that seems obviously evolved. And, as V_V mentioned, the lateral asymmetry of the brain's functionality vs the macroscopic symmetry.

Otherwise, the conclusion I would draw from this is that both theories are wrong, or that some halfway combination of them is true (say, "universal" plasticity plus a genetic set of strong priors somehow encoded in the structure).

Comment by nshepperd on MIRI's 2015 Summer Fundraiser! · 2015-08-29T05:01:19.215Z · score: 12 (12 votes) · LW · GW

I donated $10. I can't afford not to.

Comment by nshepperd on Open Thread, Apr. 20 - Apr. 26, 2015 · 2015-07-22T04:00:14.822Z · score: 0 (2 votes) · LW · GW

This is essentially just another version of the smoking lesion problem, in that there is no connection, causal or otherwise, beween the thing you care about and the action you take. Your decision theory has no specific effect on your likelyhood of dying, that being determined entirely by environmental factors that do not even attempt to predict you. All you are paying for is to determine whether or not you get a visit from the oracle.

ETA: Here's a UDT game tree (see here for an explanation of the format) of this problem, under the assumption that oracle visits everyone meeting his criteria, and uses exclusive-or:

ETA2: More explanation: the colours are states of knowledge. Blue = oracle asks for money, Orange = they leave you alone. Let's say the odds of being healthy are α. If you Pay the expected reward is α(-1000) + (1-α) DEATH; if you Don't Pay the expected reward is α 0 + (1-α) DEATH. Clearly (under UDT) paying is worse by a term of -1000α.