Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-13T19:59:19.375Z · score: 2 (1 votes) · LW · GW

Hm. Interesting; I hadn't realized you intended that to be your answer. In that case, however, the question simply gets kicked one level back:

conditional on our ability to observe anything

Why do we have this ability in the first place?

(Also, even granting that our ability to make observations implies some level of predictability--which I'm not fully convinced of--I don't think it implies the level of predictability we actually observe. For one thing, it doesn't rule out the possibility of the laws of physics changing every Sunday. I'm curious to know, on your model, why don't we observe anything like that?)

Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-07T04:51:41.328Z · score: 2 (1 votes) · LW · GW
are you asking why they are not random and unpredictable?

Yes, I am. I don't see why the fact that that's an "observation in itself" makes it an invalid question to ask. The fact of the matter is, there are many possible observation sequences, and the supermajority of those sequences contain nothing resembling structure or regularity. So the fact that we appear to be recording an observation sequence that is ordered introduces an improbability that needs to be addressed. How do you propose to address this improbability?

Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-06T01:18:03.332Z · score: 2 (1 votes) · LW · GW

My actual question was "why are our observations structured rather than unstructured?", which I don't think you actually answered; the closest you got was

Again, without a certain regularity in our observations we would not be here talking about it. Or hallucinating talking about it. Or whatever.

which isn't actually an explanation, so far as I can tell. I'd be more interested in hearing an object-level answer to the question.

Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-04T21:55:24.935Z · score: 2 (1 votes) · LW · GW

That doesn't seem to answer the question? You seem to be claiming that because any answer to the question will necessitate the asking of further questions, that means the question itself isn't worth answering. If so, I think this is a claim that needs defending.

Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-03T04:45:11.652Z · score: 2 (1 votes) · LW · GW
It's an empirical fact (a meta-observation) that they do.

Right, and I'm asking why this particular meta-observation holds, as opposed to some other meta-observation, such as e.g. the meta-observation that the laws of physics change to something different every Sunday, or perhaps the meta-observation that there exists no regularity in our observations at all.

Comment by dxu on Experimental Open Thread April 2019: Socratic method · 2019-04-03T04:08:17.443Z · score: 2 (1 votes) · LW · GW

If observations do not require a source, then why do they seem to exhibit various regularities that allow them to be predicted with a greater accuracy than chance?

Comment by dxu on Philosophy as low-energy approximation · 2019-02-08T00:14:58.800Z · score: 1 (2 votes) · LW · GW
Seeing red is more than a role or disposition. That is what you have left out.

Do you have any evidence for this claim, besides a subjective feeling of certainty?

Comment by dxu on Open Thread January 2019 · 2019-01-16T02:59:19.936Z · score: 41 (11 votes) · LW · GW
FDT says you should not pay because, if you were the kind of person who doesn't pay, you likely wouldn't have been blackmailed. How is that even relevant? You are being blackmailed.

I'm quoting this because, even though it's wrong, it's actually an incredibly powerful naive intuition. I think many people who have internalized TDT/UDT/FDT-style reasoning have forgotten just how intuitive the quoted block is. The unstated underlying assumption here (which is unstated because Schwarz most likely doesn't even realize it is an assumption) is extremely persuasive, extremely obvious, and extremely wrong:

If you find yourself in a particular situation, the circumstances that led you to that situation are irrelevant, because they don't change the undeniable fact that you are already here.

This is the intuition driving causal decision theory, and it is so powerful that mainstream academic philosophers are nearly incapable of recognizing it as an assumption (and a false assumption at that). Schwarz himself demonstrates just how hard it is to question this assumption: even when the opposing argument was laid right in front of him, he managed to misunderstand the point so hard that he actually used the very mistaken assumption the paper was criticizing as ammunition against the paper. (Note: this is not intended to be dismissive toward Schwarz. Rather, it's simply meant as an illustrative example, emphasizing exactly how hard it is for anyone, Schwarz included, to question an assumption that's baked into their model of the world.) And if even if you already understand why FDT is correct, it still shouldn't be hard to see why the assumption in question is so compelling:

How could what happened in the past be relevant for making a decision in the present? The only thing your present decision can affect is the future, so how could there be any need to consider the past when making your decision? Surely the only relevant factors are the various possible futures each of your choices leads to? Or, to put things in Pearlian terms: it's known that the influence of distant causal nodes is screened off by closer nodes through which they're connected, and all future nodes are only connected to past nodes through the present--there's no such thing as a future node that's directly connected to a past node while not being connected to the present, after all. So doesn't that mean the effects of the past are screened off when making a decision? Only what's happening in the present matters, surely?

Phrased that way, it's not immediately obvious what's wrong with this assumption (which is the point, of course, since otherwise people wouldn't find it so difficult to discard). What's actually wrong is something that's a bit hard to explain, and evidently the explanation Eliezer and Nate used in their paper didn't manage to convey it. My favorite way of putting it, however, is this:

In certain decision problems, your counterfactual behavior matters as much--if not more--than your actual behavior. That is to say, there exists a class of decision problems where the outcome depends on something that never actually happens. Here's a very simple toy example of such a problem:

Omega, the alien superintelligence, predicts the outcome of a chess game between you and Kasparov. If he predicted that you'd win, he gives you $500 in reality; otherwise you get nothing.

Strictly speaking, this actually isn't a decision problem, since the real you is never faced with a choice to make, but it illustrates the concept clearly enough: the question of whether or not you receive the $500 is entirely dependent on a chess game that never actually happened. Does that mean the chess game wasn't real? Well, maybe; it depends on your view of Platonic computations. But one thing it definitely does not mean is that Omega's decision was arbitrary. Regardless of whether you feel Omega based his decision on a "real" chess game, you would in fact either win or not win against Kasparov, and whether you get the $500 really does depend on the outcome of that hypothetical game. (To make this point clearer, imagine that Omega actually predicted that you'd win. Surprised by his own prediction, Omega is now faced with the prospect of giving you $500 that he never expected he'd actually have to give up. Can he back out of the deal by claiming that since the chess game never actually happened, the outcome was up in the air all along and therefore he doesn't have to give you anything? If your answer to that question is no, then you understand what I'm trying to get at.)

So outcomes can depend on things that never actually happen in the real world. Cool, so what does that have to do with the past influencing the future? Well, the answer is that it doesn't--at least, not directly. But that's where the twist comes in:

Earlier, when I gave my toy example of a hypothetical chess game between you and Kasparov, I made sure to phrase the question so that the situation was presented from the perspective of your actual self, not your hypothetical self. (This makes sense; after all, if Omega's prediction method was based on something other than a direct simulation, your hypothetical self might not even exist.) But there was another way of describing the situation:

You're going about your day normally when suddenly, with no warning whatsoever, you're teleported into a white void of nothingness. In front of you is a chessboard; on the other side of the chessboard sits Kasparov, who challenges you to a game of chess.

Here, we have the same situation, but presented from the viewpoint of the hypothetical you on whom Omega's prediction is based. Crucially, the hypothetical you doesn't know that they're hypothetical, or that the real you even exists. So from their perspective, something random just happened for no reason at all. (Yes, yes, if Omega used some method other than a simulation to make his prediction, the hypothetical you wouldn't have existed and wouldn't have had a perspective--but hey, that doesn't stop me from writing from their perspective, right? After all, real people write from the perspectives of unreal people all the time; that's just called writing fiction. And besides, we've already established that real or unreal, the outcome of the game really does determine whether you get the $500, so the thoughts and feelings of the hypothetical you are nonetheless important in that they partially determine the outcome of the game.)

And now we come to the final, crucial point that makes sense of the blackmail scenario and all the other thought experiments in the paper, the point that Schwarz and most mainstream philosophers haven't taken into account:

Every single one of those thought experiments could have been written from the perspective, not of the real you, but a hypothetical, counterfactual version of yourself.

When "you're" being blackmailed, Schwarz makes the extremely natural assumption that "you" are you. But there's no reason to suppose this is the case. The scenario never stipulates why you're being blackmailed, only that you're being blackmailed. So the person being blackmailed could be either the real you or a hypothetical. And the thing that determines whether it's the real you or a mere hypothetical is...

...your decision whether or not to pay up, of course.

If you cave into the blackmail and pay up, then you're almost certainly the real deal. On the other hand, if you refuse to give in, it's very likely that you're simply a counterfactual version of yourself living in an extremely low-probability (if not outright inconsistent) world. So your decision doesn't just determine the future; it also determines (with high probability) which you "you" are. And so then the problem simplifies into this: which you do you want to be?

If you're the real you, then life kinda sucks. You just got blackmailed and you paid up, so now you're down a bunch of money. If, on the other hand, you're the hypothetical version of yourself, then congratulations: "you" were never real in the first place, and by counterfactually refusing to pay, you just drastically lowered the probability of your actual self ever having to face this situation and (in the process) becoming you. And when things are put that way, well, the correct decision becomes rather obvious.

But this kind of counterfactual reasoning is extremely counterintuitive. Our brains aren't designed for this kind of thinking (well, not explicitly, anyway). You have to think about hypothetical versions of yourself that have never existed and (if all goes well) will never exist, and therefore only exist in the space of logical possibility. What does that even mean, anyway? Well, answering confused questions like that is pretty much MIRI's goal these days, so I dunno, maybe we can ask them.

Comment by dxu on What are the components of intellectual honesty? · 2019-01-16T01:54:37.319Z · score: 4 (2 votes) · LW · GW

(Posted as a comment rather than an answer because all of this is pretty rambling, and I'm not super-confident about any of the stuff I say below, even if my tone or phrasing seems to suggest otherwise.)

For the purposes of a discussion like this, rather than talk about what intellectual honesty is, I think it makes more sense to talk about what intellectual honesty is not. Specifically, I'd suggest that the kinds of behavior we consider "intellectually honest" are simply what human behavior looks like when it's not being warped by some combination of outside incentives. The reason intellectual honesty is so hard to find, then, is simply that humans tend to find themselves influenced by external incentives almost all of the time. Even absent more obvious factors like money or power, humans are social creatures, and all of us unconsciously track the social status of ourselves and others. Throw in the fact that social status is scarce by definition, and we end up playing all sorts of social games "under the table".

This affects practically all of our interactions with other people, even interactions ostensibly for some other purpose (such as solving a problem or answering a question). Unless people are in a very specific kind of environment, by default, all interactions have an underlying status component: if I say something wrong and someone corrects me on it, I'm made to seem less knowledgeable in comparison, and so that person gains status at my expense. If you're in an environment where this sort of thing is happening (and you pretty much always are), naturally you're going to divert some effort away from accomplishing whatever the actual goal is, and toward maintaining or increasing your social standing. (Of course, this behavior needn't be conscious at all; we're perfectly capable of executing status-increasing maneuvers without realizing we're doing it.)

This would suggest that intellectual honesty is most prevalent in fields that prioritize problem-solving over status, and (although confirmation bias is obviously a thing) I do think this is observably true. For example, when a mathematician finds that they've made a mistake, they pretty much always own up to it immediately, and other mathematicians don't respect them less for doing so. (Ditto physicists.) And this isn't because mathematicians and physicists have some magical personality trait that makes them immune to status games--it's simply because they're focused on actually doing something, and the thing they're doing is more important to them than showing off their own cleverness.

If you and I are working together to solve a particular problem, and both of us actually care about solving the problem, then there's no reason for me to feel threatened by you, even if you do something that looks vaguely like a status grab (such as correcting me when I make a mistake). Because I know that we're fundamentally on the same side, I don't need to worry nearly as much about what I say or do in front of you, which in turn allows me to voice my actual thoughts and opinions much more freely. The atmosphere is collaborative rather than competitive. In that situation, both of us can act "intellectually honest", but importantly, there's not even a need for that term. No one's going to compliment me on how "intellectually honest" I'm being if I quickly admit that I made a mistake, because, well, why would I be doing anything other than trying to solve the problem I set out to solve? It's a given that I'd immediately abandon any unpromising or mistaken approaches; there's nothing special about that kind of behavior, and so there's no need to give it a special name like "intellectual honesty".

The only context in which "intellectual honesty" is a useful concept is one that's already dominated by status games. Only in cases where the incentives are sharply aligned against admitting that you're wrong does it become something laudable, something unusual, something to be praised whenever someone actually does it. In practice, these kinds of situations crop up all the time because status is something humans breathe, but I still think it's useful to point out that "intellectual honesty" is really just the default mode of behavior, even if that default mode is often corrupted by other stuff.

Comment by dxu on Tentatively considering emotional stories (IFS and “getting into Self”) · 2018-12-02T07:39:20.809Z · score: 8 (4 votes) · LW · GW

Firstly, to make sure all of us are on the same page: "procrastination", as the word is the typically used, does not mean that one sits down and thinks carefully about the benefits and drawbacks of beginning to work right now as opposed to later, and then, as a result of this consideration, rationally decides that beginning to work later is a more optimal decision. Rather, when most people use the word "procrastinate", they generally mean that they themselves are aware that they ought to start working immediately--such that if you asked them if they endorsed the statement "I should be working right now", they would wholeheartedly reply that they do--and yet mysteriously, they still find themselves doing something else.

If, Said, you have not experienced this latter form of procrastination, then I'm sure you are the object of envy for many people here (including myself). If, however, you have, and this is what you were referring to when you answered "yes" to lkaxas' question, then the followup question about "internal experience" can be interpreted thusly:

Why is it that, even though you consciously believe that working is the correct thing to be doing, and would verbally endorse such a sentiment if asked, you nonetheless do not do the thing you think is correct to do? This is not merely "irrational"; it seems to defy the very concept of agency--you are unable to act on your own will to act, which seems to undercut the very notion that you choose to do things at all. What does it feel like when this strange phenomenon occurs, when your agency seems to disappear for no explicable reason at all?

To this, certain others (such as myself and, I presume, lkaxas and Kaj Sotala) would reply that there is some additional part of our decision-making process, perhaps a less conscious, less explicit part whose desires we cannot verbalize on demand and are often entirely unaware of, which does not endorse our claim that to begin working now is the best thing to do. This part of us may feel some sense of visceral repulsion when the thought of working arises, or perhaps it may simply be attracted to something else that it would rather be doing--but regardless of the cause, the effect of that hidden desire overrides our conscious will to work, and a result, we end up doing something other than working, despite the fact that we genuinely do wish to work. (Much of IFS, as I understand it, has to do with identifying these more subtle parts of our minds and promoting them to conscious attention so that they may be analyzed with the same rigor one devotes to one's normal thoughts.)

You, however, seem to have rejected this multi-agent framework, and so--assuming that you have in fact experienced "procrastination" as described above--your experience while procrastinating must describe something else entirely, something which need not invoke reference to such concepts as desires being "overridden" by deeper desires, or a different "part" of oneself that wants different things than the one does. If so, could you provide such a description?

Comment by dxu on Sam Harris and the Is–Ought Gap · 2018-11-16T21:52:05.580Z · score: 13 (5 votes) · LW · GW

This post seems relevant. (Indeed, it seems to dissolve the question entirely, and a full decade in advance.)

Comment by dxu on Tradition is Smarter Than You Are · 2018-09-20T20:19:57.901Z · score: 1 (2 votes) · LW · GW
I suspect that this is evidence in favour of slower takeoff speeds, because being as smart as humans isn't nearly enough to do as well as humans.

I don't see the connection between the latter claim and the former claim.

Comment by dxu on Resurrection of the dead via multiverse-wide acausual cooperation · 2018-09-07T04:38:50.757Z · score: 2 (1 votes) · LW · GW
I don't understand what "the underlying causality I am part of" can possibly mean, since causality is a human way to model observations. This statement seems to use the mind projection fallacy to invert the relationship between map and territory.

If you want to discount the use of causal models as merely a "human way to model observations" (one that presumably bears no underlying connection to whatever is generating those observations), then you will need to explain why they work so well. The set of all possible sequences of observations is combinatorially large, and the supermajority of those sequences admit no concise description--they contain no regularity or structure that would allow us to substantially compress their length without losing information. The fact that our observations do seem to be structured, therefore, is a very improbable coincidence indeed. The belief in an external reality is simply a rejection of the notion that this extremely improbable circumstance is a coincidence.

Comment by dxu on Decisions are not about changing the world, they are about learning what world you live in · 2018-07-29T18:40:43.010Z · score: 5 (3 votes) · LW · GW
This is a strange scenario (it seems to be very different from the sort of scenario one usually encounters in such problems), but sure, let’s consider it. My question is: how is it different from “Omega doesn’t give A any money, ever (due to a deep-seated personal dislike of A). Other agents may, or may not, get money, depending on various factors (the details of which are moot)”?
This doesn’t seem to have much to do with decision theories.

Yes, this is correct, and is precisely the point EYNS was trying to make when they said

Intuitively, this problem is unfair to Fiona, and we should compare her performance to Carl’s not on the “act differently from Fiona” game, but on the analogous “act differently from Carl” game.

"Omega doesn't give A any money, ever (due to a deep-seated personal dislike of A)" is a scenario that does not depend on the decision theory A uses, and hence is an intuitively "unfair" scenario to examine; it tells us nothing about the quality of the decision theory A is using, and therefore is useless to decision theorists. (However, formalizing this intuitive notion of "fairness" is difficult, which is why EYNS brought it up in the paper.)

I'm not sure why shminux seems to think that his world-counting procedure manages to avoid this kind of "unfair" punishment; the whole point of it is that it is unfair, and hence unavoidable. There is no way for an agent to win if the problem setup is biased against them to start with, so I can only conclude that shminux misunderstood what EYNS was trying to say when he (shminux) wrote

I note here that simply enumerating possible worlds evades this problem as far as I can tell.
Comment by dxu on Decisions are not about changing the world, they are about learning what world you live in · 2018-07-29T04:15:22.217Z · score: 4 (2 votes) · LW · GW

Say you have an agent A who follows the world-enumerating algorithm outlined in the post. Omega makes a perfect copy of A and presents the copy with a red button and a blue button, while telling it the following:

"I have predicted in advance which button A will push. (Here is a description of A; you are welcome to peruse it for as long as you like.) If you press the same button as I predicted A would push, you receive nothing; if you push the other button, I will give you $1,000,000. Refusing to push either button is not an option; if I predict that you do not intend to push a button, I will torture you for 3^^^3 years."

The copy's choice of button is then noted, after which the copy is terminated. Omega then presents the real agent facing the problem with the exact same scenario as the one faced by the copy.

Your world-enumerating agent A will always fail to obtain the maximum $1,000,000 reward accessible in this problem. However, a simple agent B who chooses randomly between the red and blue buttons has a 50% chance of obtaining this reward, for an expected utility of $500,000. Therefore, A ends up in a world with lower expected utility than B.

Q.E.D.

Comment by dxu on Bayesian Probability is for things that are Space-like Separated from You · 2018-07-24T07:48:08.199Z · score: 3 (4 votes) · LW · GW

[META] As a general heuristic, when you encounter a post from someone otherwise reputable that seems completely nonsensical to you, it may be worth attempting to find some reframing of it that causes it to make sense--or at the very least, make more sense than before--instead of addressing your remarks to the current (nonsensical-seeming) interpretation. The probability that the writer of the post in question managed to completely lose their mind while writing said post is significantly lower than both the probability that you have misinterpreted what they are saying, and the probability that they are saying something non-obvious which requires interpretive effort to be understood. To maximize your chances of getting something useful out of the post, therefore, it is advisable to condition on the possibility that the post is not saying something trivially incorrect, and see where that leads you. This tends to be how mutual understanding is built, and is a good model for how charitable communication works. Your comment, to say the least, was neither.

Comment by dxu on Beyond Astronomical Waste · 2018-06-08T18:18:50.278Z · score: 7 (2 votes) · LW · GW
I am not sure how one can talk about the observed universe and the number 3^^^3 in the same sentence, given that the maximum informational content is roughly 10^120 qubits, the rest is outside the cosmological horizon.

Where in the post do you see it suggested that our universe is capable of containing 3^^^3 of anything?

Alternatively, if we talk about the simulation argument, then the expression "practical implications" seems out of place.

How so?

Comment by dxu on Beyond Astronomical Waste · 2018-06-08T18:16:37.449Z · score: 9 (2 votes) · LW · GW
If there's some kind of measure of "observer weight" over the whole mathematical universe, we might be already much larger than 1/3^^^3 of it, so the total utilitarian can only gain so much.

Could you provide some intuition for this? Naively, I'd expect our "observer measure" over the space of mathematical structures to be 0.

Comment by dxu on The Jordan Peterson Mask · 2018-03-08T16:22:04.663Z · score: 10 (4 votes) · LW · GW

Saving the world certainly does seem to be an instrumentally convergent strategy for many human terminal values. Whatever you value, it's hard to get more of it if the world doesn't exist. This point should be fairly obvious, and I find myself puzzled as to why you seem to be ignoring it entirely.

Comment by dxu on My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms · 2018-03-08T10:18:05.933Z · score: 87 (23 votes) · LW · GW
While I liked Valentine’s recent post on kensho and its follow-ups a lot, one thing that I was annoyed by were the comments that the whole thing can’t be explained from a reductionist, third-person perspective. I agree that such an explanation can’t produce the necessary mental changes that the explanation is talking about. But it seemed wrong to me to claim that all of this would be somehow intrinsically mysterious and impossible to explain on such a level that would give people at least an intellectual understanding of what Looking and enlightenment and all that are.

Speaking as someone who's more or less avoided participating in the kensho discussion (and subsequent related discussions) until now, I think the quoted passage pretty much nails the biggest reservation I had with respect to the topic: the language used in those threads tended to switch back and forth between factual and metaphorical with very little indication as to which mode was being used at any particular moment, to the point where I really wanted to just say, "Okay, I sort of see what you're gesturing at and I'd love to discuss this with you in good faith, but before we get started on that, can we quickly step out of mythic mode/metaphor land/narrative thinking for a moment, just to make sure that we are all still on the same page as far as basic ontology goes, and agree that, for instance, physics and mathematics and logic are still true?"

But when other people in those threads (such as, for example, Said Achmiz) asked essentially the same question, it seemed to me (as in System-1!seemed) that Val and others would simply respond with "It doesn't matter what basic ontology you're using unless that ontology actually helps you Look." Which, okay, fine, but I don't really want to start trying to Look until I can confirm the absence of some fairly huge epistemic issues that typically plague this region of thought-space.

All of which is to say, I'm glad this post was made. ;-)

(although there is a part of me that can't help but wonder why this post or something like it wasn't the opener for this topic, as opposed to something that was only typed up after a couple of huge demon threads spawned)

Comment by dxu on Robustness to Scale · 2018-02-22T01:43:02.398Z · score: 0 (3 votes) · LW · GW
The purpose of this post is to communicate, not to persuade. It may be that we want to bit [sic] the bullet of the strongest form of robustness to scale, and build an AGI that is simply not robust to scale, but if we do, we should at least realize that we are doing that.
Comment by dxu on [Meta] New moderation tools and moderation guidelines · 2018-02-21T09:29:28.093Z · score: 20 (8 votes) · LW · GW
Indeed I am, and for good reason: the cost I speak of is one which utterly dwarfs all others.

This is a claim that requires justification, not bald assertion--especially in this kind of thread, where you are essentially implying that anyone who disagrees with you must be either stupid or malicious. Needless to say, this implication is not likely to make the conversation go anywhere positive. (In fact, this is a prime example of a comment that I might delete were it to show up on my personal blog--not because of its content, but because of the way in which that content is presented.)

Issues with tone aside, the quoted statement strongly suggests to me that you have not made a genuine effort to consider the other side of the argument. Not to sound rude, but I suspect that if you were to attempt an Ideological Turing Test of alkjash's position, you would not in fact succeed at producing a response indistinguishable from the genuine article. In all charitability, this is likely due to differences of internal experience; I'm given to understand that some people are extremely sensitive to status-y language, while others seem blind to it entirely, and it seems likely to me (based on what I've seen of your posts) that you fall into the latter category. In no way does this obviate the existence or the needs of the former category, however, and I find your claim that said needs are "dwarfed" by the concerns most salient to you extremely irritating.

Footnote: Since feeling irritation is obviously not a good sign, I debated with myself for a while about whether to post this comment. I decided ultimately to do so, but I probably won't be engaging further in this thread, so as to minimize the likelihood of it devolving into a demon thread. (It's possible that it's already too late, however.)

Comment by dxu on Hufflepuff Cynicism on Crocker's Rule · 2018-02-15T03:19:57.350Z · score: 7 (2 votes) · LW · GW

It's also entirely information-free, which means that as an epistemic aid it's rather... lacking.

Comment by dxu on Factorio, Accelerando, Empathizing with Empires and Moderate Takeoffs · 2018-02-04T19:12:13.855Z · score: 9 (2 votes) · LW · GW

The argument is never about how soon the future will come, always about how good the future will be. There is nothing "wrong" with any given outcome, but if we can do better, then it's worth dedicating thought to that.

Comment by dxu on A LessWrong Crypto Autopsy · 2018-01-28T21:18:19.321Z · score: 43 (16 votes) · LW · GW

I think a large part of what prevented many people from investing in Bitcoin may have been the epistemic norms commonly referred to nowadays as "the absurdity heuristic", "the outside view", "modest epistemology", etc. In other words, many of us may have held the (subconscious) belief that it's impossible to perform substantially better than the market, even in situations where the Efficient Markets Hypothesis may not fully apply. To put it another way:

Well, suppose God had decided, out of some sympathy for our project, to make winning as easy as possible for rationalists. He might have created the biggest investment opportunity of the century, and made it visible only to libertarian programmers willing to dabble in crazy ideas. And then He might have made sure that all of the earliest adapters were Less Wrong regulars, just to make things extra obvious.

I think many of us considered this, and unconsciously dismissed it due to the obvious absurdity: surely things can't be that easy, right? Sure, we may be rationalists, and sure, rationalists "ought to win", but surely winning can't be so easy that the opportunity to win literally hits us on the head, right?

I think what this points to is a fundamental inability on our part to Take Ideas Seriously. Of course, most people don't have this ability at all, and we're surely doing much better on that count--but what matters in this case isn't your relative superiority to other people, but your absolute level of skill. (I'm using the pronoun "your" here to refer to the majority of rationalists who didn't invest in Bitcoin, not the few who did.) The corresponding solution seems obvious: work to improve our ability to Take Ideas Seriously, without dismissing absurd-sounding ideas too quickly.

Easier said than done, of course.

Comment by dxu on Against Instrumental Convergence · 2018-01-28T19:11:30.636Z · score: 4 (1 votes) · LW · GW

Incidentally, I'm also interested in what specifically you mean by "random program". A natural interpretation is that you're talking about a program that is drawn from some kind of distribution across the set of all possible programs, but as far as I can tell, you haven't actually defined said distribution. Without a specific distribution to talk about, any claim about how likely a "random program" is to do anything is largely meaningless, since for any such claim, you can construct a distribution that makes that claim true.

(Note: The above paragraph was originally a parenthetical note on my other reply, but I decided to expand it into its own, separate comment, since in my experience having multiple unrelated discussions in a single comment chain often leads to unproductive conversation.)

Comment by dxu on Against Instrumental Convergence · 2018-01-28T19:09:20.540Z · score: 4 (1 votes) · LW · GW
I claim that a given random program, regardless of whether it explicitly predicts the future, is unlikely to have the kind of motivational structure that would exhibit instrumental convergence.

Yes, I understand that. What I'm more interested in knowing, however, is how this statement connects to AI alignment in your view, since any AI created in the real world will certainly not be "random".

Comment by dxu on Against Instrumental Convergence · 2018-01-28T09:05:55.325Z · score: 4 (1 votes) · LW · GW
Though I don't want to make claims about how common such programs would be.

If you don't want to make claims about how common such programs are, how do you defend the (implicit) assertion that such programs are worth talking about, especially in the context of the alignment problem?

Comment by dxu on Magic Brain Juice · 2018-01-26T18:07:32.918Z · score: 36 (12 votes) · LW · GW
We are agents who cannot simply act because every action is accompanied by self-modification.

Just want to say; this is one hell of a line.

Comment by dxu on Taking it Private: Short Circuiting Demon Threads (working example) · 2018-01-24T00:56:34.172Z · score: 8 (2 votes) · LW · GW
This is a branch of two people only, so your "step 1" doesn't really make sense

Based on personal experience, I think there's a difference between having a conversation in private, versus having it with a single person, but in a public place where anyone can read what either of you two is saying and form impressions of you based on it. If you agree that such a distinction exists in principle, then I think that suffices to address the quoted objection.

Slightly tangentially: I should also note that I do not view the thread you linked as a particularly strong example of a demon thread, if it is one at all. Of course, I only skimmed the thread in question, so it's possible that I missed something; it's also possible that because I was not a participant in that thread myself and don't possess any social connections to either of the participants, the stakes in status were harder for me to perceive. Even so, I think that if you want to talk about examples of demon threads, there are much clearer cases to point to. (Is there a specific reason you chose that particular thread to talk about, or was it simply due to said thread being fresh in your mind?)

Comment by dxu on Taking it Private: Short Circuiting Demon Threads (working example) · 2018-01-23T01:42:14.342Z · score: 5 (2 votes) · LW · GW

I'm somewhat confused about this specific point of yours, because in the article itself, you write

Step 3. They write a short summary of whatever progress they were able to make (and any major outstanding disagreements that remain). They must both endorse the summary. Writing such a summary needs to get you as much kudos / feel-good as winning an argument does.

which I interpreted to mean "both participants must give explicit verbal endorsement of the summary before it gets posted". It's possible that my interpretation is mistaken, but right now it's not entirely obvious to me how one is supposed to make sure that "they [...] both endorse the summary" without asking first.

It's also possible that what you're saying is that we should omit this part of the procedure for the time being, in order to make sure the procedure doesn't present too much of a trivial inconvenience for people to try it. If so, however, I think it's worth making this explicit in your summary of the procedure itself, perhaps with a simple edit like the following:

Step 3. They write a short summary of whatever progress they were able to make (and any major outstanding disagreements that remain). Ideally, both participants should endorse the summary before it is published, although I think this particular requirement should remain optional for the time being. Writing such a summary needs to get you as much kudos / feel-good as winning an argument does.
Comment by dxu on Taking it Private: Short Circuiting Demon Threads (working example) · 2018-01-23T01:28:43.728Z · score: 4 (1 votes) · LW · GW

I think a potential factor to consider here is that normally, even when speaking in private, there's no spoken guarantee that the conversation will remain private, e.g. it's entirely possible that at some point after having had a private conversation with you, I might offhandedly mention to someone else "Raemon said X a while back"--and the possibility of my doing so brings back the common knowledge/signaling aspect that so often leads to demon threads. Hypothetically, therefore, a private conversation where both participants agree beforehand to not make the conversation public unless both of them agree would lack this aspect entirely, and hence make it much easier to talk in good faith.

I admit that this may seem a bit like hair-splitting, though. I think most participants in this conversation have participated in enough demon threads in the past to have a fairly decent idea of what we're all referring to, and slight differences in intuition like this may not be worth bringing up. (Of course, sometimes they point to a much deeper and more fundamental inferential gap, but I'm inclined to think this isn't the case here.)

Comment by dxu on Nonlinear perception of happiness · 2018-01-08T23:48:52.516Z · score: 12 (3 votes) · LW · GW
I'm a bit confused by "identifying any function of happiness with utility seems clearly wrong to me" : Do you propose the actual utility function as you understand it, has no relation to happiness at all?

I believe what Qiaochu is saying is that not that happiness isn't a component of your utility function, but rather that it doesn't comprise the entirety of your utility function. (Normatively speaking, of course. In practical terms humans don't even behave as though they have a consistent utility function.)

Comment by dxu on Niceness Stealth-Bombing · 2018-01-08T23:15:43.686Z · score: 21 (6 votes) · LW · GW

This isn't new advice by any means, but we forget it frequently enough that reminders such as this are always welcome. Upvoted.

Comment by dxu on Hero Licensing · 2017-11-20T01:21:14.286Z · score: 21 (7 votes) · LW · GW
This added fact contains a lot of important information and Pat glosses over it.

Ah, but this fact isn't third-party visible, which is one of the requirements Pat has before they'll accept it as valid evidence. (Since, you know, anyone can say that they feel it's going well, right? What makes you feel especially confident about your own judgment as to whether or not a given project is going well?)

Comment by dxu on Status Regulation and Anxious Underconfidence · 2017-11-17T04:20:14.465Z · score: 13 (4 votes) · LW · GW

This is true, but also doesn't seem to engage with the point of the book, which is largely about when to trust yourself over others, as opposed to some random (person who may or may not be a) crackpot. (In the latter case, you can't trust that you're not being presented with deliberately filtered evidence.)

Moreover, even in the latter case, it's possible to be skeptical of someone's claims without making the further assertion that they cannot possibly know what they claim to know. It's one thing to say, "What you say appears to makes sense, but I don't know enough about the subject to be able to tell if that's because it actually makes sense, or because I just can't see where the flaw is," and quite another to say, "No, I unilaterally reject the argument you're making because you don't have the credentials to back it up."

EDIT: For some reason I can't get the site to stop mangling the second hyperlink. Although I kept said hyperlink for reference, here is the actual page address: http://squid314.livejournal.com/350090.html

Comment by dxu on An Equilibrium of No Free Energy · 2017-11-01T02:28:54.280Z · score: 12 (3 votes) · LW · GW

This is an important failure mode to consider, to be sure, but why do you think we've fallen into it? And more relevantly, is this something you're saying you've observed about LessWrong 2.0 in particular, or the rationalist movement in general?

Comment by dxu on Inadequacy and Modesty · 2017-10-29T03:05:22.765Z · score: 19 (5 votes) · LW · GW
If we agree modest folk could criticize the Bank of Japan out of deference to Nobel laureates

Point of clarification: Eliezer was not, in fact, deferring to Nobel laureates who were critical of Japan's monetary policy, or even aware that such laureates existed at the time. He was specifically deferring to econ bloggers who he happened to follow. Nor should we consider it an act of modesty ("not taking the inside view") to side with one set of experts over another; to do so is to call the opposing side wrong, after all.

Comment by dxu on There's No Fire Alarm for Artificial General Intelligence · 2017-10-16T09:54:53.139Z · score: 5 (2 votes) · LW · GW

Re 'maybe this was all a ploy / rhetorical device', I'm skeptical that that's true in any strong/unusual sense.

I don't think that there's anything particularly unusual about someone asking "Is there any evidence for claim X?" to imply that, no, there is not enough evidence for claim X. Rhetorical questions are such a common argumentative technique that you can sometimes employ them without even being consciously aware of it. That still doesn't make it the kind of style of discourse I approve of, however, and downvoting is a compact way of expressing that disapproval.

I also want to discourage treating it as normal, at the outset of a debate over some set of factual issues, to publicly speculate that the person on the other side has bad motives (in an accusatory/critical/dismissive/etc. way).

To be clear, I didn't reply to the original comment at all; my initial comment upthread was written purely in response to Zvi's allegation that the downvoters of Quanticle's comment were being unfair and/or petty. I disagreed with the notion that there was no valid reason to downvote, and I replied for that reason and that reason only. I certainly didn't intend my comment to be interpreted as "public speculation" regarding Quanticle's motives, only as an indication that the phrasing used could give the impression of bad motives, which I think is just as important as whether it was actually intended that way. More generally, however:

You said that the substance of a comment is more important than its tone, and I certainly don't disagree, but that still doesn't mean that issues relating to tone are unimportant. In fact, I'd go so far as to say that the way a commenter phrases certain things can very strongly shape the subsequent flow of discussion, and that in some cases the effects are strong enough to outweigh the actual substance of their comment entirely, especially when there's little to no substance to begin with (as in this case). Given that, I think voting based on such "ephemeral" considerations as tone and phrasing is just as valid as any other means of voting, and I take issue with the idea that you can't downvote and/or criticize someone for anything other than the purely denotative meaning of their statements.

Comment by dxu on There's No Fire Alarm for Artificial General Intelligence · 2017-10-15T19:15:59.486Z · score: 11 (4 votes) · LW · GW

This feels like an excellent place for someone to ask for the evidence

Was that really what the grandparent comment was doing, though? The impression I got was that the original commenter was simply using the question as a rhetorical device in order to reinforce the (false) impression that MIRI et al. "automatically assume" things relating to the growth curve of superintelligent AI, and that kind of rhetoric is certainly not something I want to encourage.

Comment by dxu on There's No Fire Alarm for Artificial General Intelligence · 2017-10-14T05:02:23.377Z · score: 10 (4 votes) · LW · GW

I'm generally leery of ascribing motives to people who I don't know on the Internet, since I could very well be mistaken. By "bad faith", I simply meant a comment that was not (primarily) written for the purpose of accelerating progress along some axis generally agreed to be positive, e.g. understanding, knowledge, community, etc. This doesn't, of course, imply that I know what the actual motives of the commenter were, only that I'm fairly sure that they don't fall into the specific subset of motives I consider good.

That being said, if I were forced to generate a hypothesis that fits into one of the three categories you described, I would (very tentatively) nominate the third thing--"less-than-ideal self-awareness and candor"--as closest to what I think may actually be happening.

Comment by dxu on There's No Fire Alarm for Artificial General Intelligence · 2017-10-14T04:36:12.221Z · score: 8 (4 votes) · LW · GW

There is constructive criticism, and there is non-constructive criticism. My personal heuristic for determining whether a given critic is being constructive is to look at (a) how specific they are about the issues they perceive, and (b) whether they provide any specific suggestions as to how to address those issues. The parent comment does poorly on both fronts, and that in conjunction with the heavily aggressive tone therein are sufficient to convince me that it was very much written in bad faith. Please strive to do better.

Comment by dxu on Contra double crux · 2017-10-09T00:18:28.346Z · score: 11 (4 votes) · LW · GW

Naturally, and I wasn't claiming it was. That being said, I think that when you single out a specific point of disagreement (without mentioning any others), there is an implication that the mentioned point is, if not the only point of disagreement, then at the very least the most salient point of disagreement. Moreover, I'd argue that if Duncan's only recourse after being swayed regarding sequence versus cluster thinking is "huh, then I'm not sure why we're getting such good results", then there is a sense in which sequence versus cluster thinking is the only point of disagreement, i.e. once that point is settled, Duncan has no more arguments.

(Of course, I'm speaking purely in the hypothetical here; I'm not trying to make any claims about Duncan's actual epistemic state. This should be fairly obvious given the context of our discussion, but I just thought I'd throw that disclaimer in there.)

Comment by dxu on Contra double crux · 2017-10-08T23:51:12.237Z · score: 15 (6 votes) · LW · GW

As I understand it, Duncan argues that sequence thinking is more common than cluster thinking, and you're arguing the converse.

This looks remarkably like an attempt to identify a crux in the discussion. Assuming that you're correct about double-cruxing being problematic due to encouraging sequence-like thinking: isn't the quoted sentence precisely the kind of simplification that propagates such thinking? Conversely, if it's not a simplification, doesn't that provide (weak) evidence in favor of double-cruxing being a useful tool in addressing disagreements?

Comment by dxu on Contra double crux · 2017-10-08T19:42:13.855Z · score: 7 (4 votes) · LW · GW

I upvoted this entire chain of comments for the clear and prosocial communication displayed throughout.

Comment by dxu on Contra double crux · 2017-10-08T19:21:32.981Z · score: 10 (3 votes) · LW · GW

I would argue that Thrasymachus' initial post also carried an undertone of aggression (that Duncan may have picked up on, either consciously or subconsciously), but that this was possibly obscured and/or distracted from by its very formal tone.

(Whether you prefer veiled or explicit aggression is a "pick your poison" kind of choice.)

Comment by dxu on Contra double crux · 2017-10-08T19:02:09.627Z · score: 1 (1 votes) · LW · GW

I was going to suggest that you delete this comment, but then I realized that I have no idea if that's actually possible on this site. Would someone with more familiarity with LW 2.0 than I currently have care to comment?

Comment by dxu on Contra double crux · 2017-10-08T19:00:02.212Z · score: 9 (5 votes) · LW · GW

I am censoring further things I want to say, to avoid pushing on the resonance of tribalism-fighting.

Out of curiosity, do you think that inserting an explicit disclaimer like this helps to reduce feelings of tribal offense? If so, having now written such a disclaimer, do you think it would be worth it to share more of your thoughts on the matter?

(I'll be honest; my main motivator for asking this is because I'm curious and want to read the stuff you didn't say. But even taking that into consideration, it seems to me that the questions I asked have merit.)

Comment by dxu on Open thread, Apr. 03 - Apr. 09, 2017 · 2017-04-05T22:01:26.775Z · score: 1 (1 votes) · LW · GW

One very common error people make is to treat "utilitarianism" and "consequentialism" as if they were one and the same thing. Utilitarianism makes claims about what is moral and what is not. Consequentialism makes claims about what sort of properties a moral criterion should have. Criticisms about utilitarianism, therefore, are often taken also as criticisms of consequentialism, when in fact the two are distinct concepts!

Comment by dxu on Open thread, Apr. 03 - Apr. 09, 2017 · 2017-04-05T21:38:17.837Z · score: 0 (0 votes) · LW · GW

That's fair, but I also think it largely misses the point of Elo's comment. Here, have (an attempt at) a rephrase:

No community is as prone to self-criticism as the rationalist community.