Ditto.

This site isn't too active - maybe email someone from CFAR directly?

Man, this interviewer sure likes to ask dense questions. Bostrom sort of responded to them, but things would have gone a lot smoother if LARB guy (okay, Andy Fitch) had limited himself to one or two questions at a time. Still, it's kind of shocking the extent to which Andy "got it," given that he doesn't seem to be specially selected - instead he's a regular LARB contributor and professor in an MFA program.

Hm, the format is interesting. The end product is, ideally, a tree of arguments, with each argument having an attached relevance rating from the audience. I like that they didn't try to use the pro and con arguments to influence the rating of the parent argument, because that would be too reflective of audience composition.

Infinity minus one isn't smaller than infinity. That's not useful in that way.

The thing being added or subtracted is not the mere number of hypotheses, but a measure of the likelihood of those hypotheses. We might suppose an infinitude of mutually exclusive theories of the world, but most of them are extremely unlikely - for any degree of unlikeliness, there are an infinity of theories less likely than that! A randomly-chosen theory is so unlikely to be true, that if you add up the likelihoods of every single theory, they add up to a number less than infinity.

It is for this reason that it is important when we divide our hypotheses between something likely, and everything else. "Everything else" contains infinite possibilities, but only finite likelihood.

I think this neglects the idea of "physical law," which says that theories can be good when they capture the dynamics and building-blocks of the world simply, even if they are quite ignorant about the complex initial conditions of the world.

Can't this be modelled as uncertainty over functional equivalence? (or over input-output maps)?

Hm, that's an interesting point. Is what we care about just the brute input-output map? If we're faced with a black-box predictor, then yes, all that matters is the correlation even if we don't know the method. But I don't think any sort of representation of computations as input-output maps actually helps account for how we should learn about or predict this correlation - we learn and predict the predictor in a way that seems like updating a distribution over computations. Nor does it seem to help in the case of trying to understand to what extend two agents are logically dependent on one another. So I think the computational representation is going to be more fruitful.

Interesting that resnets still seem state of the art. I was expecting them to have been replaced by something more heterogeneous by now. But I might be overrating the usefulness of discrete composition because it's easy to understand.

Plausibly? LW2 seems to be doing okay, which is gonna siphon off posts and comments.

The dust probably is just dust - scattering of blue light more than red is the same reason the sky is blue and the sun looks red at sunset (Rayleigh scattering / Mie scattering). It comes from scattering off of particles smaller than a few times the wavelength of the light - so if visible light is being scattered less than UV, we know that lots of the particles are of size smaller than ~2 um. This is about the size of a small bacterium, so dust with interesting structure isn't totally out of the question, but still... it's probably just dust.

I think people get too hung up on computers as being mechanistic. People usually think of symbol manipulation in terms of easy-to-imagine language-like models, but then try to generalize their intuitions to computation in general, which can be unimaginably complicated. It's perfectly possible to simulate a human on an ordinary classical computer (to arbitrary precision). Would that simulation of a human be conscious, if they matched the behavior of a flesh and blood human almost perfectly, and could output to you via text channel and output things like "well, I sure feel conscious"?

The reason LWers are so confident that this simulation is conscious is because we think of concepts like "consciousness," to the extent that they exist, as having something to do with the cause of us talking and thinking about consciousness. It's just like how the concept of "apples" exists because apples exist, and when I correctly think I see an apple, it's because there's an apple. Talking about "consciousness" is presumed to be a consequence of our experience with consciousness. And the things we have experience with that we can label "consciousness" are introspective phenomena, physically realized as patterns of neurons firing, that have exact analogies in the simulation. Demanding that one has to be made of flesh to be conscious is not merely chauvinism, it's a misunderstanding of what we have access to when we encounter consciousness.

Neat paper about the difficulties of specifying satisfactory values for a strong AI. h/t Kaj Sotala.

The design of social choice AI faces three sets of decisions: standing, concerning whose ethics views are included; measurement, concerning how their views are identified; and aggregation, concerning how individual views are combined to a single view that will guide AI behavior. [] Each set of decisions poses difficult ethical dilemmas with major consequences for AI behavior, with some decision options yielding pathological or even catastrophic results.

I think it's slightly lacking in sophistication about aggregation of numerical preferences, and in how revealed preferences indicate that we don't actually have incommensurable or infinitely-strong preferences, but is overall pretty great.

On the subject of the problem, I don't think we should program in values that are ad-hoc on the object level (what values to use - trying to program this by hand is destined for failure), or even the meta level (whose values to use). But I do think it's okay to use an ad-hoc process to try to learn the answers to the meta-level questions. After all, what's the worst that could happen? (irony). Of course, the ability to do this assumes the solution of other, probably more difficult philosophical/AI problems, like how to refer to peoples' values in the first place.

Yeah, whenever you see a modifier like "just" or "merely" in a philosophical argument, that word is probably doing a lot of undeserved work.

I don't, and maybe you've already been contacted, but you could try contacting him on social sites like this one (user paulfchristiano) and Medium, etc. Typical internet stalking skillset.

Ah, you mean to ask if the brain is special in a way that evades our ability to construct an analogy of the chinese room argument for it? E.g. "our neurons don't indiviually understand English, and my behavior is just the product of a bunch of neurons following the simple laws of chemistry, therefore there is nothing in my body that understands English."

I think such an argument is totally valid imitation. It doesn't necessarily bear on the Chinese room itself, which is a more artificial case, but it certainly applies to AI in general.

You say impressions, but I'm assuming this is just the "things I want changed" thread :)

Vote button visibility and responsiveness is a big one for me. Ideally, it should require one click, be disabled while it messages the server, and then change color much more clearly.

On mobile, the layout works nicely, but load / render times are too long (how much javascript is necessary to serve text? Apparently, lots) and the text formatting buttons take up far too much space.

First time, non-logged in viewers should probably not see the green messaging blob in the corner, particularly on mobile.

I agree that some kind of demarcation between comments, and between comments and "write a new comment", would be nice. Doesn't have to be 2009 internet boxes, it can be 2017 internet fading horizontal lines or something.

Well, it really is defined that way. Before doing math, it's important to understand that entropy is a way of quantifying our ignorance about something, so it makes sense that you're most ignorant when (for discrete options) you can't pick out one option as more probable than another.

Okay, on to using the definition of entropy as the sum over event-space of -P log(P) of all the events. E.g. if you only had one possible event, with probability 1, your entropy would be 1 log(1) = 0. Suppose you had two events with different probabilities. If you changed the probability assignment so their probability gets closer together, entropy goes up. This is because the function -P log(P) is concave downwards between 0 and 1 - this means that the entropy is always higher between two points than you'd get by just averaging those points (or taking any weighted average, represented by a straight line connecting the two points).. So if you want to maximize entropy, you move all the points together as far as they can go.

Moderation is basically the only way, I think. You could try to use fancy pagerank-anchored-by-trusted-users ratings, or make votes costly to the user in some way, but I think moderation is the necessary fallback.

Goodhart's law is real, but people still try to use metrics. Quality may speak for itself, but it can be too costly to listen to the quality of every single thing anyone says.

The only thing I don't like about the "2017 feel" is that it sometimes feel like you're just adrift in the text, with no landmarks. Sometimes you just want guides to the eye, and landmarks to keep track of how far you've read!

I also agree that HPMOR might need to go somewhere other than the front page. From a strategic perspective, I somehow want to get the benefits of HPMOR existing (publicity, new people finding the community) without the drawbacks (it being too convenient to judge our ideas by association).

I think votes have served several useful purposes.

Downvotes have been a very good way of enforcing the low-politics norm.

When there's lots of something, you often want to sort by votes, or some ranking that mixes votes and age. Right now there aren't many comments per thread, but if there were 100 top-level comments, I'd want votes. Similarly, as a new reader, it was very helpful to me to look for old posts that people had rated highly.

And what if the universe is probably different for the two possible copies of you, as in the case of the boltzmann brain? Presumably you have to take some weighted average of the "non-anthropic probabilities" produced by the two different universes.

Re: note. This use of SSA and SIA can also be wrong. If there is a correct method for assigning subjective probabilities to what S.B. will see when she looks at outside, it should not be an additional thing on top of predicting the world, it should be a natural part of the process by which S.B. predicts the world.

EDIT: Okay, getting a better understanding of what you mean now. So you'd probably just say that the weight on the different universes should be exactly this non-anthropic probability, assigned by some universal prior or however one assigns probability to universes. My problem with this is that when assigning probabilities in a principled, subjective way - i.e. trying to figure out what your information about the world really implies, rather than starting by assuming some model of the world, there is not necessarily an easily-identifiable thing that is the non-anthropic probability of a boltzmann brain copy of me existing, and this needs to be cleared up in a way that isn't just about assuming a model of the world. If anthropic reasoning is, as I said above, not some add-on to the process of assigning probabilities, but a part of it, then it makes less sense to think something like "just assign probabilities, but don't do that last anthropic step."

But I suspect this problem actually can be resolved. Maybe by interpreting the non-anthropic number as something like the probability that the universe is a certain way (i.e. assuming some sort of physicalist prior), conditional on there only being at least one copy of me, and then assuming that this resolves all anthropic problems?

That's not quite what I was talking about, but I managed to resolve my question to my own satisfaction anyhow. The problem of conditionalization can be worked around fairly easily.

Suppose that there is 50% ehance of there being a boltzmann brain copy of you

Actually, the probability that you should assign to there being a copy of you is not defined under your system - otherwise you'd be able to conceive of a solution to the sleeping beauty problem - the entire schtick is that Sleeping Beauty is not merely ignorant about whether another copy of her exists, but that it is supposedly a bad question.

Hm, okay, I think this might cause trouble in a different way that I was originally thinking of. Because all sorts of things are possibilities, and it's not obvious to me how ADT is able to treat reasonable anthropic possibilities different from astronomically-unlikely ones, if it throws out any measure of unlikeliness. You might try to resolve this by putting in some "outside perspective" probabilities, e.g. that an outside observer in our universe would see me as normal most of the time and me as a Boltzmann brain less of the time, but this requires making drastic assumptions about what the "outside observer" is actually outside, observing. If I really was a Boltzmann brain in a thermal universe, an outside observer would think I was more likely to be a Boltzmann brain. So postulating an outside perspective is just an awkward way of sneaking in probabilities gained in a different way.

This seems to leave the option of really treating all apparent possibilities similarly. But then the benefit of good actions in the real world gets drowned out by all the noise from all the unlikely possibilities - after all, for every action, one can construct a possibility where it's both good and bad. If there's no way to break ties between possibilities, no ties get broken.

Moral value is not an "intrinsic property" of a mathematical structure - aliens couldn't look at this mathematical structure and tell that it was morally important. And yet, whenever we compute something, there is a corresponding abstract structure. And when we reason about morality, we say that what is right wouldn't change if you gave us brain surgery, so by morality we don't mean "whatever we happen to think," we mean that abstract structure.

Meanwhile, we are actual evolved mammals, and the reason we think what we do about morality is because of evolution, culture, and chance, in that order. I'm not sure what the point is of calling this objective or not, but it definitely has reasons for being how it is. But maybe you can see how this evolved morality can also be talked about as an abstract structure, and therefore both of these paragraphs can be true at the same time.

It seems like you were looking for things with "intrinsic properties" and "objective"-ness that we don't much care about, and maybe this is why the things you were thinking of were incompatible, but the things we're thinking of are compatible.

Since we are in the real world, it is a possibility that there is a copy of me, e.g. as a boltzmann brain, or a copy of the simulation I'm in.

Does your refusal to assign probabilities to these situations infect everyday life? Doesn't betting on a coin flip require conditioning on whether I'm a boltzmann brain, or am in a simulation that replaces coins with potatoes if I flip them? You seem to be giving up on probabilities altogether.

This seems like a question about aesthetics - thia choice won't change my experience, but it will change what kind of universe I live in. I think I'd choose duplication - I put a pretty low value on tiling the universe with conscious experience, but it's larger than zero.

I totally agree. Perceived differences in kind here are largely due to the different methods we use to think about these things.

For the triangle, everybody knows what a triangle is, we don't even need to use conscious thought to recognize them. But for the key, I can't quite keep the entire shape in my memory at once, if I want to know if something is shaped like my front door key, I have to compare it to my existing key, or try it in the lock.

So it naively seems that triangleness is something intrinsic (because I perceive it without need for thought), while front-door-key-shape is something that requires an external reference (because I need external reference). But just because I perceive the world a certain way doesn't mean those are particularly important distinctions. If one wanted to make a computer program recognize triangles and my front door key shape, one could just as well use the same code for both.

What is the analogy of sum that you're thinking about? Ignoring how the little pieces are defined, what would be a cool way to combine them? For example, you can take the product of a series of numbers to get any number, that's pretty cool. And then you can convert a series to a continuous function by taking a limit, just like an integral, except rather than the limit going to really small pieces, the limit goes to pieces really close to 1.

You could also raise a base to a series of powers to get any number, then take that to a continuous limit to get an integral-analogue. Or do other operations in series, but I can't think of any really motivating ones right now.

Can you invert these to get derivative-analogues (wiki page)? For the product integral, the value of the corresponding derivative turns out to be the limit of more and more extreme inverse roots, as you bring the ratio of two points close to 1.

Are there any other interesting derivative-analogues? What if you took the inverse of the difference between points, but then took a larger and larger root? Hmm... You'd get something that was 1 almost everywhere for nice functions, except where the function's slope got super-polynomially flat or super-polynomially steep.

In problems where multiple different agents in the universe "could be you," (i.e. share information), you really don't have to do anything fancy. Just assign equal probability to all agents in the entire universe who, as far as you know, can match your current state of information.

If there are two copies of Earth, and, hypothetically, only these two copies of me in the universe, I assign 50% probability to being each. This stays the same whether these Earths are at different points in space, or at different times, or sped up or slowed down or played in reverse order due to strange entropic boundary conditions. All that matters is that on each Earth there is a copy of me that has the same information as me from their own internal perspective, and I just count these copies up.

I will be frank. This sounds like a lame deal for anyone who takes you up on the offer. "My physics is shit, but I have a great idea for a new theory of gravity. PM me if you are a professional physicist and want to coauthor a paper." "My writing is shit, but I have a clever idea for a story and would like someone to write it it for me."

First you should do 90% of the work you know about, then maybe you can find a professional to do the last 10% plus the things you didn't know about. Read the relevant philosophy! Go read wikipedia, read the stanford encyclopedia of philosophy, go to your library and check out a book or three. Do a lot of writing! Make arguments, try to find good ways to phrase things, think of counterarguments other people might use, explain how this builds on and extends the stuff you read about. Then maybe, if you put that out in the public and ask for someone to devote their time to making this idea spread, you might get takers.

Across the frozen sea around most of Antartica even in the summertime?

I'm not sure if you're actually curious, or if you think this is a "gotcha" question.

Here's a picture. As the glacier flows outward (here's measured flow rates), it begins floating on the sea and becomes an ice shelf, which then loses mass to the ocean through melting and breaking up into pieces, which then melt. This ice shelf is thick (100m - 1 km scale), because it's a really thick sheet of ice being pushed out into the water by gravity. It then encounters the sea ice, which is ~1-4 meters thick. The sea ice gets pushed out, or piled up, because there are no particular forces holding the sea ice in place.

At this point I'm tapping out of the conversation. Either you're ignorant but curious and there's no point to me typing up things you could look up, or you want to feel superior while remaining ignorant and there's no point to me typing up things you don't care about.

If the glacier is flowing off of the continent into the sea, then sea ice is in an equilibrium between melting at the edges and bottom and being replenished at the middle.

demands huge melting we don't see

"See" how? It seems to me that you don't have an involved understanding of the melting of glaciers. If we could measure the mass of the Antarctic glacier straightforwardly, then I'm sure we'd agree on the meaning of changes in that mass. But if we don't see the particular melting process you expect, perhaps you're just expectung the wrong process, and haven't uncovered a conspiracy among all the experts.

Much smaller numbers, popular now

In my experience, actually reading the ipcc review has never been popular and still isn't. I'm sure you could still find someone in the press claiming larger sea level rise, if you tried. But why pick the easiest opponent?

Neat!

Glaciers don't have to form icebergs in order to melt. It can just melt where it meets the sea.

Almost 3 Amazons still missing for the 6 meters sea rise in a century

You know, now that you mention it, 6 meters sure is a lot. Where did you get that number from? See p. 1181 for IPCC projections.

How about glacial flow? Ice doesn't move fast, but it does move. It can postpone melting until it's in contact with seawater. What do you think the ratio of mass moved by rivers vs. glaciers is in Antarctica?

Where and how some people see three Amazons on Antarctica, is a mystery to me. The amount of ice falling directly into the sea, is quite pathetic, as well.

The amazon begins distributed across brazil, as the occasional drops of rain. Then it comes together because of the shape and material of the landscape, and flows into streams, which join into rivers, which feed one big river. If global warming is causing antarctica to lose mass, do you expect the same thing to happen in antarctica, with meltwater beginning distributed across the surface, and then collecting into rivers and streams?

Why do we care about acausal trading with aliens to promote them acting with "moral reflection, moral pluralism," etc?

I think writing something like this is a bit like a rite of passage. So, welcome to LW :P

When we talk about someone's values, we're using something like Dan Dennett's intentional stance. You might also enjoy this LW post about not applying the intentional stance.

Long story short, there is no "truly true" answer to what people want, and no "true boundary" between person and environment, but there are answers and boundaries that are good enough for what people usually mean.

Well, if the acronym "POMDP" didn't make any sense, I think we should start with a simpler example, like a chessboard.

Suppose we want to write a chess-playing AI that gets its input from a camera looking at the chessboard. And for some reason, we give it a button that replaces the video feed with a picture of the board in a winning position.

Inside the program, the AI knows about the rules of chess, and has some heuristics for how it expects the opponent to play. Then it represents the external chessboard with some data array. Finally, it has some rules about how the image in the camera is generated from the true chessboard and whether or not it's pressing the button.

If we just try to get the AI to make the video feed be of a winning position, then it will press the button. But if we try to get the AI to get its internal representation of the data array to be in a winning position, and we update the internal representation to try to track the true chessboard, then it won't press the button. This is actually quite easy to do - for example, if the AI is a jumble of neural networks, and we have a long training phase in which it's rewarded for actually winning games, not just seeing winning board states, then it will learn to take into account the state of the button when looking at the image.

To our best current understanding, it has to have a model of the world (e.g. as a POMDP) that contains a count of the number of paperclips, and that it can use to predict what effect its actions will have on the number of paperclips. Then it chooses a strategy that will, according to the model, lead to lots of paperclips.

This won't want to fool itself because, according to basically any model of the world, fooling yourself does not result in more paperclips.

I think I don't know the solution, and if so it's impossible for me to guess what he thinks if he's right :)

But maybe he's thinking of something vague like CIRL, or hierarchical self-supervised learning with generation, etc. But I think he's thinking of some kind of recurrent network. So maybe he has some clever idea for unsupervised credit assignment?

Cool insight. We'll just pretend constant density of 3M/4r^3.

This kind of integral shows up all the time in E and M, so I'll give it a shot to keep in practice.

You simplify it by using the law of cosines, to turn the vector subtraction 1/|r-r'|^2 into 1/(|r|^2+|r'|^2+2|r||r'|cos(θ)). And this looks like you still have to worry about integrating two things, but actually you can just call r' due north during the integral over r without loss of generality.

So now we need to integrate 1/(r^2+|r'|^2+2r|r'|cos(θ)) r^2 sin(θ) dr dφ dθ. First take your free 2π from φ. Cosine is the derivative of sine, so substitution makes it obvious that the θ integral gives you a log of cosine. So now we integrate 2πr (ln(r^2+|r'|^2+2r|r'|) - ln(r^2+|r'|^2-2r|r'|)) / 2|r'| dr from 0 to R. Which mathematica says is some nasty inverse-tangent-containing thing.

Okay, maybe I don't actually want to do this integral that much :P

To spell it out:

Beauty knows limiting frequency (which, when known, is equal to the probability) of the coin flips that she sees right in front of her will be equal to one-half. That is, if you repeat the experiment many times (plus a little noise to determine coin flips), then you get equal numbers of the event "Beauty sees a fair coin flip and it lands Heads" and "Beauty sees a fair coin flip and it lands Tails." Therefore Beauty assigns 50/50 odds to any coin flips she actually gets to see.

You can make an analogous argument from symmetry of information rather than limiting frequency, but it's less accessible and I don't expect people to think of it on their own. Basically, the only reason to assign thirder probabilities is if you're treating states of the world given your information as the basic mutually-exclusive-and-exhaustive building block of probability assignment. And the states look like Mon+Heads, Mon+Tails, and Tues+Tails. If you eliminate one of the possibilities, then the remaining two are symmetrical.

If it seems paradoxical that, upon waking up, she thinks the Monday coin is more likely to have landed tails, just remember that half of the time that coin landed tails, it's Tuesday and she never gets to see the Monday coin being flipped - as soon as she actually expects to see it flipped, that's a new piece of information that causes her to update her probabilities.

According to SSA beauty should update credence of H to 2/3 after learning it is Monday.

I always forget what the acronyms are. But the probability of H is 1/2 after learning it's Monday, any any method that says otherwise is wrong, exactly by the argument that you can flip the coin on monday right in front of SB, and if she knows it's Monday and thinks it's not a 50/50 flip, her probability assignment is bad.

He proposes the coin toss could happen after the first awakening. Beauty’s answer ought to remain the same regardless the timing of the toss. A simple calculation tells us his credence of H must be 1/3. As SSA dictates this is also beauty’s answer. Now beauty is predicting a fair coin toss yet to happen would most likely land on T. This supernatural predicting power is a conclusive evidence against SSA.

So how do you get Beauty's prediction? If at the end of the first day you ask for a prediction on the coin, but you don't ask on the second day, then now Beauty knows that the coin flip is, as you say, yet to happen, and so she goes back to predicting 50/50. She only deviates from 50/50 when she thinks there's some chance that the coin flip has already happened.

Sometimes people absolutely will come to different conclusions. And I think you're part of the way there with the idea of letting people talk to see if they converge. But I think you'll get the right answer even more often if you set up specific thought-experiment processes, and then had the imaginary people in those thought experiments bet against each other, and say the person (or group of people all with identical information) who made money on average (where "average" means over many re-runs of this specific thought experiment) had good probabilities, and the people who lost money had bad probabilities.

I don't think this is what probabilities mean, or that it's the most elegant way to find probabilities, but I think it's a pretty solid and non-confusing way. And there's a quite nice discussion article about it somewhere on this site that I can't find, sadly.

Sorry for the slow reply.

The 8 rooms are definitely the unbiased sample (of your rooms with one red room subtracted).

I think you are making two mistakes:

First, I think you're too focused on the nice properties of an unbiased sample. You can take an unbiased sample all you want, but if we know information in addition to the sample, our best estimate might not be the average of the sample! Suppose we have two urns, urn A has 10 red balls and 10 blue balls, while urn B has 5 red balls and 15 blue balls. We choose an urn by rolling a die, such that we have a 5/6 chance of choosing urn A and a 1/6 chance of choosing urn B. Then we take a fair, unbiased sample of 4 balls from whatever urn we chose. Suppose we draw out 1 red ball and 3 blue balls. Since this is an unbiased sample, does the process that you are calling "statistical analysis" have to estimate that we were drawing from urn B?

Second, you are trying too hard to make everything about the rooms. It's like someone was doing the problem with two urns from the previous paragraph, but tried to mathematically arrive at the answer only as a function of the number of red balls drawn, without making any reference to the process that causes them to draw from urn A vs. urn B. And they come up with several different ideas about what the function could be, and they call those functions "the Two-Thirds-B-er method" and "the Four-Tenths-B-er method." When really, both methods are incomplete because they fail to take into account what we know about how we picked the urn to draw from.

To answer the last part of your statement. If beauty randomly opens 8 doors and found them all red then she has a sample of pure red. By simple statistics she should give R=81 as the estimation. Halfer and thirders would both agree on that. If they do a bayesian analysis R=81 would also be the case with the highest probability. I'm not sure where 75 comes from I'm assuming by summing the multiples of probability and Rs in the bayesian analysis? But that value does not correspond to the estimation in statistics. Imagine you randomly draw 20 beans from a bag and they are all red, using statistics obviously you are not going to estimate the bag contains 90% red bean.

Think of it like this: if Beauty opens 8 doors and they're all red, and then she goes to open a ninth door, how likely should she think it is to be red? 100%, or something smaller than 100%? For predictions, we use the average of a probability distribution, not just its highest point.

The HoTT book is pretty readable, but I'm not in a position to evaluate its actual goodness.

In your example, I think Bob is doing something unrelated to rationalist Taboo.

In the actual factual game of Taboo, you replace a word with a description that is sufficient to tell your team what the original word is. In rationalist Taboo, you replace a word with a description that is sufficient to convey the ideas you were trying to convey with the original word.

So if Bob tries to taboo "surprise" as "the feeling of observing a low-probability event," and Alice says "A license plate having the number any particular number is low probability - is it surprising?," Bob should think "Oh, the description I replaced 'surprise' with did not convey the same thing as the word 'surprise'. I need to try tabooing it differently."

This works better when you're trying to taboo the usage of a word in a specific context, because the full meaning of a word is very very complicated (though trying to make definitions can still be a fun and profitable game, I agree), but when you look at how you've used it in just one sentence, then you have some hope of pinning down what you mean by it to your satisfaction.

And yet, people, when giving examples of selfishness, don't just sample the entirety of human behavior. They point out a specific sort of behavior. Or when naming optimization functions, they might call one function "greedy," even though all functions tautologically do what they do. So clearly people have some additional criteria for everyday use of the word not captured by the extremely simple definition in this post.

First, I checked out the polling data on interracial marriage. Every 10 years the approval rating has gone up by ~15 percentage points. I couldn't find a concise presentation of the age-segregated data from now vs. in the past, but 2007 and 1991 were available, and they look consistent with over 80% of the opinion change being due to old people dying off. This surprised me, I expected to see more evidence of people changing their mind.

Now look at gay marriage.. It's gained at ~18 points per 10 years. This isn't too different from 15, so maybe this is people dying off too. And indeed it seems to be mostly the case - except the in the last 10 years, where the gains don't follow the right age pattern, indicating that of 18 points of gain, about 40% may actually involve people changing their minds.

"Refute" is usually not an objective thing - it's a social thing. You can probably prove to yourself that pi=3 is false, but if you write "pi=3" on a sheet of paper, no argument will make the ink rearrange itself to be correct.

This is one of the problems with a falsificationist idea of scientific progress, where we never prove theories true but make progress by proving them false. If evidence against a theory appears (e.g. the ability to see different stars from different parts of the earth might be thought of as "refuting" the idea of a flat earth), a proponent of that theory never has to give up on it. They can just patch the theory. Maybe light does a special little dance to make all the observations look like we're looking out at a universe, etc. If you try to refute someone, they can just refuse to be refuted and add another patch to their theory.

After doing some reading, I feel like this guy actually does a pretty admirable job of seeing open questions and admitting ignorance. For example, he doesn't know about the coriolis effect, so he calls it "a mysterious thing that happens to objects falling down mineshafts" and wonders whether it could cause an error in the readings of plumb bobs hanging down a mineshaft. Again, I think this is a good thing, though not as good as knowing about the coriolis effect before trying to understand the structure of the cosmos. The trouble seems mostly to be that he's read a lot of books that are full of shit, and believes them.