## Posts

Suspiciously balanced evidence 2020-02-12T17:04:20.516Z
"Future of Go" summit with AlphaGo 2017-04-10T11:10:40.249Z
AlphaGo versus Lee Sedol 2016-03-09T12:22:53.237Z
[LINK] "The current state of machine intelligence" 2015-12-16T15:22:26.596Z
Scott Aaronson: Common knowledge and Aumann's agreement theorem 2015-08-17T08:41:45.179Z
Group Rationality Diary, March 22 to April 4 2015-03-23T12:17:27.193Z
Group Rationality Diary, March 1-21 2015-03-06T15:29:01.325Z
Open thread, September 15-21, 2014 2014-09-15T12:24:53.165Z
Proportional Giving 2014-03-02T21:09:07.597Z
A few remarks about mass-downvoting 2014-02-13T17:06:43.216Z
[Link] False memories of fabricated political events 2013-02-10T22:25:15.535Z
[LINK] Breaking the illusion of understanding 2012-10-26T23:09:25.790Z
The Problem of Thinking Too Much [LINK] 2012-04-27T14:31:26.552Z
Harry Potter and the Methods of Rationality discussion thread, part 4 2010-10-07T21:12:58.038Z
The uniquely awful example of theism 2009-04-10T00:30:08.149Z
Voting etiquette 2009-04-05T14:28:31.031Z

Comment by gjm on Hardware for Transformative AI · 2021-06-23T10:04:04.323Z · LW · GW

If multiplying the size by 5 means multiplying the cost by 10 (and if this relationship is consistent as size continues to increase) then a 100x size increase is about 2.86 5x-ings, which means a cost increase of about 10^2.86 or about 730, which means that $12M becomes about$8.7B. (Much more than the $3B you suggest and of course much much more than the$300M OP suggests.)

[EDITED to add:] Oops, should have refreshed the page before commenting; I see that OP has already fixed this.

Comment by gjm on Tracey Davis and the Prisoner of Azkaban - Part 6 · 2021-06-23T09:55:40.431Z · LW · GW

Possible typos or other slip-ups:

"Draco could appreciate that the show" ("could appreciate the show"?)

"It still took him just as long to get anyway" ("to get in"?)

"Mistress Fireball's concert was complete secret" ("a complete secret"? "completely secret"?)

"What s the point of secrets" ("What's")

"Blood status is just a fossilized marriage involving a Dark Lord" (I suspect this isn't a mistake but I don't think it makes the least bit of sense and can't imagine anyone in any version of the HPverse saying it; it seems like a type error if taken at all literally and if it's a metaphor I can't make much sense of it; maybe I'm just too stupid to be reading this story, but you might want to consider making this a bit more explicit or something)

Comment by gjm on Open problem: how can we quantify player alignment in 2x2 normal-form games? · 2021-06-22T12:32:12.542Z · LW · GW

(To be clear, of course it may be that what you were intending to ask for is exactly what Vanessa provided, and you have every right to be interested in whatever questions you're interested in. I'm just trying to explain why the question Vanessa answered doesn't feel to me like the key question if you're asking about how well aligned one agent is with another in a particular context.)

Comment by gjm on Open problem: how can we quantify player alignment in 2x2 normal-form games? · 2021-06-22T12:30:43.554Z · LW · GW

I agree that this is measuring something of interest, but it doesn't feel to me as if it solves the problem I thought you said you had.

This describes how well aligned an individual action by B is with A's interests. (The action in question is B's choice of (mixed) strategy , when A has chosen (mixed) strategy .) The number is 0 when B chooses the worst-for-A option available, 1 when B chooses the best-for-A option available, and in between scales in proportion to A's expected utility.

But your original question was, on the face of it, looking for something that describes the effect on alignment of a game rather than one particular outcome:

In my experience, constant-sum games are considered to provide "maximally unaligned" incentives, and common-payoff games are considered to provide "maximally aligned" incentives. How do we quantitatively interpolate between these two extremes?

or perhaps the alignment of particular agents playing a particular game.

I think Vanessa's proposal is the right answer to the question it's answering, but the question it's answering seems rather different from the one you seemed to be asking. It feels like a type error: outcomes can be "good", "bad", "favourable", "unfavourable", etc., but it's things like agents and incentives that can be "aligned" or "unaligned".

When we talk about some agent (e.g., a hypothetical superintelligent AI) being "aligned" to some extent with our values, it seems to me we don't just mean whether or not, in a particular case, it acts in ways that suit us. What we want is that in general, over a wide range of possible situations, it will tend to act in ways that suit us. That seems like something this definition couldn't give us -- unless you take the "game" to be the entirety of everything it does, so that a "strategy" for the AI is simply its entire program, and then asking for this coefficient-of-alignment to be large is precisely the same thing as asking for the expected behaviour of the AI, across its whole existence, to produce high utility for us. Which, indeed, is what we want, but this formalism doesn't seem to me to add anything we didn't already have by saying "we want the AI's behaviour to have high expected utility for us".

It feels to me as if there's more to be done in order to cash out e.g. your suggestion that constant-sum games are ill-aligned and common-payoff games are well-aligned. Maybe it's enough to say that for these games, whatever strategy A picks, B's payoff-maximizing strategy yields Kosoy coefficient 0 in the former case and 1 in the latter. That is, B's incentives point in a direction that produces (un)favourable outcomes for A. The Kosoy coefficient quantifies the (un)favourableness of the outcomes; we want something on top of that to express the (mis)alignment of the incentives.

Comment by gjm on Four Components of Audacity · 2021-06-21T15:08:29.504Z · LW · GW

I think so. Or, taking the specific kind of boldness lsusr mentions -- making statements without qualifications that might make them sound weaker -- this is a thing one will very often hear from religious or political fanatics, who may well have arrived at their views by pure conformism.

Or maybe a bunch of your friends join a secret society with a scary initiation ritual, and you go ahead with it because you want to be like your friends. Or you're part of a culture in which men who reach adolescence are expected to go and hunt a tiger or stay in the wilderness for a week or something, and you go ahead and do it because that's what everybody does. (Of course you might also be doing it because if you don't the tribal elders will kill you, in which case it wouldn't count as daring.) Or you're a member of the Westboro Baptist Church and you go with everyone else to picket military funerals with signs saying GOD HATES FAGS and GOD HATES AMERICA, even though you're worried that someone in the crowd may pick a fight. (This one is marginal, because someone in that position is nonconformist relative to the culture at large but conformist relative to what directly surrounds them. I think the latter is probably the thing that's both harder and more important to escape.)

Comment by gjm on Four Components of Audacity · 2021-06-21T11:39:54.693Z · LW · GW

I am not convinced that audacity is the same thing as, or anything like a guarantee of, nonconformity. lsusr, do you have evidence for that claim?

In particular, "boldness" and "daring" seem to me as if they have very little to do with nonconformity; it may happen they they correlate with it (e.g. because similar personality-types find those various things congenial) but I think it's clear they they don't intrinsically equate to, or imply, or follow from, nonconformity, and while there might be a not-so-trivial relationship between them I think it requires some actual evidence.

I do agree that "shamelessness" seems like it should obviously reduce conformity. "Impertinence" might well do, to whatever extent conformity results from treating social superiors as arbiters of truth; my impression (though not a confident one) is that conformity is more driven by trying to align with one's peers than with accepting authority from above, which if true would limit how much "impertinence" can prevent it. The combination of shamelessness and impertinence does seem like it should reduce conformity; but if there's reason to think that (with or without "boldness" and "daring") this either suffices or is necessary to make one a nonconformist, I don't think lsusr has presented it here.

Comment by gjm on Four Components of Audacity · 2021-06-21T11:32:50.179Z · LW · GW

Most people, most of the time, don't preface statements with "I believe". Therefore, when they do so it conveys information.

(Note: the footnote-looking annotations here have a slightly different meaning from usual; see the comment before the actual footnotes, though you'd probably immediately figure out what I'm up to even without it.)

In my case, I think[1] the information it conveys is something like this: I am aware that the thing I'm stating is not practically-universally believed by people in the relevant reference class (which is something like "people whose opinion my interlocutor might pay some attention to"), or that I expect it not to be (e.g., because I am uncertain myself), and therefore even if the person I'm talking to generally trusts me they should probably remain somewhat uncertain unless on this point they think I'm especially trustworthy.

I think[2] this is approximately what "I believe ..." indicates for most speakers.

Because "I believe ..." has this meaning, and because it's common practice to use it (or some similar construction) when stating something that you know isn't universally agreed, the absence of "I believe ..." also has meaning: it indicates that you consider the opinion you're stating to be one with which a reasonable and well informed person couldn't disagree. (Again, within the relevant reference class. If I'm talking to a creationist, I might say "I believe that birds are the descendants of ancient dinosaurs" or maybe something like "Biologists are pretty much 100% agreed that ...", but if I'm not then I'll just say "Birds are the descendants of ancient dinosaurs". I don't think creationists are reasonable and well-informed, but when talking to a creationist a requirement for productive discussion is that one somewhat suspend disbelief on this point. But I digress.)

Someone who just asserts things without qualifiers like "I believe", even when they know that the opinion they're stating is open to reasonable disagreement, is defecting in the social game[3]; they are misleading their listeners (at least those who don't already know them well enough to know that they regularly do this; and perhaps even those listeners, as far as System 1 goes) by implicitly claiming that they know of no relevant disagreement on the issue.

This can backfire, if someone listening knows that there is relevant disagreement and infers that the speaker doesn't know or dismisses it wrongly. But even if it doesn't, I think[4] it's a species of dishonesty. It may well be that people who are "bold" in this sense get ahead in life compared with those who don't, but if so I think[4] they are doing so at the expense of the people they are misleading, whom they induce to (sometimes) make worse decisions than they otherwise would, by giving more credence to the bold person's opinions than they otherwise would.

Of course it's a small kind of deceit; it isn't necessarily consciously intended as deceit; most of the time it will do no harm. But it is, none the less, a kind of deceit, and I think[5] it does, on balance, do harm on average, and a given "bold" person will do it again and again and again, and the harms add up.

I hope others will not take this particular bit of lsusr's advice. If they do, they will be making the world a little bit worse[6].

----

I have marked with footnote-markers the places where I used the construction in question, or considered doing so and consciously decided not to. Some comments on each:

[1] Acknowledges that introspection is difficult, and getting a clear idea of one's own habitual behaviour is difficult, and that I may therefore be wrong about exactly what information "I believe ..." conveys when I use it. Omitting the qualification would have encouraged readers to think that I either have done some sort of careful analysis of my own usage (which I haven't) or am unaware of the difficulties of introspection (which I'm not).

[2] Acknowledges that determining what "most speakers" do is difficult, especially in cases like this where it's hard even to be sure about one's own behaviour. Omitting the qualification would have encouraged readers to think that I either have done something like going through instances of "I think", "I believe", etc., in books and blogposts and the like to see how they're used (which I haven't) or am unaware of the difficulties of extrapolation (which I'm not).

[3] I wondered about putting something like "in my opinion" here, and actually I normally would, acknowledging that this sort of claim is potentially controversial and e.g. most likely lsusr disagrees, and moving the implied criticism of lsusr from "they endorse a practice I consider harmful" (which is true) to "they endorse a practice that they know, or ought to know, is dishonest" (which is probably not true). In this case I didn't, mostly because I thought I'd go along with lsusr's suggested policy for a moment. I would be interested to know whether lsusr felt on reading that paragraph that I was making an accusation of dishonesty or malice or something of the kind, and whether they reckon it would have felt the same way if I had softened the claim with the usual qualifier.

[4] Acknowledges the things that under [3] I noted not acknowledging. (Because despite my decision not to qualify earlier, I do think it's important to be aware that the position I'm taking is potentially controversial and possibly uncomfortable for the person I'm disagreeing with.) Omitting these qualifications would have given the false impression that I think any reasonable person would agree with me that the policy lsusr proposes amounts to dishonest exploitation of other people (I actually think it's possible that that's so, but I am aware that I haven't considered the matter in enough depth to justify making such a claim) and that I think that by proposing that policy lsusr is endorsing dishonest exploitation (which I'm pretty sure they are not doing).

[5] Acknowledges that this sort of thing is really difficult to assess, and that I am not claiming to have done actual calculations showing that the opinion I express here is correct, merely saying how things look to me.

[6] I wondered about putting something like "I think" here, but considered that it would be redundant given the qualifications already present above. I think readers here are smart enough to understand that, since I acknowledge the debatability of lots of premises on which this summation is based, I acknowledge that the summation itself is debatable. In other contexts I might have qualified this statement too.

Comment by gjm on Four Components of Audacity · 2021-06-21T10:56:54.374Z · LW · GW

I find treating superiors as equals more impressive when it is accompanied by treating inferiors as equals too. Unfortunately, some audacious people don't do that bit, which has three unfortunate consequences. (1) It means they treat their "inferiors" badly, which is bad for those people. (2) It gives them a reputation for dickishness. (3) It reveals that their impertinence isn't a matter of principled not-giving-a-damn-about-status, but has some other, possibly less reputable cause such as selfishness.

Comment by gjm on Assume long serving politicians are rationally maximizing their careers · 2021-06-19T14:30:33.537Z · LW · GW

I don't think "flailing" was particularly meant to stand in specifically for changing political position. I think it was just meant to indicate "behaving in ways that seem like they ought to be bad politics".

(Considering the three examples specifically mentioned in the post: I think Pelosi is criticized more for being intransigent than for being inconsistent. Boris Johnson has changed his opinions almost as often as his women, and I don't think anyone would bet heavily on his having much in the way of principles, but since entering politics he's actually been fairly consistent. Khamenei seems to me to have been pretty consistent, but I don't know much about Iranian politics. So these don't seem like the examples one would choose if "flailing" meant making a lot of U-turns.)

Comment by gjm on Assume long serving politicians are rationally maximizing their careers · 2021-06-19T14:23:01.538Z · LW · GW

I completely agree that the dubiousness of this particular story doesn't do much to cast doubt on Clinton's (very well attested) one-on-one charisma. That's why I said "I'm not denying Clinton's one-on-one charisma, though".

Comment by gjm on Assume long serving politicians are rationally maximizing their careers · 2021-06-18T22:37:15.971Z · LW · GW

Someone writing a book about eye contact may possibly have reason to exaggerate the effectiveness of eye contact. I would not want to bet that the effect was as dramatic as Michael Elsberg says it was.

(I'm not denying Clinton's one-on-one charisma, though.)

Comment by gjm on The aducanumab approval · 2021-06-17T13:42:31.773Z · LW · GW

Your first link (the one in the "This is a linkpost for ..." intro) is broken. I think it's probably meant to go to https://blogs.sciencemag.org/pipeline/archives/2021/06/08/the-aducanumab-approval .

Comment by gjm on Open problem: how can we quantify player alignment in 2x2 normal-form games? · 2021-06-17T13:39:08.571Z · LW · GW

Sorry, I think I wasn't clear about what I don't understand. What is a "strategy profile (like stag/stag)"? So far as I can tell, the usual meaning of "strategy profile" is the same as that of "strategy", and a strategy in a one-shot game of stag hunt looks like "stag" or "hare", or maybe "70% stag, 30% hare"; I don't understand what "stag/stag" means here.

----

It is absolutely standard in game theory to equate payoffs with utilities. That doesn't mean that you have to do the same, of course, but I'm sure that's why Dagon said what he did and it's why when I was enumerating possible interpretations that was the first one I mentioned.

(The next several paragraphs are just giving some evidence for this; I had a look on my shelves and described what I found. Most detail is given for the one book that's specifically about formalized 2-player game theory.)

"Two-Person Game Theory" by Rapoport, which happens to be the only book dedicated to this topic I have on my shelves, says this at the start of chapter 2 (titled "Utilities"):

So far nothing has been said about the nature of the payoffs. [...] It is even conceivable that a man playing Checkers with a child would rather lose than win. In that case a larger payoff must be assigned to his loss than to his win. [...] the game remains undefined if we do not know what payoff magnitudes are assigned by the players to the outcomes, even if the latter are specified in terms of monetary payoffs. However, this problem is bypassed by the game theoretician, who assumes that the payoffs are given.

Unfortunately, Rapoport is using the word "payoffs" to mean two different things here. I think it's entirely clear from context, though, that his actual meaning is: you may begin by specifying monetary payoffs, but what we care about for game theory is payoffs as utilities. Here's more from a little later in the chapter:

The given payoffs are assumed to reflect the psychological worth of the associated outcomes to the player in question.

A bit later:

When payoffs are specified on an interval scale [as opposed to an "ordinal scale" where you just say which ones are better than which other ones -- gjm], they are called utilities.

and:

As has already been pointed out, these matters are not of concern to the game theoretician. His position is that if utility scales can be determined, then a theory of games can be built on a reliable foundation. If no such utility scale can be established with references to any real subjects, then game theory will not be relevant to the behaviour of people in either a normative or descriptive sense.

As I say, that's the only book of formal game theory on my shelves. Schelling's Strategy of Conflict has a little to say about such games, but not much and not in much detail, but it looks to me as if he assumes payoffs are utilities. The following sentence is informative, though it presupposes rather than stating: "But what configuration of value systems for the two participants -- of the "payoffs", in the language of game theory -- makes a deterrent threat credible?" (This is from the chapter entitled "International Strategy"; in my copy it's on page 13.)

Rapoport's "Strategy and Conscience" isn't a book of formal game theory, but it does discuss the topic, and it explicitly says: payoffs are utilities.

One chapter in Schelling's "Choice and Consequence" is concerned with this sort of game theory; he says that the numbers you put in the matrix are either arbitrary things whose relative ordering is the only thing that matters, or numbers that behave like utilities in the sense that the players are trying to maximize their expectations.

The Wikipedia article on game theory says: "The payoffs of the game are generally taken to represent the utility of individual players." (This is in the section about the use of game theory in economics and business. It does also mention applications in evolutionary biology, where the payoffs are fitnesses -- which seem to me very closely analogous to utilities, in that what the evolutionary process stochastically maximizes is something like expected fitness.)

Again, I don't claim that you have to equate payoffs with utilities; you can apply the formalism of game theory in any way you please! But I don't think there's any question that this is the usual way in which payoffs in a game matrix are understood.

----

It feels odd to me to focus on response functions, since as a matter of fact you never actually know the other player's strategy. (Aside from special cases where your opponent is sufficiently deterministic and sufficiently simple that you can "read their source code" and make reliable predictions from it. There's a bit of an LW tradition of thinking in those terms, but I think that with the possible exception of reasoning along the lines of "X is an exact copy of me and will therefore make the same decisions as I do" it's basically never going to be relevant to real decision-making agents because the usual case is that the other player is about as complicated as you are, and you don't have enough brainpower to understand your own brain completely.)

If you are not considering payouts to be utilities, then you need to note that knowing the other player's payouts -- which is a crucial part of playing this sort of game -- doesn't tell you anything until you also know how those payouts correspond to utilities, or to whatever else the other player might use to guide their decision-making.

(If you aren't considering that they're utilities but are assuming that higher is better, then for many purposes that's enough. But, again, only if you suppose that the other player does actually act as someone would act who prefers higher payouts to lower ones.)

My feeling is that you will get most insight by adopting (what I claim to be) the standard perspective where payoffs are utilities; then, if you want to try to measure alignment, the payoff matrix is the input for your calculation. Obviously this won't work if one or both players behave in a way not describable by any utility function, but my suspicion is that in such cases you shouldn't necessarily expect there to be any sort of meaningful measure of how aligned the players are.

Comment by gjm on Reply to Nate Soares on Dolphins · 2021-06-17T12:45:58.996Z · LW · GW

I agree: one can have multiple reasons for having (or professing) a belief. For that reason, to me saying "X believes Y because Z" (where Z is a disreputable reason and one would otherwise assume something less disreputable) is rather uninteresting if not accompanied by actual evidence that the other less-disreputable reasons aren't sufficient explanation for X to believe Y.

In the present case, Scott is known to be a good thinker, and has given (not particularly disreputable) reasons for believing Y; rationalists on the whole are also pretty good thinkers (Nate and Eliezer included); if you think their opinions on this point are mostly the result of political prejudice, you're entitled to think that but I don't see any good reason to agree.

Comment by gjm on Open problem: how can we quantify player alignment in 2x2 normal-form games? · 2021-06-16T18:31:22.952Z · LW · GW

I'm not 100% sure I am understanding your terminology. What does it mean to "play stag against (stag,stag)" or to "defect against cooperate/cooperate"?

If your opponent is not in any sense a utility-maximizer then I don't think it makes sense to talk about your opponent's utilities, which means that it doesn't make sense to have a payout matrix denominated in utility, which means that we are not in the situation of my second paragraph above ("The meaning generally assumed in game theory...").

We might be in the situation of my last-but-two paragraph ("Or maybe we're playing a game in which..."): the payouts might be something other than utilities. Dollars, perhaps, or just numbers written on a piece of paper. In that case, all the things I said about that situation apply here. In particular, I agree that it's then reasonable to ask "how aligned is B with A's interests?", but I think this question is largely decoupled from the specific game and is more about the mapping from (A's payout, B's payout) to (A's utility, B's utility).

I guess there are cases where that isn't enough, where A's and/or B's utility is not a function of the payouts alone. Maybe A just likes saying the word "defect". Maybe B likes to be seen as the sort of person who cooperates. Etc. But at this point it feels to me as if we've left behind most of the simplicity and elegance that we might have hoped to bring by adopting the "two-player game in normal form" formalism in the first place, and if you're prepared to consider scenarios where A just likes choosing the top-left cell in a 2x2 array then you also need to consider ones like the ones I described earlier in this paragraph -- where in fact it's not just the 2x2 payout matrix that matters but potentially any arbitrary details about what words are used when playing the game, or who is watching, or anything else. So if you're trying to get to the essence of alignment by considering simple 2x2 games, I think it would be best to leave that sort of thing out of it, and in that case my feeling is that your options are (a) to treat the payouts as actual utilities (in which case, once again, I agree with Dagon and think all the alignment information is in the payout matrix), or (b) to treat them as mere utility-function-fodder, but to assume that they're all the fodder the utility functions get (in which case, as above, I think none of the alignment information is in the payout matrix and it's all in the payouts-to-utilities mapping), or (c) to consider some sort of iterated-game setup (in which case, I think you need to nail down what sort of iterated-game setup before asking how to get a measure of alignment out of it).

Comment by gjm on Reply to Nate Soares on Dolphins · 2021-06-16T14:47:21.859Z · LW · GW

(Pedantic notes: A group doesn't have two operations, you're thinking of rings or fields or modules or algebras or other such things; one of them distributes over the other, but not the other way around.)

I agree that it would be annoying if we had different names for all the different things we currently call multiplication.

It sounds as if you might prefer a system where instead of saying "R is a ring" we have some concise way of saying what operations R has and what axioms it satisfies. Something like "R is an (01+-*,ACD) algebra", meaning it has addition & subtraction (= addition and additive inverses), multiplication but not necessarily division (so not necessarily multiplicative inverses), multplication is associative and commutative, and there's a distributive law. And you'd say that instead of "commutative ring with unity" or "commutative ring" or "ring", so there'd be a bit more verbosity and a bit more scope for not reading carefully exactly what's being assumed, but less ambiguity, and the terminology wouldn't so strongly favour a smallish set of particular types of structure that have their own names. Instead of "monoid" we'd say "0+ algebra", instead of "semigroup" we'd say "+ algebra", instead of "abelian group" we'd say "0+- algebra", instead of "group" we'd say "1*/ algebra", etc.

(My notation there assumes that it's understood that something called addition is always commutative and associative.)

There's definitely something to be said for that. On the other hand, those structures with special names have those names because those types of structures keep coming up. "Group" is only five letters and one syllable, and a lot of things are groups.

Comment by gjm on Open problem: how can we quantify player alignment in 2x2 normal-form games? · 2021-06-16T13:20:17.823Z · LW · GW

I think "X and Y are playing a game of stag hunt" has multiple meanings.

The meaning generally assumed in game theory when considering just a single game is that the outcomes in the game matrix are utilities. In that case, I completely agree with Dagon: if on some occasion you prefer to pick "hare" even though you know I will pick "stag", then we are not actually playing the stag hunt game. (Because part of what it means to be playing stag hunt rather than some other game is that we both consider (stag,stag) the best outcome.)

But there are some other situations that might be described by saying that X and Y are playing stag hunt.

Maybe we are playing an iterated stag hunt. Then (by definition) what I care about is still some sort of aggregation of per-round outcomes, and (by definition) each round's outcome still has (stag,stag) best for me, etc. -- but now I need to strategize over the whole course of the game, and e.g. maybe I think that on a particular occasion choosing "hare" when you chose "stag" will make you understand that you're being punished for a previous choice of "hare" and make you more likely to choose "stag" in future.

Or maybe we're playing an iterated iterated stag hunt. Now maybe I choose "hare" when you chose "stag", knowing that it will make things worse for me over subsequent rounds, but hoping that other people looking at our interactions will learn the rule Don't Fuck With Gareth and never, ever choose anything other than "stag" when playing with me.

Or maybe we're playing a game in which the stag hunt matrix describes some sort of payouts that are not exactly utilities. E.g., we're in a psychology experiment and the experimenter has shown us a 2x2 table telling us how many dollars we will get in various cases -- but maybe I'm a billionaire and literally don't care whether I get $1 or$10 and figure I might as well try to maximize your payout, or maybe you're a perfect altruist and (in the absence of any knowledge about our financial situations) you just want to maximize the total take, or maybe I'm actually evil and want you to do as badly as possible.

In the iterated cases, it seems to me that the payout matrix still determines alignment given the iteration context -- how many games, with what opponents, with what aggregation of per-round utilities to yield overall utility (in prospect or in retrospect; the former may involve temporal discounting too). If I don't consider a long string of (stag,stag) games optimal then, again, we are not really playing (iterated) stag hunt.

In the payouts-aren't-really-utilities case, I think it does make sense to ask about the players' alignment, in terms of how they translate payouts into utilities. But ... it feels to me as if this is now basically separate from the actual game itself: the thing we might want to map to a measure of alignedness is something like the function from (both players' payouts) to (both players' utilities). The choice of game may then affect how far unaligned players imply unaligned actions, though. (In a game with (cooperate,defect) options where "cooperate" is always much better for the player choosing it than "defect", the payouts->utilities function would need to be badly anti-aligned, with players actively preferring to harm one another, in order to get uncooperative actions; in a prisoners' dilemma, it suffices that it not be strongly aligned; each player can slightly prefer the other to do better but still choose defection.)

Comment by gjm on Reply to Nate Soares on Dolphins · 2021-06-16T10:53:15.844Z · LW · GW

If you're having to learn 300 new mathematical concepts, it seems to me that the cost of learning 300 new words is lost in the noise. (And even if it's a single page, figuring out what that single page says is going to take you days, weeks, months, or years.)

Further, you're having to learn 300 new word meanings in any case. Are you saying this is much easier when the words already have existing but completely different meanings?

Sometimes the mathematical meaning is closely related to the existing meaning. ("Set" and "category" are arguably of this kind.) In that case, using the existing words may well be a good decision. But if you tell me that somehow the concepts "ring", "field", "module" are easier to learn because those are existing English words, I'm skeptical.

I don't buy that it's much harder for most learners either, though. It's not as if, when someone writes "let F be a field", you're left wondering whether perhaps they mean that it's a patch of land suitable for agriculture.

(The ambiguities that bother me are the "internal" ones. If someone says "let R be a ring", you may not know whether or not you're supposed to be assuming that it has a multiplicative identity element.)

Comment by gjm on Reply to Nate Soares on Dolphins · 2021-06-16T01:01:32.044Z · LW · GW

Your comment seems to me to assume that Scott thinks there would be nothing very wrong with a definition of "fish" that included whales only because that's something he has to think in order to remain consistent while classifying transgender people the way they feel they should be classified.

I don't think it's at all obvious that that's so.

(Similarly, one could postulate that Zack thinks there would be something very wrong with such a definition of "fish" only because that's something he has to think in order to remain consistent while insisting that transgender people should be classified according to attributes like anatomy, chromosomes, etc.

I don't think that's obviously so, either.)

Either of those things could be true. Both of them could be true, for that matter. Or neither. But I think that in order for "rationalists as a group have changed their minds on this for political reasons" to be a better analysis than "rationalists as a group have changed their minds on this because they found Scott's arguments about King Solomon and the like convincing", there needs to be some good reason to think that Scott's arguments are bad enough that rationalists couldn't be convinced by them without political motivation, or that Scott's arguments were clearly made in bad faith, or something of the kind.

(Note: my description of Scott's and Zack's positions is brief and necessarily sketchy. E.g., Scott is writing about whether it's OK to define a word that groups what-we-call-fish together with whales and dolphins; Zack is more interested in whether it would be OK to use the specific word "fish" that way, given how it is already used; it's not clear which question Soares is really debating, given that the whole thing is shitposting anyway. Some of the gender-political issues for which this serves as an analogy match up pretty well with one of those, some not so much. Anyway, please do not take any of the above as a serious attempt to describe either Scott's or Zack's exact position.)

Comment by gjm on An Introduction to Prediction Markets · 2021-06-14T19:22:26.626Z · LW · GW

Nitpicky correction: "Metacalculus" is actually called "Metaculus". Its name is not derived from "calculus".

Comment by gjm on Which rationalists faced significant side-effects from COVID-19 vaccination? · 2021-06-14T11:19:26.266Z · LW · GW

Note that answers here, on their own, will give a one-sided view of the side-effect situation which may mislead everyone's Systems 1. I am not suggesting that everyone who had no significant side-effects comment to say so because (1) in practice that won't happen, so answers will still be greatly biased but it may be less obvious that they are and (2) if it did it would mean way the hell too many comments :-). I will remark, since I'm commenting anyway, that I had two vaccine doses (AstraZeneca) with no significant side effects. (I felt grotty for a day or two after the first dose. I noticed no side effects at all after the second.)

Comment by gjm on What are the gears of gluten sensitivity? · 2021-06-11T02:04:03.175Z · LW · GW

You can lose some gluten and still have enough to maintain the structure of your dough. I think the person I mentioned thinks that's what's going on. (But it sure seems plausible to me that the actually relevant difference between sourdough bread and other bread is something else. If gluten degradation were the mechanism then you'd think you could do equally well by using weaker flour, not kneading as much, etc.)

I've made breads that are actually leavened with ordinary baker's yeast but also contain some sourdough starter (in fact I have two loaves made that way sitting cooling on a rack right now), but it would never occur to me to call them sourdough breads. They're ordinary yeasted breads with a bit of sourdough starter in to give flavour and longevity. But I guess if you're selling bread, and your customers like the idea of sourdough bread, and you aren't too scrupulous... :-)

Comment by gjm on Other Constructions of Gravity · 2021-06-09T22:41:34.453Z · LW · GW

In what way is Idea 1 simpler than what Newtonian gravity gives you? It's (roughly) an integral of a Fourier transform of a convolution (three nested integrals) versus two nested integrals for Newton. What's simpler about the stuff inside those integrals than the stuff inside Newton's?

Well, I think we can simplify the calculations for your Idea 1. Let's have a go. Your proposed potential is

which we can write as  where subscripts indicate components.

This equals

and this is (up to some multiplicative constant whose value will depend on exactly what conventions you're using for Fourier transforms) the same thing as  which provided everything is reasonably smooth and well-behaved at infinity equals  which by Parseval's theorem equals (again maybe up to a multiplicative constant)  which if we don't mind convolving a vector function with a scalar one, with the obvious meaning, equals .

So, how could this give us "interaction terms" resembling Newtonian gravity? It seems like we need f to look like  (at least away from the origin), so that you get something that looks like  which has a local . remote . 1/r in it. Maybe if you do it right this somehow gives you exactly Newtonian gravity or something ... but, also, this is beginning to look rather similar to the original Newtonian integral, and I don't see what about it is simpler. (And it seems to me we got here by starting with your proposal and simplifying it, so I don't see how the original proposal can be simpler than Newton.)

I'm not sure I understand Idea 2, so I won't try to comment on it :-).

Comment by gjm on What are the gears of gluten sensitivity? · 2021-06-09T21:34:04.614Z · LW · GW

That looks like no sourdough recipe I've ever seen. The usual process goes like this:

• If your starter isn't being fed very frequently, take some and give it a couple of feeds at reasonably short intervals.
• Now take some starter, combine with flour and water, and leave it until it's active and bubbly -- might be 4-12 hours depending on the starter:flour ratio, starter activity, ambient temperature, etc.
• Combine with flour, water and salt.
• Bulk ferment: Over the next 4-12 hours, depending on the starter:flour ratio, starter activity, ambient temperature, etc., strengthen the dough by some combination of kneading and gentler folding. (A typical prescription might be: (wait 30 minutes, do some folds) 6 times, then leave it alone until it's increased sufficiently in size, which might be another ~2 hours.
• Shape and transfer to a proving basket.
• Prove: Leave it for maybe 1-3 hours (or substantially longer in the fridge) until it's increased in size again, passes the "poke test", etc.
• Bake.

There are various wrinkles (e.g., you might combine just flour and water some way in advance, before combining with starter and salt; this allows a process called autolysis) but that's the general technique. Maybe if you do it in warm enough conditions and with a very active starter you might get it down to 4 hours bulk fermentation + 1 hour proving. You absolutely cannot do it in 1-2 hours.

(You can do a distinctly not-normal thing that's much quicker. If you keep the bits of sourdough starter discarded at feeding in a bowl or jar in the fridge, then once you have ~1kg of that you can add a bit of flour, mix it up a bit, leave it a couple of hours, and bake it in a loaf tin. You can make a rather nice loaf this way, especially if you like your sourdough breads dense and sour. But it's not by any stretch of the imagination "the normal sourdough recipe".)

Just in case I'd somehow been living in a weird sourdough bubble, I put "sourdough recipe" into Google and looked at the first 8 hits. First: a no-knead no-fold recipe. 8-14 hours bulk fermentation + 1 hour refrigerated proving. Second: 3 hours bulk fermentation + 4-8 hours proving. Third: 3-12 hours bulk fermentation + 1/2-1 hour proving. Fourth: 3.5-7.5 hours bulk fermentation + 3-4 hours proving. Fifth: 2.5-3 hours bulk fermentation + 2.5 hours proving. Sixth: 4.5 hours bulk fermentation + overnight proving. Seventh: 4.5 hours bulk fermentation + overnight proving. Eighth: 10-12 hours bulk fermentation + 1.5-48 hours proving.

Comment by gjm on Bad names make you open the box · 2021-06-09T19:20:14.880Z · LW · GW

My intuition says that

• if it's called getPromotedPosts then it is probably fetching some information from somewhere -- maybe the internet, maybe a database -- and probably isn't doing any computation to speak of;
• if it's called promotedPosts then it is probably either computing something or just using a value it already knows and can return quickly and easily.

I am not sure there's any function name that would be perfectly neutral between (1) extremely cheap operation, probably just returning something already known, (2) nontrivial calculation, and (3) nontrivial fetching.

There's also a bit of ambiguity about whether something called getPromotedPosts is fetching the posts themselves or just cheap representations of them (e.g., ID numbers, pointers, etc.).

So I might consider names like fetchPromotedPostIDsFromDatabase, retrievePromotedPostContent, inferPromotedPostsByModel, cachedPromotedPostList, etc. Or I might prefer a brief name like promotedPosts and put information about what it does and the likely performance implications in a comment, docstring, etc.

Comment by gjm on Bad names make you open the box · 2021-06-09T19:11:22.458Z · LW · GW

To me "move" in this context would sound unnatural, perhaps because it's a verb as well as a noun.

I suspect that the suggestion of badness may have been intended when the term "regression to the mean" was first coined by Francis Galton. I think he was particularly interested in investigating exceptional people of various kinds. The OED's first citation for "regression" in this sense is from him, and the exact phrase he uses is "regression towards mediocrity", that last word being another one that generally has a somewhat negative sense.

Comment by gjm on Bad names make you open the box · 2021-06-09T16:03:41.558Z · LW · GW

Yes, if it "takes a long time to go look at a database and do some post-processing", that would be a case where (as I put it) "there is a retrieval operation involved and it might be expensive", and then we might want a name that makes it easier to guess that it might be expensive.

Comment by gjm on What are the gears of gluten sensitivity? · 2021-06-09T14:50:00.454Z · LW · GW

Is it true that sourdough fermentation alters other things but not gluten? My understanding is that the acidity of sourdough degrades gluten. I know of one person who was actually a (semi-?)professional baker before developing -- at least by her account of things -- a gluten intolerance, who found that she could eat some sourdough things whose non-sourdough things made her feel bad. But of course I have no way of knowing whether it was really the gluten that was giving her problems.

(Who adds flour to the dough after a sourdough fermentation? Other than maybe a dusting to prevent sticking while shaping or proofing, which will add little enough that I'd have thought someone who isn't outright coeliac should be OK with it.)

Comment by gjm on Bad names make you open the box · 2021-06-09T14:25:41.147Z · LW · GW

If a function returns a value then in some sense it's necessarily a get.

Things are more complicated when something both (1) does something and (2) returns a value. E.g., you might put something and then return something that indicates whether it worked or not; you might get something but the process of doing it might update a cache, having (if nothing else) an impact on performance of related future operations.

Some people advocate a principle of "command-query separation": every operation is a "command" that might change the world (and doesn't return anything) or a "query" that gives you some information (but doesn't change anything) but nothing tries to do both at once. (If some commands can fail, you either use an exception-handling mechanism or have related queries for checking whether a command worked.)

That's nice and clean but sometimes inconvenient; the standard example is a "pop" operation on a stack, which both tells you what's on the top of the stack and removes it. (If it's possible that there might be multiple concurrent things operating on the stack at once, you need either to have atomic operations like "pop" or else some explicit mechanism for claiming exclusive access to the stack while you look at its top element and then maybe remove it.)

In the present case, to me "getPromotedPosts" feels ambiguous between (1) "tell me which posts are promoted" and (2) "retrieve the promoted posts from somewhere". If the function is just called "promotedPosts" then that makes it explicit that either it's (1) or it's (2) but the retrieval is an implementation detail you aren't meant to care about, so I think I prefer "promotedPosts" unless there is a retrieval operation involved and it might be expensive or have side effects that matter.

Comment by gjm on Bad names make you open the box · 2021-06-09T14:15:52.540Z · LW · GW

I think the implication of getting worse is strong enough that (outside the technical uses in statistics) you'd never say "regress" when the change involved wasn't a worsening. E.g., if I try to imagine any of the following, I can't see anyone actually saying them. "I have good news for you: the latest scans show that your cancer has regressed somewhat." "The fifth wave of the COVID-19 pandemic is beginning to regress now." "The most recent figures show some regression in the unemployment caused by last year's financial crash."

The statistical uses -- "regression to the mean" and the practice of "regression" (meaning model-fitting), which historically is actually derived from "regression to the mean" -- are of course well enough established that once you're used to them they don't carry any connotation of things getting worse.

[EDITED to add:] On looking in the OED, I find that in fact "regression" is used about tumours and the like. But I bet that in the unfortunate event that any of us has to consult an oncologist, they will not use the word in that sense with us; I think it's for technical use only, just like the statistical sense.

Comment by gjm on Bad names make you open the box · 2021-06-09T14:10:39.500Z · LW · GW

Because (to me, at least) that would mean going all the way back to the mean, whereas regression to the mean means going some of the way back towards the mean.

(For the avoidance of doubt, I am not claiming that "regression to the mean" is the optimal name for this phenomenon; just saying why a particular other name might not be an improvement.)

Comment by gjm on Against intelligence · 2021-06-08T13:45:16.151Z · LW · GW

One key point here is reminiscent of Amdahl's law in computing: if you are trying to make some computation faster by parallelizing it, and a fraction x of it is stuff you don't have a way to parallelize, then no amount of extra hardware will make the speedup bigger than 1/(1-x).

Similarly, if you are trying to make some discovery / invention / optimization faster by doing it more intelligently, and a fraction x of the work is something other than pure thinking, then no amount of better thinking will make the speedup bigger than 1/(1-x).

I think the usual rejoinder on the "AI go foom" side is that we are likely to overestimate x by underestimating what really effective thinking can do; Eliezer's story "That Alien Message" is intended as an intuition pump in that direction. (Along with his non-fictional comments in the same post, suggesting e.g. that something sufficiently smart looking at webcam footage of a falling apple would be contemplating GR as a plausible hypothesis by the third frame.) I don't find that my own intuition is very compellingly pumped by this, for what it's worth.

Comment by gjm on Article on IQ: The Inappropriately Excluded · 2021-05-31T18:20:21.722Z · LW · GW

Sure. But that doesn't tell us anything about the distribution of IQ among people in "elite professions", and that's the key question here.

Comment by gjm on Why don't long running conversations happen on LessWrong? · 2021-05-31T11:14:40.847Z · LW · GW

The distinction Professor Quirrell makes is a useful one but I think it's a mistake to apply it at the level of people, as if some people were Average Joes who just want to kill time on the internet and some people are Serious Arguers who want to make intellectual progress. Anyone can be both at different times. I suspect most people, even those who are valuable as Serious Arguers, are more often in Average Joe mode, and I am not convinced that that's a problem.

To whatever extent that's right, the way to have more long-running discussions with Serious Arguers is to present things in a way that encourage Serious Arguer mode rather than Average Joe mode. To whatever extent it's wrong, what you need to do is to attract people who are more Serious Arguers than Average Joes.

(This may be stating the obvious, but it seems worth making it explicit.)

Comment by gjm on Why don't long running conversations happen on LessWrong? · 2021-05-31T11:05:09.141Z · LW · GW

I suspect (but am not sure) that the cases in which long-running discussion would be valuable aren't ones where a post-plus-comment system is ideal, even at the start before the discussion has petered out. They're ones where A writes something substantial (of post-length), and then B writes something similarly substantial (also of post-length), and then A writes something similarly substantial, and so forth. In that case, B should be writing a post rather than a comment.

Writing top-level posts can be intimidating, and can be a lot of work. (They can be intimidating because they're a lot of work -- one feels daunted at the prospect. They can be a lot of work because they're intimidating -- fear of writing something unsatisfactory motivates research and polishing. This parenthesis is really just for fun.) Perhaps there's space for something intermediate between a comment and a post, somehow, for exactly this situation? But I'm not sure what it would be, how it would work, or whether it would actually end up feeling like something intermediate.

In the absence of such a mechanism, I suggest: (1) If you have something to say in response to a post, and it feels as if it might turn into a discussion that will run for more than a day or two, consider making your response a post rather than a comment. (2) If a discussion in comments turns out to be substantive and potentially long-running, then (as a participant) consider making a post rather than a comment next. (It should begin with a pointer to the existing discussion and a brief summary, which will also probably help the discussion. And of course you should drop a link to the post into the discussion in comments.)

Having written this, I see that Ericf has said something similar much more briefly and adamzerner says: no, I'm talking about back-and-forth stuff. But it seems to me that quickfire back-and-forth of the sort that's better staying in comments is seldom productive when extended beyond (say) a week, both because purely mechanically it becomes difficult and unpleasant to follow in a nested comment-thread setup and also because quick back-and-forth comments are mostly about what you might call "easy" responses: A says something, B realises that it conflicts with some ideas B already has, B remarks on that, A finds that what B says conflicts with ideas A already has, etc. This can be helpful, but usually there's only so much "on the surface" that's actually productive to say and long back-and-forth discussions tend to end up with everyone feeling that the other party is saying the same thing over and over and not really understanding, as a result of which they have to keep reiterating things that the other party hasn't understood. (The other party, meanwhile, of course feels the same way.)

I think most discussions that are worth continuing at length involve, at some point, more serious engagement than that and deeper thinking, and that's a point at which making a post rather than a comment makes sense.

Comment by gjm on For Better Commenting, Take an Oath of Reply. · 2021-05-31T10:46:25.390Z · LW · GW

My interpretation is that an "Oath of Reply" is only meant to create an obligation to reply if the thing being replied to expects a reply. (Some comments don't.)

Comment by gjm on Article on IQ: The Inappropriately Excluded · 2021-05-29T16:50:09.016Z · LW · GW

The article says:

The above statistics are the result of dividing the Gaussian distribution of 126 with a standard deviation of 6.7 by the IQ distribution of the total population

and I'm sorry but this is complete nonsense unless you have good evidence that (even at the tails) the "elite professions" IQ distribution is close to being Gaussian with the mean and standard deviation you cite. I bet it isn't.

To reiterate: the point is the shape of the distribution, not its mean and standard deviation. Lots of different distributions have mean 126 and standard deviation 6.5. Some of them lead to curves like the one you plotted. Some don't.

Toy model: the "elite professions" simply reject everyone with an IQ below 120 and sample at random from those with higher IQs. This gives a mean of almost exactly 127 and a standard deviation of 6.1 -- very close to the ones you quote, and if you redid your calculation with those figures you would get an even worse "exclusion" for higher IQs. But in a world matching this model, there is no real "exclusion" at all.

So, show us your evidence that the distribution of IQs in "elite professions" is close to Gaussian or, alternatively, your evidence that your analysis still works if you use a more realistic estimate of that distribution. If you can do that, then you'll have some evidence that people with very high IQ have less success in (or maybe less interest in) entering "elite professions". Otherwise, not. The argument in your actual article, as it stands, is 100% hopeless, I'm afraid.

Comment by gjm on What should we expect from ARPA-H? · 2021-05-20T16:02:37.568Z · LW · GW

It doesn't tell us anything that Science tries to present both sides. In particular, it doesn't tell us anything that Science found someone unhappy at ARPA-H being part of the NIH. That was my point.

I'm not sure what your last paragraph is about. On the face of it it seems to be something to do with the Science article, but it doesn't seem like it lines up with that. The people quoted there are, in order,

• Liz Feld (BA in Government, journalist and politician): the one you quoted, very negative about ARPA-H as part of NIH
• Michael Stebbins (genetics PhD, career in science, science journalism and science policy): "It has tremendous potential at NIH" but various caveats
• David Walt (professor at Harvard): seems positive (though I think he's comparing it to the status quo rather than to putting ARPA-H somewhere else)
• ResearchAmerica (sciencey PR organization): seems opposed to anything ARPA-H shaped at all because it will take money away from ordinary NIH projects

The last of these seems to be the one you're talking about -- its CEO does indeed describe her background that way. But they're strongly opposed to putting ARPA-H inside the NIH.

Comment by gjm on What should we expect from ARPA-H? · 2021-05-19T22:19:28.572Z · LW · GW

I'm not sure to what extent there's such a thing as the purpose for which DARPA was founded, but it seems to me you could equally say that cancer, diabetes and Alzheimer's aren't the purpose for which ARPA-H is being founded (assuming it actually happens); they're merely current expectations for some of the first things it will work on. So I'm not sure what distinction you're drawing htere.

Perhaps if I were more familiar with the structure of the relevant US agencies it would be clearer to me, but as it is I don't see how you get from "It will be part of the NIH rather than part of the OSTP" to "it will be more politician-driven than DARPA".

The article in Science quotes some people who think ARPA-H will probably do well and some who don't. This isn't terribly surprising; I bet that if the administration had issued a similar announcement but had said that ARPA-H would be part of the OSTP, they'd instead have quoted some people saying "It makes no sense for ARPA-H not to be part of Health and Human Services; I'm worried that it will end up wasting money on projects that don't really have anything to do with health" or something of the sort. (I remark that the alternative mentioned in the Science article isn't "part of the OSTP" but "a standalone part of DHHS".)

This is looking less and less like a real question and more like you just wanted to complain that the proposed ARPA-H isn't what you'd like it to be.

(For the avoidance of doubt, the above is not a coded way of saying that what you'd like it to be wouldn't be an improvement. I don't know enough about these things to have a strong opinion on that.)

Comment by gjm on What should we expect from ARPA-H? · 2021-05-19T22:04:54.173Z · LW · GW

I don't think that's sufficient justification. (Which is not to say that you don't have sufficient justification.) If I sincerely say "I am going to climb Mount Everest" and you think I don't have the skills and resources required to do so, it would not be reasonable for you to say "Gareth pretends that he wants to climb Mount Everest" unless you think I am outright lying about my intentions, rather than just unrealistic.

If the Biden administration wants something that will be able to engage in grand and innovative projects in the realm of health but has wrong ideas about how to get that (e.g., if they aren't giving ARPA-H enough autonomy, or aren't giving it enough money, or are encouraging the people running it to focus on areas where there aren't good opportunities for such projects), it is not accurate in present-day English to say that they "pretend to want" a DARPA-like organization.

On the other hand, if you actually think that they don't actually want something DARPA-like (maybe you reckon they want something that will let them funnel large sums of money to institutions that make large political donations, or something) then "pretend to want" accurately represents your opinions.

Regardless of whether it accurately represents your opinions, I predict that as long as the text says "pretends to want" a substantial fraction of readers will be distracted from what I take to be the actual question (what is this organization likely to achieve?) by politically-charged emotions, whether positive ("ha, nice to see someone seeing the Biden administration for the bullshitters they are") or negative ("why is this guy posting a political attack piece on LW under the guise of a question?"). Assuming that your actual goal is to get good answers to that question rather than to make political attacks, less-aggressive language might serve that goal better.

Comment by gjm on What should we expect from ARPA-H? · 2021-05-19T13:29:51.216Z · LW · GW

When DARPA was founded it was explicitly under political authority. "The Agency shall be responsible for the direction or performance of such advanced projects in the field of research and development as the Secretary of Defense shall, from time to time, designate by individual project or by category." Its initial focus was on space technology, ballistic missile defence, and solid propellants, all of them things that politicians were concerned about because of Russia's recent success in the space race. (Much of that stuff got moved over to NASA when that was founded shortly afterwards.)

Is there anything that specifically suggests that ARPA-H will be more politician-driven than DARPA?

Comment by gjm on What should we expect from ARPA-H? · 2021-05-19T13:25:33.364Z · LW · GW

"Pretends" near the start of the first paragraph seems a more hostile word than you may actually intend. (I think I recall that you're not a native anglophone; "pretend" in English almost always implies specifically that the one pretending knows that what they're pretending is not so. It can also simply mean "claim" but that meaning is old-fashioned and very much not what anyone reading something in present-day English will think it means.)

I have no idea whether this ARPA-H thing will actually do anything interesting, but unless you think that the administration doesn't really want it to work like DARPA or to be more open to innovation, you should probably use a word other than "pretends". Even if you do think that, you might prefer to use a different word, in the interests of not being needlessly distracting to those who feel positively about the current US administration; politics is the mindkiller, etc.

Comment by gjm on Are PS5 scalpers actually bad? · 2021-05-18T14:53:44.360Z · LW · GW

It seems as if the following might be true. (I have no idea whether it is, but the key point is one you mentioned in the OP.) If it is, then the scalpers are hurting ordinary people's ability to acquire PS5s at non-scalper prices, even if they are prepared to spend a lot of time.

• Every time some consoles become available, they will go to the people who get there first.
• Ordinary people don't have very effective ways of getting there first.
• Scalpers, because they are buying many PS5s per drop, can afford to take measures that let them almost always get there first. (Any specific scalper won't necessarily win in a particular case, but the person who wins will almost always be a scalper.)
• Therefore, as long as there are scalpers around doing their thing, ordinary people don't have the option of trading time against money and getting PS5s in stock drops, because all the PS5s in those drops actually go to scalpers. Or, at least, almost all, so that the time-cost of trying to get a PS5 that way is much larger than you would expect.

Probably some ordinary people could implement the same sorts of bot that the scalpers use, but (1) most people don't have the necessary skills and (2) even for those who do, that again is a large investment of time given that all they want is one PS5.

Comment by gjm on Challenge: know everything that the best go bot knows about go · 2021-05-16T11:24:26.505Z · LW · GW

Robot arms and computer vision, at the level necessary for playing a game of go, are I think a sufficiently solved problem that there's no particular reason why AI researchers working on making a strong go-playing program would bother hooking them up. On its own I don't think doing that would add anything interesting; in particular, I don't think there's any sense in which it would make the program's thinking more human-like.

I don't know about the Alphas, but my laptop running KataGo uses an amount of power that's in the same ballpark as Ke Jie (more than just his brain, less than his whole body) and I'm pretty sure it would beat him very solidly. Go programs don't generally concern themselves with power consumption as such but they do have to manage time (which is roughly proportional to total energy use) -- but so far as I know the time-management code is always hand-written rather than being learned somehow.

No one is claiming that a strong go-playing program is anything like an artificial general intelligence or that it makes humans obsolete or anything like that. (Though every surprising bit of progress in AI should arguably make us think we're closer to making human-level-or-better general intelligences than we'd previously thought we were.)

Programs like AlphaGo Zero or KataGo don't see the board in terms of local patterns as a result of learning from humans who do (and actually I very much doubt that non-Zero AlphaGo can usefully be said to do that either), they see the board in terms of local patterns because the design of their neural nets encourages them to do so. At least, it does in the earlier layers; I don't know how much has been done to look at later layers and see whether or not their various channels tend to reflect local features that are comprehensible to humans.

Of course every move is a whole-board move in the sense that it is made on a whole board, its consequences may affect the whole board, and most of the time the way you chose it involves looking at the whole board. That's true for humans and computers alike. But humans and computers alike build up their whole-board thinking from more-local considerations. (By "alike" I don't mean to imply that the mechanisms are the same; the neural networks used in go programs are somewhat inspired by the structure of actual brain networks but that certainly doesn't mean they are doing the same computation.)

And yes, because the training process is aiming to optimize the network weights for evaluating positions accurately and suggesting good moves and ultimately winning games all those local things are chosen and adjusted in order to get good global results. (The same is true for humans; the patterns we learn to make and avoid are considered good or bad as a result of long experience of how they affect actual games.) In that sense, indeed local and global are entangled (but I really wouldn't say "quantumly"; so far as I can see there is no connection with quantum mechanics beyond the fact that the word "entangled" is used in both).

Some specifics about the network. I'm going to assume you don't know anything :-); apologies for any resulting redundancy.

At any point in the computation, for each point on the board you have a bunch of numbers. The numbers (or the machinery that computes them) are called "channels". At the very start you have a smallish number of channels corresponding to the lowest-level features; you can find a list in appendix A.1 of the KataGo paper, but they're things like one channel that has 1 where there's a black stone and 0 elsewhere, and one that has 1 where there's a white stone and 0 elsewhere, and then some slightly cleverer things like channels that encode whether a given location contains a stone that's part of a group with a small number of liberties or a stone that's part of a group that can be captured in a ladder-like way (i.e., group has exactly two liberties and there's a way to iterate "atari, opponent plays only escape move, group has exactly two liberties again" until eventually the group is captured; KataGo has hand-written code to look for this situation). I think some things that aren't location-specific (e.g., what's the value of komi?) are represented as channels that have the same value at every board location.

Now a single layer looks like this. It has, let's say, m input channels and n output channels. (Usually these numbers are equal, but they're different at the very start and at the very end.) So the input to this layer consists of 19x19xm numbers, and the output consists of 19x19xn numbers: one per board location per channel. Now, for each output channel, we have a 3x3xm array of numbers called weights. For each board location, overlay these on the surrounding 3x3xm region of the input. (Where it falls off the edge of the board, hallucinate an infinite sea of zeros. One input channel at the very start just has 1s at every board location so that you can identify the edge.) Multiply weights by channel values, add up the results, and force the sum to zero if it was negative; congratulations, you have computed the value for that output channel at that board location. Repeat 19x19xn times, doing a 3x3xm multiply/sum/rectify operation for each, until you have a full 19x19xn output: one number per channel per board location. This is a single convolutional layer.

The first convolutional neural networks were just stacks of convolutional layers, but a few newer tricks have been developed. One is what's called a residual network or ResNet; I believe all the current generation of go programs use these. The idea is that the network will learn better if it's encouraged to make later layers "refinements" of earlier layers rather than arbitrarily different. What this means in practice is that you build the network out of two-layer blocks as follows: you take the block's input, you apply two convolutional layers to it, and then the output of the block isn't the output of the second layer, it's the output of the second layer plus the input to the block. (This only makes sense when the block has the same number of input and output channels, but as already mentioned this is the case for almost all of the network.) This doesn't change what the network can compute in principle, but it tends to lead to more effective learning, and you need to know about it because the sizes of these networks are always quoted in blocks rather than layers. The strongest KataGo networks are 40 blocks which means 80 layers.

Another tweak is batch normalization which means that before each convolutional layer you "normalize" each of its channels by scaling and shifting all the numbers in that channel so that their mean is 0 and their variance is 1.

Another trick introduced by KataGo (it's not found in the Alphas or in Leela Zero) is global pooling. I think I described this in a comment earlier in this thread. The idea is that every now and then you have a layer in which you take some of the channels, you compute their mean and maximum values, and you then feed these as inputs into all the other channels. So e.g. if one of these channels happens to have +1 for board locations that are in some sense under black's control and -1 for ones that are under white's control then the mean gives you a sort of a score estimate. If one has 1 for a location where there seems to be a ko and 0 elsewhere then the max tells you whether there are any kos and the mean tells you how many.

OK. So we start with some number of input channels describing the basic state of the game. We have 40 blocks, each consisting of two convolutional layers, and every now and then one of these blocks does the global-pooling thing to incorporate information from the whole board into the local calculations. At the very end, we take our final set of output channels and use them to compute a whole bunch of things. The most important are (1) a "policy" value for each board location, indicating how plausible a place it is for the next move, and (2) how likely each player is to win, and the likely final score. But there are also (1') guesses at the opponent's next move, (1'') estimates of how likely each board location is to end up in the control of each player, (2') estimates of how likely each specific final score is, and (3) various other interesting things whose details you can find in appendix A.4 and A.5 of the KataGo paper.

The way we compute these outputs is a bit like the way we compute the other layers. We take the outputs of the last ordinary layer, and for the (1) things that have per-location values we compute linear combinations of those outputs (the coefficients are learned along with all the other weights in the network) and apply suitable nonlinear functions to the results for the outputs to make human sense; for the (2) things that have global values we first do a global-pooling-like step on all the per-location outputs and then compute linear combinations and apply suitable final nonlinearities. Again, details in appendix A.4 and A.5.

That's all about the network; of course the search is another very important component, but you didn't ask about that :-).

Comment by gjm on Containment Thread on the Motivation and Political Context for My Philosophy of Language Agenda · 2021-05-16T10:17:28.326Z · LW · GW

OK. I'm not sure to what extent I'm supposed to take the last comment as an insult ("you're very good at emitting sophistical bullshit" or whatever), but no matter :-).

I don't know that I was feeling optimistic, but I had had some hopes that you might be persuaded to engage with what seem like key criticisms rather than just dismissing them. But you certainly should feel obliged to engage with someone you aren't finding it worthwhile arguing with. [EDITED to add:] Er, oops, of course I mean you shouldn't feel obliged.

By the way, I see that at least one earlier comment of yours in this thread has been downvoted; it wasn't by me.

Comment by gjm on Containment Thread on the Motivation and Political Context for My Philosophy of Language Agenda · 2021-05-16T03:41:50.309Z · LW · GW

Definitions aren't generally arbitrary in communication for reasons similar to why they aren't arbitrary in cognition; if I define "woman" to mean "adult female human" (for some possibly-contentious definition of female" I will communicate more effectively than if I define it to mean "adult female human who is not called Jane, OR 4x2 lego brick" (same definition of "female"), even if everyone knows what definitions I am using. I think the distinction that's doing the actual work isn't between communication and cognition, but between proper nouns (where the question is how you assign names to specific single things) and common nouns (where there are also boundary-drawing questions). Anyway, no matter; as I said, the point of what I said about proper nouns was merely to establish that language is not used only for prediction-optimizing.

Of course people take "he" and "she" to convey sex-category information! But (because different people draw boundaries in different places) that doesn't mean that what e.g. "he" conveys is anything as specific as "part of the cluster in thingspace that contains typical men, with Zack's preferred metric on thingspace", which is what it would have to be for your use of terms like "lie" to be right.

My apologies for missing the link to TCWMFM! I retract my claim that eukaryote didn't mention it. For what it's worth, my guess is that without TCWMFM eukaryote would in fact have said much the same things as they did with TCWMFM; the idea that category boundaries are "a little bit arbitrary" did not originate there. Philosophers have been saying "well, it all depends on what you mean by ..." for ages. And, despite your many thousands of words on the subject, I remain entirely unpersuaded that it's "bonkers", and it still looks to me as if your assertions that it's "bonkers" are based on entirely unrealistic ideas about how language ought to be used.

Sure, it would be nice if the language I use to talk were perfectly optimized for thinking. But it isn't, and it never will be, and any possible conflict between tact and prediction-optimality in gender-words is far down the list of reasons why. Because the primary requirement for the language I use to talk to my friends is that I should be able to use it to talk to my friends, and it happens that the language my friends mostly talk is English, which like every other natural language is a pile of kludges and path-dependent arbitrary choices. And also because different people define things in slightly different ways, so no matter how carefully optimized my notion of (say) "woman" might be, when I'm talking with other people that doesn't matter because they are not going to mean the exact same thing by it as I do. Even if nothing we are saying has anything to do with trans people at all.

To be more precise: no, I don't "want" the language I speak with my friends to be the same as the language I use to organize my own thoughts. The way I organize my own thoughts is necessarily idiosyncratic; it has to work for my brain, thinking the kinds of thoughts that I happen to need to think, given the things that I happen to know; fortunately my friends are not my clones, and their brains and thoughts and knowledge are different; so there is no possible way for the language we speak to be optimal for all of our thoughts.

What's wrong with "adult human female" is that it just pushes whatever difficulties there might have been in "woman" over to "female". I'm perfectly happy agreeing that women are adult female humans (or adult human females, but that's an ugly locution) and I think everyone would be if that set of words hadn't been adopted by some people specifically as a way of objecting to some things trans people want to say and think. But some people would want "female" to mean "person with no Y chromosome" and some would want it to mean "person nearer to the cluster in Zack-concept-space that contains typical women" and some would want it to mean "person who considers herself female". Again, the circularity is perfectly benign; take any definition of "woman" that is anywhere near any definition any sane human being uses, apply the cluster definition, and iterate a few times, and it will converge very rapidly.

I've no idea why you say at that point "I get it, we want to be trans-inclusive"; I was referring to your preferred understanding of the word "woman" (in so far as I understand it right, which quite possibly I don't) and so far as I can tell you don't particularly want that to be trans-inclusive, to whatever extent being trans-inclusive conflicts with prediction-optimality. But, for what it's worth, I don't think that the clean way to be trans-inclusive is what you say; maybe that's one clean-ish way to be kinda-trans-inclusive but it seems to me it doesn't do either of those things terribly well and that term "sufficiently successful" has an awful lot of stuff swept under it.

Indeed sex is functionally binary (though I remark that most of the individual consequences of sex that actually matter for most human interactions are very much not binary, although if you take them collectively you can pretty much recover the binary classification them them), and indeed it is unlike (say) size in that respect. But I don't think that's relevant to the point I was making, which is simply that "woman" and "town" both have boundaries that different people draw in different places, and therefore the only circumstances in which you are going to get anything like the optimal communication you want are those where we say very explicitly where we draw the boundaries, and once we do that we are communicating far more precisely than the norm however much you dislike the specific boundaries I draw.

The rest of that paragraph only has any chance of being right if we make the assumption, which I explicitly reject, that the only valid way to delineate the meanings of words is by picking something like minimum-volume clusters in concept-space. No! It isn't! The only valid way to delineate the meanings of words, outside technical discussions where you make explicit and precise definitions, is to try to match your usage of each word to how the other people you're trying to communicate with use it. And, like it or not, the way language actually works does not give you anything like a guarantee that common usage will do a very good job of optimizing compactness in concept-space. If your goal is to communicate with other people, it doesn't matter whether your preferred usage corresponds to something you consider a natural cluster in concept-space unless everyone else is also choosing their definitions that way. Which they aren't.

(And I am very unconvinced by the assumptions you are making about the proper metrics in concept-space, for reasons I already explained about as well as I think I can explain them. If all I know about you is what gender you, or someone else, says you are, then it is most likely that the appropriate metric on concept-space for the purposes of whatever interactions we are going to have while that remains the case is one that cares mostly about things like what words you prefer me to use when talking to you. And once we start having interactions for which more subtle gender-related differences matter, what I've been told about your gender probably no longer has much impact on my predictions because I know a bunch of other things that mostly screen that off.)

"Treating someone as male or female" means, in practice, treating them in ways that they interpret as indicating that I regard them as male or female. This is not a thing with a precise definition and it probably couldn't be. But as long as we have (e.g.) different pronouns for male and female, different public toilets for men and women, different social expectations for men and women (maybe we shouldn't have those! but they are there none the less), it will be hard to avoid some degree of pigeonholing and many people will care which pigeonhole you seem to be putting them into.

Comment by gjm on Challenge: know everything that the best go bot knows about go · 2021-05-16T02:34:12.173Z · LW · GW

I too am not sure whence cometh our disagreement, but I know the point at which I first thought we had one. There was some discussion of CNN-based go programs looking at "local patterns" and you said:

Does AlphaGo rely on local patterns? Possibly, but AlphaGoZero? Where humans see a 3 phase game with maybe 320 moves, which gets broken down into opening, middle and end game, ko's, threats, exchanges, and so on, it seems likely AlphaGoZero sees the whole game as one 'thing' (and in fact sees that one game as just one variation in the likely billions of millions of trillions of games it has played with itself).

which seemed to me to be responding to "these programs look at local patterns" with "I don't believe AlphaGo Zero does, because it sees the whole game as one thing rather than looking at different phases of the game separately", and I think that in the previous discussion "local" was being used spatially (small region of the board) rather than temporally (one phase of the game, or part thereof) but your response seemed to assume otherwise.

On "locally alive", on reflection I think a more common usage is that you call a group "locally alive" when you can see two eyes for it (or an unstoppable way of getting them) locally; but it can be "not locally alive" without being "dead" because there might be a way to connect it to other useful things, or run it out into somewhere where it has space to make more eyes.

I think we are using "local ko threat" in pretty much the same way, which is reassuring :-). I think it's a bit different from other uses of "local" because the region of the board involved can potentially be very large, if e.g. the black dragon in your example stretches all the way across the board. But it's not very different; it's still about being concerned only with a subset of the board.

Comment by gjm on Challenge: know everything that the best go bot knows about go · 2021-05-15T16:47:45.196Z · LW · GW

As I said elsewhere in the thread, by "local" I mean "looking only at a smallish region of the board". A "local pattern" is one defined by reference to a small part of the board. A group is "locally alive" if there's nothing in the part of the board it occupies that would make it not-alive. A move is "the best move locally" if when you look at a given smallish region of the board it's the best move so far as you can judge from the configuration there. Etc. (There are uses of "local" that don't quite match that; e.g., a "local ko threat" is one that affects the stones that take part in the ko.)

What I mean about empty triangles is (of course) not that making an empty triangle is always bad. I mean that the judgement of whether something is an empty triangle or not is something you do by looking only at a very small region of the board; and that if some part of you winces just a little when you have to play one, that part of you is probably looking only at a small bit of the board at a time. That is: judging that something is an empty triangle (which your brain needs to do for that slight wincing reaction to occur) is a matter of recognizing what I have been calling a "local pattern".

Yes, "placing one stone next to another" is also a local thing; the notion of "attachment", for instance, is a very local one.

Yes, the computations the network does even on its lowest layers have been optimized to try to win the whole game. But what that means (in my terms, at least) is that the early layers of the network are identifying local features of the position that may have global significance for the actual task of evaluating the position as a whole and choosing the next move.

Comment by gjm on Challenge: know everything that the best go bot knows about go · 2021-05-15T16:37:31.526Z · LW · GW

Perhaps you would like to clarify how you are intending to use the word "local"?

My usage here is as follows: a "local pattern" is something whose presence or absence you can evaluate by looking at a small region of the board. (The smaller, the more local; locality comes in degrees. Presence or absence of a pattern might do, too.) So e.g. an empty triangle is an extremely local pattern; you can tell whether it is present by looking at a very small region of the board. A ponnuki is slightly less local, a table-shape slightly less local again, but these are all very local. A double-wing formation is substantially less local: to determine that one is present you need to look at (at least) a corner region and about half of one adjacent side. A ladder in one corner together with a ladder-breaker in the opposite corner is substantially less local again: to see that that's present you need to look all the way across the board.

(I should maybe reiterate that the networks aren't really computing simple binary "is there an empty triangle here?" values, at least not in later layers. But what they're doing in their earlier layers is at least a little bit like asking whether a given pattern is present at each board location.)

This seems to me to be the standard sense, but I might well be missing something. I had a quick look through some books but didn't spot any uses of the word "local" :-).

(One can talk about things other than patterns being local. E.g., you might say that a move is locally good, meaning something like "there is some, hopefully obvious, portion of the board within which this move is good for most plausible configurations of the rest of the board, but it's possible that in the actual global position it's not a good move". Or sometimes you might say the same thing meaning just "this is the best move among moves in this part of the board". Or you might say that a group is locally alive, meaning something similar: for the group to be dead there would need to be unusual things elsewhere on the board that somehow interact with it. All these things seem entirely compatible with what I'm saying about "local patterns".)

The KataGo network is not pure CNN. It does something called "global pooling", where at various points in the network the mean and max values across all board locations of some of the channels are computed and used to bias the values of the other channels in the next layer. So it learns to use those channels to compute things that are of global interest. I'm not sure how much is known about exactly what things they are, but I expect them to be things like which player is winning by how much, whether there are kos on the board or likely to be on the board soon, who has more ko threats, etc.

(In case you aren't familiar with the relevant terminology, a "channel" is one of the things computed on each layer for each location on the board.)

Comment by gjm on Containment Thread on the Motivation and Political Context for My Philosophy of Language Agenda · 2021-05-15T14:38:46.851Z · LW · GW

The examples seem relevant to me because they illustrate that language is not used only to predict, that the merits of a particular language-using strategy are not determined only by its impact on predictive accuracy. If language in general has proper goals other than predictive accuracy, why should I think that category-boundary drawing has no proper goal other than predictive accuracy?

I'm not sure exactly what distinction you're appealing to, by the way. In particular, you say "the communicative function of proper names ... the cognitive function of categories" and it's not clear to me whether (1) you're suggesting that proper names are used primarily for communication while categories are used primarily for cognition, or (2) you're saying that your complaints about talk of arbitrariness apply only when thinking about cognition as opposed to communication, or (3) something entirely different. I say that proper names and category boundaries are highly relevant to both communication and cognition, and that some of the examples that apparently bother you most seem much more about communication than about cognition (e.g., being asked to use pronouns that you find inappropriate for the people they refer to, which you say amounts to asking you to lie, clearly a communicative category more than a cognitive one). It's entirely possible that your meaning is something entirely different, of course; please feel free to enlighten me if so.

The pouncing algorithm can't literally be "pounce when people say nice things about TCWMFM" because in the particular case that sparked this particular discussion that didn't happen but you still pounced.

It probably won't surprise you that I don't agree with your description of TCWMFM as pants-on-fire mendacious, and I don't think the edit at the end undermines the foregoing material nearly as much as you say it does. There are rules, but the rules are of the form "if you want your thinking to be optimal in such-and-such a way, you have to think like this" and Scott is talking about words: how to speak, not how to think. Words and thoughts are closely related but they aren't the same, and I agree with Scott that neither rationality nor rationalism includes any obligation to optimize your words for the exact same thing as your thoughts.

For example, suppose that (1) someone decides to use the word "woman" to mean, as precisely as they are able, "adult human being who identifies as female", and that also (2) you are correct in holding that the region of person-space picked out by this definition is less useful for making predictions than one like "person whose gender-related characteristics are, collectively, more like those of the average woman than like those of the average man". (That definition is kinda-circular but in a way that does no actual harm, and it leaves open the question of what "more like" means, but let's suppose that (3) no plausible way of making "more like" concrete gives you anything close to "adult human who identifies as female", though I'm not sure that's actually the case.) Is our hypothetical person doing something rationalists should be ashamed to do?

I claim they need not be. They are (given all the stipulations in the previous paragraph) using language in a way that isn't optimized for prediction, but I see no reason why language-use should be optimized purely for prediction, because the actual uses of language are broader than that.

You might object that they are impairing their ability to think clearly: when they say or hear that someone is a woman, that will lead to certain expectations in their mind, and if they define "woman" in a gerrymandered way then those expectations will be wronger-in-expectation than they need be. It seems to me that any such effect is very small; someone using the word "woman" this way knows that they are doing so, knows that when someone else says "X is a woman" they don't necessarily mean it the same way, can when it seems appropriate think explicitly in terms of Y-chromosomes or hormones or whatever other characteristics might be relevant. The last point is important. When you have some category without a standard precise technical definition, and it's actually important to reason about its members, it usually turns out that what really matters isn't membership in the category as such but something more specific. Is So-and-So "really" a woman? In cases where there's any scope for doubt about that, you will get more useful answers by figuring out what the real question is (do I want to have sex with her? should I let her into my javelin-throwing competition? which public toilet should she be using? what dose of this drug should I give her? is it possible that she's pregnant? etc., etc., etc.) and answering that more directly. So, anyway, I think any thought-impairment here is very small, and reiterate that language has purposes other than facilitating clear thought; if our hypothetical person thinks there are other benefits that outweigh any such loss, it is far from obvious that they're wrong.

You might object that if they use language that way they are failing to communicate maximally informatively: if someone who uses "woman" that way tells you someone is a woman, the information they're giving you is less useful for your prediction-making than if they used the word in a prediction-optimized way. Well, maybe, kinda, but for at least two reasons I don't think this is something it's reasonable to get upset about. 1. Regardless of what they do, the word "woman" is going to have ambiguities around the edges unless the person using that word tells you explicitly and in detail where they draw the boundaries. (The same is true, in less politically-sensitive ways, of many other words. If I call something a mountain or a town, or say someone is stupid or beautiful, or say that a wall is painted green, you don't know exactly how "towns" shade into "villages" and "cities" for me, or what features make someone more beautiful or how "beautiful" shades into weaker terms like "pretty" or "sexy" or "elegant", or how I categorize turquoise-ish colours.) So even if I choose to use the word "woman" in a way that is in fact optimized for your predictive purposes, that's only any use if I first of all tell you exactly how I'm defining that word. Which almost never happens. 2. If I do tell you exactly how I'm defining the word, then no matter what definition I'm using my uses of "woman" are thereafter strictly more informative than if I don't give you that information; i.e., strictly more informative than say 99.9% of uses of the word; I don't think you're then entitled to complain that I could be helping you even more.

None of this is particularly specific to the word "woman". The same is true of, say, the word "fish" as in Scott's initial example. I think Scott is exactly correct when he says that "in no case can an agreed-upon set of borders or a category boundary be factually incorrect". Even though a category boundary absolutely can be not optimized for any reasonable notion of optimal prediction. (And can be bad in other ways; in practice, I think being very different from other people's boundaries is a more important failure mode for the purpose of communication.)

I should maybe add that I don't actually find the stipulations above very plausible. Most social interactions are rather shallow, which means that most gender-related predictions I need to make that might be influenced by what pronoun is being used for a given person have little to do with the fascinating ways in which typical men's and women's bodies or minds differ, which means that if all I know about someone is that they were referred to as "she" or that someone called them a "woman", quite possibly the most useful information I can get from that does concern whether that person would prefer to be treated as male or female. If at some point those other deeper questions start mattering to me, almost certainly by that point I will have met them, seen what they look like, talked to them, etc., and at that point most of the information I have about their personality and preferences and so forth doesn't come from their Official Gender at all. In which case, during the period when their Official Gender is actually of much use for my predictions, the predictions I'm needing to make are in fact mostly about "how they identify".