Comment by JBlack on [AN #157]: Measuring misalignment in the technology underlying Copilot · 2021-07-24T03:45:29.463Z · LW · GW

I suspect that "progress toward human level AI" is extremely non-linear in the outputs, not just the inputs. I think the ranges between "dog" and "human", and between "human" and "unequivocally superhuman" are pretty similar and narrow on some absolute scale: perhaps 19, 20, and 21 respectively in some arbitrary units.

We have developed some narrow AI capabilities that are superhuman, but in terms of more general intelligence we're still well short of "dog". I wouldn't be very surprised if we were at about 10-15 on the previous fictional scale, up from maybe 2-3 back in the 80's.

Superficially it doesn't look like a lot of progress is being made on human-level general intelligence, because our best efforts are in general terms still stupider than the dumbest dogs and that fact hasn't changed in the last 40 years. We've broadened the fraction of tasks where AI can do almost as well as a human or better by maybe a couple of percent, which doesn't look like much progress.

But that's exactly what we should expect. Many of the tasks we're interested in have "sharp" evaluation curves, where they require multiple capabilities and anything less than human performance in any one capability required for the task will lead to a score near zero.

If this model has any truth to it, by the time we get to 19 on this general intelligence scale we'll probably still be bemoaning (or laughing at) how dumb our AIs are. Even while in many more respects they will have superior capabilities, and right on the verge of becoming unequivocally superhuman.

Comment by JBlack on A cognitive algorithm for "free will." · 2021-07-22T06:20:36.189Z · LW · GW

If they're all interchangeable for the effects of your point, then I'm even more unsure what your point is than when I started.

You seem to be mixing up "discussing non-determinism" with "doing non-deterministic things", which seems like the essence of a map/territory confusion.

Comment by JBlack on The inescapability of knowledge · 2021-07-22T04:50:52.902Z · LW · GW

Viruses are generally very much simpler than bacteria, yes.

My possibly flawed understanding is that most viruses don't really do anything at all by themselves. Once they encounter cells with the right receptors, they get ingested and (again only for the right types of cell) the internal machinery processes them in a way that makes more viruses.

I suppose you could think of that as "sensing" cells and "acting" to get inside and hijack them, but it's a bit of a stretch and why I'm not sure that they should be included. From an information-processing point of view, I think of them more like passive info-hazards than active agents.

Comment by JBlack on bfinn's Shortform · 2021-07-22T04:29:50.339Z · LW · GW

Yes, there is an idea of "right to reproduce", but hardly anyone believes that it should hold for animals in the sense of your article. The exceptions seem mainly to apply to critically endangered species.

Rather a lot of people don't hold "right to reproduce" in unrestricted form for humans either. It certainly doesn't have the same near-universality as human right to life.

While mass forced sterilization of particular ethnic groups is absolutely a form of genocide, this again goes way beyond any analogous beliefs that animal rights people hold. Nobody is saying that all chickens and cows should be sterilized so that their species becomes extinct. The closest it gets is "stop force-breeding them".

Comment by JBlack on bfinn's Shortform · 2021-07-20T07:01:31.155Z · LW · GW

I guess one important question not raised here (but lurking in unspoken assumptions) is whether it is better for 1000 animals to be killed after about 3 months on average, or for 50 animals to live an average of 5 possibly painful years each in the wild.

If you scaled up for humans (1000 humans each killed at 3 years old versus 50 humans living to an average of 60) it seems obvious, but that's probably a false equivalency.

Comment by JBlack on bfinn's Shortform · 2021-07-20T06:45:22.035Z · LW · GW

I'm not sure what position you're taking here.

"Right to life" doesn't mean "right to have as many individuals of some group brought into existence as physically possible". It usually means more like "a living individual has a right to not be killed". If you're interpreting it as the former, then it seems to me that you're grossly misinformed about what people actually mean by the term.

In theory some position of total utilitarianism might lead to something along the lines you imply, but even that would have to be traded off against other possible ways of increasing whatever total utility function is being proposed. Most such theories are incompatible with "right to life" in either sense.

I do agree with part of your argument, but think it would have been better stated alone, and not following from and in conjunction with a complete strawman.

Comment by JBlack on Generalising Logic Gates · 2021-07-20T06:19:55.578Z · LW · GW

Even very simply stated problems in boolean circuit optimization turn out to be deeply connected with extremely hard mathematical and computational problems, and in some cases even provably undecidable problems.

I don't hold out very much hope of ever being able to find and prove solutions to most of these questions, even assuming superintelligence.

On the other hand, there are no doubt lots of practical improvements, even very substantial ones, to be made.

Comment by JBlack on Equivalent of Information Theory but for Computation? · 2021-07-20T03:41:20.795Z · LW · GW

Yes, and for this reason it's usual to consider only finite alphabets.

While any particular bound on input and output size can be processed in a single step with a large enough finite alphabet, Turing machines operate on any finite input without bound on length. Representing all of those with one symbol each would require an infinite alphabet with a correspondingly infinite state transition table.

Comment by JBlack on Equivalent of Information Theory but for Computation? · 2021-07-19T13:49:04.334Z · LW · GW

Turing machine encodings don't really "hide" much in the alphabet or state, since their information content is logarithmic. Large alphabets can be efficiently emulated with only logarithmic slowdown, that is alphabets of up to 2^N symbols can always be emulated with a TM of only 2 symbols running N times slower. Larger state tables are harder to efficiently emulate, but there are at least many universal Turing machines that can emulate any other Turing machine in time proportional to the information content of the state table.

Boolean circuits aren't really arbitrary. The exact set of operators doesn't actually matter much, since every set of operators can be (usually efficiently) emulated by every other universal set. The set {or, and, not} is familiar and can express every other boolean function. The singleton sets {nand} and {nor} are also commonly used. Either of these is also universal, with the only difference between them being the labelling of the states. In one sense, this is the simplest element of digital computation.

I'm not sure what you mean by needing a theory of computation to understand things like logical uncertainty. Can you expand on this a bit more?

Comment by JBlack on A cognitive algorithm for "free will." · 2021-07-19T08:59:10.491Z · LW · GW

I'm not sure how logical deduction is related in any way at all to physical (or even psychological) determinism. It is normal, and extremely common, to reason logically about non-deterministic systems.

Even logic itself is non-deterministic, when viewed as a sequential system. From "A and (if A then B)" you can derive "B", as usual. You can also derive "A and B", or "(A or not-B) and (if not-B then not-A)" or any of an infinite number of other sentences. Nothing says which you will logically derive, it just says what you can and what you can't.

Comment by JBlack on The inescapability of knowledge · 2021-07-19T04:27:13.109Z · LW · GW

I don't think in the context of this discussion that it does beg the question.

The point I was discussing was whether we really mean the same thing by "knowledge in a book" and "knowledge known by an agent". My argument is that the phrase "knowledge in a book" is just a notational shorthand for "knowledge some implied agents can be expected to gain from it".

If this is a reasonable position, then "knowledge in an object" is not a property of the object itself. Examining how it is present there is making a probably erroneous assumption that it is there to be found at all.

The question does remain about how knowledge is represented in agents, and I think that is the much more interesting and fruitful meat of the question.

Comment by JBlack on The inescapability of knowledge · 2021-07-19T04:05:22.658Z · LW · GW

A living, otherwise fairly typical cat? Absolutely yes, and not even near a boundary for the concept. Dead cat? I wouldn't say so.

As I see it, the term "agent" is very much broader than "sentient". It covers pretty much anything capable of taking different actions based on internal information processing and external senses. Essentially all living things are agents, and plenty of nonliving things.

So the bacteria within a dead cat (or even a living one) would qualify, but I don't think you could reasonably ascribe "actions based on internal information" to a dead cat as any sort of single entity.

It seems to me that examples in the fuzzy boundaries are more like simple thermostats and viruses than cats.

Comment by JBlack on The inescapability of knowledge · 2021-07-16T06:43:19.996Z · LW · GW

I would prefer to say that a textbook doesn't make predictions. It may encode some information in a way that allows an agent to make a prediction. I'm open to that being semantic hair-splitting, but I think there is a useful distinction to be made between "making predictions" (as an action taken by an agent), and "having some representation of a prediction encoded in it" (a possibly static property that depends upon interpretation).

But then, that just pushes the distinction back a little: what is an agent? Per common usage, it is something that can "decide to act". In this context we presumably also want to extend this to entities that can only "act" in the sense of accepting or rejecting beliefs (such as the favourite "brain in a jar").

I think one distinguishing property we might ascribe even to the brain-in-a-jar is the likelihood that its decisions could affect the rest of the world in the gross material way we're accustomed to thinking about. Even one neuron of input or output being "hooked up" could suffice in principle. It's a lot harder to see how the internal states of a lump of rock could be "hooked up" in any corresponding manner without essentially subsuming it into something that we already think of as an agent.

Comment by JBlack on Anthropic decision theory for self-locating beliefs · 2021-07-13T05:33:33.709Z · LW · GW

I'm not sure about the utility of a monetary payout in this variant of the Sleeping Beauty problem where she is killed at the end of the experiment. Is she assumed to pay for something she values in the one remaining day in which she (physically or psychologically) exists?

I can see two distinct possibilities here: whether she buys something of local utility (maybe an ice-cream), or of global utility (e.g. gifting it to someone she values). This may matter a great deal in the Tails case: even the copy altruistic average utilitarian will recognise that average utility is increased twice as much as in Heads for this case, even though each copy won't know whether or not it has.

Comment by JBlack on Agency and the unreliable autonomous car · 2021-07-13T04:41:50.743Z · LW · GW

Yes, I think this change is much better. I'm still a bit unclear about how exactly the agent reasons about itself. That doesn't seem to be well-defined.

Is it capable of distinctions between its models of the world? The reasoning in the article leads me to think not. The wording used is of the form "a proof exists of P" rather than "in hypothetical situation H in which I have world model M, my deductive rules could find a proof of P".

This may lead it to equivocation errors, where the same symbols are used to refer to different things.

Comment by JBlack on The inescapability of knowledge · 2021-07-12T10:13:58.633Z · LW · GW

  "Most importantly, we are seeking a characterization of the patterns themselves that are produced by evidence-collecting, evidence-organizing entities, and are later used to exert flexible influence over the future."

This would be very nice, but may turn out to be as difficult as seeking an explanation for how cats purr in terms of quantum field theory. It's quite possible that there are a lot of abstraction layers in between physical patterns and agentive behaviour.

Comment by JBlack on The inescapability of knowledge · 2021-07-12T09:40:39.260Z · LW · GW

Isn't the most important feature of an "internal map" that it is a conceptual and subjective thing, and not a physical thing? Obviously this smacks of dualism, but that's the price we pay for being able to communicate at all.

Any part of reality very likely corresponds with other parts of reality (to the extent that it makes sense to divide reality into parts), but that doesn't imbue them with knowledge, because they're the wrong abstraction level of thing to be maps and so their correspondence doesn't count.

Like any other abstraction, internal maps are fuzzy around the edges. We know of some things that we definitely call maps (in the sense of "correspondence between map and territory"), such as hand-waving at some aspects of whatever commonalities there are between humans when thinking about the world. This concept also seems to be applicable to behaviours of some other animals. We often ascribe internal maps to behaviour of some computer programs too. We ascribe maps to lots of classes of imaginary things. It seems that we do this for most things where we can identify some sensory input, some sort of information store, and some repertoire of actions that appear to be based on both.

We can talk metaphorically about physical objects conveying knowledge, with the help of some implied maps. For example, we may have some set of agents in mind that we expect to be able to use the physical object to update their maps to better match whatever aspects of territory we have in mind. With some shared reference class we can then talk about which objects are better by various metrics at conveying knowledge.

I do think it is is true that in principle "there is no objective difference between a book containing a painstakingly accurate account of a particular battle, and another book of carelessly assembled just-so stories about the same battle" (emphasis mine). With sufficiently bizarre coincidence of contexts, they could even be objectively identical objects. We can in practice say that in some expected class of agents (say, people from the writer's culture who are capable of reading) interacting in expected ways (like reading it instead of burning it for heat), the former will almost certainly convey more knowledge about the battle than the latter.

Comment by JBlack on Consequences of Bayesian Epistemology? · 2021-07-12T04:30:27.256Z · LW · GW

With regard to journal results, it is even worse than that.

A published result with p < 0.05 means that: if the given hypothesis is false, but the underlying model and experimental design is otherwise correct, then there is at least a 95% chance that we don't see results like this.

There are enough negations and qualifiers in there that even highly competent scientists get confused on occasion.

Comment by JBlack on Rationality Yellow Belt Test Questions? · 2021-07-12T04:15:56.484Z · LW · GW

The rest of the questions have much more reasonable explanations. It seems unfortunate that the first (and most visible) question is also the most dubious.

Comment by JBlack on Rationality Yellow Belt Test Questions? · 2021-07-12T03:52:39.340Z · LW · GW

Uh wow. The first sample question at that link has five possible answers, and all are wrong for different reasons. I agree with their selected answer in that it's least incorrect (it's possible that this is a source of disagreement), but it's still wrong (you can't logically conclude that this is a disagreement).

Response (D) is incorrect since Kim has not said or even strongly implied that medical applications are the most valuable achievements. Kim merely provides it as an example for pure research leading to saving lives. Kim may believe that other achievements are even more valuable (i.e. save far more lives), but chose medicine as an example due to the link being more direct or any number of other reasons.

So far I am not very impressed.

Comment by JBlack on Agency and the unreliable autonomous car · 2021-07-12T02:10:50.366Z · LW · GW

Yes, I'm referring to 5a/5b in conjunction with self-reflection as a deduction rule, since the agent is described as being able to use these to derive new propositions.

There is also a serious problem with 5a itself: the agent needs to try to prove that some new proposition P is "consistent with its model of the world". That is, for its existing axioms and derivation rules T, prove that T+P does not derive a contradiction.

If T is consistent, then this is impossible by Gödel's second incompleteness theorem. Hence for any P, Step 5a will always exhaust all possible proofs of up to whatever bound and return without finding such a proof. Otherwise T is inconsistent and it may be able to find such a proof, but it is also obvious that its proofs can't be trusted and not at all surprising that it will take the wrong route.

Comment by JBlack on Anthropic probabilities: answering different questions · 2021-07-10T09:03:33.195Z · LW · GW

What sort of "anthropic events" must be happening to you every second with enough weight to be non-negligible?

I'd clarify the theory of gravity statement to "might be true for all possible observers, we have no way of knowing". I would agree that our observations only support it for those who share enough of our memories.

Comment by JBlack on Sleeping Beauty Not Resolved · 2021-07-10T08:04:14.877Z · LW · GW

I like an alternative version of the problem proposed by someone whose identity escapes me.

Thirty students are enrolled in an ethically dubious study. One of them is selected at random to be awakened on the 1st of the month. For the rest, a 6-sided die is rolled. On a result of "1", each is awakened on a different day of the rest of the 30-day month. On any other result, they are not awakened at all. All participants who are woken are asked to guess whether the die rolled "1" or not.

What should they guess? Does this differ from the case where the 30 students are replaced by a single participant given amnesic drugs?

Comment by JBlack on Agency and the unreliable autonomous car · 2021-07-10T06:40:55.430Z · LW · GW

This looks full of straightforward errors.

Firstly, the agent's logical deduction system is unsound. It includes something comparable with Peano arithmetic (or else Löb's theorem can't be applied), and then adds a deduction rule "if P can be proved consistent with the system-so-far then assume P is true". But we already know that for any consistent extension T of Peano arithmetic there is at least one proposition G for which both G and ~G are consistent with T. So both of these are deducible using the rule. Now, the agent might not find the contradiction, because...

This system can only find proofs up to a length of a million characters. The requirements of Löb's theorem include a formal system that is an extension of PA and is closed under deduction: that is, derives proofs of unbounded length. So we can't apply Löb's theorem to this system anyway.

Comment by JBlack on Are coincidences clues about missed disasters? It depends on your answer to the Sleeping Beauty Problem. · 2021-07-10T04:58:00.118Z · LW · GW

I don't see why "the probability of now being Monday" is undefined. I agree that the information given in the problem isn't enough to determine it, but that's a very different objection.

Comment by JBlack on Should VS Would and Newcomb's Paradox · 2021-07-08T01:52:42.330Z · LW · GW

No, I wouldn't take the money on the table in this case.

I'm easily read, so I already gave off signs of what my decision would turn out to be. You're not very good at picking them up, but enough that if people in my position take the money then there's a 49% chance that the envelope contains a million dollars. If they don't, then there's a 51% chance that it does.

I'm not going to take $1000 if it is associated with a 2% reduction in the chance of me having a $1000000 in the envelope. On average, that would make me poorer. In the strict local causal sense I would be richer taking the money, but that reasoning is subject to Simpson's paradox: action Take appears better than Leave for both cases Million and None in the envelope, but is worse when the cases are combined because the weights are not independent. Even a very weak correlation is enough because the pay-offs are so disparate.

Comment by JBlack on [deleted post] 2021-07-07T13:21:03.754Z

There are two factors that make all diet claims frustrating.

Most Diets Work. While most people adhere to them, most people find that their weight moves in the desired direction for at least a couple of months. Pick any fad diet, and there is a good chance that despite not having any sound basis, they do the job. One study even found that just prescribing that participants think about the diet while purchasing food, without actually listing any foods that should or not not be eaten or any limits on quantities, was almost as good as a more rigorous diet plan.

Most Diets Fail. Beyond a few months, the majority of people who began with unhealthy weight don't adhere long-term to any particular dietary structure, and usually return to their previous eating habits for long enough to return to their previous weight. There are of course also plenty of exceptions, and lots of inconclusive studies as to why most diets fail.

All sorts of claims are made about foundations that will make This Diet Work in the first sense, without addressing the fact that Most Diets Work. The second part is usually glossed over, at best getting some lip service like "well, you should be able to sustain it for the rest of your life. It works, so why wouldn't you?"

Comment by JBlack on Should VS Would and Newcomb's Paradox · 2021-07-07T03:26:12.756Z · LW · GW

"After you put it in your pocket I put 1000 dollars on the table. Do you suggest not taking the 1000 dollar will make you richer?"

Unlike the Omega problem, this is way too underspecified to make a sensible answer. It depends upon the details of how you get your 51% success rate.

Do you always predict they're going to take two boxes, and only 51% of people actually did? Then obviously I will have $1000 instead of $0, and always be richer.

Or maybe you just get these feelings about people sometimes. In later carefully controlled tests it turns out that you get this feeling for about 2% of "easily read" people, you're right about 90% of the time in both directions for them, and it isn't correlated with how certain they themselves are about taking the money. This is more definite than any real-world situation will ever be, but illustrates the principle. In this scenario, if I'm in the 98% then your "prediction" is uncorrelated with my eventual intent, and I will be $1000 richer if I take the money.

Otherwise, I'm in the 2%. If I choose to take the money, there's a 90% chance that showed up in some outward signs before you gave me the envelope, and I get $1000. There's a 10% chance that it didn't, and I get $1001000 for an expected payout of $101000. Note that this is an expected payout because I don't know what is in the envelope. If I choose not to take the money, the same calculation gives $901000 expected payout.

Since I don't know whether I'm easily read or not, I'm staking a 98% chance of a $1000 gain against a 2% chance of a $800000 loss. This is a bad idea, and on balance loses me money.

Comment by JBlack on Should VS Would and Newcomb's Paradox · 2021-07-06T00:51:15.839Z · LW · GW

Neither analysis is correct. Both are incomplete. The outsider's viewpoint is less wrong.

The first person argument presented here finds a local maximum and stops there. Yes, if they ignore Omega's abilities and climb the two-box hill then they can get 1000 dollars more. No mention of whether they're on the right hill, and no analysis of whether this is reasonable given their knowledge of Omega.

The outsider's view as stated fails to account for a possibly limited ability of the decision-maker to choose what sort of decision-maker they can be. Omega knows (in this infallible predictor version), but the decision-maker might not. Or they might know but be powerless to update (not all possible agents have even local "free will").

Comment by JBlack on Should VS Would and Newcomb's Paradox · 2021-07-05T10:12:13.460Z · LW · GW

There is a problem, but it is not quite an inconsistency. The problem is the assumption that Omega is a perfect predictor. That is: for the system Omega + Agent + everything else, Omega can always find a fixed point such that Omega's prediction is always correct for the state of the system some time after making that prediction and subsequent evolution of the system based on that. Even in a completely deterministic universe, this is asking too much. Some systems just don't have fixed points.

The problem becomes much more reasonable when you consider an almost perfect predictor. It ruins the purity of the original question, but the space of possibilities becomes much broader.

It becomes more like a bluffing game. Omega can observe tells with superhuman acuity, you don't know what those tells are, and certainly don't know how to mask them. If your psychology was such that you'd take both boxes, you were already showing signs of it before you went into the room. Note that this is not supernatural, retro-causal, nor full of anthropic paradoxes but something people already do. Omega is just very much better at it than other humans are.

In this viewpoint, the obvious solution is to be so good at hiding your tells that Omega thinks you're a one-boxer while actually being a two-boxer. But you don't know how to do that, nobody else who tried succeeded, and you have to think you've got a 99.9% chance of success for it to be more worthwhile than obviously taking one box.

Comment by JBlack on The Unexpected Hanging Paradox · 2021-06-29T01:30:52.095Z · LW · GW

Yes, it does. The paradox is that if you accept "A and you cannot prove A" as a premise, and logically derive A, then you arrive at a contradiction: A is true and you can prove it. By the usual laws of logic, this means that you should refute the premise. That is, conclude that the judge is lying.

But if the judge is lying, then what basis do you have for proving that A is true?

Of course, this whole argument may not apply to someone who uses a different basis for reasoning than classical logic with the Law of Excluded Middle.

Comment by JBlack on How can there be a godless moral world ? · 2021-06-21T23:07:04.380Z · LW · GW

This says to me that you already accept the premise that it is possible that a godless moral world can exist, but feel that it would be unsatisfying if that possibility were true. I agree that it could be unsatisfying, but don't see that as a basis for whether it is actually true or not.

Comment by JBlack on How can there be a godless moral world ? · 2021-06-21T13:01:17.397Z · LW · GW

First I need to ask, what does the phrase "moral world" mean, or more specifically mean to you? Is it much the same concept as a "just world"? I know what is intended by someone who says that an action is moral (though there are major and vigorous disagreements about which actions are moral). I know what is meant by a person (or deity) being moral.

I do not know what is meant by a world being moral. Does it just mean "an absolute standard for morality exists"? I'm not sure how that could conceivably be a property of a world though, regardless of whether that world has a god or not.

Comment by JBlack on Eli's shortform feed · 2021-06-21T01:05:35.029Z · LW · GW

Utility functions are especially problematic in modeling behaviour for agents with bounded rationality, or those where there are costs of reasoning. These include every physically realizable agent.

For modelling human behaviour, even considering the ideals of what we would like human behaviour to achieve, there are even worse problems. We can hope that there is some utility function consistent with the behaviour we're modelling and just ignore cases where there isn't, but that doesn't seem satisfactory either.

Comment by JBlack on Variables Don't Represent The Physical World (And That's OK) · 2021-06-21T00:44:55.334Z · LW · GW

The way I use "extensibility" here is between two different models of reality, and just means that one can be obtained from the other merely by adding details to it without removing any parts of it. In this case I'm considering two models, both with abstractions such as the idea that "fish" exist as distinct parts of the universe, have definite "weights" that can be "measured", and so on.

One model is more abstract: there is a "population weight distribution" from which fish weights at some particular time are randomly drawn. This distribution has some free parameters, affected by the history of the tank.

One model is more fine-grained: there are a bunch of individual fish, each with their own weights, presumably determined by their own individual life circumstances. The concept of "population weight distribution" does not exist in the finer-grained model at all. There is no "abstract" population apart from the actual population of 100 fish in the tank.

So yes, in that sense the "population mean" variable does not directly represent anything in the physical world (or at least our finer-grained model of it). This does not make it useless. Its presence in the more abstract model allows us to make predictions about other tanks that we have not yet observed, and the finer-grained model does not.

Comment by JBlack on Conditional offers and low priors: the problem with 1-boxing Newcomb's dilemma · 2021-06-19T00:42:55.702Z · LW · GW

It is not at all surprising or interesting that when you make up a problem superficially similar to Newcomb's paradox but with totally different conditions, that you end up with different outcomes.

Comment by JBlack on Variables Don't Represent The Physical World (And That's OK) · 2021-06-17T04:32:56.532Z · LW · GW

I don't see any implications for determinism here, or even for complexity.

It's just a statement that these abstract models (and many others) that we commonly use are not directly extensible into the finer-grained model.

One thing to note is that in both cases, there are alternative abstract models with variables that are fully specified by the finer-grained reality model. They're just less convenient to use.

Comment by JBlack on Shall we count the living or the dead? · 2021-06-15T03:32:15.751Z · LW · GW

There is a serious underlying causal model difference here that cannot be addressed in a purely statistical manner.

The video proposes a model in which some fraction of the population have baseline allergies (yielding a 1% baseline allergic reaction rate over the course of the study), and independently from that the treatment causes allergic reactions in some fraction of people (just over 1% during the same period). If this model is universally correct, then the argument is reasonable.

Do we know that the side effects are independent in this way? It seems to me that we cannot assume this model of independence for every possible combination of treatment and side effect. If there is any positive correlation, then the assumption of independence will yield an extrapolated risk that is too low.

Putting this in terms of causal models, the independence model looks like: Various uncontrolled and unmeasured factors E in the environment sometimes cause some observed side effect S, as observed in the control group. Treatment T also sometimes causes S, regardless of E.

This seems too simplistic. It seems much more likely that E interacts with unknown internal factors I that vary per-person to sometimes cause side effect S in the control group. Treatment T also interacts with I to sometimes produce S.

If you already know that your patient is in a group observed to be at extra risk of S compared with the study population, it is reasonable to infer that this group has a different distribution of I putting them at greater risk of S than the studied group, even if you don't know how much or what the gears-level mechanism is. Since T also interacts with I, it is reasonable as a general principle to expect a greater chance of T causing the side effect in your patient than independence would indicate.

So the question "shall we count the living or the dead" seems misplaced. The real question is to what degree side effects are expected to have common causes or susceptibilities that depend upon persistent factors.

Allergic reactions in particular are known to be not independent, so an example video based on some other side effect may have been better. I can't really think of one though, which does sort of undermine the point of the article.

Comment by JBlack on Paper Review: Mathematical Truth · 2021-06-06T15:14:21.186Z · LW · GW

The main difference between mathematics and most other works of fiction is that mathematics is based on what you can derive when you follow certain sets of rules. The sets of rules are in principle just as arbitrary as any artistic creation, but some are very much more interesting in their own right or useful in the real world than others.

As I see it, the sense in which 2+2 "really" equals 4 is that we agree on a foundational set of definitions and rules taught at a very young age in today's cultures, following those rules leads to that result, and that such rules have been incredibly useful for thousands of years in nearly every known culture.

There are "mathematical truths" that don't share this history and aren't talked about in the same way.

Comment by JBlack on The Nature of Counterfactuals · 2021-06-06T06:29:04.980Z · LW · GW

A "counterfactual" seems to be just any output of a model given by inputs that were not observed. That is, a counterfactual is conceptually almost identical to a prediction. Even in deterministic universes, being able to make predictions based on incomplete information is likely useful to agents, and ability to handle counterfactuals is basically free if you have anything resembling a predictive model of the world.

If we have a model that Omega's behaviour requires that anyone choosing box B must receive 10 utility, then our counterfactuals (model outputs) should reflect that. We can of course entertain the idea that Omega doesn't behave according to such a model, because we have more general models that we can specialize. We must have, or we couldn't make any sense of text such as "let's suppose Omega is programmed in such a way...". That sentence in itself establishes a counterfactual (with a sub-model!), since I have no knowledge in reality of anyone named Omega nor of how they are programmed.

We might also have (for some reason) near-certain knowledge that Amy can't choose box B, but that wasn't stated as part of the initial scenario. Finding out that Amy in fact chose box A doesn't utterly erase the ability to employ a model in which Amy chooses box B, and so asking "what would have happened if Amy chose box B" is still a question with a reasonable answer using our knowledge about Omega. A less satisfactory counterfactual question might be "what would happen if Amy chose box A and didn't receive 5 utility".

Comment by JBlack on AllAmericanBreakfast's Shortform · 2021-06-02T08:06:47.352Z · LW · GW

Would you prefer "for nearly all purposes, any bounds there might be are irrelevant"?

Comment by JBlack on Jemist's Shortform · 2021-06-01T08:56:42.626Z · LW · GW

To me it seems more likely that this person is misreporting their motive than that they really oppose this allocation of a patch of grass on biodiversity grounds. I would expect grounds like "I want to use it myself" or slightly more general "it should be available for a wider group" to be very much more common, for example if I had to rank likelihood of motives after hearing that someone objects, but before hearing their reasons. I'd end up with more weight on "playing social games" than on "earnestly believes this".

On the other hand it would not surprise me very much that at least one person somewhere might truly hold this position. Just my weight for any particular person would be very low.

Comment by JBlack on Paper Review: Mathematical Truth · 2021-05-31T03:45:31.126Z · LW · GW

I see no reason why the semantic uniformity property should hold for mathematics. Having sentences of similar form certainly does not require that they have the same semantics even for sentences that have nothing to do with mathematics. English has polysemous words, and the meaning of "exists" in reference to mathematical concepts is different from the meaning of "exists" in reference to physical entities.

If this was for some reason a big problem, then an obvious solution would be to use different words or sentence structure for mathematics. To some degree this is already the usual convention: Mathematics has its own notations, which are translated loosely into English and other natural languages. There are many examples of students taking these vaguer translations overly literally to their own detriment.

Comment by JBlack on Agency in Conway’s Game of Life · 2021-05-13T02:50:56.506Z · LW · GW

My strong expectation is that in Life, there is no configuration that you can put in the starting area that is robust against randomly initializing the larger area.

There are very likely other cellular automata which do support arbitrary computation, but which are much less fragile versus evolution of randomly initialized spaces nearby.

Comment by JBlack on Zero to One: A Minimum Viable Review · 2021-05-08T03:21:08.128Z · LW · GW

It seems to me that the positions labelled "definite" vs "indefinite" here are really closer to "self-determination" vs "fatalism". It may be true that at the scale of societies these are highly predictive of each other, but I'm not sure that they are really interchangeable.

In particular the coin toss society thought experiment wasn't convincing to me at all.

Comment by JBlack on [deleted post] 2021-05-05T01:38:19.807Z

Despite the form, statement (b) is not actually a logical conjunction. It is a statement about the collective of both parents.

This becomes clearer if we strengthen the statement slightly to "Alvin's mother and father are responsible for all of his genome". It's much more clear now that it is not a logical conjunction. If it were, it would mean "Alvin's mother is responsible for all of his genome and Alvin's father is responsible for all of his genome", both of which are false.