Posts
Comments
Building a paperclipper is low-value (from the point of view of total utilitarianism, or any other moral view that wants a big flourishing future) because paperclips are not sentient / are not conscious / are not moral patients / are not capable of flourishing. So filling the lightcone with paperclips is low-value. It maybe has some value for the sake of the paperclipper (if the paperclipper is a moral patient, or whatever the relevant category is) but way less than the future could have.
Your counter is that maybe building an aligned AI is also low-value (from the point of view of total utilitarianism, or any other moral view that wants a big flourishing future) because humans might not much care about having a big flourishing future, or might even actively prefer things like preserving nature.
If a total utilitarian (or someone who wants a big flourishing future in our lightcone) buys your counter, it seems like the appropriate response is: Oh no! It looks like we're heading towards a future that is many orders of magnitude worse than I hoped, whether or not we solve the alignment problem. Is there some way to get a big flourishing future? Maybe there's something else that we need to build into our AI designs, besides "alignment". (Perhaps mixed with some amount of: Hmmm, maybe I'm confused about morality. If AI-assisted humanity won't want to steer towards a big flourishing future then maybe I've been misguided in having that aim.)
Whereas this post seems to suggest the response of: Oh well, I guess it's a dice roll regardless of what sort of AI we build. Which is giving up awfully quickly, as if we had exhausted the design space for possible AIs and seen that there was no way to move forward with a large chance at a big flourishing future. This response also doesn't seem very quantitative - it goes very quickly from the idea that an aligned AI might not get a big flourishing future, to the view that alignment is "neutral" as if the chances of getting a big flourishing future were identically small under both options. But the obvious question for a total utilitarian who does wind up with just 2 options, each of which is a dice roll, is Which set of dice has better odds?
Is this calculation showing that, with a big causal graph, you'll get lots of very weak causal relationships between distant nodes that should have tiny but nonzero correlations? And realistic sample sizes won't be able to distinguish those relationships from zero.
Andrew Gelman often talks about how the null hypothesis (of a relationship of precisely zero) is usually false (for, e.g., most questions considered in social science research).
A lot of people have this sci-fi image, like something out of Deep Impact, Armageddon, Don't Look Up, or Minus, of a single large asteroid hurtling towards Earth to wreak massive destruction. Or even massive vengeance, as if it was a punishment for our sins.
But realistically, as the field of asteroid collection gradually advances, we're going to be facing many incoming asteroids which will interact with each other in complicated ways, and whose forces will to a large extent balance each other out.
Yet doomers are somehow supremely confident in how the future will go, foretelling catastrophe. And if you poke at their justifications, they won't offer precise physical models of these many-body interactions, just these mythic stories of Earth vs. a monolithic celestial body.
They're critical questions, but one of the secret-lore-of-rationality things is that a lot of people think criticism is bad, because if someone criticizes you, it hurts your reputation. But I think criticism is good, because if I write a bad blog post, and someone tells me it was bad, I can learn from that, and do better next time.
I read this as saying 'a common view is that being criticized is bad because it hurts your reputation, but as a person with some knowledge of the secret lore of rationality I believe that being criticized is good because you can learn from it.'
And he isn't making a claim about to what extent the existing LW/rationality community shares his view.
Seems like the main difference is that you're "counting up" with status and "counting down" with genetic fitness.
There's partial overlap between people's reproductive interests and their motivations, and you and others have emphasized places where there's a mismatch, but there are also (for example) plenty of people who plan their lives around having & raising kids.
There's partial overlap between status and people's motivations, and this post emphasizes places where they match up, but there are also (for example) plenty of people who put tons of effort into leveling up their videogame characters, or affiliating-at-a-distance with Taylor Swift or LeBron James, with minimal real-world benefit to themselves.
And it's easier to count up lots of things as status-related if you're using a vague concept of status which can encompass all sorts of status-related behaviors, including (e.g.) both status-seeking and status-affiliation. "Inclusive genetic fitness" is a nice precise concept so it can be clear when individuals fail to aim for it even when acting on adaptations that are directly involved in reproduction & raising offspring.
The economist RH Strotz introduced the term "precommitment" in his 1955-56 paper "Myopia and Inconsistency in Dynamic Utility Maximization".
Thomas Schelling started writing about similar topics in his 1956 paper "An essay on bargaining", using the term "commitment".
Both terms have been in use since then.
On one interpretation of the question: if you're hallucinating then you aren't in fact seeing ghosts, you're just imagining that you're seeing ghosts. The question isn't asking about those scenarios, it's only asking what you should believe in the scenarios where you really do see ghosts.
My updated list after some more work yesterday is
96286, 9344, 107278, 68204, 905, 23565, 8415, 62718, 83512, 16423, 42742, 94304
which I see is the same as simon's list, with very slight differences in the order
More on my process:
I initially modeled location just by a k nearest neighbors calculation, assuming that a site's location value equals the average residual of its k nearest neighbors (with location transformed to Cartesian coordinates). That, along with linear regression predicting log(Performance), got me my first list of answers. I figured that list was probably good enough to pass the challenge: the sites' predicted performance had a decent buffer over the required cutoff, the known sites with large predicted values did mostly have negative residuals but they were only about 1/3 the size of the buffer, there were some sites with large negative residuals but none among the sites with high predicted values and I probably even had a big enough buffer to withstand 1 of them sneaking in, and the nearest neighbors approach was likely to mainly err by giving overly middling values to sites near a sharp border (averaging across neighbors on both sides of the border) which would cause me to miss some good sites but not to include any bad sites. So it seemed fine to stop my work there.
Yesterday I went back and looked at the residuals and added some more handcrafted variables to my model to account for any visible patterns. The biggest was the sharp cutoff at Latitude +-36. I also changed my rescaling of Murphy's Constant (because my previous attempt had negative residuals for low Murphy values), added a quadratic term to my rescaling of Local Value of Pi (because the dropoff from 3.15 isn't linear), added a Shortitude cutoff at 45, and added a cos(Longitude-50) variable. Still kept the nearest neighbors calculation to account for any other location relevance (there is a little but much less now). That left me with 4 nines of correlation between predicted & actual performance, residuals near zero for the highest predicted sites in the training set, and this new list of sites. My previous lists of sites still seem good enough, but this one looks better.
Did a little robustness check, and I'm going to swap out 3 of these to make it:
96286, 23565, 68204, 905, 93762, 94408, 105880, 9344, 8415, 62718, 80395, 65607
To share some more:
I came across this puzzle via aphyer's post, and got inspired to give it a try.
Here is the fit I was able to get on the existing sites (Performance vs. Predicted Performance). Some notes on it:
Seems good enough to run with. None of the highest predicted existing sites had a large negative residual, and the highest predicted new sites give some buffer.
Three observations I made along the way.
First (which is mostly redundant with what aphyer wound up sharing in his second post):
Almost every variable is predictive of Performance on its own, but none of the continuous variables have a straightforward linear relationship with Performance.
Second:
Modeling the effect of location could be tricky. e.g., Imagine on Earth if Australia and Mexico were especially good places for Performance, or on a checkerboard if Performance was higher on the black squares.
Third:
The ZPPG Performance variable has a skewed distribution which does not look like what you'd get if you were adding a bunch of variables, but does look like something you might get if you were multiplying several variables. And multiplication seems plausible for this scenario, e.g. perhaps such-and-such a disturbance halves Performance and this other factor cuts performance by a quarter.
My current choices (in order of preference) are
96286, 23565, 68204, 905, 93762, 94408, 105880, 8415, 94304, 42742, 92778, 62718
What's "Time-Weighted Probability"? Is that just the average probability across the lifespan of the market? That's not a quantity which is supposed to be calibrated.
e.g., Imagine a simple market on a coin flip, where forecasts of p(heads) are made at two times: t1 before the flip and t2 after the flip is observed. In half of the cases, the market forecast is 50% at t1 and 100% at t2, for an average of 75%; in those cases the market always resolves True. The other half: 50% at t1, 0% at t2, avg of 25%, market resolves False. The market is underconfident if you take this average, but the market is perfectly calibrated at any specific time.
Have you looked at other ways of setting up the prior to see if this result still holds? I'm worried that they way you've set up the prior is not very natural, especially if (as it looks at first glance) the Stable scenario forces p(Heads) = 0.5 and the other scenarios force p(Heads|Heads) + p(Heads|Tails) = 1. Seems weird to exclude "this coin is Headsy" from the hypothesis space while including "This coin is Switchy".
Thinking about what seems most natural for setting up the prior: the simplest scenario is where flips are serially independent. You only need one number to characterize a hypothesis in that space, p(Heads). So you can have some prior on this hypothesis space (serial independent flips), and some prior on p(Heads) for hypotheses within this space. Presumably that prior should be centered at 0.5 and symmetric. There's some choice about how spread out vs. concentrated to make it, but if it just puts all the probability mass at 0.5 that seems too simple.
The next simplest hypothesis space is where there is serial dependence that only depends on the most recent flip. You need two numbers to characterize a hypothesis in this space, which could be p(Heads|Heads) and p(Heads|Tails). I guess it's simplest for those to be independent in your prior, so that (conditional on there being serial dependence), getting info about p(Heads|Heads) doesn't tell you anything about p(Heads|Tails). In other words, you can simplify this two dimensional joint distribution to two independent one-dimensional distributions. (Though in real-world scenarios my guess is that these are positively correlated, e.g. if I learned that p(Prius|Jeep) was high that would probably increase my estimate of p(Prius|Prius), even assuming that there is some serial dependence.) For simplicity you could just give these the same prior distribution as p(Heads) in the serial independence case.
I think that's a rich enough hypothesis space to run the numbers on. In this setup, Sticky hypotheses are those where p(Heads|Heads)>p(Heads|Tails), Switchy are the reverse, Headsy are where p(Heads|Heads)+p(Heads|Tails)>1, Tails are the reverse, and Stable are where p(Heads|Heads)=p(Heads|Tails) and get a bunch of extra weight in the prior because they're the only ones in the serial independent space of hypotheses.
Try memorizing their birthdates (including year).
That might be different enough from what you've previously tried to memorize (month & day) to not get caught in the tangle that has developed.
My answer to "If AI wipes out humanity and colonizes the universe itself, the future will go about as well as if humanity had survived (or better)" is pretty much defined by how the question is interpreted. It could swing pretty wildly, but the obvious interpretation seems ~tautologically bad.
Agreed, I can imagine very different ways of getting a number for that, even given probability distributions for how good the future will be conditional on each of the two scenarios.
A stylized example: say that the AI-only future has a 99% chance of being mediocre and a 1% chance of being great, and the human future has a 60% chance of being mediocre and a 40% chance of being great. Does that give an answer of 1% or 60% or something else?
I'm also not entirely clear on what scenario I should be imagining for the "humanity had survived (or better)" case.
The time on a clock is pretty close to being a denotative statement.
Batesian mimicry is optimized to be misleading, "I"ll get to it tomorrow" is denotatively false, "I did not have sexual relations with that woman" is ambiguous as to its conscious intent to be denotatively false.
Structure Rebel, Content Purist: people who disagree with me are lying (unless they say "I think that", "My view is", or similar)
Structure Rebel, Content Neutral: people who disagree with me are lying even when they say "I think that", "My view is", or similar
Structure Rebel, Content Rebel: trying to unlock the front door with my back door key is a lie
How do you get a geocentric model with ellipses? Venus clearly does not go in an ellipse around the Earth. Did Riccioli just add a bunch of epicycles to the ellipses?
Googling... oh, it was a Tychonic model, where Venus orbits the sun in an ellipse (in agreement with Kepler), but the sun orbits the Earth.
Kepler's ellipses wiped out the fully geocentric models where all the planets orbit around the Earth, because modeling their orbits around the Earth still required a bunch of epicycles and such, while modeling their orbits around the sun now involved a simple ellipse rather than just slightly fewer epicycles. But it didn't straightforwardly, on its own wipe out the geoheliocentric/Tychonic models where most planets orbit the sun but the sun orbits the Earth.
Here is Yudkowsky (2008) Artificial Intelligence as a Positive and
Negative Factor in Global Risk:
Friendly AI is not a module you can instantly invent at the exact moment when it is first needed, and then bolt on to an existing, polished design which is otherwise completely unchanged.
The field of AI has techniques, such as neural networks and evolutionary programming, which have grown in power with the slow tweaking of decades. But neural networks are opaque—the user has no idea how the neural net is making its decisions—and cannot easily be rendered unopaque; the people who invented and polished neural networks were not thinking about the long-term problems of Friendly AI. Evolutionary programming (EP) is stochastic, and does not precisely preserve the optimization target in the generated code; EP gives you code that does what you ask, most of the time, under the tested circumstances, but the code may also do something else on the side. EP is a powerful, still maturing technique that is intrinsically unsuited to the demands of Friendly AI. Friendly AI, as I have proposed it, requires repeated cycles of recursive self-improvement that precisely preserve a stable optimization target.
The most powerful current AI techniques, as they were developed and then polished and improved over time, have basic incompatibilities with the requirements of Friendly AI as I currently see them. The Y2K problem—which proved very expensive to fix, though not global-catastrophic—analogously arose from failing to foresee tomorrow’s design requirements. The nightmare scenario is that we find ourselves stuck with a catalog of mature, powerful, publicly available AI techniques which combine to yield non-Friendly AI, but which cannot be used to build Friendly AI without redoing the last three decades of AI work from scratch.
Also, chp 25 of HPMOR is from 2010 which is before CFAR.
There are a couple errors in your table of interpretations. For "actual score = subjective expected", the second half of the interpretation "prediction = 0.5 or prediction = true probability" got put on a new line in the "Comparison score" column instead of staying together in the "Interpretation" column, and similarly for the next one.
I posted a brainstorm of possible forecasting metrics a while back, which you might be interested in. It included one (which I called "Points Relative to Your Expectation") that involved comparing a forecaster's (Brier or other) score with the score that they'd expect to get based on their probability.
Sign error: "Tyler Cowen fires back that not only is this inevitable"
--> "not inevitable"
I don't understand why it took so long to seriously considered the possibility that orbits are ellipses.
It seems that a circle is the simplest, most natural, most elegant hypothesis for the shape of an orbit, and an ellipse is the second-most simple/natural/elegant hypothesis. But instead of checking if an ellipse fit the data, everyone settled for 'a lot like a circle, but you have to include a bunch of fudge factors to match the observational data'.
Apparently Kepler had a similar view. April 1605 is when he figured out that the orbit of Mars was an ellipse around the sun; two years earlier when he was already in the process of trying to figure out what sort of ovalish shape fit the data, he said that he had considered and rejected the ellipse hypothesis because if the answer was that simple then someone else would've figured it out already. This incorrect inadequacy analysis is from a July 1603 letter that Kepler wrote to David Fabricius: "I lack only a knowledge of the geometric generation of the oval or face-shaped curve. [...] If the figure were a perfect ellipse, then Archimedes and Apollonius would be enough."
I could make some guesses about why it didn't happen sooner (e.g. the fact that it happened right after Brahe collected his data suggests that poor data quality was a hindrance), but it feels pretty speculative. I wonder if there have been / could be more quantitative analyses of this question, e.g. do we have the data sets that the ancient Greeks used to fit their models, and can we see how well ellipses fit those data sets?
The suspects have since retreated to attempting to sue datacolada (the investigators).
It's just one of the suspects (Gino) who is suing, right?
I like the examples of quickly-resolved disagreements.
They don't seem that Aumannian to me. They are situations where one person's information is a subset of the other person's information, and they can be quickly resolved by having the less-informed person adopt the belief of the more-informed person. That's a special case of Aumann situations which is much easier to resolve in practice than the general case.
Aumann-in-general involves reaching agreement even between people who have different pieces of relevant information, where the shared posterior after the conversation is different from either person's belief at the start of the conversation. The challenge is: can they use all the information they have to reach the best possible belief, when that information is spread across multiple people? That's much less of a challenge when one person starts out with the full combined set of information.
(Note: my understanding of Aumann's theorem is mostly secondhand, from posts like this and this.)
I initially misread the title because "defer judgment" is often used to mean putting off the judgment until later. But here you meant "defer" as in taking on the other person's value judgment as your own (and were arguing against that), rather than as waiting to make the value judgment later (and arguing for that).
I guess "defer" is an autoantonym, in some contexts. When someone makes a claim which you aren’t in a good position to evaluate immediately, then to decide what you think about it you can either defer (to them) or defer (till later).
I don't understand what question you're trying to answer here. e.g., I could imagine questions of the form "If I see someone forecast 39.845%, should I treat this nearly the same as if they had forecast 40%?" Most mathematical ways of dealing with forecasts do (e.g. scoring rules, formulas for aggregating multiple people's forecasts, decision rules based on EV). But that doesn't seem like what you're interested in here.
Also not sure if/how these graphs differ from what they'd look like if they were based on forecasts which were perfectly calibrated with arbitrary precision.
Also, the description of Omar sounds impossible. If (e.g.) his 25% forecasts come true more often than his 5% forecasts, then his 15% forecasts must differ from at least one of those two (though perhaps you could have too little data to be able to tell).
Multiple related problems with Alice's behavior (if we treat this as a real conversation):
- Interfering with Bob's boundaries/autonomy, not respecting the basic background framework where he gets to choose what he does with his life/money/etc.
- Jumping to conclusions about Bob, e.g., insisting that the good he's been doing is just for "warm fuzzies", or that Bob is lying
- Repeatedly shifting her motive for being in the conversation / her claim about the purpose of the conversation (e.g., from trying to help Bob act on his values, to "if you’re lying then it’s my business", to what sorts of people should be accepted in the rationalist community)
- Cutting off conversational threads once Bob starts engaging with them to jump to new threads, in ways that are disorienting and let her stay on the attack, and don't leave Bob space to engage with the things that have already come up
These aren't merely impolite, they're bad things to do, especially when combined and repeated in rapid succession. It seems like an assault on Bob's ability to orient & think for himself about himself.
The spoiler tag only works for me if I type it, not if I copy-paste it.
In this scenario, Omega is described as predicting your actions and choosing how much money goes in the boxes, which is different from HPMOR where timelines settle into a self-consistent equilibrium in more complicated ways. And with this plan it should be straightforward for Omega to carry out its usual process, since you're not trying to generate a contradiction or force events through a weird narrow path, you're just entangling your box-opening decision with an observation about the world which you haven't made yet, but which omniscient Omega should know. For Omega, this is no different than you deciding that your second-box-opening decision will be contingent on the results of yesterday's Yankees game, which you haven't heard yet but which you will look up on your phone after you open the first box.
Extra credit
If you make your decision about whether to open the second box contingent on some fact about the future, then the contents of the first box let you predict that fact in advance. e.g., If the New York Yankees win their baseball game today then I will open the second box after their game, and if they do not win then I will not open the second box. Then if I open the first box and see $100, I can bet a lot of money on the Yankees to win, and if I see $0 then I can bet against the Yankees winning. If Omega is less patient then you'll need to pick something that resolves quicker.
Actively searching for counterexamples to the post's danger+safety theory... What about when people who are in love with each other get giggly around each other? (As an example of laughter, not humor.) I mean the sickly-sweet they're-in-their-own-world-together kind of giggling, not the nervous unsure-of-oneself laughter. Doesn't seem like there's danger.
Similarly, people laughing more when they're on drugs such as weed. (Though that seems more open to alternative physiological explanations.)
Do the stoners and giggly lovers fit the sketch of a theory that I'm maybe building towards? There is playing around with frames in both of them, I think. People on drugs see things fresh, or from a different perspective from usual; lovers-in-their-own-world have stepped out of the default social framework for interacting with the social world into their own little world. There is definitely syncing on frames with the lovers, though often not with the people on drugs. And there's an importance thing going on in the relationship, and a perception-of-importance thing with drug use. So mostly fits, except with an exception on the syncing bit.
I think I'm much more of an incongruity theorist than this (though haven't read the literature). More specifically: Humor generally involves playing around with frames / scripts, often crashing different frames together in interesting ways. Laughter involves distancing yourself from a frame you're engaging with, so that you're kinda acting with that frame but not fully immersed in it.
This fits the playfighting evolutionary context, which is engaging largely within the "fight" frame while keeping some detachment from it.
There is also a thing about social syncing on a shared frame, when the frame isn't the obvious default (we both understand that this is not a real fight, just a play fight, hahaha). Which seems related to how "getting the joke" is such a central aspect of humor. And to how humor is involved in social bonding.
My experience of coming up with in-context jokes matches this theory much more than your theory. It generally starts by noticing some alternative interpretation or frame on something that just happened (e.g. something that someone said), then gets refined into a specific thing to say by some combination of trying to find the most interesting interplay / biggest tension between the two frames, and trying to best bring in the new frame, and fitting it all to the people around me and what they'll understand & like. I'm not trying to track the amount of danger or the amount of safety. (Though maybe sometimes I'm tracking something like the level of physiological arousal - at some point you swapped that in for "danger" and it does resonate more.)
For a concrete example to talk about, I looked through recent SMBC comics and picked out the one I found funniest - this one. It is a good example of crashing two frames together, the Disneyfied frame on wild animals interacting with humans and a more realistic one, brought together by a picture that can be seen from either viewpoint. The phrase "tickborne diseases" really makes the realistic frame pop. Though there's also definitely some danger vs safety stuff happening here, so it's not a counterexample to your theory.
This other recent SMBC is also funny to me, and also has the playing around with frames thing without any obvious danger. So maybe is a counterexample? Though not sure if trying to find counterexamples is an important exercise; your theory seems more like it's incomplete than like it's totally wrong.
Trying my own hand at theorizing about this danger/arousal/whatever thing... Seems like there's something about the content (at least one of the frames) being important to the person. So danger, sex, taboo things. Pulling on some sort of relevance / importance / salience system in the brain.
Rationalists: If you write your bottom line first, it doesn't matter what clever arguments you write above it, the conclusion is completely useless as evidence.
The classic take is that once you've written your bottom line, then any further clever arguments that you make up afterwards won't influence the entanglement between your conclusion and reality. So: "Your effectiveness as a rationalist is determined by whichever algorithm actually writes the bottom line of your thoughts."
That is not saying that "the conclusion is completely useless as evidence."
Robust Cooperation in the Prisoner's Dilemma
and other MIRIish writing under the tags to that post
I believe the extinction year question was asking for a median, not an expected value. In one place in the paper it is paraphrased as asking "by what year humanity is 50% likely to go extinct".
This year, the big NFL controversy is over a play that the Philadelphia Eagles were repeatedly successful with last season. When the Eagles offense only needed a short gain, they would have their quarterback run the ball with 1 or 2 of his teammates pushing him from behind like in a rugby scrum. Despite the arguments that this play is a loophole that is "not football", the NFL decided not to change the rules for the upcoming season, so we'll probably see a lot more of this play by other teams for at least 1 more year.
You may want to look into the crisis in personality psychology which was sparked by Walter Mischel's (1968) book "Personality and assessment". There were a lot of studies, and arguments between researchers, about questions like these.
Mischel's challenge: there are often low-seeming correlations between broad personality measures and specific behaviors.
Part of the response was that the correlations are much larger if you aggregate across many behaviors, e.g. instead of the correlation between an abstract rating of conscientiousness and how much a person engages in a single specific conscientious-related behavior, look at the correlation between an abstract rating of conscientiousness and the average of how much a person engages in 50 specific conscientiousness-related behaviors. Which suggests that there is some sort of broad trend that the person carries, even if any one behavior depends on a mix of things beyond that broad trend.
Mischel argued for also looking for narrower patterns that are more stable for a person rather than just these broad traits, e.g. a person might pretty consistently be talkative with their friends, even if they don't consistently engage in some other extraversion-related behaviors.
Interesting that this essay gives both a 0.4% probability of transformative AI by 2043, and a 60% probability of transformative AI by 2043, for slightly different definitions of "transformative AI by 2043". One of these is higher than the highest probability given by anyone on the Open Phil panel (~45%) and the other is significantly lower than the lowest panel member probability (~10%). I guess that emphasizes the importance of being clear about what outcome we're predicting / what outcomes we care about trying to predict.
The 60% is for "We invent algorithms for transformative AGI", which I guess means that we have the tech that can be trained to do pretty much any job. And the 0.4% is the probability for the whole conjunction, which sounds like it's for pervasively implemented transformative AI: AI systems have been trained to do pretty much any job, and the infrastructure has been built (chips, robots, power) for them to be doing all of those jobs at a fairly low cost.
It's unclear why the 0.4% number is the headline here. What's the question here, or the thing that we care about, such that this is the outcome that we're making forecasts for? e.g., I think that many paths to extinction don't route through this scenario. IIRC Eliezer has written that it's possible that AI could kill everyone before we have widespread self-driving cars. And other sorts of massive transformation don't depend on having all the infrastructure in place so that AIs/robots can be working as loggers, nurses, upholsterers, etc.
Is the TikTok graph just showing that it takes time for viral videos to get their views, so many of the recent videos that will eventually get to 10MM views haven't gotten there yet?
Contrast with Zvi's The One Mistake Rule.
It seems there is a thing called ‘exercise non-responders’ who have very low returns to exercise, both in terms of burning off calories and in terms of building muscle.
The research I've seen on exercise non-responders has been pretty preliminary. A standard study gives a bunch of people an exercise routine, and measures some fitness-related variable before and after. The amount of improvement on that fitness-related variable has some distribution, obviously, because there are various sources of noise, context, individuals differences between people. You can look at the left tail of that distribution - the people who had little or no or negative improvement on this fitness measurement over the course of the study - and say that they were "non-responders" to this exercise intervention.
One hypothesis for why that happened is that there is some stable & general individual difference, something about their physiology, which makes it so that they wouldn't respond much to pretty much any exercise routine that they tried.
In order to see if that's what's going on, rather than the various other things which could cause this measure of a person's fitness to fluctuate, or attempts at this exercise routine to have more or less of an effect, you'd have to do some more research to understand what's going on with these people in more detail. But for the most part that additional research doesn't get done, people just talk about the people whose fitness measurement didn't change much from this intervention as if being a non-responder or a low-responder is a stable trait of theirs. That's the case with the first study cited in this twitter discussion.
In the replies to that tweet, someone links to one paper which did continue to study the non-responders to an exercise intervention, just trying more dakka (more training sessions per week), and found that every person ended up with some improvement.
What do you mean by 'average' I can think of at least four possibilities.
The standard of fairness "both players expect to make the same profit on average" implies that the average to use here is the arithmetic mean of the two probabilities, (p+q)/2. That's what gives each person the same expected value according to their own beliefs.
This is easy to verify in the example in the post. Stakes of 13.28 and 2.72 involve a betting probability of 83%, which is halfway between the 99% and 67% that the two characters had.
You can also do a little arithmetic from the equal expected values to derive that this is how it works in general: the ratio of the amounts at stake should be (p+q):(2-p-q), which is the ratio mean(p,q):[1-mean(p,q)]. Using the arithmetic mean to set the betting odds gives both parties the same subjective expected value.
If there's a simple intuitive sketch of why this has to be true, I don't have it.
The extra work involved in the method in the post is to make it incentive-compatible for each person to state their true probability by allowing the stakes to vary.
So: To make the bet fair (equal EV), the betting odds should be based on the arithmetic mean of the probabilities. If you need to make it strategy-proof, then you can use the algorithm in this post to decide how much money to bet at those odds.
I was recently looking at Yudkowsky's (2008) "Artificial Intelligence as a Positive and
Negative Factor in Global Risk" and came across this passage which seems relevant here:
Friendly AI is not a module you can instantly invent at the exact moment when it is first needed, and then bolt on to an existing, polished design which is otherwise completely unchanged.
The field of AI has techniques, such as neural networks and evolutionary programming, which have grown in power with the slow tweaking of decades. But neural networks are opaque—the user has no idea how the neural net is making its decisions—and cannot easily be rendered unopaque; the people who invented and polished neural networks were not thinking about the long-term problems of Friendly AI. Evolutionary programming (EP) is stochastic, and does not precisely preserve the optimization target in the generated code; EP gives you code that does what you ask, most of the time, under the tested circumstances, but the code may also do something else on the side. EP is a powerful, still maturing technique that is intrinsically unsuited to the demands of Friendly AI. Friendly AI, as I have proposed it, requires repeated cycles of recursive self-improvement that precisely preserve a stable optimization target.
The most powerful current AI techniques, as they were developed and then polished and improved over time, have basic incompatibilities with the requirements of Friendly AI as I currently see them. The Y2K problem—which proved very expensive to fix, though not global-catastrophic—analogously arose from failing to foresee tomorrow’s design requirements. The nightmare scenario is that we find ourselves stuck with a catalog of mature, powerful, publicly available AI techniques which combine to yield non-Friendly AI, but which cannot be used to build Friendly AI without redoing the last three decades of AI work from scratch.
Notice that this was over a 25% response rate
15% (1162/7704)
Only like 10% are perfect scores. Median of 1490 on each of the two old LW surveys I just checked.
See also the heuristics & biases work on framing effects, e.g. Tversky and Kahneman's Rational Choice and the Framing of Decisions
Alternative descriptions of a decision problem often give rise to different preferences, contrary to the principle of invariance that underlines the rational theory of choice. Violations of this theory are traced to the rules that govern the framing of decision and to the psychological principles of evaluation embodied in prospect theory. Invariance and dominance are obeyed when their application is transparent and often violated in other situations. Because these rules are normatively essential but descriptively invalid, no theory of choice can be both normatively adequate and descriptively accurate.
A hypothesis for the negative correlation:
More intelligent agents have a larger set of possible courses of action that they're potentially capable of evaluating and carrying out. But picking an option from a larger set is harder than picking an option from a smaller set. So max performance grows faster than typical performance as intelligence increases, and errors look more like 'disarray' than like 'just not being capable of that'. e.g. Compare a human who left the window open while running the heater on a cold day, with a thermostat that left the window open while running the heater.
A Second Hypothesis: Higher intelligence often involves increasing generality - having a larger set of goals, operating across a wider range of environments. But that increased generality makes the agent less predictable by an observer who is modeling the agent as using means-ends reasoning, because the agent is not just relying on a small number of means-ends paths in the way that a narrower agent would. This makes the agent seem less coherent in a sense, but that is not because the agent is less goal-directed (indeed, it might be more goal-directed and less of a stimulus-response machine).
These seem very relevant for comparing very different agents: comparisons across classes, or of different species, or perhaps for comparing different AI models. Less clear that they would apply for comparing different humans, or different organizations.