Posts

What if "friendly/unfriendly" GAI isn't a thing? 2022-04-07T16:54:05.735Z
Study: In giving charity, let not your right hand... 2014-08-22T22:23:34.215Z
Democracy and rationality 2013-10-30T12:07:24.727Z
Does quantum mechanics make simulations negligible? 2011-08-13T01:53:17.622Z
Overcoming bias in others 2011-08-12T15:38:32.282Z

Comments

Comment by homunq on How BoMAI Might fail · 2022-04-09T01:52:48.075Z · LW · GW

Is "do whatever action you predict to maximize the electricity in this particular piece of wire" really "general"? You're basically claiming that the more intelligent someone is, the more likely they are to wirehead. With humans, in my experience, and for a loose definition of "wirehead", the pattern seems to be the opposite; and that seems to me to be solid enough in terms of how RL works that I doubt it's worth the work to dig deep enough to resolve our disagreement here.

Comment by homunq on I discovered LessWrong... during Good Heart Week · 2022-04-08T19:18:47.877Z · LW · GW

I've posted on LW before, but I posted again here after a long hiatus because of recent AI news, and entirely unaware of the good heart thing; then made several comments after reading the original post, but thinking it was a joke. Now I understand why the site was so strangely active.

Comment by homunq on Playing with DALL·E 2 · 2022-04-08T19:10:34.123Z · LW · GW

"An animal looking curiously in the mirror, but the reflection is a different kind of animal; in digital style."

"A cat looking curiously in the mirror, but the reflection is a different kind of animal; in digital style."

"A cat looking curiously in the mirror, but the reflection is a dog; in digital style."

Curious to see how it handles modified-reflection and lack-of-specificity.

Comment by homunq on Why Agent Foundations? An Overly Abstract Explanation · 2022-04-08T01:20:20.777Z · LW · GW

Another thing whose True Name is probably a key ingredient for alignment (and which I've spent a lot of time trying to think rigorously about): collective values.

Which is interesting, because most of what we know so far about collective values is that, for naive definitions of "collective" and "values", they don't exist. Condorcet, Arrow, Gibbard and Satterthwaite, and (crucially) Sen have all helped show that.

I personally don't think that means that the only useful things one can say about "collective values" are negative results like the ones above. I think there are positive things to say; definitions of collectivity (for instance, of democracy) that are both non-trivial and robust. But finding them means abandoning the naive concepts of "collective values".

I think that this is probably a common pattern. You go looking for the True Name of X, but even if that search ever bears fruit, you'd rarely if ever look back and say "Y is the True Name of X". Instead, you'd say something like "(long math notation) is the True Name of itself, or for short, of Y. Though I found this by looking for X, calling it 'X' was actually a misnomer; that phrase has baked-in misconceptions and/or red herrings, so from now on, let's call it 'Y' instead."

Comment by homunq on What an actually pessimistic containment strategy looks like · 2022-04-07T23:35:01.287Z · LW · GW

I think this post makes sense given the premises/arguments that I think many people here accept: that AG(S)I is either amazingly good or amazingly bad, and that getting the good outcome is a priori vastly improbable, and that the work needed to close the gap between that prior and a good posterior is not being done nearly fast enough.

I don't reject those premises/arguments out of hand, but I definitely don't think they're nearly as solid as I think many here do. In my opinion, the variance in goodness of reasonably-thinkable post-AGSI futures is mind-bogglingly large, but it's still probably a bell curve, with greater probability density in the "middle" than in super-heaven or ultra-hell. I also think that just making the world a better place here and now probably usually helps with alignment.

This is probably not the place for debating these premises/arguments; they're the background of this post, not its point. But I do want to say that having a different view on that background is (at least potentially) a valid reason for not buying into the "containment" strategy suggested here.

Again, I think my point here is worthwhile to mention as one part of the answer to the post's question "why don't more people think in terms of containment". I don't think that we're going to resolve whether there's space in between "friendly" and "unfriendly" right here, though.

Comment by homunq on What if "friendly/unfriendly" GAI isn't a thing? · 2022-04-07T22:54:03.890Z · LW · GW

Sure, humans are effectively ruthless in wiping out individual ant colonies. We've even wiped out more than a few entire species of ant. But our ruthfulness about our ultimate goals — well, I guess it's not exactly ruthfulness that I'm talking about...

...The fact that it's not in our nature to simply define an easy-to-evaluate utility function and then optimize, means that it's not mere coincidence that we don't want anything radical enough to imply the elimination of all ant-kind. In fact, I'm pretty sure that for a large majority of people, there's no utopian ideal you could pitch and they'd buy into, that's so radical enough that getting there would imply or even suggest actions that would kill all ants. Not because humanity wouldn't be capable of doing that, just that we're not capable of wanting that, and that fact may be related to our (residual) ruthfulness and to our intelligence itself.  And metaphorically, from a superintelligence's perspective, I think that humanity-as-a-whole is probably closer to being Formicidae than it is to being one species of ant.

...

This post, and its line of argument, is not about saying "AI alignment doesn't matter". Of fucking course it does. What I'm saying is: "it may not be the case that any tiny misalignment of a superintelligence is fatal/permanent". Because yes, a superintelligence can and probably will change the world to suit its goals, but it won't ruthlessly change the whole world to perfectly suit its goals, because those goals will not, themselves, be perfectly coherent. And in that gap, I believe there will probably still be room for some amount of humanity or posthumanity-that's-still-commensurate-with-extrapolated-human-values having some amount of say in their own fates.

The response I'm looking for is not at all "well, that's all OK then, we can stop worrying about alignment". Because there's a huge difference between future (post)humans living meagerly under sufferance in some tiny remnant of the world that a superintelligence doesn't happen to care about coherently enough to change, or them thriving as an integral part of the future that it does care about and is building, or some other possibility better or worse than those. But what I am arguing is that I think the "win big or lose big are the only options" attitude I see as common in alignment circles (I know that Eleizer isn't really cutting edge anymore, but, look at his recent April Fools' "joke" for an example) may be misguided. Not every superintelligence that isn't perfectly friendly is terrifyingly unfriendly, and I think that admitting other possibilities (without being complacent about them) might help useful progress in pursuing alignment.

...

As for your points about therapy: yes, of course, my off-the-cuff one-paragraph just-so-story was oversimplified. And yes, you seem to know a lot more about this than I do. But I'm not sure the metaphor is strong enough to make all that complexity matter here. 

Comment by homunq on What if "friendly/unfriendly" GAI isn't a thing? · 2022-04-07T18:21:08.424Z · LW · GW

I guess we're using different definitions of "friendly/unfriendly" here. I mean something like "ruthlessly friendly/unfriendly" in the sense that humans (neurotic as they are) aren't. (Yes, some humans appear ruthless, but that's just because their "ruths" happen not to apply. They're still not effectively optimizing for future world-states, only for present feels.)

I think many of the arguments about friendly/unfriendly AI, at least in the earlier stages of that idea (I'm not up on all the latest) are implicitly relying on that "ruthless" definition of (un)friendliness.  

You (if I understand) mean "friendly/unfriendly" in a weaker sense, in which humans can be said to be friendly/unfriendly (or neither? Not sure what you'd say about that, but it probably doesn't matter.)

As for the "smart people going to dumb therapists" argument, I think you're going back to a hidden assumption of ruthlessness: if the person knew how to feel better in the future, they would just do that. But what if, for instance, they know how to feel better in the future, but doing that thing wouldn't make them feel better right now unless they first simplify it enough to explain it to their dumb therapist? The dumb therapist is still playing a role.

My point is NOT to say that non-ruthless GASI isn't dangerous. My point is that it's not an automatic "game over" because if it's not ruthless it doesn't just institute its (un)friendly goals; it is at least possible that it would not use all its potential power. 

Comment by homunq on How BoMAI Might fail · 2022-04-07T17:57:59.803Z · LW · GW

Why does the AI even "want" failure mode 3? If it's a RL agent, it's not "motivated to maximize its reward", it's "motivated to use generalized cognitive patterns that in its training runs would have marginally maximized its reward". Failure mode 3 is the peak of an entirely separate mountain than the one RL is climbing, and I think a well-designed box setup can (more-or-less "provably") prevent any cross-peak bridges in the form of cognitive strategies that undermine this. 

That is to say: yes, it can (or at least, it it's not provable that it can't) imagine a way to break the box, and it can know that the reward it would actually get from breaking the box would be "infinite", but it can be successfully prevented from "feeling" the infinite-ness of that potential reward, because the RL procedure itself doesn't consider a broken-box outcome to be a valid target of cognitive optimization.

Now, this creates a new failure mode, where it hacks its own RL optimizer. But that just makes it unfit, not dangerous. Insofar as something goes wrong to let this happen, it would be obvious and easy to deal with, because it would be optimizing for thinking it would succeed and not for succeeding.

(Of course, that last sentence could also fail. But at least that would require two simultaneous failures to become dangerous; and it seems in principle possible to create sufficient safeguards and warning lights around each of those separately, because the AI itself isn't subverting those safeguards unless they've already failed.)

Comment by homunq on LessWrong 2.0 · 2015-12-11T14:47:26.542Z · LW · GW

One way of dividing up the options is: fix the current platform, or find new platform(s). The natural decay process seems to be tilting towards the latter, but there are downsides: the diaspora loses cohesion, and while the new platforms obviously offer some things the current one doesn't, they are worse than the current one in various ways (it's really hard to be an occasional lurker on FB or tumblr, especially if you are more interested in the discussion than the "OP").

If the consensus is to fix the current platform, I suggest trying the simple fixes first. As far as I can tell, that means, break the discussion/main dichotomy, and do something about "deletionist" downvoting. Also, making it clearer how to contribute to the codebase, with a clearer owner. I think that these things should be tried and given a chance to work before more radical stuff is attempted.

If the consensus is to find something new, I suggest that it should be something which has a corporation behind it. Something smallish but on the up-and-up, and willing to give enough "tagging" capability for the community to curate itself and maintain itself reasonably separate from the main body of users of the site. It should be something smaller than FB but something willing to take the requests of the community seriously. Reddit, Quora, StackExchange, Medium... this kind of thing, though I can see problems with each of those specific suggestions.

Comment by homunq on Taking Effective Altruism Seriously · 2015-07-14T00:52:05.873Z · LW · GW

I disagree. I think the issue is whether "pro-liberty" is the best descriptive term in this context. Does it point to the key difference between things it describes and things it doesn't? Does it avoid unnecessary and controversial leaps of abstraction? Are there no other terms which all discussants would recognize as valid, if not ideal? No, no, and no.

Comment by homunq on Taking Effective Altruism Seriously · 2015-07-12T23:08:06.629Z · LW · GW

Whether something is a defensible position, and whether it should be embedded in the very terms you use when more-neutral terms are available, are separate questions.

If you say "I'm pro-liberty", and somebody else says "no you're not, and I think we could have a better discussion if you used more specific terms", you don't get to say "why won't you accept me at face value".

Comment by homunq on Taking Effective Altruism Seriously · 2015-07-12T23:01:55.933Z · LW · GW

When you say "Nothing short of X can get you to Y", the strong implication is that it's a safe bet that X will at least not move you away from Y, and sometimes move you toward it. So OK, I'll rephrase:

The OP suggests that colonization is in fact a proven way to turn at least some poor countries into more productive ones.

Comment by homunq on Taking Effective Altruism Seriously · 2015-06-15T20:57:35.994Z · LW · GW

Note that my post just above was basically an off-the-cuff response to what I felt was a ludicrously wrong assumption buried in the OP. I'm not an expert on African history, and I could be wrong. I think that I gave the OP's idea about the level of refutation it deserved, but I should have qualified my statements more ("I'd guess..."), so I certainly didn't deserve 5 upvotes for this (5 points currently; I deserve 1-3 at most).

Comment by homunq on Taking Effective Altruism Seriously · 2015-06-06T15:50:17.224Z · LW · GW

I think that it's worth being more explicit in your critique here.

The OP suggests that colonization is in fact a proven way to turn poor countries into productive ones. But in fact, it does the opposite. Several parts of Africa were at or above average productivity before colonization¹, and well below after; and this pattern has happened at varied enough places and times to be considered a general rule. The examples of successful transitions from poor countries to rich ones—such as South Korea—do not involve colonization.

¹Note that I'm considering the triangular trade as a form of colonization; even if it didn't involve proconsuls, it involved an external actor explicitly fomenting a hierarchical and extractive social order.

Comment by homunq on Taking Effective Altruism Seriously · 2015-06-06T15:43:09.527Z · LW · GW

I think you can make this critique more pointed. That is: "pro-liberty" is flag-waving rhetoric which makes us all stupider.

I dislike the "politics is a mind-killer" idea if it means we can't talk about politically touchy subjects. But I entirely agree with it if it means that we should be careful to keep our language as concrete and precise as possible when we approach these subjects. I could write several paragraphs about all the ways that the term "pro-liberty" takes us in the wrong direction, but I expect that most of you can figure all that out for yourselves.

Comment by homunq on Announcing the Complice Less Wrong Study Hall · 2015-03-04T03:05:36.174Z · LW · GW

It appears that you need to be logged in from FB or twitter to be fully non-guest. That seems like a... strange... choice for an anti-akrasia tool.

(Tangentially related to above, not really a reply)

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T18:22:53.325Z · LW · GW

Fair enough. Thanks. Again, I agree with some of your points. I like blemish-picking as long as it doesn't require open-ended back-and-forth.

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T17:53:47.622Z · LW · GW

You're raising some valid questions, but I can't respond to all of them. Or rather, I could respond (granting some of your arguments, refining some, and disputing some), but I don't know if it's worth it. Do you have an underlying point to make, or are you just looking for quibbles? If it's the latter, I still thank you for responding (it's always gratifying to see people care about issues that I think are important, even if they disagree); but I think I'll disengage, because I expect that whatever response I give would have its own blemishes for you to find.

In other words: OK, so what?

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T17:10:32.685Z · LW · GW

Full direct democracy is a bad idea because it's incredibly inefficient (and thus also boring/annoying, and also subject to manipulation by people willing to exploit others' boredom/annoyance). This has little or nothing to do with whether people's preferences correlate with their utilities, which is the question I was focused on. In essence, this isn't a true Goldilocks situation ("you want just the right amount of heat") but rather a simple tradeoff ("you want good decisions, but don't want to spend all your time making them").

As to the other related concepts... I think this is getting a bit off-topic. The question is, is energy (money) spent on pursuing better voting systems more of a valid "saving throw" than when spent on pursuing better individual rationality. That's connected to the question of the preference/utility correlation of current-day, imperfectly-rational voters. I'm not seeing the connection to rule of law &c.

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T16:48:34.215Z · LW · GW

(small note: the sentence you quote from me was unclear. "because" related to "presume", not "saying". But your response to what I accidentally said is still largely cogent in relation to what I meant to say, so the miscommunication isn't important. Still, I've corrected the original. Future readers: lumifer quoted me correctly.)

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T16:44:51.362Z · LW · GW

The model is not easy to subject to full, end-to-end testing. It seems reasonable to test it one part at a time. I'm doing the best I can to do so:

  • I've run an experiment on Amazon Mechanical Turk involving hundreds of experimental subjects voting in dozens of simulated elections to probe my strategy model.

  • I'm working on getting survey data and developing statistical tools to refine my statistical model (mostly, posterior predictive checks; but it's not easy, given that this is a deeper hierarchical model than most).

  • In terms of the utilitarian assumptions of my model, I'm not sure how those are testable rather than just philosophical / assumed axioms. Not that I regard these assumptions as truly axiomatic, but that I think they're pretty necessary to get anywhere at all, and in practice unlikely to be violated severely enough to invalidate the work.

  • I haven't started work on testing / refining my media model (other than some head-scratching), but I can imagine how to do at least a few spot checks with posterior predictive checks too.

  • The assumptions that preference and utility correlate positively, even in an environment where candidates are strategic about exploiting voter irrationality, are certainly questionable. But insofar as these are violated, it would just make democracy a bad idea in general, not invalidate the fact that plurality is still a worse idea than other voting systems such as approval. Also, I think it would be basically impossible to test these assumptions without implausibly accurate and unbiased measurements of true utility. Finally, call me a hopeless optimist, but I do actually have faith that democracy is a good idea because "you can't fool all the people all the time".

tl;dr: I'm working on this.

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-24T12:09:27.833Z · LW · GW

I presume you're saying that utility-based simulations are not credible. I don't think you're actually trying to say that they're not numerical estimates. So let me explain what I'm talking about, then say what parts I'm claiming are "credible".

I'm talking about monte-carlo simulations of voter satisfaction efficiency. You use some statistical model to generate thousands of electorates (that is, voters with numeric utilities for candidates); a media model to give the voters information about each other; and a strategy model to turn information, utilities, and choice of voting system into valid ballots for that voting system. Then, you see who wins each time, and calculate the average overall utility of that winners. Clearly, there are a lot of questionable assumptions in terms of the statistical, media, and strategy models, but the interesting thing is that exploring various assumptions in all of those cases shows that the (plurality-dictatorship)≈(good system-plurality) equation is pretty robust, with various systems such as approval, condorcet, majority judgment, score, or SODA in place of "good system".

There are certainly various ways to criticize the above.

  • "Don't believe it": If you think that I've messed up my math or not done a good job with the sensitivity analysis, of course you'd question my conclusions. But if you want to play with my code to check it, it's here.

  • "Utilitarianism is a bad metric": It may not be perfect, but as far as I can tell it's the only rational way to put numbers on things.

  • "Democracy is a bad idea": In other words, if you think that the average voter's estimate of their utility for a candidate has 0 or negative correlation with their true utility of that candidate winning, then this simulation is garbage. I'd respond with the old saying about democracy being the worst system except all the others.

  • "The advantages of democracy over dictatorship aren't in terms of who's in charge": if you think that democracy's clear superiority to dictatorship in terms of human welfare comes from something other than choosing better leaders (such as, for instance, reducing the prevalence of civil wars), then improving the voting system might not be able to have comparable payoff as instituting a voting system to begin with. I'd respond that this critique is probably partially right, but on the other hand, better leadership could credibly have better responses to crises (financial, environmental, and/or existential-risk) which could indeed be on the same order as the democracy dividend.

All in all, taking a more outside view, I see how the combination of the above objections would reduce your estimate of the expected "voting system dividend". Still, when I "shut up and multiply" I get: $80 trillion world GDP plausible (conservative) effect size in a good year of 2% .1 plausible portion of good years over time .5 plausible portion of good years over space (some country's economies might already be immune to the kind of harm this could prevent) .5 chance you trust my simulations .1 correlation of voter preference with utility .5 probability leadership makes any difference = about $2 billion/year potential payoff in expected value, even without compounding. That seems to me like (a) quite a conservative choice of factors, (b) not a totally implausible end result, and (c) still big enough to care about. Of course, it's incredibly back-of-the-envelope, but I invite you to try doing the estimation yourself.

Comment by homunq on 2014 Survey Results · 2015-02-23T01:53:00.725Z · LW · GW

[ ] Wow, these people are smart. [ ] Wow, these people are dumb. [ ] Wow, these people are freaky. [ ] That's a good way of putting it, I'll remember that.

(For me, it's all of the above. "Insight porn" is probably the biggest, but it doesn't dominate.)

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-23T00:27:04.868Z · LW · GW

Electology is an organization dedicated to improving collective decision making — that is, voting. We run on a shoestring; somewhere in the lowish 5 digits $ per year. We've helped get organizations such as the German Pirate Party and the various US stat Libertarian Parties to use approval voting, and gotten bills brought up in several states (no major victories so far, but we're just starting.)

Is a better voting system worth it, even if most people still vote irrationally? I'd say emphatically yes. Plurality voting is just a disaster as a system, filled with pathological results, perverse incentives, and pernicious equilibria. Credible numerical estimates (utility-based simulations) suggest that better systems such as approval voting offer as much improvement again as the move from dictatorship to democracy was.

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-23T00:07:34.173Z · LW · GW

In terms of “saving throws” one can buy for a humanity that may be navigating tricky situations in an unknown future, improvements to thinking skill seem to be one of the strongest and most robust.

Improvements to collective decision making seem to be potentially an even bigger win. I mean, voting reform; the kind of thing advocated by Electology. Disclaimer: I'm a board member.

Why do I think that? Individual human decisionmaking has already been optimized by evolution. Sure, that optimization doesn't fit perfectly with a modern need for rationality, but it's pretty darn good. However, democratic decisionmaking is basically still using the first system that anybody ever thought of, and monte carlo utility simulations show that we can probably make it at least twice as good (using a random dictator as a baseline).

On the other hand, achieving voting reform requires a critical mass, while individual rationality only requires individuals. And electology is not as far along in organizational growth as CFAR. But it seems to me that it's a complementary idea, and that it would be reasonable for an effective altruist to diversify their "saving throw" contributions. (We would also welcome rationalist board members or volunteers.)

Comment by homunq on CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype · 2015-02-22T23:53:47.191Z · LW · GW

One idea for measurement in a randomized trial:

In order to apply, you have to list 4 people who would definitely know how awesome you're being a year from now, and give their contact info. Then, choose 1 of those people 6 months later and 1 person a year later and ask them how awesome the person is being. When you ask, include a "rubric" of various stories of various awesomeness levels, in which the highest levels are not always just $$$ but sometimes are. Ask the people you're asking to please not contact the person specifically to check awesomeness, because that could introduce bias ("this person is checking, that makes me remember the workshop I did, and feel awesome").

The 4 people should probably include no couples. Your family, long-term friends...

The one way this breaks down is facebook. I mean, if your interaction with each person is separate, and the workshop makes you seem more awesome to each of 4 people, it is working. But if it just makes you post more upbeat things on Facebook, that might not translate to actual awesomeness. But I think that's a really minor factor.

Sure, it's gonna be a noisy and imperfect measurement. You will have to look at standard deviations and calculate power (including burning all 4 contacts for some people to see the within-subject variance). Also, correct for demographic info on contacts, and various other tricks to increase power. But one way or another, you'll get a posterior distribution of the causal impact.

Comment by homunq on The Importance of Sidekicks · 2015-02-22T14:30:14.131Z · LW · GW

I think you've misunderstood the question. As I understand it, it's not "is the distribution of startup values a power law" but "do startups distribute their profits to employees according to a power law".

Comment by homunq on The Importance of Sidekicks · 2015-02-22T12:55:00.155Z · LW · GW

Wish I could both up- and down- vote this comment. +1 for interesting, cogent observation; -1 for followinng that up with facile beakering. So instead I upvoted this comment and downvoted your reply below ( which deserves the downvote in its own right)

(I just made up the word "beakering". It means doing TV science, with beakers and bafflegab, in real life. A lot of amateur evo-something and neuro-something involve beakering.)

Comment by homunq on Improving The Akrasia Hypothesis · 2015-02-15T20:22:28.328Z · LW · GW

Would be better if you didn't say whom you ended up agreeing with. Most people here have either a halo or horns on Eliezer, and discounting that is distracting.

Comment by homunq on Roles are Martial Arts for Agency · 2014-08-22T16:17:13.787Z · LW · GW

That's simpler to say, but not at all simpler to do.

Comment by homunq on Why the tails come apart · 2014-08-22T16:04:38.881Z · LW · GW

Bump.

(I realize you're busy, this is just a friendly reminder.)

Also, I added one clause to my comment above: the bit about "imperfectly measured", which is of course usually the case in the real world.

Comment by homunq on Why the tails come apart · 2014-08-02T17:58:39.590Z · LW · GW

Great article overall. Regression to the mean is a key fact of statistics, and far too few people incorporate it into their intuition.

But there's a key misunderstanding in the second-to-last graph (the one with the drawn-in blue and red "outcome" and "factor"). The black line, indicating a correlation of 1, corresponds to nothing in reality. The true correlation is the line from the vertical tangent point at the right (marked) to the vertical tangent point at the left (unmarked). If causality indeed runs from "factor" (height) to "outcome" (skill), that's how much extra skill an extra helping of height will give you. Thus, the diagonal red line should follow this direction, not be parallel to the 45 degree black line. If you draw this line, you'll notice that each point on it has equal vertical distance to the top and bottom of the elliptical "envelope" (which is, of course, not a true envelope for all the probability mass, just an indication that probability density is higher for any point inside than any point outside).

Things are a little more complex if the correlation is due to a mutual cause, "reverse" causation (from "outcome" to "factor"), or if "factor" is imperfectly measured. In that case, the line connecting the vertical tangents may not correspond to anything in reality, though it's still what you should follow to get the "right" (minimum expected squared error) answer.

This may seem to be a nitpick, but to me, this kind of precision is key to getting your intuition right.

Comment by homunq on A critique of effective altruism · 2014-03-16T12:43:09.439Z · LW · GW

No argument here. It's hard to build a good social welfare function in theory (ie, even if you can assume away information limitations), and harder in practice (with people actively manipulating it). My point was that it is a mistake to think that Arrow showed it was impossible.

(Also: I appreciate the "thank you", but it would feel more sincere if it came with an upvote.)

Comment by homunq on A critique of effective altruism · 2013-12-21T19:29:23.346Z · LW · GW

I think you've done better than CarlShulman and V_V at expressing what I see as the most fundamental problem with EA: the fact that it is biased towards the easily- and short-term- measurable, while (it seems to me) the most effective interventions are often neither.

In other words: how do you avoid the pathologies of No Child Left Behind, where "reform" becomes synonymous with optimizing to a flawed (and ultimately, costly) metric?

This issue is touched by the original post, but not at all deeply.

Comment by homunq on A critique of effective altruism · 2013-12-21T19:23:09.761Z · LW · GW

Note: Arrow's Impossibility Theorem is not actually a serious philosophical hurdle for a utilitarian (though related issues such as the Gibbard-Satterthwaite theorem may be). That is to say: it is absolutely trivial to create a social utility function which meets all of Arrow's "impossible" criteria, if you simply allow cardinal instead of just ordinal utility. (Arrow's theorem is based on a restriction to ordinal cases.)

Comment by homunq on A critique of effective altruism · 2013-12-21T19:15:16.091Z · LW · GW

Upvoted because I think this is a real issue, though I'm far from sure whether I'd put it at "worst".

Comment by homunq on A critique of effective altruism · 2013-12-21T13:01:43.148Z · LW · GW

... And that is not a new idea either. "Allow me to play the devil's advocate for a moment" is a thing people say even when they are expressing support before and after that moment.

Comment by homunq on What Can We Learn About Human Psychology from Christian Apologetics? · 2013-11-03T11:08:30.744Z · LW · GW

Can anyone explain why the parent was downvoted? I don't get it. I hope there's a better reason than the formatting fail.

Comment by homunq on Democracy and rationality · 2013-10-31T19:58:39.249Z · LW · GW

This is a key question. The general answer is:

  1. For realistic cases, there is no such theorem, and so the task of choosing a good system is a lot about choosing one which doesn't reward strategy in realistic cases.

  2. Roughly speaking, my educated intuition is that strategic payoffs grow insofar as you know that the distinctions you care about are orthogonal to what the average/modal/median voter cares about. So insofar as you are average/modal/median, your strategic incentive should be low; which is a way of saying that a good voting system can have low strategy for most voters in most elections.

2a. It may be possible to make this intuition rigorous, and prove that no system can make strategy non-viable for the orthogonal-preferenced voter. However, that would involve a lot of statistics and random variables.... I guess that's what I'm learning in my PhD so eventually I may be up to taking on this proof.

  1. The exception, the realistic case where there are a number of voters who have an interest that's orthogonal to the average voter, is a case called the chicken dilemma, which I'll talk about a lot more in section 6. Chicken strategy is by far the trickiest realistic strategy to design away.
Comment by homunq on Democracy and rationality · 2013-10-31T19:52:54.962Z · LW · GW

Yup. That's what people say. I don't know what the general rule is, but it's definitely right for this case.

Comment by homunq on Democracy and rationality · 2013-10-31T02:23:08.953Z · LW · GW

I, too, hope that our disagreement will soon disappear. But as far as I can see, it's clearly not a semantic disagreement; one of us is just wrong. I'd say it's you.

So. Say there are 3 voters, and without loss of generality, voter 1 prefers A>B>C. Now, for every one of the 21 distinct combinations for the other two, you have to write down who wins, and I will find either an (a priori, determinative; not mirror) dictator or a non-IIA scenario.

ABC ABC: A

ABC ACB: A

ABC BAC: ?... you fill in these here

ABC BCA: ?

ABC CAB: .

ABC CBA: .

ACB ACB: .

ACB BAC:

ACB BCA:

ACB CAB:

ACB CBA:

BAC BAC:

BAC BCA:

BAC CAB:

BAC CBA:

BCA BCA:

BCA CAB: .... this one's really the key, but please fill in the rest too.

BCA CBA:

CAB CAB:

CAB CBA:

CBA CBA:

Once you've copied these to your comment I will delete my copies.

Comment by homunq on Democracy and rationality · 2013-10-30T20:25:58.266Z · LW · GW

I'm sorry, you really are wrong here. You can't make up just one scenario and its result and say that you have a voting rule; a rule must give results for all possible scenarios. And once you do, you'll realize that the only ones which pass both unanimity and IIA are the ones with an a priori dictatorship. I'm not going to rewrite Arrow's whole paper here but that's really what he proved.

Comment by homunq on Democracy and rationality · 2013-10-30T20:22:15.474Z · LW · GW

Under Arrow's terms, this still counts as a dictator, as long as the other ballots have no effect. (Not "no net effect", but no effect at all.)

In other words: if I voted for myself, and everyone else voted for Kanye, and my ballot happened to get chosen, then I would win, despite being 1 vote against 100 million.

It may not be the traditional definition of dictatorship, but it sure ain't democracy.

Comment by homunq on Democracy and rationality · 2013-10-30T18:29:57.501Z · LW · GW

Again, you're simply not understanding the theorem. If a system fails non-dictatorship, that really does mean that there is an a priori dictator. That could be that one vote is chosen by lot after the ballots are in, or it could be that everybody (or just some special group or person) knows beforehand that Mary's vote will decide it. But it's not that Mary just happens to turn out to be the pivotal voter between a sea of red on one side and blue on the other.

I realize that this is counterintuitive. Do you think I have to be clearer about it in the post?

Comment by homunq on Democracy and rationality · 2013-10-30T18:25:13.889Z · LW · GW

Wait until I get to explaining SODA; a voting system where you can vote for one and still get better results.

As for comparing different societies: there are of course societies with different electoral systems, and I think some systems do tend to lead to better governance than in the US/UK, but the evidence is weak and VERY confounded. It's certainly impossible to clearly demonstrate a causal effect; and would be, even assuming such an effect existed and were sizeable. I will talk about this more as I finish this post.

Comment by homunq on Democracy and rationality · 2013-10-30T17:28:35.727Z · LW · GW

Thanks, I'll work on that.

Comment by homunq on What Can We Learn About Human Psychology from Christian Apologetics? · 2013-10-30T17:26:59.801Z · LW · GW

Your probability theory here is flawed. The question is not about P(A&B), the probability that both are true, but about P(A|B), the probability that A is true given that B is true. If A is "has cancer" and B is "cancer test is positive", then we calculate P(A|B) as P(B|A)P(A)/P(B); that is, if there's a 1/1000 chance of cancer and and the test is right 99/100, then P(A|B) is .99.001/(.001.99+.999.01) which is about 1 in 10.

Comment by homunq on Democracy and rationality · 2013-10-30T16:19:33.852Z · LW · GW

I'll certainly have more content that addresses these questions as the post develops. For now, I'll simply respond to your misunderstanding about Arrow. The problem is not that there will always be an a posteori pivotal voter, but that (to satisfy the other criteria) there must be an a priori dictator. In other words, you would get the same election result by literally throwing away all ballots but one without ever looking at them. This is clearly not democracy.

Comment by homunq on Democracy and rationality · 2013-10-30T16:15:11.821Z · LW · GW

This is still in-progress, and I'm going to get to some of that later. Here's my defense of the current summary:

  • First, it's just a summary. If it could include all the subtleties of the article, I wouldn't need to write the article.
  • Second, even if the public voting systems (muni, state, and national) wherever you happen to live continue to be stupid ones, understanding voting systems better is useful knowledge. You should understand bad voting systems if they affect you, and good voting systems if you're in organizations that could use them.
  • Third, I don't agree that changing voting systems is a negligible priority. For instance: various cities nationwide, including most of the SF bay area, use IRV for city elections (though this isn't actually the best system, it is certainly a change from 15 years ago.) A number of states (at least 10 to my knowledge) have revamped their primary systems in this time. An approval voting initiative for primaries is currently in the signature stage in Oregon, and legislative study commissions of approval voting are underway in Rhode Island and Arizona, with Colorado considering one. States representing 136 electoral votes have signed the National Popular Vote interstate compact, which is about halfway to it taking effect. Obviously, these various facts affect a small minority of Americans, but that small minority is still millions of people. So I'd estimate that a nationwide change (accomplished at the state-by-state level and NOT through a constitutional amendment) is an outside chance but not a negligible one.

As to telling you how to find truth, how to win, and how to vote: obviously the goal here is not to tell you which way to vote, but to help deepen your understanding of the utility of voting mechanisms, both at the public scale and in private contexts.

....

On the other hand, I understand that you're telling me that this sounds grating to you, like overblown rhetoric. I'll see what I can do to improve that while keeping it succinct and intriguing. So thank you.

Comment by homunq on Trusting Expert Consensus · 2013-10-30T15:18:50.704Z · LW · GW

It's easy, but not helpful, to use "postmodern" as a shorthand for "bad ideas" of some kind. Something like Sturgeon's law ("90% of everything is crap") applies to postmodernism as to everything else, and I'd even agree that it's a kind of thinking that is more likely than average to come unmoored from reality, but that doesn't mean that it's barren of all insight. Especially today, at least 20 years after its heydey, and considering that even in its heyday it was a very rare academic department indeed where drinking the kool-aid was either mandatory or an excuse for stupidity (as opposed to wrongness, which it certainly did excuse; but again, Sturgeon), beating up on postmodernism seems like worrying about crack babies; slightly anachronistic and unhelpful.