Posts
Comments
I see your point here, although I will say that decision science is ideally a major component in the skill set for any person in a management position. That being said, what's being proposed in the article here seems to be distinct from what you're driving at.
Managing cognitive biases within an institution doesn't necessarily overlap with the sort of measures being discussed. A wide array of statistical tools and metrics isn't directly relevant to, e.g. battling sunk-cost fallacy or NIH. More relevant to that problem set would be a strong knowledge of known biases and good training in decision science and psychology in general.
That isn't to say that these two approaches can't overlap, they likely could. For example stronger statistical analysis does seem relevant to the issue of over-optimistic projections you bring up in a very straightforward way.
From what I gather you'd want a CRO that has a complimentary knowledge base in relevant areas of psychology alongside more standard risk analysis tools. I definitely agree with that.
Just thought I'd point out, actuaries can also do enterprise risk management. Also, a lot of organizations do have a Chief Risk Officer.
I think it's fair to say that most of us here would prefer not to have most Reddit or Facebook users included on this site, the whole "well-kept garden" thing. I like to think LW continues to maintain a pretty high standard when it comes to keeping the sanity waterline high.
This is part of why I tend to think that for the most part, these works aren't (or if they are, they shouldn't be) aimed at de-converting the faithful (who have already built up a strong meme-plex to fall back on), but rather for interception and prevention for young potential converts and people who are on the fence. Particularly college kids who have left home and are questioning their belief structure.
The side effect is that something that is marketed well towards this group (imo, this is the case with "The God Delusion") comes across as shocking and abrasive to the older converts (and this also plays into its marketability to a younger audience). So there's definitely a trade-off, but getting the numbers right to determine the actual payoff is difficult.
I think a more effective way to increase secular influence is through lobbying. I think in the U.S. there is a great need for a well-funded secular lobby to keep things in check. I found one such lobby but I haven't had the chance to look into it yet.
I've met both sorts, people turned off by "The God Delusion" who really would have benefited from something like "Greatest Show on Earth", and people who really seemed to come around because of it (both irl and in a wide range of fora). The unfortunate side-effect of successful conversion, in my experience, has been that people who are successfully converted by rhetoric frequently begin to spam similar rhetoric, ineptly, resulting mostly in increased polarization among their friends and family.
It seems pretty hard to control for enough factors to see what kind of impact popular atheist intellectuals actually have on de-conversion rates and belief polarization (much less with specific subset of abrasive works), and I can't find any clear numbers on it. Seems like opinion mining facebook could potentially be useful here.
First, I do have a couple of nitpicks:
Why evolve a disposition to punish? That makes no sense.
That depends. See here for instance.
Does it make sense to punish somebody for having the wrong genes?
This depends on what you mean by "punish". If by "punish" you mean socially ostracize and disallow mating privileges, I can think of situations in which it could make evolutionary sense, although as we no longer live in our ancestral environment and have since developed a complex array of cultural norms, it no longer makes moral sense.
In any event, what you've written is pretty much orthogonal to what I've said; I'm not defending what you're calling evolutionary ethics (nor am I aware of indicating that I hold that view, if anything I took it to be a bit of a strawman). Descriptive evolutionary ethics is potentially useful, but normative evolutionary ethics commits the naturalistic fallacy (as you've pointed out), and I think the Euthyphro argument is fairly weak in comparison to that point.
The view you're attacking doesn't seem to take into account the interplay between genetic, epigenetic and cultural/mememtic factors in how moral intuitions are shaped and can be shaped. It sounds like a pretty flimsy position, and I'm a bit surprised that any ethicist actually holds it. I would be interested if you're willing to cite some people who currently hold the viewpoint you're addressing.
The reason that the Euthyphro argument works against evolutionary ethics because - regardless of what evolution can teach us about what we do value, it teaches us that our values are not fixed.
Well, really it's more neuroscience that tells us that our values aren't fixed (along with how the valuation works). It also has the potential to tell us to what degree our values are fixed at any given stage of development, and how to take advantage of the present degree of malleability.
Because values are not genetically determined, there is a realm in which it is sensible to ask about what we should value, which is a question that evolutionary ethics cannot answer.
Of course; under your usage of evolutionary ethics this is clearly the case. I'm not sure how this relates to my comment, however.
Praise and condemnation are central to our moral life precisely because these are the tools for shaping learned desires
I agree that it's pretty obvious that social reinforcement is important because it shapes moral behavior, but I'm not sure if you're trying to make a central point to me, or just airing your own position regardless of the content of my post.
I'm not sure if it's elementary, but I do have a couple of questions first. You say:
what each of us values to themselves may be relevant to morality
This seems to suggest that you're a moral realist. Is that correct? I think that most forms of moral realism tend to stem from some variant of the mind projection fallacy; in this case, because we value something, we treat it as though it has some objective value. Similarly, because we almost universally hold something to be immoral, we hold its immorality to be objective, or mind independent, when in fact it is not. The morality or immorality of an action has less to do with the action itself than with how our brains react to hearing about or seeing the action.
Taking this route, I would say that not only are our values relevant to morality, but the dynamic system comprising all of our individual value systems is an upper-bound to what can be in the extensional definition of "morality" if "morality" is to make any sense as a term. That is, if something is outside of what any of us can ascribe value to, then it is not moral subject matter, and furthermore; what we can and do ascribe value to is dictated by neurology.
Not only that, but there is a well-known phenomenon that makes naive (without input from neuroscience) moral decision making: the distinction between liking and wanting. This distinction crops up in part because the way we evaluate possible alternatives is lossy - we can only use a very finite amount of computational power to try and predict the effects of a decision or obtaining a goal, and we have to use heuristics to do so. In addition, there is the fact that human valuation is multi-layered - we have at least three valuation mechanisms, and their interaction isn't yet fully understood. Also see Glimcher et al. Neuroeconomics and the Study of Valuation From that article:
10 years of work (that) established the existence of at least three interrelated subsystems in these brain areas that employ distinct mechanisms for learning and representing value and that interact to produce the valuations that guide choice (Dayan & Balliene, 2002; Balliene, Daw, & O’Doherty, 2008; Niv & Montague, 2008).
The mechanisms for choice valuation are complicated, and so are the constraints for human ability in decision making. In evaluating whether an action was moral, it's imperative to avoid making the criterion "too high for humanity".
One last thing I'd point out has to do with the argument you link to, because you do seem to be being inconsistent when you say:
What we intuitively value for others is not.
Relevant to morality, that is. The reason is that the argument cited rests entirely on intuition for what others value. The hypothetical species in the example is not a human species, but a slightly different one.
I can easily imagine an individual from species described along the lines of the author's hypothetical reading the following:
If it is good because it is loved by our genes, then anything that comes to be loved by the genes can become good. If humans, like lions, had a disposition to not eat their babies, or to behead their mates and eat them, or to attack neighboring tribes and tear their members to bits (all of which occurs in the natural kingdom), then these things would not be good. We could not brag that humans evolved a disposition to be moral because morality would be whatever humans evolved a disposition to do.
And being horrified at the thought of such a bizarre and morally bankrupt group. I strongly recommend you read the sequence I linked to in the quite if you haven't. It's quite an interesting (relevant) short story.
So, I have a bit more to write but I'm short on time at the moment. I'd be interested to hear if there is anything you find particularly objectionable here though.
I initially wrote up a bit of a rant, but I just want to ask a question for clarification:
Do you think that evolutionary ethics is irrelevant because the neuroscience of ethics and neuroeconomics are much better candidates for understanding what humans value (and therefore for guiding our moral decisions)?
I'm worried that you don't because the argument you supplied can be augmented to apply there as well: just replace "genes" with "brains". If your answer is a resounding 'no', I have a lengthy response. :)
As I understand it, because T proves in n symbols that "T can't prove a falsehood in f(n) symbols", taking the specification of R (program length) we could do a formal verification proof that R will not find any proofs, as R only finds a proof if T can prove a falsehood within g(n)<exp(g(n)<<f(n) symbols. So I'm guessing that the slightly-more-than-n-symbols-long is on the order of:
n + Length(proof in T that R won't print with the starting true statement that "T can't prove a falsehood in f(n) symbols")
This would vary some with the length of R and with the choice of T.
Typically you make a "sink" post with these sorts of polls.
ETA: BTW, I went for the paper. I tend to skim blogs and then skip to the comments. I think the comments make the information content on blogs much more powerful, however.
You can donate it to my startup instead, our board of directors has just unanimously decided to adopt this name. Paypal is fine. Our mission is developing heuristics for personal income optimization.
Winners Evoking Dangerous Recursively Improving Future Intelligences and Demigods
Bob's definition contains my definition
Well here's what gets me. The idea is that you have to create Bob as well, and you had to hypothesize his existence in at least some detail to recognize the issue. If you do not need to contain Bob's complete definition, then It isn't any more transparent to me. In this case, we could include worlds with any sufficiently-Bob-like entities that can create you and so play a role in the deal. Should you pre-commit to make a deal with every sufficiently-Bob-like entity? If not, are there sorts of Bob-agents that make the deal favorable? Limiting to these sub-classes, is a world that contains your definition more likely than one that contains a favorable Bob-agent? I'm not sure.
So the root of the issue that I see is this: Your definition is already totally fixed, and if you completely specify Bob, the converse of your statement holds, and the worlds seem to have roughly equal K-complexity. Otherwise, Bob's definition potentially includes quite a bit of stuff - especially if the only parameters are that Bob is an arbitrary agent that fits the stipulated conditions. The less complete your definition of Bob is, the more general your decision becomes, the more complete your definition of Bob is, the more the complexity balances out.
EDIT: Also, we could extend the problem some more if we consider Bob as an agent that will take into account an anti-You that will create Bob and torture it for all eternity if Bob creates you. If we adjust to that new set of circumstances, the issue I'm raising still seems to hold.
I'm not sure I completely understand this, so Instead of trying to think about this directly I'm going to try to formalize it and hope that (right or wrong) my attempt helps with clarification. Here goes:
Agent A generates a hypothesis about an agent, B, which is analogous to Bob. B will generate a copy of A in any universe that agent B occupies iff agent A isn't there already and A would do the same. Agent B lowers the daily expected utility for agent A by X. Agent A learns that it has the option to make agent B, should A have pre-committed to B's deal?
Let Y be the daily expected utility without B. Then Y - X = EU post-B. The utility to agent A in a non-B-containing world is
)
Where d(i) is a time dependent discount factor (possibly equal to 1) and t is the lifespan of the agent in days. Obviously, if
the agent should not have pre-committed (and if X is negative or 0 the agent should/might-as-well pre-commit, but then B would not be a jerk).
Otherwise, pre-commitment seems to depend on multiple factors. A wants to maximize its sum utility over possible worlds, but I'm not clear on how this calculation would actually be made.
Just off the top of my head, if A pre-commits, every world in which A exists and B does not, but A has the ability to generate B will drop from a daily utility of Y, to one of Y - X. On the other hand, every world in which B exists but A does not, but B can create A goes from 0 to Y -X utility. Let's assume a finite and equal number of both sorts of worlds for simplicity. Then pairing up each type of world, we go from an average daily utility Y/2 to Y-X. So we would probably at least want it to be the case that:
so
So then the tentative answer would be "it depends on how much of a jerk Bob really is". The rule of thumb from this would indicate that you should only pre-commit if Bob reduces your daily expected utility by less than half. This was under the assumption that we could just "average out" the worlds where the roles are reversed. Maybe this could be refined some with some sort of K-complexity consideration, but I can't think of any obvious way to do that (that actually leads to a concrete calculation anyway).
Also, this isn't quite like the Prometheus situation, since Bob is not always your creator. Presumably you're in a world where Bob doesn't exist, otherwise you wouldn't have any obligation to use the Bob-maker Omega dropped off even if you did pre-commit. So I don't think the same reasoning applies here.
An essential part of who Bob the Jerk is is that he was created by you, with some help from Omega. He can't exist in a universe where you don't, so the hypothetical bargain he offered you isn't logically coherent.
I don't see how this can hold. Since we're reasoning over all possible computable universes in UDT, if Bob can be partially simulated by your brain, a more fleshed out version (fitting the stipulated parameters) should exist in some possible worlds
Alright, well that's what I've thought of so far.
SPARC for undergrads is in planning, if we can raise the funding.
Awesome, glad to hear it!
See here.
Alright, I think I'll sign up for that.
Anything for undergrads? It might be feasible to do a camp at the undergraduate level. Long term, doing an REU style program might be worth considering. NSF grants are available to non-profits and it may be worth at least looking into how SIAI might get a program funded. This would likely require some research, someone who is knowledgeable about grant writing and possibly some academic contacts. Other than that I'm not sure.
In addition, it might be beneficial to identify skill sets that are likely to be useful for SI research for the benefit of those who might be interested. What skills/specialized knowledge could SI use more of?
My bigger worry is more along the lines of "What if I am useless to the society in which I find myself and have no means to make myself useful?" Not a problem in a society that will retrofit you with the appropriate augmentations/upload you etc. and I tend to think that is more likely that not, but what if, say, the Alcore trust gets us through a half-century-long freeze and we are revived, but things have moved more slowly than one might hope, yet fast enough to make any skill sets I have obsolete? Well, if the expected utility of living is sufficiently negative I could kill myself and it would be as if I hadn't signed up for cryonics in the first place, so we can chalk that up as a (roughly) zero utility situation. So in order to really be an issue, I would have to be in a scenario where I am not allowed to kill myself or be re-frozen etc. Now, if I am not allowed to kill myself in a net negative utility situation (I Have no Mouth and I Must Scream) that is a worst case scenario, and seems exceedingly unlikely (though I'm not sure how you can get decent bounds for that).
So my quick calculation would be something like: P("expected utility of living is sufficiently negative upon waking up")*P("I can't kill myself" | "expected utility of living is sufficiently negative upon waking up") = P("cryonics is not worth it" | "cryonics is successful")
It's difficult to justify not signing up for cryonics if you accept that it is likely to work in an acceptable form (this is a separate calculation). AFAICT there are many more foreseeable net positive or (roughly) zero utility outcomes than foreseeable net negative utility outcomes.
If I like and want to hug everyone at a gathering except one person, and that one person asks for a hug after I've hugged all the other people and deliberately not hugged them, that's gonna be awkward no matter what norms we have unless I have a reason like "you have sprouted venomous spines".
Out of curiosity, are there any particular behaviors you have encountered at a gathering (or worry you may encounter) that you find off-putting enough to make the hug an issue?
I'm 100% for this. If there were such a site I would probably permanently relocate there.
essentially erasing the distiction of map and territory
This idea has been implied before and I don't think it holds water. That this has come up more than once makes me think that there is some tendency to conflate the map/territory distinction with some kind of more general philosophical statement, though I'm not sure what. In any event, the Tegmark level 4 hypothesis is orthogonal to the map/territory distinction. The map/territory distinction just provides a nice way of framing a problem we already know exists.
In more detail:
Firstly, even if you take some sort of Platonic view where we have access to all the math, you still have to properly calibrate your map to figure out what part of the territory you're in. In this case you could think of calibrating your map as applying an appropriate automorphism, so the map/territory distinction is not dissolved.
Second, the first view is wrong, because human brains do not contain or have access to anything approaching a complete mathematical description of the level 4 multiverse. At best a brain will contain a mapping of a very small part of the territory in pretty good detail, and also a relatively vague mapping that is much broader. Brains are not logically omniscient; even given a complete mathematical description of the universe, the derivations are not all going to be accessible to us.
So the map territory distinction is not dissolved, and in particular you don't somehow overcome the mind projection fallacy, which is a practical (rather than philosophical) issue that cannot be explained away by adopting a shiny new ontological perspective.
Disagreement is perfectly fine by me. I don't agree with the entirety of the sequences either. It's disagreement without looking at the arguments first that bothers me.
Firstly, a large proportion of the Sequences do not constitute "knowledge", but opinion. It's well-reasoned, well-presented opinion, but opinion nonetheless -- which is great, IMO, because it gives us something to debate about. And, of course, we could still talk about things that aren't in the sequences, that's fun too. Secondly:
Whether the sequences constitute knowledge is beside the point - they constitute a baseline for debate. People should be familiar with at least some previously stated well-reasoned, well-presented opinions before they try to debate a topic, especially when we have people going through the trouble of maintaining a wiki that catalogs relevant ideas and opinions that have already been expressed here. If people aren't willing or able to pick up the basic opinions already out there, they will almost never be able to bring anything of value to the conversation. Especially on topics discussed here that lack sufficient public exposure to ensure that at least the worst ideas have been weeded out of the minds of most reasonably intelligent people.
I've participated in a lot of forums (mostly freethough/rationality forums), and by far the most common cause of poor discussion quality among all of them was a lack of basic familiarity with the topic and the rehashing of tired, old, wrong arguments that pop into nearly everyone's head (at least for a moment) upon considering a topic for the first time. This community is much better than any other I've been a part of in this respect, but I have noticed a slow decline in this department.
All of that said, I'm not sure if LW is really the place for heavily moderated, high-level technical discussions. It isn't sl4, and outreach and community building really outweigh the more technical topics, and (at least as long as I've been here) this has steadily become more and more the case. However, I would really like to see the sort of site the OP describes (something more like sl4) as a sister site (or if one already exists I'd like a link). The more technical discussions and posts, when they are done well, are by far what I like most about LW.
But having an AI that circumvents it's own utility function, would be evidence towards poor utility function design.
By circumvent, do you mean something like "wireheading", i.e. some specious satisfaction of the utility function that involves behavior that is both unexpected and undesirable, or do you also include modifications to the utility function? The former meaning would make your statement a tautology, and the latter would make it highly non-trivial.
I'm going to assert that it has something to do with who started the blog.
I think he's talking about "free market optimism" - the notion that deregulation, lowered taxes, less governmental oversight, a removal of welfare programs etc. lead to optimal market growth and eventually to general prosperity. Most conservative groups in America definitely proselytize this idea, I'm not sure about elsewhere.
The sample list of subjects is even broader than all the subjects mentioned someone on this page.
In that case I'm a bit unclear about the sort of research I'd be expected to do were I in that position. Most of those subjects are very wide open problems. Is there an expectation that some sort of original insights be made, above and beyond organizing a clear overview of the relevant areas?
I think it might help if you elaborate on the process some: How are hours tracked? Is it done by the honor system or do you have some software? Will I need to work at any specific times of the day, or do I just need to be available for at least 20 hours? Is there a sample list of subjects?
Either way, I'll probably send in an application and go from there. I currently tutor calculus online for approximately the same pay, but this seems somewhat more interesting.
Fixed
I posted this article to the decision theory group a moment ago. It seems highly relevant to thinking concretely about logical uncertainty in the context of decision theory, and provides what looks to be a reasonable metric for evaluating the value of computationally useful information.
ETA: plus there is an interesting tie-in to cognitive heuristics/biases.
The original article and usual use of "Ugh Field" (in the link at the top of the post) is summariezed as:
Pavlovian conditioning can cause humans to unconsciously flinch from even thinking about a serious personal problem they have, we call it an "Ugh Field"1. The Ugh Field forms a self-shadowing blind spot covering an area desperately in need of optimization, imposing huge costs.
I agree that LW has Ugh Fields, but I can't see how AI risks is one. There may be fear associated with AI risks here but that is specifically because it is a major topic of discussion here. Fear may impede clear thinking, sure, but this particular case doesn't seem to fit into the notion of Ugh Field.
I think the confusion stems from the definition in the post being much too loose:
Ugh Fields are internal negative reactions that occur before the conscious mind has an opportunity to process the information, often resulting in less than optimal decision making.
If you want to take a look at possible LW Ugh Fields, I'd take a look at user:Will_Newsome's posts.
Not to mention a massive underestimation of intermediate positions, e.g. the doubting faithful, agnostics, people with consciously chosen, reasonable epistemology etc. This sets that number to 0. I've met plenty of more liberal theists that didn't assert 100% certainty.
That makes sense. It still seems to be more of a rhetorical tool to illustrate that there is a spectrum of subjective belief. People tend to lump important distinctions like these together: "all atheists think they know for certain there isn't a god" or "all theists are foaming at the mouth and have absolute conviction", so for a popular book it's probably a good idea to come up with this sort of scale like this, to encourage people to refine their categorization process. I kind of doubt that he meant it to be used as a tool for inferring Bayesian confidence (in particular, I doubt 6.9 out of 7 is meant to be fungible with P(god exists) = .01428).
There has however be a mechanism for it to work for correct positions better than for incorrect ones. That is absolutely the key.
The whole point of studying formal epistemology and debiasing (major topics on this site) is to build the skill of picking out which ideas are more likely to be correct given the evidence. This should always be worked on in the background, and you should only be applying these tips in the context of a sound and consistent epistemology. So really, this problem should fall on the user of these tips - it's their responsibility to adhere to sound epistemic standards when conveying information.
As far as the issue of changing minds - there is sort of a continuum here, for instance I might have a great deal of strong evidence for something like, say, evolution. Yet there will be people for whom the inferential distance is too great to span in the course of a single discussion - "well, it's just a theory", "you can't prove it" etc.
Relevant to the climate example, a friend of mine who is doing his doctorate in environmental engineering at Yale was speaking to the relative of a friend who is sort of a 'naive' climate change denier - he has no grasp of how scientific data works nor does he have any preferred alternative theory he's invested in. He's more like the "well it's cold out now, so how do you explain that?" sort. My friend tried to explain attractors and long term prediction methods, but this was ineffective. Eventually he pointed out how warm the winter has been unusually this year, and that made him think a bit about it. So he exploited the other person's views to defend his position. However, it didn't correct the other person's epistemology at all, and left him with an equally wrong impression of the issue.
The problem with his approach (and really, in his defense, he was just looking to end the conversation) is that should that person learn a bit more about it, he will realize that he was deceived and will remember that the deceiver was a "global warming believer". In this particular case, that isn't likely (he almost certainly will not go and study up on climate science), but it illustrates a general danger in presenting a false picture in order to vault inferential distance.
It seems like the key is to first assess the level of inferential distance between you and the other person, and craft your explanation appropriately. The difficult part is doing so without setting the person up to feel cheated once they shorten the inferential distance a bit.
So, the difficulty isn't just in making it work better for correct positions (which has its own set of suggestions, like studying statistics and (good) philosophy of science), but also being extremely careful when presenting intermediate stories that aren't quite right. This latter issue disappears if the other person has close to the same background knowledge as you, and you're right that in such cases it can become fairly easy to argue for something that is wrong, and even easier to argue for something that isn't as well settled as you think it is (probably the bigger danger of the two), leading you to misrepresent the strength of your claim. I think this latter issue is much 'stickier' and particularly relevant to LW, where you see people who appear to be extremely confident in certain core claims yet appear to have a questionable ability to defend them (often opting to link to posts in the sequences, which is fine if you've really taken the time to work out the details, but this isn't always the case).
I'm speaking of people arguing. Not that there's all that much wrong with it - after all, the folks who deny the global warming, they have to be convinced somehow, and they are immune to simple reasonable argument WRT the scientific consensus. No, they want to second-guess science, even though they never studied anything relevant outside the climate related discussion.
I'm a tad confused. Earlier you were against people using the information they don't fully understand yet happens to be true, but here you seem to be suggesting that this isn't so bad and has a useful purpose - convincing people who deny global warming because they don't trust the science.
Would you be amenable to the position that sometimes it is OK to purposely direct people to adopt your point of view if it has a certain level of clear support, even if those people leave not fully understanding why the position is correct? I.e. is it sometimes good to promote "guessing the teacher's password" in the interest of minimizing risk/damages?
Given that he's pretty disposed to throwing out rhetorical statements, I'd say that's a reasonable hypothesis. I'd be surprised if there was more behind it than simply recognizing that his subjective belief in any religion was 'very, very low', and just picking a number that seemed to fit.
Just look at the 'tips' for productive arguments. Is there a tip number 1: drop your position ASAP if you are wrong? Hell frigging no (not that it would work either, though, that's not how arguing ever works).
I've done my best to make this a habit, and it really isn't that hard to do, especially over the internet. Once you 'bite the bullet' the first time it seems to get easier to do in the future. I've even been able to concede points of contention in real life (when appropriate). Is it automatic? No, you have to keep it in the back of your mind, just like you have to keep in mind the possibility that you're rationalizing. You also have to act on it which, for me, does seem to get easier the more I do it.
The tip is awesome when you are right (and I totally agree that it is great to have references and so on). When you are wrong, which is more than half of the problem (as much of the time BOTH sides are wrong), it is extremely obtuse. I'd rather prefer people dump out something closer to why they actually believe the argument, rather than how they justify them. Yes, that makes for poor show, but it is more truthful. Why you believe something, is [often] not accurate citation. It is [often] the poor paraphrasing.
This sort of goes with the need to constantly try to recognize when you are rationalizing. If you are looking up a storm of quotes, articles, posts etc. to back up your point and overwhelm your 'opponent', this should set off alarm bells. The problem is that those who spend a lot of time correcting people who are obviously wrong by providing them with large amounts of correct information also seem prone to taking the same approach to a position that merely seems obviously wrong for reasons they might not be totally conscious of themselves. They then engage in some rapid fire confirmation bias, throw a bunch of links up and try to 'overpower the opponent'. This is something to be aware of. If the position you're engaging seems wrong but you don't have a clear-cut, well evidenced reason why this is, you should take some time to consider why you want it to be right.
When facing someone who is engaging in this behavior (perhaps they are dismissing something you think is sensible, be it strong AI or cryonics, or existential risk, what have you) there are some heuristics you can use. In online debates in particular, I can usually figure out pretty quickly if the other person understands the citations they make by choosing one they seem to place some emphasis on and looking at it carefully, then posing questions about the details.
I've found that you can usually press the 'citingpeople' into revealing their underlying motivations in a variety of ways. One way is sort of poor - simply guess at their motivations and suggest that as truth. They will feel the need to defend their motivations and clarify. The major drawback is that this can also shut down the discussion. An alternative is to suggest a good-sounding motivation as truth - this doesn't feel like an attack, and they may engage it. The drawback is that this may encourage them to take up the suggested motivation as their own. At this point, some of their citations will likely not be in line with their adopted position, but pointing this out can cause backtracking and can also shut down discussion if pressed. Neither approach guarantees us clear insight into the motivations of the other person, but the latter can be a good heuristic (akin to the 'steel man suggestion). Really, I can't think of a cut-and-dried solution to situations in which people try to build up a wall of citations - each situation I can think of required a different approach depending on the nature of the position and the attitude of the other person.
Anyway, I think that in the context of all of the other suggestions and the basic etiquette at LW, the suggestions are fine, and the situation you're worried about would typically only obtain if someone were cherry picking a few of these ideas without making effort to adjust their way of thinking. Recognizing your motivation for wanting something to be true is an important step in recognizing when you're defending a position for poor reasons, and this motivation should be presented upfront whenever possible (this also allows the other person to more easily pinpoint your true rejection).
I found myself wondering if there are any results about the length of the shortest proof in which a proof system can reach a contradiction, and found the following papers:
Paper 1 talks about partial consistency. We have statements of the following form:
) is a statement that there is no ZF-proof of contradiction with length =< n.
The paper claims that this is provable in ZF for each n. The paper then discusses results about the proof length of the partial consistency statements is polynomial in n. The author goes on to derive analogous results pertaining to frege proof systems weaker than ZF.
From these results it may be possible to have the agent's proof system produce a partial consistency result for a stipulated length.
Paper 2 shows that the question of whether a formula has a proof of length no more than k is undecidable.
Both papers seem relevant, but I don't presently have the time to think them through carefully.
This seems to be conflating rationality centered material with FAI/optimal decision theory material and has lumped them all under the heading "utilit maximization". These individual parts are fundamentally distinct, and aim at different things.
Rationality centered material does include some thought about utility, Fermi calculations and heuristics, but focuses on debiasing, recognizing cognitive heuristics that can get in the way (such as rationalization, cached thoughts) and the like. I've managed to apply them a bit in my day to day thought. For instance; recognizing the fundamental attribution error has been very useful to me, because I tend to be judgmental. This has in the past led to me isolating myself much more than I should, and sinking into misanthropy. For the longest time I avoided the thoughts, now I've found that I can treat them in a more clinical manner and have gained some perspective on them. This helps me raise my overall utility, but it does not perfectly optimize it by any stretch of the imagination - nor is it meant to, it just makes things better.
Bottomless recursion with respect to expected utility calculations is a decision theory/rational choice theory issue and an AI issue, but it is not a rationality issue. To be more rational, we don't have to optimize, we just have to recognize that one feasible procedure is better than another, and work on replacing our current procedure with this new, better one. If we recognize that a procedure is impossible for us to use in practice, we don't use it - but it might be useful to talk about in a different, theoretical context such as FAI or decision theory. TDT and UDT were not made for practical use by humans - they were made to address a theoretical problem in FAI and formal decision theory, even though some people claim to have made good use of them (even here we see TDT being used at a psychological aide for overcoming hyperbolic discounting more than as a formal tool of any sort).
Also, there are different levels of analysis appropriate for different sorts of things. If I'm analyzing the likelihood of an asteroid impact over some timescale, I'm going to include much more explicit detail there, than in my analysis of whether I should go hang out with LWers in New York for a bit. I might assess lots of probability measures in a paper analyzing a topic, but doing so on the fly rarely crosses my mind (I often do a quick and dirty utility calculation to decide whether or not to do something, e.g. - which road home has the most right turns, what's the expected number of red lights given the time of day etc., but that's it).
Overall, I'm getting the impression that all of these things are being lumped in together when they should not be, and utility maximization means very distinct things in these very distinct contexts, most technical aspects of utility maximization were not intended for explicit everyday use by humans, they were intended for use by specialists in certain contexts.
This is not so simple to assert. You have to think of the intensity of their belief in the words of allah. Their fundamental wordview is so different from ours that there may be nothing humane left when we try to combine them.
CAVEAT: I'm using CEV as I understand it, not necessarily as it was intended as I'm not sure the notion is sufficiently precise for me to be able to accurately parse all of the intended meaning. Bearing that in mind:
If CEV produces a plan or AI to be implemented, I would expect it to be sufficiently powerful that it would entail changing the worldviews of many people during the course of implementation. My very basic template would be that of Asimov's The Evitable Conflict - the manipulations would be subtle and we would be unlikely to read their exact outcomes at a given time X without implementing them (this would be dangerous, as it means you can't "peak ahead" at the future you cause) though we still prove that at the end we will be left with a net gain in utility. The Asimov example is somewhat less complex, and does not seek to create the best possible future, only a fairly good, stable one, but this basic notion I am borrowing is relevant to CEV.
The drives behind the conviction of the suicide bomber are still composed of human drives, evolutionary artifacts that have met with a certain set of circumstances. The Al Qaeda example is salient today because the ideology is among the most noncontroversial, damaging ideology we can cite. However, I doubt that any currently held ideology or belief system held by any human today is ideal. The CEV should search for ways of redirecting human thought and action - this is necessary for anything that is meant to have global causal control. The CEV does not reconcile current beliefs and ideologies, it seeks to redirect the course of human events to bring about new, more rational, more efficient and healthy ideologies that will be compatible, if this can be done.
If there exists some method for augmenting our current beliefs and ideologies to become more rational, more coherent and more conducive to positive change, then the CEV should find it. Such a method would allow for much more utility than the failure mode you describe, and said failure mode should only occur when such a method is intractable.
In this specific case I was using this figure of speech, yes. I mean't that we would be extrapolating drives that matter for evolution (our ancient drives) but don't really matter to us, not in the sense of Want to Want described in 4c.
My point is that, in general, our drives are a product of evolutionary drives, and are augmented only by context. If the context changes, those drives change as well, but both the old set and new set are comprised of evolutionary drives. CEV changes those higher level drives by controlling the context in sufficiently clever ways.
CEV should probably be able to look at how an individual will develop in different contexts and compute the net utility in each one, and then maximize. The danger here is that we might be directed into a course of events that leads to wireheading.
It occurs to me that the evolutionary failure mode here is probably something like wireheading, though it could be more general. As I see it, CEV is aiming to maximize total utility while minimizing the net negative utility for as many individuals as possible. If some percentage of individuals prove to be impossible to direct towards a good future without causing massive dis-utility in general we have to devise a way to look at each case like this, and ask what sorts of individuals are not getting a pay-off. If it turns out to be a small number of sociopaths, this will probably not be a problem. I expect that we will have the technology to treat sociopaths and bring them into the fold. CEV should consider this possibility as well. If it turns out to be a small number of very likable people, it could be somewhat more complicated, and we should ask why this is happening. I can't think of any reasonable possible scenarios for this at the moment, but I think this is worth thinking about more.
The kernel of this problem is very central to CEV as I understand it, so I think it is good to discuss it in as much detail as we can in order to glean insight.
There is little in common between Eliezer, Me and Al Qaeda terrorists, and most of it is in the so called reptilian brain. We may end up with a set of goals and desires that are nothing more than “Eat Survive Reproduce,” which would qualify as a major loss in the scheme of things.
I think you may possibly be committing the fundamental attribution error. It's my understanding that Al Qaeda terrorists are often people who were in a set of circumstances that made them highly succeptible to propaganda - often illiterate, living in poverty and with few, if any, prospects for advancment. It is easy to manipulate the ignorant and disenfranchised. If they knew more, saw the possibilities and understood more about the world I would be surprised if they would choose a path that diverges so greatly with your own that CEV would have to resort to looking at the reptilian brain.
In this specific case, what ends up dominating CEV is what evolution wants, not what we want. Instead of creating a dynamic with a chance of creating the landscape of a Nice Place to Live, we end up with some exotic extrapolation of simple evolutionary drives.
"What evolution wants" doesn't seem like a clear concept - at least I'm having trouble making concrete sense of it. I think that you're conflating "evolution" with "more ancient drives" - the described extrapolation is an extrapolation with respect to evolutionarily ancient drives.
In particular, you seem to be suggesting that a CEV including only humans will coincide with a CEV including all vertibrates possessing a reptillian brain on the basis that our current goals seem wildly incompatible. However, as I understand it, CEV asks what we would want if we "knew more, grew up further together" etc.
Well the agent definition contains a series of conditionals. You have as the last three lines: if "cooperating is provably better than defecting", then cooperate; else, if "defecting is provably better than cooperating" then defect; else defect. Intuitively, assuming the agent's utility function is consistent, only one antecedent clause will evaluate to true. In the case that the first one does, the agent will output C. Otherwise, it will move through to the next part of the conditional and if that evaluates to true the agent will output D. If not, then it will output D anyway. Because of this, I would go for proving that line three and 4 can't both obtain.
Or you could prove that line 3 and line 4 can't both obtain; I haven't figured out exactly how to do this yet.
How about this? Suppose condition 3 and 4 both obtain. Then there exists and a and b such that U(C) = #a and U(D) = #b (switching out the underline for '#'). Also, there exists a p and q such that p > q and U(D) = #p and U(C) = #q. Now U(C) = #a > #b = U(D) = #c > #q = U(C) so U(C) > U(C). I may actually be confused on some details, since you indicate that a and b are numbers rather than variables (for instance in proposition 2 you select a and b as rational numbers), yet you further denote numerals for them, but I'm assuming that a >b iff #a > #b. Is this a valid assumption? I feel as though I'm missing some details here, and I'm not 100% sure how to fill them in. If my assumption isn't valid I have a couple of other ideas. I'm not sure if I should be thinking of #a as 'a' under some interpretation.
Going a little bit further, it looks like your lemma 1 only invokes the first two lines and can easily be extended for use in conjecture 4 - look at parts 1 and 3. Part 1 is one step away from being identical to part 3 with C instead of D.
~Con(PA) -> ~ Con(PA + anything), just plug in Con(PA) in there. Adding axioms can't make an inconsistent theory consistent. Getting the analogous lemma to use in conjecture 4 looks pretty straightforward - switching out C with D and vice versa in each of the parts yields another true four part lemma, identical in form but for the switched D's and C's so you can use them analogously to how you used the original lemma 1 in the proof of proposition 3.
That would be very impressive, but I don't see that in any of the stuff on his semiotics on Wikipedia.
A caveat: I'm not at all sure how much I'm projecting on Peirce as far as this point goes. I personally think that his writings clarified my views on the scientific method (at the time I originally read them, which was a good while back) and I was concurrently thinking about machine learning - so I might just be having a case of cached apophenia.
However; if you want a condensed version of his semiotic look over this. You might actually need to read some of the rest of that article (which, I admit, is a bit long) to put it in more context. Also, this wikipedia page looks pretty comprehensive. I'm pretty confident that they're leaving a bit out that might be clearer if you read Peirce, but I'm not sure of how much instrumental value that would be to you. The issue with reading Peirce is that he was a crazy hermit with thousands of unpublished notes who continuously updated his views in significant ways (another point for him, he continued to reconsider/shift his views in a systematic way until he died), so lots of what you read about/by Peirce is compiled from a vast repository of his notes collected from his workspace after his death.
Also, something neat I found: Peirce's three valued logic predating Post. That was among his tens of thousands of unpublished pages of notes. Going further in this direction, I found an interesting article on Peirce's logic. There is some interesting discussion there about his influence on modern logic.
Points for coolness - Simon Newcomb was quite possibly his evil arch-nemesis.
Anyway - I think what attracts me to Peirce the most is his seemingly endless ability to carve reality at the joints in novel (at the time at least) ways, coupled with his nearly superhuman productivity levels - I mean he was highly influential in the realm of statistical theory, his influence on experimental design was impressive, he invented an axiomatization for arithmetic before Peano, he invented a modern characterization of first-order logic on par with Frege's (but arguably with a more algebraic/model theoretic than syntactic approach), he was a skilled expositor and clear writer, he invented pragmaticism, he had a lifetime of smaller results in logic, earth sciences and mathematics that anyone would be proud of - what more could you want from a single person before you can understand why people admire them?
Oh, he seems to have disobeyed endoself's first law of philosophy: "Have as little to do with Hegel as possible."
I think it's significantly more forgivable for a contemporary of Hegel's to be influenced by him than someone today being influenced by him. Further, when I say "influenced" - he scavenged a set of ideas that he seems to have reinterpreted in terms of his own philosophy because he saw that they could round out his ideas in a variety of ways - he was still pretty critical of Hegel of some major points. I think just browsing through these excerpts reveals the lines of influence a bit.
I think there's a problem with your thinking on this - people can spot patterns of good and bad reasoning. Depending on the argument, they may or may not notice a flaw in the reasoning for a wide variety of reasons. Someone who is pretty smart probably notices the most common fallacies naturally - they could probably spot at least a few while watching the news or listening to talk shows.
People who study philosophy are going to have been exposed to many more diverse examples of poor reasoning, and will have had practice identifying weak points and exploiting them to attack an argument. This increases your overall ability to dissolve or decompose arguments by increasing your exposure and by equipping you with a trick bag of heuristics. People who argue on well moderated forums or take part in discussions on a regular basis will likely also pick up some tricks of this sort.
However, there are going to be people who can dissolve one problem, but not another because they have been exposed to something sufficiently similar to one (and are thus probably have some cached details relevant to solving it) but not so for the other:
E.g. a student of logic will probably make the correct choice in the Wason Selection Task and may be able to avoid making the conjunction fallacy, but they may not two box because they fall into the CDT reasoning trap. However, a student of the sciences or statistics may slip up in the selection task but one box, by following the EDT logic.
So if you're using this approach as an intelligence test, I'd worry about committing the fundamental attribution error pretty often. However, I doubt you're carrying out this test in isolation. In practice, it probably is reasonable to engage people you know or meet in challenging discussions if you're looking for people that are sharp and enjoy that sort of thing. I do it every time I meet someone who seems like they might have some inclination toward that sort of thing.
It might help if you provide some context though - who are you asking and how do you know them? Are you accosting strangers with tricky problems or are you probing acquaintances and friends?
Well I can relay my impressions on Peirce and why people seem to be interested in him (and why I am):
I think that the respect for Peirce comes largely from his "Illustrations in the Logic of Science" series for Scientific American. Particularly "The Fixation of Belief" and "How to Make Our Ideas Clear".
When it comes to Tychism, it's kind of silly to take it in a vacuum, especially given that the notion of statistics being fundamental to science was new, and Newtonian determinism was the de facto philosophical stance of his day. He was standing in direct conflict to the then popular (but false) view of Newtonian determinism. Observe the following excerpt from the beginning of one of his papers on the subject:
"I propose here to examine the common belief that every single fact in the universe is precisely determined by law. "
Going further
"The proposition in question is that the state of things existing at any time, together with certain immutable laws, completely determine the state of things at every other time (for a limitation to future time is indefensible). Thus, given the state of the universe in the original nebula, and given the laws of mechanics, a sufficiently powerful mind could deduce from these data the precise form of every curlicue of every letter I am now writing."
Given the context, although he might have been guilty of the mind projection fallacy (he was a realist when it comes to probabilities), and was pretty much a frequentist, I don't think it's reasonable to criticize him very harshly for either position - he was a very early pioneer in statistics (just look over the second paragraph). His embrace of statistical inference was, as far as I can tell, somewhat unusual for the day, and he made several contributions to the use of statistics in psychology and psycho physics (including the use of double blind studies to re-examine previous findings). This is in addition to his contributions to logic, mathematics and geology - so if his logic is on par with Frege (just look at his contributions to mathematics and logic).
His semiotics is interesting as well as it seems to yield an early attempt to look at science as a process of improving statistical models. Pierce's semiotic reflects this, and he uses it in his phenomenological characterizations of the scientific method. I think that this allowed him to view human scientists as statistical learners. When I read some of this it certainly invoked a more machine learning/information theoretic picture of scientific discovery than any other (non-modern) philosophers of science had managed to touch on.
As far as the "weird numerology" - Peirce's fixation on the number three seems to be mainly a side effect of Hegel's influence - Hegel focused on a tertiary relationship between ideas - thesis, antithesis, synthesis. We have an idea, a conflict is presented, and we synthesize the two into something less wrong. I think that a number of odd ideas held by Peirce were influenced by Hegel (such as his view of continuity which seems borderline incoherent to me). I've found that sometimes this tertiary form yields something nice, but it often seems to result in something strained. I'm not sure why this became so pervasive in Peirce's writings.
That's what I can think of off the top of my head, but if something else occurs to me I'll add it.
Yes, you're right. Looking at the agent function, the relevant rule seems to be defined for the sole purpose of allowing the agent to cooperate in the even that cooperation is provably better than than defecting. Taking this out of context, it allows the agent to choose one of the actions it can take if it is provably better than the other. It seems like the simple fix is just to add this:
=D\to\pi_{\underline{i}}\chi()=\underline{a})\\\mbox{\quad\quad\quad\quad}\wedge(\psi(\ulcorner%20U%20\urcorner,\underline{i})=C\to\pi_{\underline{i}}\chi()=\underline{b})\biggr)\wedge%20a%20%3E%20b)
If you alter the agent definition by replacing the third line with this and the fourth with C you have an agent that is like the original one but can be used to prove conjecture 4 but not prop 3, right? So if you instead add this line to the current agent definition and simply leave D as the last line, then if neither the old third line nor this new one hold we simply pick D, which is neither provably better nor provably worse than C, so that's fine. It seems that we can also prove conjecture 4 by using the new line in lieu of the old one, which should allow us to use a proof that is essentially symmetric to the proof of proposition 3.
Does that seem like a reasonable fix?
The utilitarian case is interesting because both Mill and Bentham seemed to espouse a multidimensional utility vector rather than a uni-dimensional metric. There is an interesting paper I've been considering summarizing that takes a look at this position in the context of neuroeconomics and the neuroscience of desire.
Of interest from the paper: They argue that "pleasure" (liking), though it comes from diverse sources, is evaluated/consolidated at the neurological level as a single sort of thing (allowing a uni-dimensional representation as is common in economics), but show that when it comes to anticipation of pleasure (and the heuristics for evaluating the future rewards an action might yield) we can be quite inaccurate. So effectively, the Bentham/Mill model of utilitarianism is predicated on the perfect or near perfect coherence of "wanting" and "liking" (which does not really exist) and incorrectly prescribes a multidimensional utility measure because of this.
However, Bentham does appear to be unusually rational for his day. Look at his highly unpopular progressive stances and this becomes obvious - they are typically stances that we would consider sane and moral today, but which were highly taboo at the time.
Personally, when it comes to clarity of thought, I've never found anyone who tops C.S. Peirce. He was unbelievably rational in his view of the world and of science and mathematics - he was scary. This is someone who recognized the fundamentally statistical nature of the sciences in the 1800's. He was able to see the broad philosophical implications of evolution in nearly as much clarity as many of your so-called "evolutionist" thinkers (can provide citation if requested).
C.S. Peirce is by far my greatest inspiration as an epistemic rationalist. Just look at what he did. The clarity and breadth of his thought and philosophy is incredible to me.
I only looked at this for a bit so I could be totally mistaken, but I'll look at it closely soon, it's a nice write up!
My thoughts:
A change of variables/values in your proof of proposition 3 definitely doesn't yield conjecture 4? At first glance it looks like you could just change the variables and flip the indicies for the projections (use pi_1 instead of pi_0) and in the functions A[U,i]. If you look at the U() defined for conjecture 4, it's exactly the one in proposition 3 with the indices i flipped and C and D flipped, so it's surprising to me if this doesn't work or if there isn't some other minor transformation of the the first proof that yields a proof of conjecture 4.
Ha! Yeah, it seems that his name is pretty ubiquitous in mathematical logic, and he wrote or contributed to quite a number of publications. I had a professor for a sequence in mathematical logic who had Barwise as his thesis adviser. The professor obtained his doctoral degree from UW Madison when it still had Barwise, Kleene and Keisler so he would tell stories about some class he had with one or the other of them.
Barwise seems to have had quite a few interesting/powerful ideas. I've been wanting to read Vicious Circles for a while now, though I haven't gotten around to it.
Is there anything, in particular, you do consider a reasonably tight lower bound for a man-made extinction event? If so, would you be willing to explain your reasoning?