Posts
Comments
I agree that's a possible way things could be. However, I don't see how it's compatible with accepting the arguments that say we should assume that alignment is a hard problem. I mean absent such arguments why expect you have to do anything special beyond normal training to solve alignment?
As I see the argumentative landscape the high x-risk estimates depend on arguments that claim to give reason to believe that alignment is just a generally hard problem. I don't see anything in those arguments that distinguishes between these two cases.
In other words our arguments for alignment difficulty don't depend on any specific assumptions about capability of intelligence so we should currently assign the same probability to an AI being unable to save it's alignment problem as we do to us being unable to solve it.
I don't mean alignment with human concerns. I mean that the AI itself is engaged in the same project we are: building a smarter system than itself. So if it's hard to control the alignment of such a system then it should be hard for the AI. (In theory you can imagine that it's only hard at our specific level of intelligence but in fact all the arguments that AI alignment is hard seem to apply equally well to the AI making an improved AI as to us making an AI).
See my reply above. The AI x-risk arguments require the assumption that superintelligence necessarily entails the agent try to optimize some simple utility function (this is different than orthogonality which says increasing intelligence doesn't cause convergence to any particular utility function). So the doesn't care option is off the table since (by orthogonality) it's super unlikely you get the one utility function which says just maximize intelligence locally (even global max isn't enough bc some child AI who has different goals could interfere).
Those are reasonable points but note that the arguments for AI x-risk depend on the assumption that any superintelligence will necessarily be highly goal directed. Thus, either the argument fails because superintelligence doesn't imply goal directed,
And given that simply maximizing the intelligence of future AIs is merely one goal in a huge space it seems highly unlikely that (especially if we try to avoid this one goal) we just get super unlucky and the AI has the one goal that is compatible with improvement.
I like the idea of this sequence, but -- given the goal of spelling out the argument in terms of first principles -- I think more needs to be done to make the claims precisce or acknowledge they are not.
I realize that you might be unable to be more precisce given the lack of precision in this argument generally -- I don't understand how people have invested so much time/mondy on research to solve the problem and so little on making the argument for it clear and rigorous -- but if that's the case I suggest you indicate where the definitions are insufficient/lacking/unclear.
I'll list a few issues here:
Defining Superintelligence
Even Bostrom's definition of superintelligence is deeply unclear. For instance, would an uploaded human mind which simply worked at 10x the speed of a normal human mind qualify as a superintelligence? Intuitively the answer should be no, but per the definition the answer is almost certainly yes (at least if we imbue that upload with extra patience). After all, virtually all cognitive tasks of interest benefit from extra time -- if not at the time of performance then extra time to practice (10x the practice games of chess would make you a better player). And if it did qualify it undermines the argument about superintelligence improvement (see below).
If you require a qualitative improvement rather than merely speeding up the rate of computation to be a superintelligence then the definition risks being empty. In many important cognitive tasks humans already implement the theoretically optimal algorithm or nearly do so. Lots of problems (eg search on unordered data on classical TM) have no better solution than just brute force and this likely includes quite a few tasks we care quite a bit about (maybe even in social interactions). Sure, maybe an AI could optimize away the part where our slow human brain slogs through (tho arg we have as well w/ computers) but that just sounds like increased processing speed.
Finally, does that superiority measure resource usage? Does a superintelligence need to beat us on a watt for watt comparison or could it use the computing capacity of the planet.
These are just a few concerns but they illustrate the inadequacy of the definition. And it's not just nitpicking. This loose way of talking about superintelligence invites us, w/o adequate argument, to assume the relationship we will have to AI is akin to the relationship you have with your dumb family members/friends. And even if that was the relationship, remember that your dumb friends wouldn't seem so easily dominated if they hadn't decided not to put in much effort into intellectual issues.
Self-improvement
When it comes to talking about self-improvement the discussion is totally missing any notion of rate, extent or qualitative measure. The tendency is for people to assume that since technology seems to happen fast somehow so will this self-improvement but why should that be?
I mean we are already capable of self-improvement. We change the culture we pass down over time and as a result a child born today ends up learning more math, history and all sorts of problem solving tools in school that an ancient Roman kid wouldn't have learned [1]. Will AI self-improvement be equally slow? If it doesn't improve itself any faster than we improve our intelligence no problem. So any discussion of this issue that seeks to draw any meaningful conclusions needs to make some claim about the rate of improvement and even defining such a quantitative measure seems extremely difficult.
And it's not just the instantaneous rate of self-improvement that matters but also the shape of the curve. You seem to grant that figuring out how to improve AI intelligence will take the AI some time to figure out -- it's gotta do the same kind of trial and error we did to build it in the first place -- and won't be instantaneous. Ok, how does that time taken scale with increasing intelligence? Maybe an AI with a 100 SIQ points can build one with 101 SIQ after a week of work. But then maybe it takes 2 weeks for the 101 SIQ AI to figure out how to reach 102 and so on. Maybe it even asymptotes.
And what does any of this even mean? Is it getting much more capable or marginally capable? Why assume the former? Given the fact that there are mathematical limits on the most efficient possible algorithms shouldn't we expect an asymptote in ability? Indeed, there might be good reasons to think humans aren't far from it.
1: Ofc, I know that people will try and insist that merely having learned a bunch of skills/tricks in school that help you solve problems doesn't qualify as improving your intelligence. Why not? If it's just a measure of ability to solve relevant cognitive challenges such teaching sure seems to qualify. I think the temptation here is to import the way we use intelligence in human society as a measure of raw potential but that relies on a kind of hardware/software distinction that doesn't obviously make sense for AI (and arguably doesn't make sense for humans over long time scales -- Flynn effect).
Maybe this question has already been answered but I don't understand how recursive self-improvement of AIs is compatible with the AI alignment problem being hard.
I mean doesn't the AI itself face the alignment problem when it tries to improve/modify itself substantially? So wouldn't a sufficently intelligent AI refuse to create such an improvement for fear the goals of the improved AI would differ from its own?
I'd just like to add that even if you think this piece is completly mistaken I think it certainly shows we are definitely not knowledgeable enough about what and how values and motives work in us much less AI to confidently make the prediction that AIs will be usefully described with a single global utility function or will work to subvert their reward system or the like.
Maybe that will turn out to be true but before we spend so many resources on trying to solve AI alignment let's try to make the argument for the great danger much more rigorous first...usually best way to start anyway.
This is one of the most important posts ever on LW though I don't think the implications have been fully drawn out. Specifically, this post raises serious doubts about the arguments for AI x-risk as a result of alignment mismatch and the models used to talk about that risk. It undercuts both Bostrom's argument that an AI will have a meaningful (self-aware?) utility function and Yudkowsky's reward button parables.
The role these two arguments play in convincing people that AI x-risk is a hard problem is to explain why, if you don't anthropomorphize should a program that's , say, excellent at conducting/scheduling interviews to ferret out moles in the intelligence community try to manipulate external events at all not just think about them to better catch moles? I mean it's often the case that ppl fail to pursue their fervent goals outside familiar context. Why will AI be different? Both arguments conclude that AI will inevitably act like it's very effectively maximizing some simple utility function in all contexts and in all ways.
Bostrom tries to convince us that as creatures get more capable they tend to act more coherently (more like they are governed by a global utility function). This is of course true for evolved creatures but by offering a theory of how value type things can arise this theory predicts that if you only train your AI in a relatively confined class of circumstances (even if that requires making very accurate predictions about the rest of the world) it isn't going to develop that kind of simple global value but, rather, would likely find multie shards in tension without clear direction if forced to make value choices in very different circumstances. Similarly, it exains why the AI won't just wirehead itself by pressing it's rewaes button.
I absolutely think that the future of online marketing g involves more asking ppl for their prefs. I know I go into my settings on good to active curate what they show me.
Indeed, I think Google is leaving a fucking pile of cash on the table by not adding a "I dislike" button and a little survey on their ads.
I feel there is something else going on here too.
Your claimed outside view asks us to compare a clean codebase with an unclean one and I absolutely agree that it's a good case for using currentDate when initially writing code.
But you motivated this by considering refactoring and I think things go off the rails there. If the only issue in your codebase was you called currentDate yyymmdd consistently or even had other consistent weird names it wouldn't be a message it would just have slightly weird conventions. Any coder working on it for a non-trivial length of time would start just reading yyymmdd as current date in their head.
Tge codebase is only messy when you inconsistently use a bunch of different names for a concept that aren't very descriptive. But now refactoring faces exactly the same problem working with the code does..the confusion coders experience seeing the variable and wondering what it does becomes ambiguity which forces a time intensive refactor.
Practically the right move is probably better stds going forward and to encourage coders to fix variable names in any piece of code they touch. But I don't think it's really a good example of divergent intuitions once you are talking about the same things.
I don't think this is a big problem.. The people who use ad blockers are both a small fraction of internet users and the most sophisticated ones so I doubt they are a major issue for website profit. I mean sure, Facebook is eventually going to try to squeeze out the last few percent of users if they can do so with an easy countermeasure but if this was really a big concern websites would be pushing to get that info back from the company they use to host ads. Admittedly when I was working on ads for Google (I'm not cut out to be out of academia so I went back to it) I never really got into this part of the system so I can't comment on how it would work out but I think if this mattered enough companies serving ads would figure out how to report back to the page about ad blockers.
I'm sure some sites resent ad blockers and take some easy countermeasures but at an economic level I'm skeptical this really matters.
What this means for how you should feel about using ad blockers is more tricky but since I kinda like well targeted ads I don't have much advice on this point.
Interesting, but I think it's the other end of the equation where the real problem lies: voting. Given the facts that
1) A surprisingly large fraction of the US population has tried hard drugs of one kind or another.
2) Even those who haven't almost surely know people who have and seem to find it interesting/fascinating/etc.. not horrifying behavior that deserves prison time.
So why is it that people who would never dream of sending their friend who tried coke to prison or even the friend who sold that friend some of his stash how do we end up with draconian drug laws?
I don't have an easy answer. I'm sure the overton window and a desire to signal that they themselves are not pro-drug or drug users is part of the answer. It's like lowering the age of consent for sex. As long as the loudest voices arguing it should be legal for 40 year olds to sleep with 16 year olds are creeps few people will make that argument no matter how good.
But this doesn't really seem like enough to explain the phenomena.
So your intent here is to diagnose the conceptual confusion that many people have with respect to infinity yes? And your thesis is that: people are confused about infinity because they think it has a unique referant while in fact positive and negative infinity are different?
I think you are on to something but it's a little more complicated and that's what gets people are confused. The problem is that in fact there are a number of different concepts we use the term infinity to describe which is why it so super confusing (and I bet there are more).
1. Virtual Points that are above or below all other values in an ordered ring (or their positive component) which we use as shorthand to write limits and reason about how they behave.
2. The background idea of the infinite as meaning something that is beyond all finite values (hence why a point at infinity is infinite).
3. The cardinality of sets which are bijectable with a proper subset of themselves, i.e., infinite. Even here there is an ambiguity between the sets with a given cardinality and the cardinal itself.
4. The notion of absolute mathematical infinity. If this concept makes sense it does have a single reference which is taken to be 'larger' (usually in the sense of cardinality) than any possible cardinal, i.e. the height of the true hierarchy of sets.
5. The metaphorical or theological notion of infinity as a way of describing something beyond human comprehension and/or without limits.
The fact that some of these notions do uniquely refer while others don't is a part of the problem.
Stimulants are an excellent short term solution. If you absolutely need to get work done tonight and can't sleep amphetamine (i.e. Adderall) is a great solution. Indeed, there are a number of studies/experiments (including those the airforce relies on to give pilots amphetamines) backing up the fact that it improves the ability to get tasks done while sleep deprived.
Of course, if you are having long term sleep problems it will likely increase those problems.
There is a lot of philosophical work on this issue some of which recommends taking conditional probability as the fundamental unit (in which case Bayes theorem only applies for non-extremal values). For instance, see this paper
Computability is just \Delta^0_1 definability. There are plenty of other notions of definability you could try to cash out this paradox in terms of. Why pick \Delta^0_1 definability?
If the argument worked in any particular definability notion (e.g. arithmetic definability) it would be a problem. Thus, the solution needs to explain why the argument shouldn't convince you that with respect to any concrete notion of definable set the argument doesn't go through.
But that's not what the puzzle is about. There is nothing about computability in it. It is supposed to be a paradox along Russell's set of all sets that don't contain themselves.
The response about formalizing exactly what counts as a set defined by an English sentence is exactly correct.
Yah, enumerable means something different than computably enumerable.
This is just the standard sleeping beauty paradox and I’d suggest that the issue isn’t unique to FNC.
However, you are a bit quick in concluding it is time inconsistent as it’s not altogether clear that one is truly referring to the same event before and after you have the observation. The hint here is that in the standard sleeping beauty paradox the supposed update involves only information you already were certain you would get.
Id argue that what’s actually going on is that you are evaluating slightly different questions in the two cases
Don't. At least outside of Silicon Valley where oversubscription may actually be a problem. It's a good intention but it inevitably will make people worry they aren't welcome or aren't the right sort of people . Instead, describe what one does or what one talks about in a way that will appeal to the kind of people who would enjoy coming
Given that you just wrote a whole post to say hi and share your background with everyone I'm pretty confident you'll fit right in and won't have any problems being too shy. Writing a post like this rather than just commenting is such a less wrong kind of thing to do so I think you'll be right at home.
Searle can be any X?? WTF? That's a bit confusingly written.
The intuition Searle is pumping is that since he, as a component of the total system doesn't understand Chinese it seems counterintuitive to conclude that the whole system understands Chinese. When Searle says he is the system he is pointing to the fact that he is doing all the actual interpretation of instructions and is seems weird to think that the whole system has some extra experiences that let it understand Chinese even though he does not. When Searle uses the word understand he does not mean demonstrate the appropriate input output behavior he is presuming it has that behavior and asking about the system's experiences.
Searle's view from his philosophy of language is that our understanding and mening is grounded in our experiences and what makes a person count as understanding (as opposed to merely dumbly parroting) Chinese is that they have certain kinds of experiences while manipulating the words. When Searle asserts the room doesn't understand Chinese he is asserting that it doesn't have the requisite experiences (because it's not having any experiences) that someone would need to have to count as understanding Chinese.
Look, I've listened to Searle explain this himself multiple times during the 2 years of graduate seminars on philosophy of mind I took with him and have discussed this very argument with him at some length. I'm sorry but you are interpreting him incorrectly.
I know I'm not making the confusion you suggest because I've personally talked with him at some length about his argument.
I essentially agree with you that science can't bridge the is-ought gap (see caveats) but it's a good deal more complicated than the arguments you give here allow for (they are a good intro but I felt it's worth pointing out the complexities).
- When someone claims to have bridged the is-ought gap they aren't usually claiming to have analytically identified (i.e. identified as a matter of definition) ought with some is statements. That' s a crazily high bar and modern philosophers (and Sam Harris was trained as a philosopher) tend to feel true analytic identities are rare but are not the only kind of necessary truths. For instance, the fact that "water is H20" is widely regarded as a necessary truth that isn't analytic (do a search if you want an explanation) and there are any number of other philosophical arguments that are seen as establishing necessary truths which don't amount to the definitional relationship you demand.
I think the standard Harris is using is much weaker even than that.
-
You insist that to be an ought it must be motivating for the subject. This is a matter of some debate. Some moral realists would endorse this while others would insist that it need only motivate certain kinds of agents who aren't too screwed up in some way. But, I tend to agree with your conclusion just suggest it be qualified by saying we presuming the standard sense of moral realism here.
-
One has to be really careful with what you mean by 'science' here. One way people have snuck around the is-ought gap before is using terms like cruel which are kinda 'is' facts that back in an ought (to be cruel requires that you immorally inflict suffering etc..).
-
It's not that Harris is purely embedded in some kind of dialectical tradition. He was trained as an analytic philosopher and they invented the is-ought gap and are no strangers to the former mode of argumentation. IT's more that Carrol is a physicist and doesn't know the terminology that would let him pin Harris down in terms he would understand and keep him from squirming off the point.
However, I'm pretty sure (based on my interaction with Harris emailing him over what sounded like a similarly wrongheaded view in the philosophy of mind) that Harris would admit that he hasn't bridged Hume's is-ought gap as philosophers understand it but instead explain that he means to address the general public's sense that science has no moral insight to offer.
In that sense I think he is right. Most people don't realize how much science can inform our moral discussions...he's just being hyperbolic to sell it.
I agree with your general thrust except your statement that "you longtermists can simply forgo your own pleasure wireheading and instead work very hard on the whole growth and reproduction agenda" if we are able to wirehead in an effective manner it might be morally obligatory to force them into wireheading to maximize utility.
Also, your concern about some kind of disaster caused by wireheading addiction and resulting deaths and damage is pretty absurd.
Yes, people are more likely to do drugs when they are more available but even if the government can't limit the devices that enable wireheading from legal purchase it will still require a greater effort to put together your wireheading setup than it currently does to drive to the right part of the nearest city (discoverable via google) and purchasing some heroin. Even if it did become very easy to access it's still not true that most people who have been given the option to shoot up heroin do so and the biggest factor which deters them is the perceived danger or harm. If wireheading is more addictive/harmful it will discourage use.
Moreover, for wireheading to pose a greater danger than just going to buy heroin it would have to give greater control over brain stimulation (i.e. create more pleasure etc..) and the greater our control over the brain stimulation the greater the chance we can do so in a way that doesn't create damage.
Indeed, any non-chemical means of brain stimulation is almost certain to be crazily safe because once monitoring equipment detects a problem you can simply shut off the intervention without the concern of long-halflife drugs remaining in the system continuing the effect.
You make a lot of claims here that seem unsupported and based on nothing but vague analogy with existing primitive means of altering our brain chemisty. For instance a key claim that pretty most of your consequences seem to depend on is this: "It is great to be in a good working mood, where you are in the flow and every task is easy, but if one feels “too good”, one will be able only to perform “trainspotting”, that is mindless staring at objects.
Why should this be true at all? The reason heroin abusers aren't very productive (and, imo, heroin isn't the most pleasurable existing drug) is because of the effects opiates have as depressants making them nod off etc.. The more control we achieve over brain stimulation the less likely wireheading will have the kind of side-effects which limit functioning. Now one might have a more subtle argument that suggests the ability of even a directly stimulated brain to feel pleasure will be limited and thus if we directly stimulate too much pleasure we will no longer have the appropriate rewards to incentivize work but it seems equally plausible that we will be able to seperate pleasure and motivation/effort and actually enhance our inclination to work while instilling great pleasure.
Skimming the paper I'm not at all impressed. In particular, they make frequent and incautious use of all sorts of approximations that are only valid up to a point under certain assumptions but make no attempt to bound the errors introduced or justify the assumptions.
This is particularly dangerous to do in the context of trying to demonstrate a failure of the 2nd law of thermodynamics as the very thought experiment that might be useful will generally break those heuristics and approximations. Worse, the 2nd law is only a statistical regularity not a true exceptionless regularity so what one actually needs to show.
Even worse this piece seems to be trying to use a suspicious mixture of quantum and classical notions, e.g., using classical notions to define a closed system then analyzing it as a quantum system
Not everyone believes that everything is commesurable and people often wish to be able to talk about these issues without implicitly presuming that fact.
Moreover, values suggests something that is desirable because it is a moral good. A priority can be something I just happen to selfishly want. For instance, I might hold diminishing suffering as a value yet my highest current priority might be torturing someone to death because they killed a loved one of mine (having that priority is a moral failing on my part but doesn't make it impossible).
Two thoughts.
First, as a relatively in shape person who walks a ton (no car living in the midwest) I can attest that I often wish I had a golf cart/scooter solution. They don't need to be a replacement for walking (though good that they can be) they might also appeal to those of us who like to walk a lot but need a replacement for a car when it gets really hot or we need to carry groceries (motorcycle style scooters require licenses and can't always be driven on campuses or parks). It would be great if these became less socially disapproved of for the non-disabled.
Second, aren't there stable, fast scooters with decent torque and larger battery packs? Why do you think the crappy scooter will become super popular? Is it that much cheaper? Or are you just saying even a crappy scooter provides these advantages.
Except if you actually go try and do the work people's pre-theoretic understanding of rationality doesn't correspond to a single precise concept.
Once you step into Newcomb type problems it's no longer clear how decision theory is supposed to correspond to the world. You might be tempted to say that decision theory tells you the best way to act...but it no longer does that since it's not that the two-boxer should have picked one box. The two-boxer was incapable of so picking and what EDT is telling you is something more like: you should have been the sort of being who would have been a one boxer not that *you* should have been a one boxer.
Different people will disagree over whether their pre-theoretic notion of rationality is one in which it is correct to say that it is rational to be a one/two boxer. Classic example of working with a imprecisely defined concept.
First, let me say I 100% agree with the idea that there is a problem in the rationality community of viewing rationality as something like momentum or gold (I named my blog rejectingrationality after this phenomena and tried to deal with it in my first post).
However, I'm not totally sure everything you say falls under that concept. In particular, I'd say that rationality realism is something like the belief that there is a fact of the matter about how best to form beliefs or take actions in response to a particular set of experiences and that many facts about this (going far beyond don't be dutch booked). With the frequent additional belief that what is rational to do in response to various kind of experiences can be inferred by a priori considerations, e.g., think about all the ways that rule X might lead you wrong in certain possible situations so X can't be rational.
When I've raised this issue in the past the response I've gotten from both Yudkowsky and Hanson is: "But of course we can try to be less wrong," i.e., have less false beliefs. And of course that is true but that's a very different notion than the notion of rationality used by rationality realists and misses the way that much of the rationality's community's talk about rationality isn't about literally being less wrong but about classify rules for reaching beliefs into rational and irrational even when they don't disagree in the actual world.
In particular, if all I'm doing is analyzing how to be less wrong I can't criticize people who dogmatically believe things that happen to be true. After all, if god does exist, than dogmatically believing he does makes the people who do less wrong. Similarly the various critiques of human psychological dispositions as leading us to make wrong choices in some kinds of cases isn't sufficient if those cases are rare and cases where it yields better results are common. However, those who are rationality realists suggest that there is some fact of the matter which makes these belief forming strategies irrational and thus appropriate to eschew and criticize. But, ultimately, aside from merely avoiding getting dutch booked, no rule for belief forming can assure it is less wrong than another in all possible worlds.
If you want to argue against that piece of reasoning give it a different name because it's not the Chinese room argument. I took multiple graduate classes with professor Searle and, while there are a number of details Said definitely gets the overall outline correct and the argument you advanced is not his Chinese room argument.
That doesn't mean we can't talk about your argument just don't insist it is Searle's Chinese room argument.
To the extent they define a particular idealization it's one which isn't interesting/compelling. What one would want to have to say there was a well defined question here is a single definition of what a rational agent is that everyone agreed on which one could then show favored such and such decision theory.
To put the point differently you and I can agree on absolutely every fact about the world and mathematics and yet disagree about which is the best decision theory because we simply mean slightly different things by rational agent. Moreover, there is no clear practical difference which presses us to use one definition or another like the practical usefulness of the aspects of the definition of rational agreement which yield the outcomes that all the theories agree on.
Obviously you can and if you define that NEW idealization an X-agent (or more likely redefine the word rationality in that situation) and then there may be a fact of the matter about how an X-agent will behave in such situations. What we can't do is assume that there is a fact of the matter about what a rational agent will do that outstrips the definition.
As such it doesn't make sense to say CDT is right or TDT or whatever before introducing a specific idealization relative to which we can prove they give the correct answer. But that idealization has to come first and has to convince the reader that it is a good idealization.
But the rhetoric around these decision theories misleadingly tries to convince us that there is some kind of pre-existing notion of rational agent and they have discovered that XDT gives the correct answer for that notion. That's what makes people view these claims as interesting. If the claim was nothing more than 'here is one way you can make decisions corresponding to the following assumptions" it would be much more obscure and less interesting.
Your criticism of the philosophy/philosophers is misguided on a number of accounts.
1. You're basing those criticisms on the presentation in a video designed to present philosophy to the masses. That's like reading some phys.org arg article claiming that electrons can be in two locations at once and using that to critisize the theory of Quantum Mechanics.
2. The problem philosophers are interested in addressing may not be the one you are thinking of. Philosophers would never suggest that the assumption of logical omniscience prevents one from using Bayesianism as a practical guide towards reasoning or that it's not often a good idealization to treat degrees of belief as probabilities. However, I believe the question that this discussion is in relation to is in giving a theory that explains the fundamental nature of probability claims and here the fact that we really aren't logically omniscient prevents us from identifying probabilities with something like rational degrees of belief (though that proposal has other problems too).
3. It's not like philosophers haven't put in plenty of effort looking for probability like systems that don't presume logical omniscience. They have developed any number of them but none seem particularly useful and I'm not convinced that the paper you link about this will be much different (not that they are wrong just not that useful).
In particular, I'd argue that the paradoxical aspects of Newcomb's problem result from exactly this kind of confusion between the usual agent idealization and the fact that actual actors (human beings) are physical beings subject to the laws of physics. The apparent paradoxical aspects results because we are used to idealizing individual behavior in terms of agents where that formalism requires we specify the situation in terms of a tree of possibilities with each path corresponding to an outcome and with the payoff computed by looking at the path specified by all agent's choices (e.g. there is a node where the demon player chooses what money to put in the boxes and then there is a node where the human player, without knowledge of demon player's choices, decides to take both boxes or neither). The agent formalization (where 1 or 2 boxing is modeled as a subsequent choice) simply doesn't allow the content of the boxes to depend on whether or not the human agent chooses 1 or 2 boxes.
Of course, since actual people aren't ideal agents one can argue that something like the newcomb demon is physically possible but that's just a way of specifying that we are in a situation where the agent idealization breaks down.
This means there is simply no fact of the matter about how a rational agent (or whatever) should behave in newcomb type situations because the (usual) rational agent idealization is incompatible with the newcomb situation (ok, more technically you can model it that way but the choice of how to model it just unsatisfactorily builds in the answer by specifying how the payoff depends on 1 vs two boxing).
To sum up what the answer to the newcomb problem is depends heavily on how you preciscify the question. Are you asking whether humans who are disposed to decide in way A end up better of than humans disposed to behave in way B? In that case it's easy. But things like CDT, TDT etc.. don't claim to be producing facts of that kind but rather saying something about ideal rational agents of some kind which then just boringly depends on a ambiguities in what we mean by that.ideal rational agents.
It doesn't really make sense to talk about the agent idealization at the same time as talking about effective precommitment (i.e. deterministic/probabilistic determination of actions).
The notion of an agent is an idealization of actual actors in terms of free choices, e.g., idealizing individuals in terms of choices of functions on game theoretic trees. This is an incompatible idealization with thinking of such actors as being deterministically or probabilistically committed to actions for those same 'choices.'
Of course, ultimately, actual actors (e.g. people) are only approximated by talk of agents but if you try and simultaneously use the agent idealization while regarding those *same* choices as being effectively precommited you risk contradiction and model absurdity (of course you can decide to reduce the set of actions you regard as free choices in the agent idealization but that doesn't seem to be the way you are talking about things here).
You are getting the statement of the Chinese room wrong. The claim isn't that the human inside the room will learn Chinese. Indeed, it's a key feature of the argument that the person *doesn't* ever count as knowing Chinese. It is only the system consisting of the person plus all the rules written down in the room etc.. which knows Chinese. This is what's supposed to (but not convincingly IMO) be an unpalatable conclusion.
Secondly, no one is suggesting that there isn't an algorithm that can be followed which makes it appear as if the room understands Chinese. The question is whether or not there is some conscious entity corresponding to the system of the guy plus all the rules which has the qualitative experience of understanding the Chinese words submitted etc.. As such the points you raise don't really address the main issue.
We can identify places we know (inductively) tend to lead us astray and even identify tricks that help us avoid being affected by common fallacies which often aflict humans. However, it's not at all clear if this actually makes us more rational in any sense.
If you mean act-rationality we'd have to study if this was a good life choice. If you mean belief rationality you'd have to specify some kind of measure/notion of importance to decide when it really matters you believed the true thing. After all if it's just maximizing the number of times you believe the truth the best way to be rational is just to memorize giant tables of dice rolls. If it's minimizing false beliefs you might want to avoid forming any at all. Even if you find some more appropriate function to maximize some beliefs obviously should count more than others. I mean you don't want to spend your time memorizing lists of dice roles and forget the facts about being killed by buses if you walk into the street.
But once you realize this point then who knows. It could be the most rational thing in the world to have a totally dogmatic, evidence irresponsive, belief in the existence of some beardy dude in the sky because it's the belief that matters the most and the rule "always believe in God Beard" will thus maximize getting important beliefs right.
I know what you mean. You mean something like avoiding the kind of fallacies that people who always talk about fallacies care about avoiding. But why should those be the most important fallacies to combat etc..
The inveresly proportional thing is a bad move. Sorting through potential charitable causes is itself charitable work and it's just crazy inefficient to do that by everyone voting on what tugs at their heartstrings but by paying someone smart to consider all the various pet causes and evaluate them. Worse, the causes that are least well known are often unknown for very good reason but will now get special attention.
The reason you are right about cases like the doctor example is that when you are actually in a location that then gets hit you *are* leveraging your superior knowledge of how to get things done there or even just understanding what's happened. Thought, truthfully, the real reason it makes sense is the easier psychological motivation.
Note that your whole delegation argument rests on the idea that you have (and know you have) some kind of superior knowledge (or virtue) about what needs to get done and you're just searching for the best way to get it done. The reason it made sense to stay involved in the local campaign was because you had the special advantage of being the person who knew the right way to improve the city so you could offer something more than any other equally virtuous person you might hand the money to instead..
In contrast, in the village case you *don't* have any special knowledge. If we just assigned everyone randomly to someone else who got to spend their charitable givings on the causes they favored (absent fraud) we would expect the world to be no better or worse. Just picking *any* reputable (easy to find online) EA charity or cause or even person and send them all your money won't make things worse and by amassing money from many people they avoid all the transaction costs of everyone trying to do the calculations.
Why assume whatever beings simulated us evolved?
Now I'm sure you're going to say well a universe where intelligent beings just pop into existence fully formed is surely less simple than one where they evolve. However, when you give it some more thought that's not true and it's doubtful if Occam's razor even applies to initial conditions.
I mean supposed for a moment the universe is perfectly deterministic (newtonian or no-collapse interp). In that case the Kolmogorov complexity of a world starting with a big bang that gives rise to intelligent creatures can't be much less and probably is much more than one with intelligent creatures simply popping into existence fully formed After all, I can always just augment the description of the big bang initial conditions with 'and then run the laws of physics for x years' when measuring the complexity.
No, because we want the probability of being a simulation conditional on having complex surroundings not the probability of having complex surroundings conditional on a simulation. The fact that a very great number of simulated beings are created in simple universes doesn't mean that none is ever simulated in a complex one or tell us anything about whether being such a simulation is more likely than being in a physical universe.
Ok, this is a good point. I should have added a requirement that the true solution is C infinity on the part of the manifold that isn't in our temporal past. The backward's heat equation is ill-posed for this reason on...it can't be propogated arbitrarily far forward (i.e. back).
Which way you think this goes probably depends on just how strongly you think Occam's razor should be applied. We are all compelled to let the probability of a theory's truth go to zero as it's kolmogorov complexity goes to infinity but there is no prima facia reason to think it drops off particularly fast or slow. If you think , as I do, that there is only relatively weak favoring of more simple scientific laws while intelligent creatures would favor simplicity as a cognitive technique for managing complexity quite strongly you get my conclusion. But I'll admit the other direction isn't implausible.
The problem with this kind of analysis is that one is using the intuition of a physical scenario to leverage an ambiguity in what we mean by agent and decision.
Ultimately, the notion of decisions and agents are idealizations. Any actual person or AI only acts as the laws of physics dictate and agents, decisions or choices don't appear in any description in terms of fundamental physics. Since people (and programs) are complex systems that often make relatively sophisticated choices about their actions we introduce the idealization of agents and decisions.
That idealization is basically what one sees in the standard formulation of game theory in terms of trees, visibility conditions and payoffs with decisions simply being nodes on the tree and agents being a certain kind of function from visible outcomes and nodes to children of those nodes. The math is all perfectly clear and there is nothing paradoxical or troubling.
What makes it seem like there is a problem is when we redescribe the situation in terms of guarantees the other player will have predicted your choice in a certain way or the like. Formally, that doesn't really make sense...or at least it corresponds to a radically different game, e.g., restricting the tree so that only those outcomes are allowed. However, because we have this other non-formal notion of choice and agent stuck in our heads (choice is something like picking what socks to wear agent is something like a person) we don't realize that our idealization just changed drastically even though in common language we are still playing the same game.
In other words there are no extra facts to be found about which decision theory is best. There are facts about what actual physical systems will do and there are mathematical facts about trees and functions on them but there isn't any room for further facts about what kind of decision theory is the true one.
I worry such a plan will face significant legal hurdles. As suggested the building would probably not fall into the exceptions to the federal fair housing act (is that right) for choosing roommates (it's not a single family dwelling but a group of apartments in some sense).
But you EXACTLY want to choose who lives there based on political/religious beliefs (almost by definition it's impossible to be a rationalist and a dogmatic unquestioning conservative christian). Also by aspects of family makeup in that you don't want people living in this community to import a large extended family to live with them if that family doesn't share the values/concerns of the other people living in the community.
Basically, I think the degree of control you want over who lives in the building may be incompatible with various non-discrimination laws. However, one could probably find 20 families that could jointly purchase the building as condos to avoid this problem.
But I don't see any way around these problems in the long run. As the original group breaks up it will be hard to replace them without legally problematic screening.
Sorry, but you can't get around the fact that humans are not well equipped to compute probabilities. We can't even state what our priors are in any reasonable sense much less compute exact probabilities.
As a result using probabilities has come to be associated with having some kind of model. If you've never studied the question and are asked how likely you think it is there are intelligent aliens you say something like "I think it's quite likely". You only answer with a number if you've broken it down into a model (chance life evolves average time to evolve intelligencechance of disaster*..).
Thus, saying something like "70% chance" indicates to most people that you are claiming your knowledge is the result of some kind of detailed computation and can thus be seen as an attempt to claim authority. You can't change this rule on your own.
Thankfully, there are easy verbal alternatives. "Ehh, I guess I would give 3:1 odds on it" and many others. But use of chance/probability language isn't it.
Uhh, why not just accept that you aren't and can never be perfectly rational and use those facts in positive ways.
Bubbles are psychologically comforting and help generate communities. Rationalist bubbling (which ironically includes the idea that they don't bubble) probably does more to build the community and correct other wrong beliefs than almost anything else.
Until and unless rationalist take over society the best strategy is probably just to push for a bubble that actively encourages breaking other (non-rationalist) bubbles.
So the equations should be (definition of vaccine efficacy from wikipedia)
.6 * p(sick2) = p(sick2) - p(sick1)
p(sick1) - .4 p(sick2) = 0
.
i.e. efficacy is the difference be the unvaccinated and vacinated rates of infection divided by the unvaccinated rate. You have to assume there is no selective pressure in terms of who gets the vaccine (they have the same risk pool as the normal population for flu which is surely untrue) to get your assumtion that
.42 p(sick1) + .58p(sick2) = .1 p(sick1) + 1.38p(sick2) = .238
or 1.78 p(sick2) = .238
p(sick2)=.13 (weird I getting a different result) p(sick1) = .05
Did I solve wrong or did you. I do math so I can't actually manipulate numbers very well but I not seeing the mistake.
Not with respect to their revealed preferences for working in high risk jobs I understand. There are a bunch of economic papers on this but it was a surprisingly low number.