Posts
Comments
An extremely low prior distribution of life is an early great filter.
Done. Thank you for running these.
Check out the previous discussion Luke linked to: http://lesswrong.com/lw/c45/almost_every_moral_theory_can_be_represented_by_a/
It seems there's some question about whether you can phrase deontological rules consequentially-- to make this more formal that needs to be settled. My first thought is that the formal version of this would say something along the lines of "you can achieve an outcome that differs by only X%, with a translation function that takes rules and spits out a utility function, which is only polynomially larger." It's not clear to me how to define a domain in such a way as to allow you to compute that X%.
...unfortunately, as much as I would like to see people discuss the moral landscape instead of the best way to describe it, I have very little time lately. :/
(Sorry for slow response. Super busy IRL.)
If a consequentialism talks about murder being bad, they mean that it's bad if anybody does it.
Not necessarily. I'm not saying it makes much sense, but it's possible to construct a utility function that values agent X not having performed action Y, but doesn't care if agent Z performs the same action.
It is technically true that all of these ethical systems are equivalent, but saying which ethical system you use nonetheless carries a lot of meaning.
a) After reading Luke's link below, I'm still not certain if what I've said about them being (approximately) isomorphic is correct... b) Assuming my isomorphism claim is true enough, I'd claim that the "meaning" carried by your preferred ethical framework is just framing.
That is, (a) imagine that there's a fixed moral landscape. (b) Imagine there are three transcriptions of it, one in each framework. (c) Imagine agents would all agree on the moral landscape, but (d) in practice differ on the transcription they prefer. We can then pessimistically ascribe this difference to the agents preferring to make certain classes of moral problems difficult to think about (i.e., shoving them under the rug).
Deontology and virtue ethics don't care about getting things done.
I maintain that this is incorrect. The framework of virtue ethics could easily have the item "it is virtuous to be the sort of person who gets things done." And "Make things happen, or else" could be a deontological rule. (Just because most examples of these moral frameworks are lame doesn't mean that it's a problem with the framework as opposed to the implementation.)
If indeed the frameworks are isomorphic, then actually this is just another case humans allowing their judgment to be affected by an issue's framing. Which demonstrates only that there is a bug in human brains.
I think so. I know they're commonly implemented without that feedback loop, but I don't see why that would be a necessary "feature".
Which is why I said "in the limit". But I think, if it is true that one can make reasonably close approximations in any framework, that's enough for the point to hold.
Hm, thanks.
Are you saying that some consequentialist systems don't even have deontological approximations?
It seems like you can have rules of the form "Don't torture... unless by doing the torture you can prevent an even worse thing" provides a checklist to compare badness ...so I'm not convinced?
How does it change the numbers if you condition on the fact that Alcor has already been around for 40 years?
Absolutely, granted. I guess I just found this post to be an extremely convoluted way to make the point of "if you maximize the wrong thing, you'll get something that you don't want, and the more effectively you achieve the wrong goal, the more you diverge from the right goal." I don't see that the existence of "marketing worlds" makes maximizing the wrong thing more dangerous than it already was.
Additionally, I'm kinda horrified about the class of fixes (of which the proposal is a member) which involve doing the wrong thing less effectively. Not that I have an actual fix in mind. It just sounds like a terrible idea--"we're pretty sure that our specification is incomplete in an important, unknown way. So we're going to satisfice instead of maximize when we take over the world."
90% agree, one other thing you may not know: both dropbox and google drive have options to automatically upload photos from your phone, and you don't have to sync your desktop with them. So it's not clear that they merely double the needed space.
I think your expanded point #6 fails to consider alternative pressures for hard drive & flash memory. Consider places like dropbox; they represent a huge demand for cheap storage. People probably (?) won't want huge(er) drives in their home computers going forward, but they are quite likely to want cloud storage if it comes down another order of magnitude in price. Just because people don't necessarily directly consume hard drives doesn't mean there isn't a large demand.
Consider also that many people have high MP digital cameras, still and video. Those files add up quickly.
It sounds like, "the better you do maximizing your utility function, the more likely you are to get a bad result," which can't be true with the ordinary meanings of all those words. The only ways I can see for this to be true is if you aren't actually maximizing your utility function, or your true utility function is not the same as the one you're maximizing. But then you're just plain old maximizing the wrong thing.
Hypocrisy is only a vice for people with correct views. Consistently doing the Wrong Thing is not praiseworthy.
Unfortunately, it's much easier to demonstrate inconsistency than incorrectness.
Ah, thank you for the explanation. I have complained about the proposed method in another comment. :)
http://lesswrong.com/lw/jao/siren_worlds_and_the_perils_of_overoptimised/aso6
The IC correspond roughly with what we want to value, but differs from it in subtle ways, enough that optimising for one could be disastrous for the other. If we didn't optimise, this wouldn't be a problem. Suppose we defined an acceptable world as one that we would judge "yeah, that's pretty cool" or even "yeah, that's really great". Then assume we selected randomly among the acceptable worlds. This would probably result in a world of positive value: siren worlds and marketing worlds are rare, because they fulfil very specific criteria. They triumph because they score so high on the IC scale, but they are outnumbered by the many more worlds that are simply acceptable.
Implication: the higher you set your threshold of acceptability, the more likely you are to get a horrific world. Counter-intuitive to say the least.
TL;DR: Worlds which meet our specified criteria but fail to meet some unspecified but vital criteria outnumber (vastly?) worlds that meet both our specified and unspecified criteria.
Is that an accurate recap? If so, I think there's two things that need to be proven:
There will with high probability be important unspecified criteria in any given predicate.
The nature of the unspecified criteria is such that it is unfulfilled in a large majority of worlds which fulfill the specified criteria.
(1) is commonly accepted here (rightly so, IMO). But (2) seems to greatly depend on the exact nature of the stuff that you fail to specify and I'm not sure how it can be true in the general case.
EDIT: The more I think about this, the more I'm confused. I don't see how this adds any substance to the claim that we don't know how to write down our values.
EDIT2: If we get to the stage where this is feasible, we can measure the size of the problem by only providing half of our actual constraints to the oracle AI and measuring the frequency with which the hidden half happen to get fulfilled.
Possibly of interest: Help Teach 1000 Kids That Death is Wrong. http://www.indiegogo.com/projects/help-teach-1000-kids-that-death-is-wrong
(have not actually looked in detail, have no opinion yet)
I think you're getting downvoted for your TL;DR, which is extremely difficult to parse. May I suggest:
TL;DR: Treating "computers running minds" as discrete objects might cause a paradox in probability calculations that involve self-location.
I dunno. I'd be pretty happy with a system that produced reasonable output when staffed with idiots, because that seems like a certainty. I actually think that's probably why democracy seems to be better than monarchies-- it has a much lower requirement for smarts/benevolence. "Without suffering" may be a high bar, but the universe is allowed to give us problems like that! (And I don't think that democracy is even close to a complete solution.)
EDIT: Also, perhaps the entirety of the system should be to make sure that an "utter genius with leet rationality skillz" is in the top position? I'd be very happy with a system that caused that even when staffed by morons.
I think we just mean different things by "human level"-- I wouldn't consider "human level" thought running at 1/5th the speed of a human or slower to actually be "human level". You wouldn't really be able to have a conversation with such a thing.
And as Gurkenglas points out, the human brain is massively parallel-- more cores instead of faster cores is actually desirable for this problem.
Ah, I see. Yeah, 1 bit in input bitstream != 1 bit of bayesian evidence.
I think you mean BB(1000) bits of evidence?
1) Yes, brains have lots of computational power, but you've already accounted for that when you said "human-level AI" in your claim. A human level AI will, with high probability, run at 2x human speed in 18 months, due to Moore's law, even if we can't find any optimizations. This speedup by itself is probably sufficient to get a (slow-moving) intelligence explosion.
2) It's not read access that makes a major difference, it's write access. Biological humans probably will never have write access to biological brains. Simulated brains or AGIs probably will have or be able to get write access to their own brain. Also, DNA is not the source code to your brain, it's the source code to the robot that builds your brain. It's probably not the best tool for understanding the algorithms that make the brain function.
3) As said elsewhere, the question is whether the speed at which you can pick the low hanging fruit dominates the speed at which increased intelligence makes additional fruit low-hanging. I don't think this has an obviously correct answer either way.
I see, thanks!
You can't "count every possible program equally".
I did know this and should have phrased my sentence hypothetically. :)
The only programs allowed in the Solomonoff distribution are ones that don't have any extended versions that produce the same output observed so far.
Did not know that! It seems like that would leave some probability mass unassigned, how do you rebalance? Even if you succeed, it seems likely that (for large enough outputs) there'll be lots of programs that have epsilon difference--that are basically the same, for all practical purposes.
I have been thinking that the universal prior is tautological. Given a program, there are an infinite number of longer programs which perform the same computation (or an indistinguishable variation) but only a finite number of shorter programs having this characteristic. If you count every possible program equally, you'll find that each short program represents a host of longer programs. However, now that I write this down, I'm no longer sure about it. Can someone say why/if it's wrong?
I'd like to agree with you, but how do I know you're not a concern troll?
I believe if you read my previous comments, you'll see that they all are attempts to do exactly this. I will bow out of this conversation now.
(Meta: you're tripping my troll sensers. I'm sorry if it's unintentional on your part. I'm just not getting the sense that you're trying to understand me. Or it's the case that the two of us just really cannot communicate in this forum. Either way, it's time to call it quits.)
EDIT: Your response to this has caused my P(you're trolling me) to rise from ~60% to ~95%.
Ah, you're disagreeing with the model and phrasing it as "if that model were true, no one would sell you btc, but people are willing to sell, therefore that model is false." Do I understand?
If so, I do not agree that "if that model were true, no one would sell you btc" is a valid inference.
You have mentioned the Mathematical Universe hypothesis several times, and Tegmark's is a name very much associated with it, ...
Right, I know what the MUH is, I know who Tegmark is, I just don't recognize terms that are a combination of his name and (im)materialism. Please taboo your terms! I don't know what they mean to you!
If anything, let's call my position "non-distinctionalism"-- I maintain that there's no other coherent models for the word "exist" than the one I mentioned earlier, and people who use the word "exist" without that model are just talking gibberish. There's no distinction between "material existence" and "immaterial existence" in the sense that the first clause of this sentence is meaningless noise.
You second sentence doens't follow from your first. Someone can define "material existence" as existence in your sense, plus some additional constraint, such as the "world" in which the pattern is found being a material world.
Yes, if you look at the last paragraph, I (tried to) say explicitly that is my desired state of the terms, it's just not the standard usage, afaik.
So, if I were writing the terms, "material existence" = exists in our sub-branch of the MUH, "immaterial existence" = exists in a sub-branch of the MUH inaccessible to us (except via simulation). However, I don't think people ordinarily use the word "material" to mean this.
Actually, now that I think about it, I think that's a bad way to think of things. Really, the concept those terms would express is accessibility, not existence. When I hear other people say things like "immaterial existence", my brain interprets that as "exists, but not in a branch of the MUH," which I think is gibberish.
What Vaniver said. Also, emperically, you can look at the current price/order book on an exchange and see that people are in fact willing to sell you these things. If my holdings represented a life altering sum of money it would be time to take less risk and I would be one of those people.
The computer doesn't hold cash (clearly), it has account # and password of a bank account (or private key to bitcoin addresses if it's a particularly risky computer). The two thin computers therefore only have half as much money to bet. (Or they'll overdraw their account.)
Bitcoin is plenty liquid right now unless you're throwing around amounts > $1 mil or so.
Thanks for editing-- I'm still puzzled.
I also don't know what "Tegmarkian immaterialism" is and I'm not arguing for or against it. I do not know what "immaterialism" is and I'm also not arguing for or against that. (Meta: stop giving either sides of my arguments names without first giving the names definitions!)
If anything, let's call my position "non-distinctionalism"-- I maintain that there's no other coherent models for the word "exist" than the one I mentioned earlier, and people who use the word "exist" without that model are just talking gibberish. There's no distinction between "material existence" and "immaterial existence" in the sense that the first clause of this sentence is meaningless noise. I can be disproved by being informed of another coherent model. I maintain that my thought experiment shows that it's difficult to hold any distinction between exists-in-reality and exists-in-the-mathematical-multiverse.
(If I were king of the world, "immaterial existence" would mean "exists in an inaccessible place of the mathematical universe", but for me to use the term that way would currently be idiosyncratic and just confuse everyone further.)
I think we are having very significant communication difficulties. I am very puzzled that you think that I think that I'm arguing against MUH. I think something like MUH is likely true. I do not know what "Tegmarkian materialism" is and I'm not defending or attacking it. I also cannot make sense of some of your sentences, there seems to be some sort of editing problem.
So far, none of this tells us what immateriality is. But then it isn't easy to say what matter is either.
Yeah. There's supposedly two mysterious substances. My claim is that I can't see a reason to claim they're separate, and this thought experiment is (possibly) a demonstration that they're in fact same thing. Then we still have one mysterious substance, and I'm not claiming to make that any less mysterious with this argument.
There are more than two options. If you had evidence of a bitstring corresponding to a billions of years of biological development involving trillions of organims -- amuch more comple bitstring than a mind, but not a mind,-- it might well be most probable to assign the production of a mind to that.
I don't know if you realise it, but your argument was Paleyian
Whoa, I think you understood something pretty different from what I was trying to say. I was definitely not claiming a deity of some sort must have been responsible! Let me repeat with unambiguous labels:
(1) Given that you've found a bitstring in a simulation that represents mind M existing, taking actions, and feeling things, ... it's therefore (4) vastly more probable that mind M actually performed computations in the course of producing that bitstring, than it is that you found it randomly.
Yes, of course mind M will have been the result of some evolutionary process in any universe I can imagine finding via simulation, but that dosen't make mind M less real. I suppose you could probably see this as a special case/expansion of the Anti Zombie Principal-- any system that produces the outputs of a mind (at least) contains that mind.
(If you meant something else by the Paleyian comment, you'll have to spell it out for me. I'm not the one that downvoted you and I appreciate the continued interaction.)
What would "immaterial existence" even mean?
I don't know exactly, but if "material existence" means something, so does "immaterial existence".
Hm. I don't think "material existence", if it's a thing, has a unique opposite.
I guess I'd define exists-in-the-real-world as equivalent to a theoretical predicate function that takes a model of a thing, and a digitized copy of our real world, and searches the world for patterns that are functionally isomorphic (given the physics of the world) to the model, and returns true iff it finds one or more.
This model of existence doesn't work if you don't supply the real world (or at least a world) as an argument. I'm interpreting "immaterial existance" as "exists, but not in a world" which seems like a logical impossibility to me. Of course, this is a function of how I've defined "exists", but I don't know of a better way to define it.
Rather than assuming that there are infinite numbers of real but immaterial people floaitng around somewhere, I prefer to assume that "memories" are just data that don't have any intrinsic connection to prior events.. Ie, a memory proper is a record of an event, but neurons can be configured as if there were a trace of an even that never happened.
OK, that's a reasonable position. I'll adjust my argument. My claim now is:
(1) Given that you've found a bitstring in a simulation that represents a mind existing, taking actions, and feeling things,
(2) this bitstring is quite astonishingly long,
(3) most long bitstrings do not similarly describe minds by any reasonable mapping function,
it's therefore (4) vastly more probable that a mind actually ran to produce that bitstring, than it is that you found it randomly.
Basically, I'm treating the outputs of a mind as being something like a proof that there was a mind running. Similarly to the idea that publishing an SHA-256 hash of some data is proof that you had that data at the time that you published the hash.
What would "immaterial existence" even mean?
I think my claim is that the above argument shows that whatever that might be, it's equivalent to epistemological objectivism.
Specifically, to believe that they're separate, given the scenario where you simulate universes until you find a conscious mind and then construct a replica in your own universe, you have to believe both of the following at the same time:
(1) Mind X didn't have real memories/experiences until you simulated it in the "real" world (i.e., yours), and (2) proof of mind X's running existed previously to you computing it (in the form of an execution history).
To me, accepting both points requires me to believe something like "Proofs that I don't know about aren't true", and I'll be happy if you can show me why that's not true.
Perhaps this isn't obvious, but note that fully validating nodes do not need to store the entire block chain history, currently, just the set of unspent transaction outputs, and there are proposals to eliminate the need for validators to store even that. If it's just a gentleman's agreement then it would be doable but wouldn't really have any teeth against a motivated attacker.
That's a good point. To make this work, it'd probably make the most sense to treat the pre-published hash the same as unspent outputs. It can't be free to make these or you could indeed DoS bitcoin.
I did not know you could have zero value outputs. I'll look into that. (And don't worry, I wasn't planning on destroying any coins!)
Infinitely is a bit of an overstatement, especially if there's a fee to store a hash. I agree it might still be prudent to have a time limit, though. Miners can forget the hash once it's been referenced by a transaction.
The ability to pay to store a hash in the blockchain could be interesting for other reasons, like proving knowledge at a particular point in the past. There's some hacky ways to do it now that involve sending tiny amounts of BTC to a number of addresses, or I suppose you could also hash your data as if it were a pubkey and send 1e-8 btc to that hash-- but that's destructive.
Thanks for the explanation, it seems like I'm not wildly misreading wikipedia. :)
It seems like the more qubits are required for this attack, the more likely we are to have a long warning time to prepare ourselves. The other attack of just cracking the pubkey when a transaction comes through and trying to beat the transaction, seems vastly more likely to be an actual problem.
Do you have any idea how I'd go about estimating the number of qubits required to implement just the SHA256(SHA256(...)) steps required by mining?
They don't.
Suppose you publish hash HT1 of a transaction T1 spending address A, and then several blocks later when you publish T1 itself, someone hacks your pubkey and publishes transaction T2 also spending address A. Miners would hypothetically prefer T1 to T2, because there's proof that T1 was made earlier.
In the case where someone had even earlier published hash HT0 of transaction T0 also spending address A, but never bothers to publish T0 (perhaps because their steal bot--which was watching for spends from A--crashed), well, they're out of luck, because A no longer has funds as soon as T1 is included in the blockchain.
The pre-published hashes would be used only to break ties among transactions not yet included in any blocks.
Also, in this hypothetical scenario, the steal-bot from above means you shouldn't trust funds that haven't yet been moved into a quantum-computer-safe address; you wouldn't know who all might have a old hash published with a bot just waiting to spend your funds as soon as you try to move them.
I always forget about the RIPEMD-160 step. The bitcoin wiki claims that it's strictly more profitable to mine than to search for collisions, but it seems to me that that's a function of block reward vs. value of address you're trying to hack, so I don't know if I believe that. https://en.bitcoin.it/wiki/Technical_background_of_Bitcoin_addresses
It's unclear to me how you would actually implement this in a quantum computer; do you have to essentially build a set of quantum gates that implement RIPEMD-160(SHA256(pubkey))? Does this imply you need enough qubits to hold all stages of the calculation in memory at the same time? I haven't been able to figure out from wikipedia how I'd arrange qubits to do a single hashing operation. Given that it's a lot of sequential steps, perhaps it actually takes a lot of qubits chained together?
I think you misunderstood me-- the transaction could still be rejected when you try to get it included in a subsequent block if it's not valid. The hash of the transaction is just to prove that the transaction is the first spend from the given address; the transaction doesn't/can't get checked when the hash is included in the blockchain. Miners wouldn't be able to do it for free-- the protocol addition would be that you pay (from a quantum-safe address) to include a hash of a transaction into the blockchain. You publish the actual transaction some number of blocks later.
It really only makes sense as an emergency "get your coins into an address that's quantum computer proof" sort of addition. Hopefully the problem is solved and everyone moves their funds before it becomes an emergency.
Me three-- I thought I was the only one, where are we all hiding? :)
That's true, but there's some precedent for picking a block that everyone agrees upon (that exists before quantum computers) and changing the rules at that block to prevent someone from rewriting the blockchain. A lot depends on how much warning we have.
It looks like making a cryptocoin that can survive quantum computers might be a high value activity.
I think a gigahertz quantum computer would have the hashing power of the current bitcoin network.
My math agrees with you. Looks like I was underestimating the effect of quantum computers.
Difficulty is currently growing at 30-40% per month. That won't last forever, obviously, but we can expect it to keep going up for a while, at least. (https://blockchain.info/charts/hash-rate) Still, it looks like you'd need an unrealistic amount of ASICs to match the output of just 1000 quantum computers.
Given that, there'll probably be a large financial incentive to mine bitcoins with the first quantum computers. The incentive goes away if you destroy the network in the process, though. It seems difficult to predict what will happen. Single pools have controlled > 50% of the mining briefly in the past.
...given such a disruption, I don't see much point to salvaging the old currency, rather than starting over.
A lot of unhappy btc holders may see a point, though. :)
This is not completely true-- since only hashes of the public key are posted until funds are spent from an address
So what? Then the attacker waits till someone spents his funds and double spends them and gives the double spending transaction a high processing fee.
This can be fixed by a protocol addition, which can be implemented as long as there's warning. (First publish a hash of your transaction. Once that's been included in a block, publish the transaction. Namecoin already does something like this to prevent that exact attack.)