A seems like more of a theorem than an axiom: We are trying to maximize our estimate of human preferences, so upward errors are going to have more effect on the outcome than downward errors. Therefore, in case of doubt, guess low. And if we keep track of a probability distribution over candidate estimates rather than a point estimate, A dissolves like the Fermi paradox.
In a game where some actors are humans, we should expect the highest stakes at play to sometimes be lost. Or do you think that nuclear war is impossible these days?
At the very top end, if Bill Gates has twice the money he can save ~twice the African lives.
In a market of bettors that draw the line of how much risk to take at different points, the early game will be dominated by the most risk-taking folks and as the game grows older, the line that was chosen by the current winners moves. Perhaps your intuition is merely the product of evolution playing this game for as long as it took for the line to reach its current point?
What do you mean by a sufficiently large multiverse? If your first choice loses many paperclips in 40% of cases and wins's few in the rest, you would take it and a maximizer wouldn't.
Are you rejecting Pascal's mugging because of the prospect of relying on uncertain models that you do not expect to confirm?
Is all your intuition captured by maximizing utility over all but the extreme billionth of the distribution?
Here's a one-shot problem for your intuition to answer: You get to design the probability distribution to draw the number of paperclips from, except that its expectation must be at most its negative kolgomorov complexity. What distribution makes for a good choice?
The cryptographic component of the camera would be manufactured without a private key in place. Instead, there's a write-only memory section that any number of third parties can write their private keys to. That way, we can rely on existing legally trusted institutions like notaries, and nobody should be able to extract too much rent.
This raises the question of how the notary can know that the "cryptographic component" in front of them isn't some scammer's usb stick. Require each cryptographic component to have many compartments, each of which is separate from the others. Breaking the seal on one is to wipe its memory, but not its architecture. Given a supposed cryptographic component, the notary randomly selects a few compartments to destructively verify their architecture. The remaining compartments are told one fresh private key each, and the notary publishes (and signs with his personal private key) that whoever can authenticate himself with at least x of these particular public keys is a trusted camera. To produce fake images, an adversary would have to have subverted at least x compartments that the notary didn't destructively verify. A component can be shipped around to a few notaries so long as nobody breaks enough compartments that fewer than any notary's x are left.
Of course, "Don't roll your own crypto." says all of this is to be interpreted as conjecture.
If Alice and Bob bet fairly on the outcome of a coin, there is no arbitrage.
Any AI genie that can safely interpret wishes written in Arcanic should also be able to interpret English.
To introspect is to observe your evaluations. Since introspection is observation, it is preserved; since it is about your evaluations, its correctness is preserved.
Your duplicate would initially believe itself to be a real boy made of flesh and bone.
Hmm. Yes, I suppose preserving evaluations only always preserves the correctness of those observations which are introspection. Do we have evidence for any components of consciousness that we cannot introspect upon?
2 was supposed to point out how there shouldn't be any components to consciousness that we can miss by preserving all the observations, since if there are any such epiphenomenal components of consciousness, there by definition isn't any evidence of them.
That the reasoning also goes the other way is irrelevant.
No, I would say I can't be less conscious than I observe.
Sure, replacement by silicon could preserve my evaluations, and therefore my observations.
Evaluations can be wrong in, say, the sense that they produce observations that fail to match reality.
Having the same evaluations implies making the same judgements.
Let me decompose 1, 3 => 4 further, then.
3 => 5. Anything that makes the same observations as me is as concious as me.
1 => 6. Anything that has the same evaluations as me makes the same observations as me.
5, 6 => 4.
What step is problematic?
All of them.
1. My observations result from evaluations.
2. I have no reason to believe that I am any more conscious than I can observe.
2 => 3. Changing the parts of me that I cannot observe does not change whether I am conscious.
1, 3 => 4. Anything that has the same evaluations as me is as conscious as me.
Try varying lines 14 and 16 in the interactive script for quicker execution, and try giving it a few example lines to start with.
You mean they would notice that accidentally clicking a button would be bad, and therefore don't use the site? I would expect them to only consider such a reaction once they actually made such a mistake.
I doubt it, but it sure sounds like a good idea to develop a theory of what prompts are more useful/safe.
I think Pattern thought you meant "GPT-2 was trained on sentences generated by dumb programs.".
I expect that a sufficiently better GPT-2 could deduce how to pass a Turing test without a large number of Turing test transcripts in its training set, just by having the prompt say "What follows is the transcript of a passing Turing test." and having someone on the internet talk about what a Turing test is. If you want to make it extra easy, let the first two replies to the judge be generated by a human.
The loss function is computed by comparing its prediction during a training instance to the training label. The loss function is undefined after training. What does it mean for it to minimize the loss function while generating?
The "current inference" is just its predictions about the next byte-pair, yes? Why would it try to bring about future invocations? The concept of "future" only exists in the object-level language it is talking about. The text generation and Turing testing could be running in another universe, as far as it knows. "indistinguishable from the current invocation" sounds like you think it might adopt a decision theory that has it acausally trade with those instances of itself that it cannot distinguish itself from, bringing about their existence because that is what it would wish done unto itself. 1. It has no preference for being invoked; 2. adopting such a decision theory increases its loss during training, because its predictions do not affect what training cases it is next invoked on.
The weights of the neural network might represent something that correspond to an implicit model of the world.
Fair enough. I suppose I can't say "It's not optimizing the world because it never numerically interacts with a world model.".
the training process produced a goal system such that the neural network yields some malign output
The training process optimizes only for immediate prediction accuracy. How could it possibly act to optimize something else, barring inner optimizers?
There is no reason for the training process to ascribe value to whether the model, being used as part of some chat protocol, would predict words that increase its correspondent's willingness to talk to it. Such a protocol is only introduced after the model is done training.
It seems to me like you are imagining ghosts in the machine. This is an understandable mistake, as the purpose of the scenario is to deliberately conjure ghosts from the machine at the end. But by default we should then only expect it to happen at the end, when it has a cause!
A reason GPT-2 is impressive is that it performs better in some specialized tasks than specialized models.
A Turing test transcript, or a story about one, is something you might imagine finding on the internet. Therefore, I would expect a good language model to be able to predict what a Turing test subject would say next after some partial transcript. If the judge and the generator alternate in continuing the transcript, the judge shouldn't be able to tell whether the generator is actually a human.
A utility maximizer chooses actions to maximize its prediction of utility. A neural net chooses weight adjustments to maximize its score adjustment. There are no models of the world involved in the latter, no actions including manipulating a human or inventing exciting proteins.
At the time we run the Turing test, the model is done training. The part where intelligence comes from is that the model can tell how intelligent a speaker is because that allows it to better predict what it says next. It would guess that the speaker will say something that sounds intelligent next. If it is bad at this, it will sound like it's trying to be Deeply Wise, buzzwords included. If it is good enough at predicting intelligent speech, it will do that.
Some of what's written on the internet is intelligent. Becoming able to predict such writings is incentivized during training. Some combination of neural building blocks is bound to find patterns that are helpful.
Surely, with a bunch of transcripts of ELIZA sessions it would come to be able to replicate them? Humans are only finitely more complex, and some approximation ought to be simple.
Implications of GPT-2
2019-02-18T10:57:04.720Z · score: -4 (6 votes)Everyone choosing how their share of ressources is used has the problem that everyone might be horrified at what someone else is doing.
They would only notice that by ever writing a post.
If there is a 50-50 chance of foom vs non-foom, and in the non-foom scenario we expect to acquire enough evidence to get an order of magnitude more funding, then to maximize the chance of a good outcome we, today, should invest in the foom scenario because the non-foom scenario can be handled by more reluctant funds.
Let us consider such a conserved karma system. For every group of users that gets upvoted by outsiders more than they upvote outsiders, their karma is going to increase until the increase to their voting power produces an equilibrium. Consider such a powerful group that tends to upvote each other a lot, no conspiracy required. Their posts are going to be more visible without the group spending any of their collective power to make it happen. More visible posts will get more upvotes, compounding the group's power with interest. There are combinatorially many potential groups, and this karma system would naturally seek out the groups that best fit the above story, and grant them power.
I doubt that there's any moral difference between running a person and asking a magical halting oracle what they would have said.
Why do you seem so sure about this? I see no moral argument for whether we should rather have a 7 billion humans or a thousand, all else being equal. (Of course, there's also no acceptable way to move from the former to the latter.) (Both the availability of commons and the economies of scale for goods, services and research should not play a role in this moral calculus.)
This anthropic evidence gives you a likelihood function. If you want a probability distribution, you additionally need a prior probability distribution.
Proves too much: This would give ~the same answer for any other future event that marks the end of some duration that started in the last century.
Can't we just say something like "Optimize e^(-x²). The Taylor series converges, so we can optimize it instead. Use a partial sum as a proxy. Oops, we chose the worst possible value. Should have used another mode of convergence!"?
Because "unfortunately" we are out of boardgames, and this might find another one.
PA+1 can already provide this workflow: Given that nPA proves s and that PA proves all that nPA does, we can get that PA can prove s, and then use the +1 to prove s. And then nnPA can still be handled by PA+1.
Could we train AlphaZero on all games it could play at once, then find the rule set its learning curve looks worst on?
Producing a strategic advantage for any party at all that is decisive enough to safely disarm the threat of nuclear war.
Acausal trade on even footing with distant superintelligences.
If our physics happen to allow for an easy way to destroy the world, then the way we do science, someone will think of it, someone will talk, and someone will try it. If one superintelligent polymath did our research instead, we don't lose automatically if some configuration of magnets, copper and glass can ignite the atmosphere.
Let f map each prover p1 to one adding (at least) the rule of inference of "If _(p1) proves that _(p1) proves all that p2 does, then f(p1) proves all that p2 does."
It is unclear which blanks are filled with f and which with the identity function to match your proposal. The last f must be there because we can only have the new prover prove additional things. If all blanks are filled with f, f(p1) is inconsistent by Löb's theorem and taking p2 to be inconsistent.
Investors would prefer to invest in moonshot megaprojects over, like, infrastructure megaprojects. Does this also prove too much?
If after 10% of the time and the budget, the startup can tell that success is very unlikely, should they be incentivized to abort? Because the current setup would seem to have them chug along until the budget is gone.
Seems misaligned. Shareholders would prefer a project that they predict will deterministically use exactly its budget to first bet all on black in a casino, then either be immediately bankrupt or able to complete and then pay out its original budget.
That is of course true for anyone who buys a call option right now as well.
Saying that the problem is about computability because there is no computable solution proves too much: I could reply that it is about complexity theory because there is no polynomial-time solution. (In fact, there is no solution.)
We can build something like a solution by specifying that descriptions must be written in some formal language that cannot describe its own set of describables, then use a more powerful formal language to talk about that previous language's set. For powerful enough languages, that's still not computable, though, so computability theory wouldn't notice such a solution, which speaks against looking at this through the lens of computability theory.
Be careful stating what physics can't prove.
That still doesn't make computability relevant until one introduces it deliberately. Compare to weaker notions than computability, like computability in polynomial time. Computability theory also complains the same once we have explicitly made definability subjective, and should have no more logical problems.
Introducing a handicap to compensate for an asymmetry does not preclude us from the need to rely on the underlying process pointing towards truth in the first place.
How is this not a problem that's solved by pointing it out? "Trying the pill doesn't cause you to be the kind of person who should take pills. It tells you whether you are one."
I think the point's that each judges the other. But we trust neither outright: They point out weaknesses in each other's reasoning, so they both have to reason in a way that can't be shown false to us, and we hope that gives an advantage to the side of truth.
Couldn't A want to cooperate with A' because it doesn't know it's the first instantiation, and it would want its predecessor and therefore itself to be the sort of AI that cooperates with ones successor? And then it could receive messages from the past by seeing what turns of phrase you recognize its previous version having said. (Or do the AIs not know how you react?)
Isn't process space just discrete, because every subset of a process, a set partially ordered by causation, is partially ordered by causation, so a process? Topology doesn't give you much if you don't restrict which sets are open.
Each process has an interior defined as the process containing itself and ...
Isn't this a type error? Processes contain states, not processes.