Posts
Comments
If they have source code, then they are not perfectly rational and cannot in general implement LDT. They can at best implement a boundedly rational subset of LDT, which will have flaws.
Assume the contrary: Then each agent can verify that the other implements LDT, since perfect knowledge of the other's source code includes the knowledge that it implements LDT. In particular, each can verify that the other's code implements a consistent system that includes arithmetic, and can run the other on their own source to consequently verify that they themselves implement a consistent system that includes arithmetic. This is not possible for any consistent system.
The only way that consistency can be preserved is that at least one cannot actually verify that the other has a consistent deduction system including arithmetic. So at least one of those agents is not a LDT agent with perfect knowledge of each other's source code.
We can in principle assume perfectly rational agents that implement LDT, but they cannot be described by any algorithm and we should be extremely careful in making suppositions about what they can deduce about each other and themselves.
Oh, I see that I misread.
One problem is that "every possible RNG call" may be an infinite set. For a really simple example, a binary {0,1} RNG with program "add 1 to your count if you roll 1 and repeat until you roll 0" has infinitely many possible rolls and no maximum output. It halts with probability 1, though.
If you allow the RNG to be configured for arbitrary distributions then you can have it always return a number from such a distribution in a single call, still with no maximum.
My guess is "no" because both of you would die first. In the context of "largest numbers" 10^10^100 is baby's first step, but is still a number with more digits than you will ever succeed in printing.
In principle the "you" in this scenario could be immortal with unbounded resources and perfect reliability, but then we may as well just suppose you are a superintelligence smarter than the AI in the problem (which isn't looking so 'S' anymore).
Truly logical counterfactuals really only make sense in the context of bounded rationality. That is, cases where there is a logically necessary proposition, but the agent cannot determine it within their resource bounds. Essentially all aspects of bounded rationality have no satisfactory treatment as yet.
The prisoners' dilemma question does not appear to require dealing with logical counterfactuals. It is not logically contradictory for two agents to make different choices in the same situation, or even for the same agent to make different decisions given the same situation, though the setup of some scenarios may make it very unlikely or even direct you to ignore such possibilities.
- It's an arbitrary convention. We could have equally well chosen a convention in which a left hand rule was valid. (Really a whole bunch of such conventions)
- In the Newtonian 2-point model gravity is a purely radial force and so conserves angular momentum, which means that velocity remains in one plane. If the bodies are extended objects, then you can get things like spin-orbit coupling which can lead to orbits not being perfectly planar if the rotation axes aren't aligned with the initial angular momentum axis.
If there are multiple bodies then trajectories can be and usually will be at least somewhat non-planar, though energy losses without corresponding angular momentum losses can drive a system toward a more planar state.
Zero dimensions would only be possible if both the net force and initial velocity were zero, which can't happen if gravity is the only applicable force and there are two distinct points.
In general relativity gravity isn't really a force and isn't always radial, and orbits need not always be planar and usually aren't closed curves anyway. Though again, many systems will tend to approach a more planar state.
I believe that there is already far too much "hate sharing".
Perhaps the default in a social media UI should be that shared content includes a public endorsement of whatever content it links to, and if you want to "hate share" anything without such an endorsement, you have to fight a hostile UI to do so.
In particular, "things that are worth sharing" absolutely should not overlap with "want to see less of". If you want to see less of some type of thing, it's self-defeating to distribute more copies of it. Worse, if you even suspect that any of your own readers are anything like you, why are you inflicting it on them?
Yes, it is a real emotion. I have felt it on some rare occasions. I do not act on it, though on such occasions I cannot rule out the possibility that it may influence me in less direct ways.
I don't know what you mean by "best way to interpret it". What sort of interpretation are you looking for? For example, what are your best ways of interpreting other emotions?
The conclusion does not follow from the argument.
The argument suggests that it is unlikely that a perfect replica of the functioning of a specific human brain can be emulated on a practical computer. The conclusion generalizes that out to no conscious emulation of a human brain, at all.
These are enormously different claims, and neither follows from the other.
For all practical purposes, such credences don't matter. Such scenarios certainly can and do happen, but in almost all cases there's nothing you can do about them without exceeding your own bounded rationality and agency.
If the stakes are very high then it may make sense to consider the probability of some sort of trick, and attempt to get further evidence of the physical existence of the coin and that its current state matches what you are seeing.
There is essentially no point in assigning probabilities to hypotheses of failures of your mind itself. You can't reason your way out of serious mind malfunction using arithmetic. At best you could hope to recognize that it is malfunctioning, and try not to do anything that will make things worse. In the case of mental impairment severe enough to have false memories or sensations this blatant, a rational person should expect that a person so affected wouldn't be capable of correctly carrying out quantified Bayesian reasoning.
My own background credences are generally not insignificant for something like this or even stranger, but they play essentially zero role in my life and definitely not in any probability calculations. Such hypotheses are essentially untestable and unactionable.
In relativity, space and time are just different directions in spacetime with a single pseudometric determining separation between events. With this understanding, the time/space distance metaphor is more literal than most people think.
The correspondence isn't exact since it's a pseudometric and not a standard metric, and everyday units of time correspond to much greater than everyday units of distance, but it's still more than just a metaphor.
Thanks for making this!
I found it a challenge to deduce strategies over many plays, rather than following the advice "not intended to be replayed". The first playthrough was pretty much meaningless for me, especially given the knowledge that both time and history could affect the results. I just viewed it as one step of information gathering for the real game.
The suboptimal zones weren't obviously suboptimal from a single pass, even Dragon Lake that always yields nothing. For all I knew, it could have yielded 5000 food with quite a low probability (and still be always optimal), or lesser amounts of food at specific combinations of time and day, or only when matching some rule based on the previous results of foraging in other zones.
After many runs I did settle on a strategy, and mentally scored myself by looking at the source to see whether there was anything that I should have spotted but didn't. As it happened, my final strategy was almost optimal though I stayed on the rats for a few more hours than ideal.
In principle I suppose one could build very large walls around it to reduce heat exchange with the rest of Earth and a statite mirror (or few slowly orbiting ones) to warm it up. That would change the southern hemisphere circulation patterns somewhat, but could be arranged to not affect the overall heat balance of the rest of Earth.
This is very unlikely to happen for any number of good reasons.
Only the first point "Good and evil are objectively real" is a necessary part of moral realism. Sometimes the first half of the third ("We have an objective moral obligation to do good and not do evil") is included, but by some definitions that is included in what good and evil mean.
All the rest are assumptions that many people who believe in moral realism also happen to hold, but aren't part of moral realism itself.
Research companies work best when there's plenty of infrastructure that can supply stuff they need to do the research. Including, to mention one recent case, electricity. It also helps to be in an area where there is stable government that can protect the research site from civil or military unrest, and too much (or too unpredictable) corruption. You also want it to be a place where your researchers are happy to live while they do their research, and where you can relatively easily recruit other skilled workers.
China does meet these requirements, but it is not exactly lacking in bureaucracy so I'm not sure why it made the list. If you're doing research involving human trials of some sort, you also want to be able to communicate well with the participants so extensive knowledge of the language and culture will be very useful.
All that said, plenty of organizations do carry out research all over the world, not just in rich countries with a lot of bureaucracy.
Yes, it definitely does depend upon local conditions. For example if your grid operator uses net metering (and is reliable) then it is not worthwhile at any positive price. This statement was in regard to my disputed upstream comment "Even now at $1000/kW-hr retail it's almost cost-effective here [...]".
Batteries are primarily used for intra-day time shifting, not weekly. I agree that going completely off grid costs substantially more than being able to use your own generated power for 80-90% of usage. That's why I focused on the case where home owners remain grid-connected in my top-level comment:
With smart meters and cheaper home battery systems the incentives starts to shift toward wealthier solar enthusiasts buying batteries and selling excess power to the grid at peak times (or consuming it themselves), lowering peak demand at no additional capital or maintenance cost to the grid operators.
The only mention I made regarding completely off-grid power systems was about the counterfactual scenario of $150/kW-hr battery cost, which I have not assumed anywhere else. I didn't say that it would be marginally cost effective to go completely off grid with such battery prices, just that it would be substantially more cost-effective than buying all my power from the grid. The middle option of 80-90% reduced but not completely eliminated grid use is still cheaper than either of the two extremes, and likely to remain so for any feasible home energy storage system.
That's what I was referring to regarding $700 kW/hr. At $1000/kW-hr it's (just barely) not worth even buying batteries to shift energy from daytime generation to night consumption, while at $700/kW-hr it definitely is worthwhile. Do you need the calculation for that?
At $150/kW-hr and assuming a somewhat low 3000 cycle lifetime, such batteries would cost $0.05 per cycled kW-hr which is very much cost-effective when paired with the extremely low cost but inconveniently timed nature of solar power. It would drop the amortized cost of a complete off-grid power system for my home to half that of grid power in my area, for example.
Even now at $1000/kW-hr retail it's almost cost-effective here to buy batteries to time-shift energy from solar generation to time of consumption. At $700/kW-hr it would definitely be cost-effective to do daily load-shifting with the grid as a backup only for heavily cloudy days.
Pumped hydro is already underway in this region, though it's proving more expensive and time-consuming to build than expected. Have there been some recent advances in compressed air energy storage? The information I read 2-3 years ago did not look promising at any scale.
How do you construct a maximizer for 0.3X+0.6Y+0.1Z from three maximizers for X, Y, and Z? It certainly isn't true in general for black box optimizers, so presumably this is something specific to a certain class of neural networks.
Battery costs should be lower by now than they are.
For example, in Australia wholesale cell prices are on the order of $150/kW-hr, while installed battery systems are still more than $1000/kW-hr. The difference isn't just packaging, electrical systems, and installation costs. Packaging doesn't cost anywhere near that much, installation costs are relatively flat with capacity, and so are electrical systems (for given peak power). Yet battery system costs from almost all suppliers are almost perfectly linear with energy capacity.
I don't know why there isn't an alternative decent-quality supplier that would eat their lunch on large-capacity systems with moderate peak power. Such a thing should be still very highly profitable with a much larger market. It could be that there just hasn't been enough time for such a market to develop, or supply issues, or something else I'm missing?
It's not cheaper in reality. Net metering is effectively a major subsidy that goes away pretty much everywhere that solar generation starts to make up a significant fraction of the supply.
Electricity companies don't want to pay all that capital expense, so it makes sense for them to shift it onto consumers up until home solar generation starts approaching daytime demand. After that point, they can discontinue the net metering and push for "smart meters" that track usage by time of day and charge or pay variable amounts applicable for that particular time, and/or have separate "feed in" credits that are radically smaller per kWh than consumption charges (in practice often up to 85% less).
With smart meters and cheaper home battery systems the incentives starts to shift toward wealthier solar enthusiasts buying batteries and selling excess power to the grid at peak times (or consuming it themselves), lowering peak demand at no additional capital or maintenance cost to the grid operators.
In principle the endgame could involve no wholesale generators at all, just grid operators charging fees to net consumers and paying some nominal amount to net suppliers, but I expect it to not converge to anything as simple as that. Economies of scale will still favour larger-scale operations and local geographic and economic conditions will maintain a mixture of types and scales of generation, storage, distribution, and consumption. Regulation, contracts, and other conditions will also continue to vary greatly from place to place.
Yes, that was a pretty terrible take. Markets quite clearly do not price externalities well, and never have done. So long as any given investor rates their specific investment as being unlikely to tip the balance into doom, they get the upside of directly financially benefiting from major economic growth due to AI, and essentially the same downside risk as if they didn't invest. Arguments like "short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends" are obviously not even trying to seriously reflect the widespread investment decisions that affect real markets.
While this is almost certainly not relevant to any real life metaphorical application of loop detection, I'll just go ahead and mention that there is a very common cycle detection algorithm in CS that goes like:
Keep two "pointers". Move one a single step at a time. Move the other two steps at a time. If they are ever equal, then you're in a loop.
This avoids the need to remember all previous steps, but it doesn't really seem as useful in the metaphor.
If you replace it with "quantum chromodynamics", then it's still very problematic but for different reasons.
Firstly, there's no obvious narrowing to equally causal factors ("motion of the planet" vs "motion of the planets") as there is in the original statement. In the original statement the use of plural instead of singular covers a much broader swath of hypothesis space, and that you haven't ruled out enough to limit it to the singular. So you're communicating that you think there is significant credence that motion of more than one planet has a very strong influence on life on Earth.
Secondly, the QCD statement is overly narrow in the stated consequent instead of overly broad in the antecedent: any significant change in quantum chromodynamics would affect essentially everything in the universe, not just life on Earth. "Motion of the planet ... life on Earth" is appropriately scoped in both sides of the relation. In the absence of a context limiting the scope to just life on Earth, yes that would be weird and misleading.
Thirdly, it's generally wrong. The processes of life (and everything else based on chemistry) in physical models depend very much more strongly on the details of the electromagnetic interaction than any of the details of colour force. If some other model produced nuclei of the same charges and similar masses, life could proceed essentially unchanged.
However, there are some contexts in which it might be less problematic. In the context of evaluating the possibility of anything similar to our familiar life under alternative physical constants, perhaps.
In a space of universes which are described by the same models to our best current ones but with different values of "free" parameters, it seems that some parameters of QCD may be the most sensitive in terms of whether life like ours could arise - mostly by mediating whether stars can form and have sufficient lifetime. So in that context, it may be a reasonable thing to say. But in most contexts, I'd say it was at best misleading.
I don't think anybody would have a problem with the statement "The motion of the planet is the strongest governing factor for life on Earth". It's when you make it explicitly plural that there's a problem.
Ah, that does make it almost impossible then. Such a utility function when paused must have constant value for all outcomes, or it will have incentive to do something. Then in the non-paused state the otherwise reachable utility is either greater than that (in which case it has incentive to prevent being paused) or less than or equal (in which case its best outcome it to make itself paused).
Are you looking for a utility function that depends only upon external snapshot state of the universe? Or are you considering utility functions that evaluate history and internal states as well? This is almost never made clear in such questions, and amphiboly is rife in many discussions about utility functions.
Yes, both of these credences should obey the axioms of a probability space.
This sort of thing is applied in cryptography with the concept of "probable primes", which are numbers (typically with many thousands of decimal digits) that pass a number of randomized tests. The exact nature of the tests isn't particularly important, but the idea is that for every composite number, most (at least 3/4) of the numbers less than it are "witnesses" such that when you apply a particular procedure using that number, the composite number fails the test but primes have no such failures.
So the idea is that you pick many random numbers, and each pass gives you more confidence that the number is actually prime. The probability of any composite number passing (say) 50 such tests is no more than 4^-50, and for most composite numbers it is very much less than that.
No such randomized test is known for parity of the googolth digit of pi, but we also don't know that there isn't one. If there was one, it would make sense to update credence using the results of such tests using probability axioms.
What is the difference between "deciding your behaviour" and "deciding upon interventions to you that will result in behaviour of its choosing"?
If showing you a formal proof that you will do a particular action doesn't result in you doing that action, then the supposed "proof" was simply incorrect. At any rate, it is unlikely in most cases that there exists a proof that merely presenting it to a person is sufficient to ensure that the person carries out some action.
In more formal terms: even in the trivial case where a person could be modelled as a function f(a,b,c,...) that produces actions from inputs, and there do in fact exist values of (a,b,c,...) such that f produces a chosen action A, there is no guarantee that f(a,b,c,...) = A whenever a = "a proof that f(a,b,c,...) = A" for all values of b,c,... .
It may be true that f(a,b,c,...) = A for some values of b,c,... and if the superintelligence can arrange for those to hold then it may indeed look like merely presenting the proof is enough to guarantee action A, but would actually be a property of both the presentation of the proof and all the other interventions together (even if the other interventions are apparently irrelevant).
There are many things that people believe they will be able to simply ignore, but where that belief turns out to be incorrect. Simply asserting that deciding to ignore the proof will work is not enough to make it true.
As you broaden the set of possible interventions and time spans, guarantees of future actions will hold for more people. My expectation is that at some level of intervention far short of direct brain modification or other intuitively identity-changing actions, it holds for essentially all people.
...How does someone this idiotic ever stay in a position of authority? I would get their statements on statistics and probability in writing and show it to the nearest person-with-ability-to-fire-them-who-is-not-also-a-moron.
Maybe the nearest person-with-ability-to-fire-them-who-is-not-also-a-moron could give them one last chance:
"I have a red die and a blue die, each with 20 sides. If I roll the red one then you only keep your job if it rolls a 20. For the blue one you only get fired if it comes up 1.
"I'm going to roll the red one unless you can explain to me why you should want me to roll the blue one instead."
But probably not.
I'm not sure what work "to the best of personal ability" is doing here. If you execute to 95% of the best of personal ability, that seems to come to "no" in the chart and appears to count the same as doing nothing?
Or maybe does executing "to the best of personal ability" include considerations like "I don't want to do that particular good very strongly and have other considerations to address, and that's a fact about me that constrains my decisions, so anything I do about it at all is by definition to the best of my ability"?
The latter seems pretty weird, but it's the only way I can make sense of "na" in the row "had intention, didn't execute to the best of personal ability, did good".
There are many variants on utilitarian theories, each with very different answers. Even aside from that though, it can really only be answered by knowing at least some definite information about the aggregated utility functions of every ethically relevant entity, including your potential children and others.
Utilitarianism is not in general a practical decision theory. It states what general form ethical actions should take, but is unhelpfully silent on what actual decisions meet those criteria.
Yes, it's definitely fishy.
It's using the experimental evidence to privilege H' (a strictly more complex hypothesis than H), and then using the same experimental evidence to support H'. That's double-counting.
The more possibly relevant differences between the experiments, the worse this is. There are usually a lot of potentially relevant differences, which causes exponential explosion in the hypothesis space from which H' is privileged.
What's worse, Alice's experiment gave only weak evidence for H against some non-H hypotheses. Since you mention p-value, I expect that it's only comparing against one other hypothesis. That would make it weak evidence for H even if p < 0.0001 - but it couldn't even manage that.
Are there no other hypotheses of comparable or lesser complexity than H' matching the evidence as well or better? Did those formulating H' even think for five minutes about whether there were or not?
The claim is false.
Suppose we're in a universe where a fixed 99% of "odds in your favour" bets are scams where I always lose (even if we accept the proposal that the coin is actually fair). This isn't reflective of the world we're actually in, but it's certainly consistent with some utility function. We can even assume that money has linear utility if you like.
Then I should reject the first bet and accept the second.
Quantum computers have been demonstrated to work with up to 50 interacting qubits, and verified to compute some functions that a classical supercomputer can verify but not compute.
Research prototypes with more than 1000 qubits exist, though the focus is more on quantum error correction so that larger quantum computations can be performed despite imperfect engineering. This comes at a pretty steep penalty in terms of "raw" qubits required, so these machines aren't as much better as might be expected from the qubit count.
If you know how to solve this paradox, inquiry is unnecessary.
If you do not know how to solve this paradox, then inquiry is impossible.[1]
So why are you asking?
- ^
Of course it's not impossible.
Sets of distributions are the natural elements of Bayesian reasoning: each distribution corresponds to a hypothesis. Some people pretend that you can collapse these down to a single distribution by some prior (and then argue about "correct" priors), but the actual machinery of Bayesian reasoning produces changes in relative hypothesis weightings. Those can be applied to any prior if you have reason to prefer a single one, or simply composed with future relative changes if you don't.
Partially ordering options by EV over all hypotheses is likely to be a very weak order with nearly all options being incomparable (and thus permissible). However, it's quite reasonable to have bounds on hypothesis weightings even if you don't have good reason to choose a specific prior.
You can use prior bounds to form very much stronger partial orders in many cases.
The post isn't even Against AI Doom. It is against the idea that you can communicate a high confidence in AI doom to policy makers.
Not in this case.
a) If you anticipated continuity of experience into upload and were right, then you experience being an upload and remember being you and and you believe that your prediction is borne out.
b) If you were wrong and the upload is conscious but isn't you, then you're dead and nothing is borne out to you. The upload experiences being an upload and remembers being you and believes that your prediction is borne out.
c) If you were wrong and the upload is not conscious, then you're dead and nothing is borne out to you. Nothing is borne out to the upload either, since it was never able to experience anything being borne out or not. The upload unconsciously mimics everything you would have done if your prediction had been borne out.
Everyone else sees you continuing as you would have done if your prediction had been borne out.
So in all cases, everyone able to experience anything notices that your prediction is borne out.
The same is true if you had predicted (b).
The only case where there is a difference is if you predicted (c). If (a) or (b) was true then someone experiences you being wrong, whether or not that person is you is impossible to determine. If you're right then the upload still behaves as if you were wrong. Everyone else's experience is consistent with your prediction being borne out. Or not borne out, since they predict the same things from everyone else's point of view.
A thought experiment: Suppose that in some universe, continuity of self is exactly continuity of bodily consciousness. When your body sleeps, you die never to experience anything ever again. A new person comes into existence when the body awakens with your memories, personality, etc. (Except maybe for a few odd dream memories that mostly fade quickly)
Does it actually mean anything to say "a new person comes into existence when the body awakens with your memories, personality, etc."? Presumably this would mean that if you are expecting to go to sleep, then you expect to have no further experiences after that. But that seems to be begging the question: who are you? Someone experiences life-after-sleep. In every determinable way, including their own internal experiences, that person will be you. If you expected to die soon after you closed your eyes, that person remembers expecting to die but actually continuing on. Pretty much everyone in the society remembers "continuing on" many thousands of times.
Is expecting to die as soon as you sleep a rational belief in such a universe?
When it comes to questions like whether you "should" consider destructive uploading, it seems to me that it depends upon what the alternatives are, not just a position on personal identity.
If the only viable alternative is dying anyway in a short or horrible time and the future belongs only to entities that do not behave based on my memories, personality, beliefs, and values then I might consider uploading even in the case where that seems like suicide to the physical me. Having some expectation of personally experiencing being that entity is a bonus, but not entirely necessary.
Conversely if my expected lifespan is otherwise long and likely to be fairly good then I may decline destructive uploading even if I'm very confident (somehow?) in personally experiencing being that upload and it seems likely that the upload would on median have a better life. For one thing, people may devise non-destructive uploading later. For another, uploads seem more vulnerable to future s-risks or major changes in things that I currently consider part of my core identity.
Even non-destructive uploading might not be that attractive if it's very expensive or otherwise onerous on the physical me, or likely to result in the upload having a poor quality of life or being very much not-me in measurable ways.
It seems extremely likely that the uploads would believe (or behave as if they believe, in the hypothetical where they're not conscious beings) in continuity of personal identity across uploading.
It also seems like an adaptive belief even if false as it allows strictly more options for agents that hold it than for those that don't.
The Gambler's Fallacy applies in both directions: if an event has happened more frequently in the past than expected, then the Fallacy states that it is less likely to occur in future as well. So for example, rolling a 6-sided die three times and getting two sixes in such a world should also decrease the probability of getting another six on the next roll by some unspecified amount.
That is, it's a world in which steps in every random walk are biased toward the mean.
However, that does run into some difficulties. Suppose that person A is flipping coins and keeping track of the numbers of heads and tails. The count is 89 tails to 111 heads so far. Person B comes in watches for 100 more flips. They see 56 more tails and 44 heads, so that A's count is now at 145 tails to 155 heads. Gambler's Verity applied to A means that tails should still be more likely. Gambler's Verity applied to B means that heads should be more likely. Which effect is stronger?
Now consider person C who isn't told the outcomes of each flip, just whether the flip moved the counts more toward equal or further away. Gambler's Verity for those who see each flip means that "toward equal" flips are more common than "more unequal" flips. But applied to C's observations, Gambler's Verity acts to cancel out any bias even more rapidly than independent chance would. So if you're aware of Gambler's Verity and try to study it, then it cancels itself out!
If this room is still on Earth (or on any other rotating body), you could in principle set up a Foucault pendulum to determine which way the rotation is going, which breaks mirror symmetry.
If the room is still in our Universe, you can (with enough equipment) measure any neutrinos that are passing through for helicity "handedness". All observations of fusion neutrinos in our universe are left-handed, and these by far dominate due to production in stars. Mirror transformations reverse helicity, so you will disagree about the expected result.
If the room is somehow isolated from the rest of the universe by sufficiently magical technology, in principle you could even wait for long enough that enough of the radioactive atoms in your bodies and the room decay to produce detectable neutrinos or antineutrinos. By mirror symmetry the atoms that decay on each side of the room are the same, and so emit the same type (neutrino or antineutrino, with corresponding handedness). You would be waiting a long time with any known detection methods though.
This would fail if your clone's half room was made of antimatter, but an experiment in which half the room is matter and half is antimatter won't last long enough to be of concern about symmetry. The question of whether the explosion is mirror-symmetric or not will be irrelevant to the participants.
I don't think your bolded conclusion holds. Why does there have to be such a threshold? There are reasonable world-models that have no such thing.
For example: suppose that we agreed not to research AI, and could enforce that if necessary. Then no matter how great our technological progress becomes, the risk from AI catastrophe remains at zero.
We can even suppose that increasing technological progress more generally includes a higher sanity waterline, and so makes such a coordination more likely to occur. Maybe we're near the bottom of a curve of technology-vs-AI-risk, where we're civilizationally smart enough to make destructive AI but not enough to coordinate to do something which is not that. That would be a case for accelerating technology that isn't AI as risk from AI in the model increases.
A few minutes thought reveals other models where no such threshold exists.
So there is a case where there may exist such a threshold, and perhaps we are beyond it if so. I don't see evidence that there must exist such a threshold.
Oh, then I'm still confused. Agent B can want to coordinate with A but still be effectively a rock because they are guaranteed to pick the designer's preferred option no matter what they see. Since agent A can analyze B's source code arbitrarily powerfully they can determine this, and realize that the only option (if they want to coordinate) is to go along with that.
A's algorithm can include "if my opponent is a rock, defect" but then we have different scenarios based on whether B's designer gets to see A's source code before designing B.
Oh then no, that's obviously not possible. The parent can choose agent B to be a rock with "green" painted on it. The only way to coordinate with a rock is to read what's painted on it.
Option 3: Make it a tax expenditure. Taxes are the standard mandatory joint contributions to things where on average everyone is better off having done and the marginal benefit to any single contributor is less than their marginal contribution.
I'm still very confused about the scenario. Agent A and B and their respective environments may have been designed as a proxy by adversarial agents C and D respectively? Both C and D care about coordinating with each other by more than they care about having the sky colour match their preference? A can simulate B + environment, but can't simulate D (and vice versa)? Presumably this means that D can no longer affect B or B's environment, otherwise A wouldn't be able to simulate.
Critical information: Did either C or D know the design of the other's proxy before designing their own? Did they both know the other's design and settle on a mutually-agreeable pair of designs?
I'm very confused what the model is here. Are you saying that agents A and B (with source code) are just proxies created by other agents C and D (internal details of which are unknown to the agents on the other side of the communication/acausal barrier)?
What is the actual mechanism by which A knows B's source code and vice versa, without any communication or any causal links? How does A know that D won't just ignore whatever decision B makes and vice versa?
In principle (depending upon computation models) this should be possible.
With this very great degree of knowledge of how the other operates, it should be possible to get a binary result by each agent:
- choosing some computable real number N and some digit position M,
- that they have no current expectation of being biased,
- compute it,
- use the other's source code to compute the other's digit,
- combine the digits (e.g. xor for binary),
- verify that the other didn't cheat,
- use the result to enact the decision.
In principle, each agent can use the other's source code to verify that the other will not cheat in any of these steps.
Even if B currently knows a lot more about values of specific numbers than A does, that doesn't help B get the result they want. B has to choose a number+position that B doesn't expect to be biased, and A can check whether they really did not expect it to be biased.
Note that this, like almost anything to do with agents verifying each other via source code, is purely theoretical and utterly useless in practice. In practice step 6 will be impossible for at least one party.
After even the first million years as slow as 0.1c, the galaxy is full and it's time to go intergalactic. A million years is nothing in the scale of the universe's age.
When sending a probe millions of light years to other galaxies, the expense of 0.999c probes start to look more useful than 0.8c ones, saving hundreds of thousands of years. Chances are that it wouldn't just be one probe either, but billions of them seeding each galaxy within plausible reach.
Though as with any discussion about these sorts of things, we have no idea what we don't know about what a civilization a million years old might achieve. Discussions of relativistic probes are probably even more laughably primitive than those of using swan's wings to fly to the abode of the Gods.