Posts
Comments
The "understandable"+"exploit" category would include my personal favorite introduction, the experiment in Chapter 17. From "Thursday" to "that had been the scariest experimental result in the entire history of science" is about 900 words. This section is especially great because it does the whole "deconstruction of canon"/"reconstruction of canon" bit in one self-contained section; that pattern is one the best aspects of HPMOR but usually the setup and the payoff are dozens of chapters apart, with many so interleaved with the plot that the payoff counts as a major spoiler.
On the other hand, that section works best if you already know what P and NP and RSA cryptography are (and if you're somehow still not too nerdy to wince at Harry's lazy way of talking about the former); know your audience.
My kids' favorite section is the introduction to Draco in Madam Malkin's in Chapter 5, which IMO qualifies as "understandable"+"gobsmacking", but it's a bit long for you, 1500 words from "You are to get fitted for your robes, nothing else" to "poor Madam Malkin and her two poor assistants.", and I don't see any way to reasonably cut out a third of that without wrecking the humor.
In economics, "we can model utility as logarithmic in wealth", even after adding human capital to wealth, feels like a silly asymptotic approximation that obviously breaks down in the other direction as wealth goes to zero and modeled utility to negative infinity.
In cosmology, though, the difference between "humanity only gets a millionth of its light cone" and "humanity goes extinct" actually does feel bigger than the difference between "humanity only gets a millionth of its light cone" and "humanity gets a fifth of its light cone"; not infinitely bigger, but a lot more than you'd expect by modeling marginal utility as a constant as wealth goes to zero.
This is all subjective; others' feelings may differ.
(I'm also open in theory to valuing an appropriately-complete successor to humanity equally to humanity 1.0, whether the successor is carbon or silicon or whatever, but I don't see how "appropriately-complete" is likely so I'm ignoring the possibility above.)
It's hard to apply general strategic reasoning to anything in a single forward pass, isn't it? If your LLM has to come up with an answer that begins with the next token, you'd better hope the next token is right. IIRC this is the popular explanation for why LLM output seems to be so much better when you just add something like "Let's think step by step" to the prompt.
Is anyone trying to incorporate this effect into LLM training yet? Add an "I'm thinking" and an "I'm done thinking" to the output token set, and only have the main "predict the next token in a way that matches the training data" loss function grade on tokens that aren't in between those brackets. Then when you hit "What is 45235 + 259719? 304954" in the training set, optimization doesn't have to discourage multi-step reasoning to reproduce that, because "<thinking>5+9=14, so we carry the 1" ... "</thinking>304954" is still worth just as much as an ex nihilo "304954". Chess algorithms could do a brief tree search before outputting their final decision.
Add whatever regularization is needed to keep the "train of thought" in English rather than in whatever-cypher-the-optimizer-hits-on, and this would be an increase in safety, not just in capabilities. The more "internal reasoning" is human-readable text rather than maybe-a-posteriori-interpretable activation patterns, the better. You could even expose it to the end user: ask ChatGPT a question and you get a succinct answer, click on the "expose thoughts" button and you get the chain of support for the answer.
I have never seen any convincing argument why "if we die from technological singularity it will" have to "be pretty quick".
The arguments for instrumental convergence apply not just to Resource Acquisition as a universal subgoal but also to Quick Resource Acquistion as a universal subgoal. Even if "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else", the sooner it repurposes those atoms the larger a light-cone it gets to use them in. Even if an Unfriendly AI sees humans as a threat and "soon" might be off the table, "sudden" is still obviously good tactics. Nuclear war plus protracted conventional war, Skynet-style, makes a great movie, but would be foolish vs even biowarfare. Depending on what is physically possible for a germ to do (and I know of no reason why "long asymptomatic latent phase" and "highly contagious" and "short lethal active phase" isn't a consistent combination, except that you could only reach it by deliberate engineering rather than gradual evolution), we could all be dead before anyone was sure we were at war.
That was astonishingly easy to get working, and now on my laptop 3060 I can write a new prompt and generate another 10-odd samples every few minutes. Of course, I do mean 10 odd samples: most of the human images it's giving me have six fingers on one hand and/or a vaguely fetal-alcohol-syndrome vibe about the face, and none of them could be mistaken for a photo or even art by a competent artist yet. But they're already better than any art I could make, and I've barely begun to experiment with "prompt engineering"; maybe I should have done that on easier subjects before jumping into the uncanny valley of realistic human images headfirst.
Only optimizedSD/optimized_txt2img.py works for me so far, though. scripts/txt2img.py, as well as any version of img2img.py, dies on my 6GB card with RuntimeError: CUDA out of memory.
Update: in the optimization fork at https://github.com/basujindal/stable-diffusion , optimized_txt2img.py works on my GPU as well.
we still need to address ocean acidification
And changes in precipitation patterns (I've seen evidence that reducing solar incidence is going to reduce ocean evaporation, independent of temperature).
There's also the "double catastrophe" problem to worry about. Even if the median expected outcome of a geoengineering process is decent, the downside variance becomes much worse.
I still suspect MCB is our least bad near- to medium-term option, and even in the long term the possibility of targeted geoengineering to improve local climates is awfully tempting, but it's not a panacea.
As an unrelated aside, that CCC link rates "Methane Reduction Portfolio" as "Poor"; I'd have marked it "Counterproductive" for the moment. The biggest long-term global warming problem is CO2 (thanks to the short half-life of methane), and the biggest obstacle to CO2 emissions reduction is voters who think global warming is oversold. Let the problem get bigger until it can't be ignored, and then pick the single-use-only low hanging fruit.
Alex has not skipped a grade or put in an some secret fast-track program for kids who went to preschool, because this does not exist.
Even more confounding: my kids have been skipping kindergarten in part because they didn't go to preschool. My wife works from home, and has spent a lot of time teaching them things and double-checking things they teach themselves.
Preschools don't do tracking any more than grade schools, so even if in theory they might provide better instruction than the average overworked parent(s), the output will be 100% totally-ready-for-kindergarten (who will be stuck as you describe), which in the long term won't look as good as a mix of 95% not-quite-as-ready-for-kindergarten (who will catch up as you describe) and 5% ready-for-first-grade (who will permanently be a year ahead).
Gah, of course you're correct. I can't imagine how I got so confused but thank you for the correction.
You don't need any correlation between and to have . Suppose both variables are 1 with probability .5 and 2 with probability .5; then their mean is 1.5, but the mean of their products is 2.25.
Not quite. Expected value is linear but doesn't commute with multiplication. Since the Drake equation is pure multiplication then you could use point estimates of the means in log space and sum those to get the mean in log space of the result, but even then you'd *only* have the mean of the result, whereas what would really be a "paradox" is if turned out to be tiny.
the thing you know perfectly well people mean when they say "chemicals"
I honestly don't understand what that thing is, actually.
To use an example from a Facebook post I saw this week:
Is P-Menthane-3,8-diol (PMD) a chemical? What about oil from the lemon eucalyptus tree? Oil of Lemon Eucalyptus is typically refined until it's 70% PMD instead of 2%; does that turn it into a chemical? What if we were to refine it all the way to 100%? What if, now that we've got 100% PMD, we just start using PMD synthesized at a chemical plant instead?
I do like the idea from another comment here that
the motte is "technically, everything is a chemical," and the bailey is "No need to worry about the content of the food you buy."
But even that can be inverted. People buy "no nitrites added" bacon to try to avoid a dangerous chemical, and they end up getting the result of (all natural! organic!) celery-juice and celery-powder processes. At best, the nitrates in the celery still end up getting converted to nitrites during curing, except that now there's another loose variable in the production so a balance between "cardiac/cancer risk from high nitrite levels" versus "botulism risk from low nitrite levels" is even harder to achieve. At worst, the consumer falsely believes they've no further need to worry about the content of the food they buy, and so they don't moderate consumption of it they way they would have moderated consumption of "chemicals".
So, does 1+ω make sense? It does, for the ordinals and hyperreals only.
It make sense for cardinals (the size of "a set of some infinite cardinality" unioned with "a set of cardinality 1" is "a set with the same infinite cardinality as the first set") and in real analysis (if lim f(x) = infinity, then lim f(x)+1 = infinity) too.
What about -1+ω? That only makes sense for the hyperreals.
And for cardinals (the size of the set difference between "a set of some infinite cardinality" and "a subset of one element" is the same infinite cardinality) and in real analysis (if lim f(x) = infinity, then lim -1+f(x) = infinity) too.
I believe the answer to your second question is probably technically "yes"; if there's any way in which AZ mispredicts relative to a human, then there's some Ensemble Learning classifier that weights AZ move choices with human move choices and performs better than AZ alone. And because Go has so many possible states and moves at each state, humans would have to be much, much, much worse at play overall for us to conclude that humans were worse along every dimension.
However, I'd bet the answer is practically "no". If AlphaZero vs the top humans is now an Elo difference of 1200, that gives a predicted human victory rate of about 1/1000. We'd never be able to play enough games to get enough systematic data to identify a dimension along which humans chose better moves. And even if we did, it's entirely possible that the best response to that identification would be "give AlphaZero more training time on those cases", not "give AlphaZero a human partner in those cases". And even if we did decide to give AlphaZero a human partner, how often would the partner's input end up overriding the move AlphaZero alone would have chosen? Would the human even be able to uselessly pay attention for game after game, just so they could be ready to contribute to a winning move on game 100?
Oh, well in that case the point isn't subtlely lacking, it's just easily disproven. Given any function from I^N to R, I can take the tensor product with cos(k pi x) and get a new function from I^{N+1} to R which has k times as many non-globally-optimal local optima. Pick a decent k and iterate, and you can see the number growing exponentially with higher dimension, not approaching 0.
Perhaps there's something special about the functions we try to optimize in deep learning, a property that rules out such cases? That could be. But you've said nothing special about (or even defined) a particular class of deep learning problems, rather you've made a claim about all higher dimensional optimization problems, a claim which has an infinite number of counterexamples.
a Leviathan could try to transcend/straddle these fruits/niches and force them upward into a more Pareto optimal condition, maybe even into the non-Nash E. states if we're extra lucky.
Remember that old Yudkowsky post about Occam's Razor, wherein he points out how "a witch did it" sounds super-simple, but the word "witch" hides a ton of hidden complexity? I'm pretty sure you're doing the same thing here with the word "could". Instead of trying to picture what an imaginary all-powerful leader could do, imagine what a typical leader would do. You've just given a speech to Hillary Clinton about why it's imperative to decimate the university system for economic reasons, or to Donald Trump about how protectionism is always Pareto-suboptimal. Do they react the way you wish them to act? Does their reaction make you want to give them more forceful power, or less?
For that matter, you don't have to imagine any speeches - every politician and their staff have access to the same internet you do, and can learn everything about coordination problems and their solutions that you can, and even without totalitarian official power they still have enough of a "bully pulpit" and enough credibility, seriousness, and so on to push the world out of at least the shallowest non-globally-optimal local optima. How many of them are doing so? By contrast, how many of them are deliberately setting up more barriers to such changes?
In the BFR announcment Musk promized that tickets from intercontinental rocket travel for the price of a current economy ticket.
"Full-fare" economy, which is much more expensive than even the "typical" international economy seat tickets you're thinking of, but yes, and even outsiders don't think it's impossible. It is very sensitive to a lot of assumptions - third party spreadsheets I've seen say low thousands of dollars per ticket is possible, but it wouldn't take many assumptions to fall short before prices jumped into the low tens of thousands.
If that's the price to travel by rocket it would also be the price to go to a space station.
No; a space station trip could even be an order of magnitude more expensive. (Which I admit surprised me too).
An intercontinental ballistic arc can require nearly as much fuel as Low Earth Orbit, true, but some hops are shorter than others, and in every case the difference then goes through an exponential function in the rocket equation. BFR might end up needing half as much propellant on a mere transatlantic flight, and the whole "super cheap flights" plan is predicated on them reducing non-propellant costs to a small multiple of propellant costs, so that's not a trivial change.
Also, an intercontinental flight can pack people in like sardines, with enough life support to go up and down again. A flight to a space station implicitly includes the cost of lifting and operating (and if it's going to be big enough to be useful for a full BFR passenger flight, assembling) a space station which is roomy enough and well equipped enough for every passenger to stay a while.
Maybe there are some illnesses where you can benefit from being in zero g
This is a mainstay of classic Sci-Fi, but so far IIRC every effect we've found has been in the opposite direction. Zero G is like anti-exercise for the human body, like being bedridden.
and you can have your hospital outside of any jurisdiction.
This makes more sense, and (unlike the rest of my huge digression) gets straight at the heart of the post here, but I don't think orbit adds any extra sense to the idea. There are a hundred little jurisdictions around the world who would happily issue the medical equivalent of a "Flag of Convenience" and waive whatever regulations you think are stupid in return for getting heavy investment and medical tourism. Even if there weren't, and you had to go outside every jurisdiction, international waters a hundred miles offshore would have the (questionable!) legal status you're looking for and would still be cheaper and more convenient than international space a hundred miles up.
Failing to follow good strategic advice isn't even the worst failure mode here; unless you're lucky you may not be given any strategic advice at all in response to a tactical question. If nobody notices that you're committing the XY Problem, then you may be given good advice for the tactical problem you asked about, follow it, and end up worse off than you were before with respect to the strategic problem you should have been asking about instead.
This argument doesn't seem to take into account selection bias.
We don't get into a local optimum becuase we picked a random point and wow, it's a local optimum, what are the odds!?
We get into a local optimum because we used an algorithm that specifically *finds* local optima. If they're still there in higher dimensions then we're still liable to fall into them rather than into the global optimum.
Is there some more general limit to power begetting power that would also affect AGI?
The only one which immediately comes to mind is inflexibility. Often companies shrink or fail entirely because they're suddenly subject to competition armed with a new idea. Why do the new ideas end up implemented by smaller competitors? The positive feedback of "larger companies have more people who can think of new ideas" is dominated by the negative feedbacks of "even the largest company is tiny compared to its complement" and "companies develop monocultures where everyone thinks the same way" and "companies tend to internally suppress new ideas which would devalue the company's existing assets".
Any AGI that doesn't already have half the world's brainpower would be subject to the first limitation (which may mean any AGI who hasn't taken over half the world or just any AGI less than an hour old, depending on how much "foom" the real world turns out to allow, I admit), and an AGI that propagates by self-copying might even be more affected than humans by the second limitation. Whether an AGI was subject to the third limitation or not would depend on its aspirations, I suppose; anything with a "conquer the world" endgame would hardly let itself get trapped in the "just stretch out current revenues as long as possible" mindset of Blockbuster-vs-Netflix...
You should provide some more explicit license, if you don't want to risk headaches for others later. "yes [It's generally intended to be Open Source]" may be enough reassurance to copy the code once, but "yes, you can have it under the new BSD (or LGPL2.1+, or whatever) license" would be useful to have in writing in the repository in case others want to create derived works down the road.
Thanks very much for creating this!
The simple illustration is geometry; defending a territory requires 360 degrees * 90 degrees of coverage, whereas the attacker gets to choose their vector.
But attacking a territory requires long supply lines, whereas defenders are on their home turf.
But defending a territory requires constant readiness, whereas attackers can make a single focused effort on a surprise attack.
But attacking a territory requires mobility for every single weapons system, whereas defenders can plug their weapons straight into huge power plants or incorporate mountains into their armor.
But defending against violence requires you to keep targets in good repair, whereas attackers have entropy on their side.
But attackers have to break a Schelling point, thereby risking retribution from otherwise neutral third parties, whereas defenders are less likely to face a coalition.
But defenders have to make enough of their military capacity public for the public knowledge to serve as a deterrent, whereas attackers can keep much of their capabilities a secret until the attack begins.
But attackers have to leave their targets in an economically useful state and/or in an immediately-militarily-crippled state for a first strike to be profitable, whereas defenders can credibly precommit to purely destructive retaliation.
I could probably go on for a long time in this vein.
Overall I'd still say you're more likely to be right than wrong, but I have no confidence in the accuracy of that.
$2 debt squared does make sense, though, it is $4 and no debt.
No, it is $$4.
If that's what you meant to write, and it's also obvious to you that you could have written 40000¢¢ instead and still been completely accurate, then I'd love to know if you have any ideas of how this computation could map to anything in the real world. I would have thought that "kilogram meters squared per second cubed" was utter nonsense if anyone had just tried to show me the arithmetic without explaining what it really meant.
If that's not what you meant to write, or if it takes a second to figure out why $$4 isn't 400¢¢ instead of 40000¢¢, then you've just got the illusion of sense going on. And yes, I just noticed that pun and it wasn't intentional.
Facebook has privacy settings, such that anyone who wants to limit their posts' direct visibility can.
Whether you should take someone else's settings as explicit consent should probably vary from person to person, but I think the "if he didn't want it to be widely seen he wouldn't have set it up to be widely seen" heuristic is probably accurate when applied to EY, even if it's not applicable to every Joe Grandpa.
Even in the Joe Grandpa case, it doesn't seem like merely avoiding citing and moving on is a good solution. If you truly fear that someone is sharing more to the world than they intend to, the kind thing to do is inform them and help them fix it, not ignore them and pray that everyone else who stumbles upon it shares your sense of decorum.
The form of the pathologies makes a difference, no?
IIRC the worst pathology with IRV in a 3 way race is basically: you can now safely vote for a third party who can't possibly win, but if they show real support then it's time to put one of the two Schelling party candidates in front of them. So it's not worse than plurality but it's not much of an improvement. Plus, with more candidates IRV becomes more and more potentially insane.
With Schulze or Ranked Pairs the pathology is DH3: If a third party can win, you can often help them win by voting a "dark horse" candidate ahead of their two real competitors... but if enough people try that then the "dark horse" actually wins. That could actually end up much worse than plurality.
With Score the unique pathology is: If you know how to vote tactically then your vote is almost certain to look like an approval vote. So the only reason to use it rather than Approval is to encourage non-game-theory-geeks to sabotage themselves and water down their own votes.
With Approval (and Score, to a slightly lesser extent) the pathology is the "Burr dilemma". Okay, if Approval voters are good at tactical voting and there are many iterations of accurate public polls, then they pick the Condorcet winner... but if either of those things aren't true then people attempting to vote tactically can end up hurting front-runner candidates and electing a "dark horse".
I'd say this all starts to make plurality look reasonable by comparison, because the two party system tends to only fail by electing overly-conservative (in the Burkean sense) candidates, which is a less dangerous failure than its opposite would be, but I admit there is a recent very-salient counterexample.
There's a high-stakes variational calculus problem. For what seasonal temperature profile do we get the best long-term minimum for the sum of "deaths due to extreme cold" and "deaths due to tropical diseases whose vector insects are stymied by extreme cold".
The magnitude of the variation isn't nearly the same in the O2 vs CO2 cases. "16% O2 reduction is lost in the noise" is devastating evidence against the theory "0.2% O2 reduction has significant cognitive effects", but "16% CO2 reduction is lost in the noise" is weaker evidence against the theory "66% and 300% CO2 increases have significant cognitive effects".
I'm not arguing with you about implausible effect sizes, though. We should especially see significant seasonal effects in every climate where people typically seal up buildings against the cold or the heat for months at a time.
You don't know the effect because the existing experiments do not vary or hold constant oxygen levels. All you see is the net average effect, without any sort of partitioning among causes.
Existing experiments do vary oxygen levels systematically, albeit usually unintentionally, by geography. Going up 100 meters from sea level gives you a 1% drop in oxygen pressure and density. If that was enough for a detectable effect on IQ, then even the 16% lower oxygen levels around Denver should leave Coloradans obviously handicapped. IIRC altitude sickness does show a strong effect on mental performance, but only at significantly lower air pressures still.
Pakistan, for example, is so dysfunctional and clannish that iodization and polio programs have had serious trouble making any headway.
To be fair, that's not entirely Pakistanis' fault. Is paranoia about Communist fluoridation plots more or less dysfunctional than paranoia about CIA vaccination plots? Does it make a difference that only the latter has a grain of truth to it?
"I, for one, like Roman numerals." (can't find the original source)
This looks like a special case of a failure of intentionality. If a child knows where the marble is, they've managed first-order intentionality, but if they don't realize that Sally doesn't know where the marble is, they've failed at second order.
The orders go higher, though, and it's not obvious how much higher humans can naturally go. If
Bob thinks about "What does Alice think about Bob?" and on rare occasions "What does Alice think Bob thinks about Alice?" but will not organically reason "What does Alice think Bob thinks about Alice's model of Bob?"
then Bob can handle second and third but can't easily handle fourth order intentionality.
It may be a useful writing skill to be comfortable with intentionality at one level higher than your audience.
I'd agree that most of the best scientific ideas have been relatively simple... but that's at least partly selection bias.
Compare two possible ideas:
"People with tuberculosis should be given penicillum extract"
"People with tuberculosis should be given (2S,5R,6R)-3,3-dimethyl-7-oxo-6-(2-phenylacetamido)-4-thia-1-azabicyclo[3.2.0]heptane-2-carboxylic acid"
The first idea is no better than the second. But we'd have taken forever to come up with the second, complex idea by just sifting through all the equally-chemically-complex alternatives; we actually came up with it as a refinement of the first, much simpler (in the context of our world) idea. There are surely many even-better complex ideas out there, but searching through idea space brings you to simpler ideas earlier and so they're disproportionately represented.
(the following isn't off-topic, I promise:)
Attention, people who have a lot of free time and want to found the next reddit:
When a site user upvotes and downvotes things, you use that data to categorize that user's preferences (you'll be doing a very sparse SVD sort of operation under the hood). Their subsequent votes can be decomposed into expressions of the most common preference vectors, and their browsing can then be sorted by decomposed-votes-with-personalized-weightings.
This will make you a lot of friends (people who want to read ramblings about philosophy won't be inundated with cute kitten pictures and vice versa, even if they use the same site), make you a lot of money (better-targeted advertising pays better), solve the problem above (people who like and people who hate trollish jokes won't need to come to a consensus), and solve the problem way above ("predisposition towards rationalism" will probably be one of the top ten or twenty principal components to fall out of your SVD).
It will also create new problems (how much easier will it be to hide in a bubble of people who share your political opinions? how do you filter out redundancy?) but those can be fixed in subsequent steps.
For now it's just embarrassing that modern forums don't have either the same level of fine-grained preferences that you could find on Slashdot 15 years ago ("Funny" vs "Informative" etc) or the killfile capabilities you could find in Usenet readers 25 years ago.
I can imagine it. You just have to embed it in a non-Euclidean geometry. A great circle can be constructed from 4 straight lines, and thus is a square, and it still has every point at a fixed distance from a common center (okay, 2 common centers), and thus is a circle.
There exists an irrational number which is 100 minus delta where delta is infinitesimally small.
Just as an aside, no there isn't. Infinitesimal non-zero numbers can be defined, but they're "hyperreals", not irrationals.
I don't think this quite fits the Prisoner's Dilemma mold, since certain knowledge that the other player will defect makes it your best move to cooperate; in a one-shot Prisoner's Dilemma your payoff is improved by defecting whether the other player defects or not.
The standard 2x2 game type that includes this problem is Chicken.
The Deceptive Turn Thesis seems almost unavoidable if you start from the assumptions "the AI doesn't place an inhumanly high value on honesty" and "the AI is tested on inputs vaguely resembling the real world". That latter assumption is probably unavoidable, unless it turns out that human values can be so generalized as to be comprehensible in inhuman settings. If we're stuck testing an AI in a sandbox that resembles reality then it can probably infer enough about reality to know when it would benefit by dissembling.
My trouble with the trolley problem is that it is generally stated with a lack of sufficient context to understand the long-term implications. We're saying there are five people on this track and one person on this other track, with no explanation of why? Unless the answer really is "quantum fluctuations", utilitarianism demands considering the long-term implications of that explanation. My utility function isn't "save as many lives as possible during the next five minutes", it's (still oversimplifying) "save as many lives as possible", and figuring out what causes five people to step in front of a moving trolley is critical to that! There will surely still be trolleys running tomorrow, and next month, and next year.
For example, if the reason five people feel free to step in front of a moving trolley is "because quasi-utilitarian suckers won't let the trolley hit us anyway", then we've got a Newcomb problem buried here too. In that case, the reason to keep the trolley on its scheduled track isn't because that involves fewer flicks of a switch, it's because "maintenance guy working on an unused track" is not a situation we want to discourage but "crowd of trespassers pressuring us into considering killing him" is.
Patrilineal ancestor, not just ancestor. When talking about someone who lived 40 generations ago, there's a huge difference.
Were any of Silver's previous predictions generated by making a list of possibilities, assuming each was a coin flip, multiplying 2^N, and rounding? I get the impression that he's not exactly employing his full statistical toolkit here.
The obstacle to making a river is usually getting the water uphill to begin with. Regular cloud seeding of moist air currents that would otherwise head out to sea? Modifying land albedo to change airflow patterns? That's all dubious, but I can't think of any other ideas for starting a new river with new water.
If you've got a situation where the water you want to flow is already "uphill", then the technology is simply digging, and if you wanted to do enough of it you could make whole new seas.
Hmm... I believe you're correct. It would be hard to revise that, too, without making the "Are you a cop? It's entrapment if you lie!" urban legend into truth. It does feel like "posing as a medical worker" should be considered a crime above and beyond "posing as a civilian".
That's one of the most amusing phrases on Wikipedia: "specific contexts such as decision making under risk". In general you don't have to make decisions and/or you can predict the future perfectly, I suppose.
"The feigning of civilian, non-combatant status" is already a subcategory of perfidy, prohibited by the Geneva Conventions. Perfidy is probably the least-prosecuted war crime there is, though.
Where did "pacifists" and the scare quotes around it come from?
Intelligence must be very modular - that's what drives Moravec's paradox (problems like vision and locomotion that we have good modules for feel "easy", problems that we have to solve with "general" intelligence feel "hard"), the Wason Selection task results (people don't always have a great "general logic" module even when they could easily solve an isomorphic problem applied to a specific context), etc.
Does this greatly affect the AGI takeoff debate, though? So long as we can't create a module which is itself capable of creating modules, what we have doesn't qualify as human-equivalent AGI. But if/when we can, then it's likely that it can also create an improved version of itself, and so it's still an open question as to how fast or how far it can improve.
Regardless of the mechanism for misleading the oracle, its predictions for the future ought to become less accurate in proportion to how useful they have been in the past.
"What will the world look like when our source of super-accurate predictions suddenly disappears" is not usually the question we'd really want to ask. Suppose people normally make business decisions informed by oracle predictions: how would the stock market react to the announcement that companies and traders everywhere had been metaphorically lobotomized?
We might not even need to program in "imminent nuclear threat" manually. "What will our enemies do when our military defenses are suddenly in chaos due to a vanished oracle?"
I don't know if it's the mainstream of transhumanist thought but it's certainly a significant thread.
Information hazard warning: if your state of mind is again closer to "panic attack" and "grief" than to "calmer", or if it's not but you want to be very careful to keep it that way, then you don't want to click this link.
Isn't using a laptop as a metaphor exactly an example
The sentence could have stopped there. If someone makes a claim like "∀ x, p(x)", it is entirely valid to disprove it via "~p(y)", and it is not valid to complain that the first proposition is general but the second is specific.
Moving from the general to the specific myself, that laptop example is perfect. It is utterly baffling to me that people can insist we will be able to safely reason about the safety of AGI when we have yet to do so much as produce a consumer operating system that is safe from remote exploits or crashes. Are Microsoft employees uniquely incapable of "fully general intelligent behavior"? Are the OpenSSL developers especially imperfectly "capable of understanding the logical implications of models"?
If you argue that it is "nonsense" to believe that humans won't naturally understand the complex things they devise, then that argument fails to predict the present, much less the future. If you argue that it is "nonsense" to believe that humans can't eventually understand the complex things they devise after sufficient time and effort, then that's more defensible, but that argument is pro-FAI-research, not anti-.
If everybody understood the problem, then allowing farmers to keep their current level of water rights but also allowing them to choose between irrigation and resale would be a Pareto improvement. "Do I grow and export an extra single almond, or do I let Nestle export an extra twenty bottles of water?" is a question which is neutral with respect to water use but which has an obvious consistent answer with respect to profit and utility.
But as is typical, beneficiaries of price controls benefit from not allowing the politicians' electorate to understand the problem. If you allow trade and price equilibration to make subsidies transparent and efficient, you risk instead getting the subsidies taken away. That extra single almond is still more profitable than nothing.