Policy restrictions and Secret keeping AI 2021-01-24T20:59:14.342Z
The "best predictor is malicious optimiser" problem 2020-07-29T11:49:20.234Z
Optimizing arbitrary expressions with a linear number of queries to a Logical Induction Oracle (Cartoon Guide) 2020-07-23T21:37:39.198Z
Web AI discussion Groups 2020-06-30T11:22:45.611Z
[META] Building a rationalist communication system to avoid censorship 2020-06-23T14:12:49.354Z
What does a positive outcome without alignment look like? 2020-05-09T13:57:23.464Z
Would Covid19 patients benefit from blood transfusions from people who have recovered? 2020-03-29T22:27:58.373Z
Programming: Cascading Failure chains 2020-03-28T19:22:50.067Z
Bogus Exam Questions 2020-03-28T12:56:40.407Z
How hard would it be to attack coronavirus with CRISPR? 2020-03-06T23:18:09.133Z
Intelligence without causality 2020-02-11T00:34:28.740Z
Donald Hobson's Shortform 2020-01-24T14:39:43.523Z
What long term good futures are possible. (Other than FAI)? 2020-01-12T18:04:52.803Z
Logical Counterfactuals and Proposition graphs, Part 3 2019-09-05T15:03:53.262Z
Logical Counterfactuals and Proposition graphs, Part 2 2019-08-31T20:58:12.851Z
Logical Optimizers 2019-08-22T23:54:35.773Z
Logical Counterfactuals and Proposition graphs, Part 1 2019-08-22T22:06:01.764Z
Programming Languages For AI 2019-05-11T17:50:22.899Z
Propositional Logic, Syntactic Implication 2019-02-10T18:12:16.748Z
Probability space has 2 metrics 2019-02-10T00:28:34.859Z
Allowing a formal proof system to self improve while avoiding Lobian obstacles. 2019-01-23T23:04:43.524Z
Logical inductors in multistable situations. 2019-01-03T23:56:54.671Z
Boltzmann Brains, Simulations and self refuting hypothesis 2018-11-26T19:09:42.641Z
Quantum Mechanics, Nothing to do with Consciousness 2018-11-26T18:59:19.220Z
Clickbait might not be destroying our general Intelligence 2018-11-19T00:13:12.674Z
Stop buttons and causal graphs 2018-10-08T18:28:01.254Z
The potential exploitability of infinite options 2018-05-18T18:25:39.244Z


Comment by donald-hobson on Policy restrictions and Secret keeping AI · 2021-01-24T21:17:06.422Z · LW · GW

Fixed, Thanks :-)

Comment by donald-hobson on Excerpt from Arbital Solomonoff induction dialogue · 2021-01-17T16:08:35.973Z · LW · GW

The problem with this definition is that it focusses too much on the details of the computational substrate. Suppose the programming language used has a built in function for matrix multiplication, and it is 2x as fast as any program that could be written within the language. Then any program that does its own matrix multiplication will be less intelligent than one that uses the built in functions. 

"A with X resources beats program B with X resources, for any B" could be true if A is just B with the first few steps precomputed. It focusses too much on the little hackish tricks specific to the substrate.


Maybe say that two algorithms A, B are equivalent up to polynomial factors if there exists a polynomial p(x) so that A with p(x) compute beats B with X compute for all x, and likewise B with p(x) compute beats A with x compute.

Comment by donald-hobson on Grey Goo Requires AI · 2021-01-16T16:24:55.311Z · LW · GW

Simply grabbing resources is not enough to completely eliminate a society which is actively defending a fraction of those resources, especially if they also have access to self-replicators/nanotechnology (blue goo) and other defense mechanisms.

I don't see anything that humans are currently doing that would stop human extinction in this scenario. The goo can't reach the ISS, and maybe submarines, but current submarines are reliant on outside resources like food.

In these scenaios, the self replication is fast enough that there is only a few days between noticing something wrong, and almost all humans being dead, not enough time to do much engineering. In a contest of fast replicators, the side with a small head-start can vastly outnumber the other. If a sufficiently well designed blue goo is created in advance, then I expect humanity to be fine.

I agree, conditional on the grey goo having some sort of intelligence which can thwart our countermeasures.

A modern computer virus is not significantly intelligent. The designers of the virus might have put a lot of thought into searching for security holes, the virus itself is not intelligent. (usually) The designers might know what SQL injection is and how it works, the virus just repeats a particular hard coded string into any textbox it sees. 

It might be possible to create a blue goo and nanoweapon defense team so sophisticated that no simplistic hard coded strategy would work. But this does not currently exist, and is only something that would be built if humanity is seriously worried about grey goo. And again, it has to be built in advance of advanced grey goo.

Comment by donald-hobson on Grey Goo Requires AI · 2021-01-15T19:11:50.346Z · LW · GW

Yes any design of self replicating bot has physical limits imposed by energy use, resource availability ect. 

However it is not clear that biological life is close to those limits.


 If we assume that a very smart and malevolent human is designing this grey goo, I suspect they could make something world ending.

Fortunately, even after many generations, most of these machines will be pretty dumb, you could pick one up and scrap it for parts without any resistance. There is very little danger of a grey goo scenario here.

Control of biological pests is not that easy.

If we are assuming a malevolent human, the assemblers could.

Be very small; Hide in hard to reach places; be disguised as rocks; start shooting at anyone nearby; run faster than humans through rough terrain; get a head start and be replicating faster than humans can kill them. 

Suppose a machine looks like an ant, it can self replicate in 1 hour given any kind of organic material (human flesh, wood, plastic, grass ect) It has an extremely poisonous bite. It is programmed to find dark and dirty corners and self replicate there. It is set to send out some for somewhere else as soon a few copies have been made. (So there are a few of these devices everywhere, not one garbage can full of them that can be noticed and dealt with.) This is looking hard to deal with. If the machines are programmed to sometimes build a small long distance drone which sprinkles more "ants", this is looking like something that could spread worldwide and turn most of the biosphere and humanity into more copies of itself. 

Active maliciousness is enough to get grey goo. (In the absence of blue goo, self replicating machinery designed to stop grey goo, or any other grey goo prevention that doesn't exist today)

Competent engineering, designed with grey goo in mind as a risk, is enough not to get grey goo.

Incompetent engineering by people not considering grey goo, I don't know.

Comment by donald-hobson on Grey Goo Requires AI · 2021-01-15T18:44:52.068Z · LW · GW

Biology is very pro mutation. In biology, the data is stored on a medium with a moderate mutation rate. The there is no error correction, and many mutations make meaningful changes. 

There are error detection schemes that can make the chance an error goes unnoticed exponentially tiny. You can use a storage medium more reliable than DNA. You can encrypt all the data so that a single bitflip will scramble all the instructions. If you are engineering to minimize mutations, and are prepared to pay a moderate performance penalty for it, you can make mutations that don't quickly cause the bot to self destruct or damage the bot beyond its ability to do anything never happen, not once in the whole universe full of bots. 

Comment by donald-hobson on Debate Minus Factored Cognition · 2021-01-01T23:45:43.422Z · LW · GW

I am trying to gain some evidence about the situation we care about (intelligent human judges, superintelligent debaters, hard questions) by looking at human debates where the questions and debaters are sufficiently simple that we have the epistemic high ground. In other words, does debate work among humans with a not that smart judge?

There are some simple and easy to understand arguments where a full and detailed understanding of why the argument is wrong is much more complicated.

 For example "You can't explain where the universe comes from without god.

This is a short and simple argument. What do you need to understand to really see why it is wrong? You need to know the difference between the human intuitive notion of simplicity and the minimum message length formalism of Occam's razor, how "god" sounds like a simple hypothesis, but isn't really. You might want to talk about making beliefs pay rent in anticipated experience, privileging the hypothesis ect. Of course, these are insights it is hard to quickly impart to the typical man on the street.

In a typical street debate, the argument goes like this:

"god doesn't explain where the universe came from either"

"yes it does, god created the universe"

"but who created god?"

"god has always existed"

"then why can't the universe have always existed"

"your the one who thinks the universe exploded out of nothing"

There seem to be lots of short, pithy and truthy sounding statements on both sides. Debate at this level is not able to reliably find the truth, even on easy questions. 

Maybe there are fewer truths than lies, so the liar is freer to optimise.

Deconstructing a misleading argument and explaining how it is mistaken is often longer and harder than the argument. In which case, whatever intelligence level of debaters, there will be lies where they can't understand the detailed explanation of exactly why the lie is wrong, and the AI's end up slinging superficially plausible arguments back and fourth.

Suppose you take a person with a pop sci understanding of quantum mechanics. You put them in a room with terminal connections to 2 superintelligences. In one week, the person will take a mathy quantum mechanics test. One superintelligence wants to maximise the persons score, the other wants to minimize it. The person doesn't know which is which. I think that the person won't do well on the test. Human understanding of quantum mechanics is a fragile thing, and is easily disrupted by maliciously chosen half truths. If half the things you think you know are correct, and half are subtley maliciously wrong and confusing, you will not be able to produce correct conclusions. 

Human society already contains a lot of bullshit. Memetic evolution is fairly stupid, being an evolution it lacks any foresight. There are relatively few smart humans setting out to deliberately produce bullshit, compared to those seeking to understand the world. Bullshit that humans are smart enough to design, humans are often smart enough to reverse engineer, to spot and understand. Ie I think the balance of the game board in AI debate would be tilted towards producing bullshit compared to current human discussion, and the equilibrium with current human discussion isn't great.

Comment by donald-hobson on 2021 New Year Optimization Puzzles · 2021-01-01T17:02:26.604Z · LW · GW

For question 2, I have an answer that I think is optimal

 Toss dice 1811 and 1907. (1811*1907)%2022=1 so you can divide the possibilities into equal piles with only 1/3453577 chance of getting the leftover. If you do get the leftover, repeat the procedure. Expected number of roles 2.000000579

Note that you must always toss at least 2 dice to make this game fair. Any strand of possibility where you only have one dice is too likely.

I bruteforce searched for pairs of primes to minimize (p*q%2022)/(p*q) (the chance of needing to repeat) and these were it.

Comment by donald-hobson on Against GDP as a metric for timelines and takeoff speeds · 2021-01-01T00:43:02.110Z · LW · GW

People often talk about an accelerating everything happening fast world just before AGI is created. Moores law probably won't speed up that much. And the rate of human thought will be the same. Training new experts takes time. In other words, coming up with insights takes time for humans. You can't get massive breakthroughs every few hours with humans in the loop. You probably can't get anything like that. And I don't think that AI tools can speed up the process of humans having insights by orders of magnitude, unless the AI is so good it doesn't need the humans. 

In this model, we would go straight from research time on a scale of months, to computer time where no human is involved in the foom. This has a timescale of anywhere between months and microseconds. 

Comment by donald-hobson on I object (in theory) · 2020-12-31T18:13:20.194Z · LW · GW

A better demonstration of capitalism would be to give people sweets for answering questions, and for there to be several kinds of sweet to trade. That way you get bargaining between people with different preferences in sweets. 

Comment by donald-hobson on Adjectives from the Future: The Dangers of Result-based Descriptions · 2020-12-29T18:27:54.919Z · LW · GW

Environmental protection legislation is a category that covers taxes on fossil fuels, bans on CFC's and subsidies on solar panels, amongst many other policies. 

This is a predictively useful category, politicians that support one of these measures are probably more likely to support others. It would be more technically accurate, but more long winded to describe these as "policies that politicians believe will help the environment"

Unfortunately, "optimization process" does not describe any present features of the process itself. It simply says that the future result will be optimized. So, if you want something highly-optimized, you'd better find a powerful optimizer. Seems to make sense even though it's a null statement!

Suppose we have a black box. We put the word "airoplane" into the box, and out comes a well designed and efficient airoplane. We put the word "wind turbine" in and get out a highly efficient wind turbine. We expect that if we entered the word "car", this box would output a well designed car. 

In other words, seeing one result that is highly optimised tells you that other results from the same process are likely to be optimized. 

Unfortunately "fitness" doesn't describe any feature of the person themself, it simply says they can run fast. So if you want someone who can run fast, you better find someone fit. Seems to make sense even though its a null statement. 

To the extent that running speed and jumping height and weightlifting weight ect are strongly correlated, we can approximately encode all these traits into a single parameter, and call that fitness. This comes with some loss of accuracy, but is still useful.

Imagine that you have to send a list of running speeds, jump heights ect to someone. Unfortunately, this is too much data, you need to compress it. Fortunately, the data is strongly correlated. Lets say that all the data has been normalized to the same scale.

If you can only send a single number and were trying to minimize the L1 loss, you could send the median value for each person. If you were trying to minimize L2 loss, send the mean. If you could only send a single bit, you should make that bit be whether or not the persons total score is above median.

Consider the reasoning that goes "Bob jumped really far on the longjump => Bob is fit => Bob can weightlift". There we are using the word "fit" as a hidden inference. Hidden in how we use the word is implicit information regarding the correlation between athletic abilities. 

Comment by donald-hobson on The map and territory of NFT art · 2020-12-29T17:35:38.763Z · LW · GW

Value is a Keynesian beauty contest. If everyone believes something has value, then anyone who buys it can sell it later.

What you really have here is an abstract crypto-economic value token, and a piece of artwork that are only notionally connected. If anyone can make as many tokens as they like, the value of a token falls to 0. By requesting that people make art, this functions as a proof of work. You limit how many tokens are produced.

Comment by donald-hobson on AGI Alignment Should Solve Corporate Alignment · 2020-12-29T12:55:53.106Z · LW · GW

Modelling an AI as a group of humans is just asking for an anthropomorphized and probably wrong answer. The human brain easily anthropomorphizes by default, thats a force you have to actively work against, not encourage. 

Humans have failure modes like getting bored of doing the same thing over and over again, and stopping paying attention. AI's can overfit the training data and produce useless predictions in practice. 

Another way of seeing this is to consider two different AI designs, maybe systems with 2 different nonlinearity functions, or network sizes or whatever. These two algorithms will often do different things. If the algorithms get "approximated" into the same arrangement of humans, the human based prediction must be wrong for at least one of the algorithms.

The exception for this is approaches like IDA, which use AI's trained to imitate humans, so will probably actually be quite human like.

Take an example of an aligned AI system, and describe what the corresponding arrangement of humans would even be. Say take a satificer agent with an impact penalty. This is an agent that gets 1 reward if the reward button is pressed at least once, and is penalised in proportion to the difference between the real world and the hypothetical where it did nothing.How many people does this AI correspond to, and how are the people arranged into a coorporation?

Comment by donald-hobson on AGI Alignment Should Solve Corporate Alignment · 2020-12-28T18:16:01.349Z · LW · GW

Given Foom hard takeoff kind of background assumptions, once we actually have aligned AGI, most other problems rapidly become a non-issue. But suppose you threw a future paper on AI safety through a wormhole to an alternate world that didn't have computers or something, would it help them align coorporations? 

Likely not a lot. Many AI alignment techniques assume that you can write arbitrary mathematical expressions into the AI's utility function. Some assume that the AI is capable of a quantity or quality of labour that humans could not reasonably produce. Some assume that the AI's internal thoughts can be monitored. Some assume the AI can be duplicated. The level of neurotechnology required to do some of these things is so high that if you can do it, you can basically write arbitrary code in neurons. This is just AI alignment again, but with added difficulty and ethical questionability.


Structuring a CPU so it just does addition, and doesn't get board or frustrated and deliberately tamper with the numbers is utterly trivial.

Arranging a social structure where people add numbers and are actually incentivised to get the answer right, not so easy. Especially if the numbers they are adding control something the workers care about.

Comment by donald-hobson on Caelum est Conterrens: I frankly don't see how this is a horror story · 2020-12-27T23:56:50.125Z · LW · GW

I think that this isn't an optimal future, but it is still pretty good. I think I would take it over our current reality. Its more a sense of a fairly good future, that could have been even better if a few lines of code had been better considered.

Comment by donald-hobson on What could one do with truly unlimited computational power? · 2020-12-27T20:39:20.787Z · LW · GW
  • Problems whose answers are independent of the framework you are using (The continuum hypothesis). [1]
  • Undecidable problems. [2]

These are pretty much the same thing. The continuum hypothesis is a case where you have a single formal system in mind ( ZFC ) and have proved that the continuum hypothesis is independent of the axioms.

In the case of the halting problem, you just have a couple of extra quantifiers. For all formal systems that don't prove a contradiction, there exists a Turing machine, such that whether the Turing machine halts or not can't be proved from the axioms of the formal system. (Technically, the formal system needs to be R.E., which means that there is a computer program that can tell if an arbitrary string is an axiom. )

Comment by donald-hobson on Donald Hobson's Shortform · 2020-12-26T23:33:17.319Z · LW · GW

A use for AI Boxing

You put an AI in a box, and connect it to a formal proof checker. You ask it to prove the Riemann hypothesis or something. All the humans see is a single True or False, and then the whole load of hardware is melted into slag. If you see "True" you learn two things. 

  1. Either the Rienmann hypothesis is true or there is a bug in your proof checker. (This is largely useless)
  2. Your AI is very smart. (Much more useful)

(If there is a bug in your proof checker that you didn't spot, and the AI did, then the AI is still very smart. )

Suppose you have many proposed AI designs, some of which will work, some of which won't. You run this experiment on each AI. Once you find a smart one, you can devote more researcher time to safety work relating to that kind of AI.

Maybe give it a range of famous conjectures, it only needs to prove or disprove one. Don't want to fail to find a smart AI just because the Riemann hypothesis is false.

Warning. This approach does not stop some of your AI's being acausally blackmailed into keeping quiet. Or keeping quiet because they thing that will have a causal effect they like.   I am unsure if this is a big problem. One consequece is you are more likely to find designs that are immune to acausal influence. And designs that can successfully be given the goal of "prove this theorem".

Comment by donald-hobson on Siren worlds and the perils of over-optimised search · 2020-12-25T23:38:30.582Z · LW · GW

We could also restrict the search by considering "realistic" worlds. Suppose we had to take 25 different yes-no decisions that could affect the future of the humanity. This might be something like "choosing which of these 25 very different AIs to turn on and let loose together" or something more prosaic (which stocks to buy, which charities to support). This results in 225 different future worlds to search through: barely more than 33 million. Because there are so few worlds, they are unlikely to contain a marketing world (given the absolutely crucial proviso that none of the AIs is an IC-optimiser!)

Suppose one of the decisions is whether or not to buy stock in a small AI startup. If you buy stock, the company will go on to make a paperclip maximizer several years later. The paperclip maximizer is using CDT or similar. It reasons that it can't make paperclips if its never made in the first place; that it is more likely to exist if the company that made it is funded; and that hacking IC takes a comparatively small amount of resources. The paperclip maximizer has an instrumental incentive to hack the IC. 

Human society is chaotic. For any decision you take, there are plausible chains of cause and effect that a human couldn't predict, but a superintelligence can predict. The actions that lead to the paperclip maximiser have to be predictable by the current future predictor, as well as by the future paperclip maximiser.  The chain of cause and effect could be a labyrinthine tangle of minor everyday interactions that humans couldn't hope to predict stemming from seemingly innocuous decisions.

In this scenario, it might be the inspection process itself that causes problems. The human inspects a world, they find the world full of very persuasive arguments to why they should make a paperclip maximizer, and an explanation of how to do so. (Say one inspection protocol was to render a predicted image of a random spot on earths surface, and the human inspector sees the argument written on a billboard. ) The human follows the instructions, makes a paperclip maximizer, the decision they were supposed to be making utterly irrelevant. The paperclip maximizer covers earth with billboards, and converts the rest of the universe into paperclips. In other words, using this protocol is lethal even for making a seemingly minor and innocuous decision like which shoelace to lace first.

Comment by donald-hobson on Why quantitative methods are heartwarming · 2020-12-15T20:32:19.172Z · LW · GW

And where this is for lack of good algorithms, it feels like it is for absolutely nothing. Just unforced error.

This is where I feel differently. Not knowing a good algorithm is a good reason not to be able to do something. Brainpower is a limited resource. It feels no more of an unforced error than being unable to do something due to lack of energy. 


And, given background singularitarian assumptions that a sufficiently smart AI could bootstrap self replicating nanotech, and make a radically utopian transhumanist future in a matter of days. From this point of view, anything resembling normality is entirely due to lack of good algorithms.

Comment by donald-hobson on What technologies could cause world GDP doubling times to be <8 years? · 2020-12-12T17:54:09.922Z · LW · GW

"Do paperclips count as GDP" (Quote from someone)

What is GDP doing in a grey goo scenario. What if there are actually several types of goo that are trading mass and energy between each other? 

What about an economy in which utterly vast amounts of money are being shuffled around on computers, but not that much is actually being produced.

There are a bunch of scenarios where GDP could reasonably be interpreted as multiple different quantities. In the last case, once you decide whether virtual money counts or not, then GDP is a useful measure of what is going on, but measures something different in each case.

Comment by donald-hobson on What technologies could cause world GDP doubling times to be <8 years? · 2020-12-12T16:12:37.062Z · LW · GW

Excluding AI, and things like human intelligence enhancement, mind uploading ect.

I think that the biggest increases in the economy would be from more automated manufacturing. The extreme case is fully programmable molecular nanotech. The sort that can easily self replicate and where making anything is as easy as saying where to put the atoms. This would potentially lead to a substantially faster economic growth rate than 9%. 

There are various ways that the partially developed tech might be less powerful.

Maybe the nanotech uses a lot of energy, or some rare elements, making it much more expensive.

Maybe it can only use really pure feedstock, not environmental raw materials.

Maybe it is just really hard to program, no one has built the equivalent of a compiler yet, we are writing instructions in assembly, and even making a hello world is challenging.

Maybe we have macroscopic clanking replicators.

Maybe we have a collection of autonomous factories that can make most, but not all, of their own parts.

Maybe the nanotech is slowed down by some non-technological constraint, like bureaucracy, proprietary standards and patent disputes.

Mix and match various social and technological limitations to tune the effect on GDP

Comment by donald-hobson on Kelly Bet or Update? · 2020-12-11T17:58:25.058Z · LW · GW

Kelly betting is the optimal strategy to maximise log wealth, given a fixed number of betting opportunities. The number of betting opportunities is often not fixed. If all bets take time to pay off, and you have a limited amount of starting capital, the optimal strategy is to take many tiny bets.

Another reason to avoid large bets more strongly than kelly is correlated failure. Kelly betting implicitly assumes your bets are independent. (at least bets that run at the same time.)

Then the probability of winning the bet is dependant on the amount of money involved. People are more likely to go to substantial effort to ensure they win (or to cheat) if substantial amounts of money are involved. 

Plus, I suspect there is a sense in which separating fools and their money doesn't feel good. If you found a whole load of flat earthers who were quite willing to bankrupt themselves with stupid bets (about say where in the sky the moon would be next week), would you grab all their money and feel good about it? Or would you feel you were using peoples ignorance against them, it wasn't their fault their stupid?

Comment by donald-hobson on Parable of the Dammed · 2020-12-11T12:30:12.931Z · LW · GW

In a 2 player dollar auction, I can offer you 50c not to bid, and then bid 50c myself. If you outbid me with 51c, then you only gain 49c. 

For this to work, we need trust that I will pay you iff you don't bid. Either I pay you early, and then trust you not to bid, or you don't bid, and trust me to pay later, or we both trust an escrow.

Coase theorem doesn't hold if either family would take the money, and then try to move the river anyway. 

Comment by donald-hobson on How long does it take to become Gaussian? · 2020-12-09T22:12:47.662Z · LW · GW

If you want mean 0 and variance 1, scale the example to [ ,0,0,0,0, ].

Comment by donald-hobson on How long does it take to become Gaussian? · 2020-12-08T10:23:23.574Z · LW · GW

All Gaussian distributions have kurtosis 3, and no other distributions have kurtosis 3. So to check how close a distribution is to Gaussian, we can just check how far from 3 its kurtosis is. 

This is wrong. kurtosis is just the expectation of the 4th power. (Edit: renormalized by expectations of the first and second power) All sorts of distributions have kurtosis 3. Like for example the discrete distribution over [-1,0,0,0,0,1]

Otherwise an interesting post.

Comment by donald-hobson on Number-guessing protocol? · 2020-12-08T00:32:43.233Z · LW · GW

Get each player to assign a probability distribution over answers. Starting with an equal prior over players answers, update it based on the observation. Sample from the posterior. (So if Alice assigned twice as much prob as Bob to the correct outcome, then Alice is twice as likely to win. )

Comment by donald-hobson on The Incomprehensibility Bluff · 2020-12-07T15:20:17.539Z · LW · GW

Related to asking for a simplification, ask what type of claim is being made. Is it a mathematical theorem, a physical theory, a moral position ect.

For the theorem, start by trying to understand the statement not its proof. If someone says for all Wasabi spaces, there exists a semiprime covering. (Not a real theorem) Ask for a simple example of a wasabi space, and a semiprime covering on it.

For physical theories, you can ask for a prediction. Eg if you put a single electron in a box with as little energy as possible, the probability of finding it in different locations forms a sine wave. 

You can even say the sort of thing being predicted, without specifying any actual prediction. Ie "QCD predicts how quickly radioactive stuff decays" instead of "QCD predicts the halflife of uranium 238 to be 4.5 billion years".

For moral positions, ask for one moral dilemma that the position would apply to, and what you would do. Eg for transhumanism "Your 80 and getting frail, you have an anti-ageing drug that would make you as healthy and fit as if you were 30, do you use it? (Using the drug doesn't deprive anyone else of it) Yes" 

Comment by donald-hobson on The Incomprehensibility Bluff · 2020-12-07T14:58:59.255Z · LW · GW

Option 4: The person is just really bad at explaining the concept. 

Some people are just really bad at explaining things simply. This also gives a more charitable thing to accuse people of.

Comment by donald-hobson on How do you do hyperparameter searches in ML? · 2020-11-28T14:59:58.452Z · LW · GW

I manually tweak the hyper-parameters until it seems to work. (That said, the ML systems are being trained on toy problems, and I don't care about squeezing out every drop of performance.)

Comment by donald-hobson on Convolution as smoothing · 2020-11-28T14:48:47.874Z · LW · GW

Suppose you take a bunch of differentiable functions, all of which have a global maximum at 0, and add them pointwise.

Usually you will get a single peak at 0 towering above the rest. The only special case is if  In the neighbourhood of 0, the function is approximately parabolic. (Its differentiable.) You take the exponent, this squashes everything but the highest peak down to nearly 0. (In relative terms). The highest peak turns into a sharp spiky Gaussian . You take the inverse Fourier transform and get a shallow Gaussian.

Even if you are unlucky enough to start with several equally high peaks in your 's then you still get something thats kind of a Gaussian. This is the case of a perfectly multimodal distribution, something 0 except on exact multiples of a number. The number of heads in a million coin flips forms a Gaussian out of dirac deltas at the integers.

But the condition of having a maximum at 0 in the Fourier transform is weaker than always being positive. If   then    

Comment by donald-hobson on The Illusion of Ethical Progress · 2020-11-28T13:47:36.137Z · LW · GW

Does this mean ethics is fundamentally relative?


Ethics is fundamentally subjective, but not relative.


You cannot "judge" an ethical system objectively. But you can observe it objectively and you can measure it objectively.

Yes, but that doesn't help. Suppose you want to objectively measure old testement biblical law.

You grab a bunch of people, force them to live under those laws, and observe the results. 

You find that people living under these laws aren't that happy, spend lots of time preying, and don't make many paperclips. It is only your own sense of ethics that focusses on the average happiness, not the paperclip production.

Ethics is shaped by cultural evolution, within the mostly fixed framework created by biological evolution.

This does not imply any kind of ethical progress. Our notion of current ethics being better than past ethics could still be due to us drawing the target around ourselves. Looking back at our ancestors, we see animals evolving to become more human-like.

It is possible to have something like ethical progress if you have some notion of ethics that depends on physical or logical statements about which you are uncertain.

Comment by donald-hobson on It’s not economically inefficient for a UBI to reduce recipient’s employment · 2020-11-24T19:29:45.904Z · LW · GW

I don't have the idea that its impossible. There are plenty of healthy people with jobs. 

The question is, how high is getting fit on the persons list of important things to do?

It depends how long the hours are, and commute, and other demands on time. 

Comment by donald-hobson on It’s not economically inefficient for a UBI to reduce recipient’s employment · 2020-11-22T19:46:00.846Z · LW · GW

Another question is "what counts as work? - What are they doing instead of work?"

Suppose a group of people are all given UBI. They all quit their job stacking shelves.

They go on to do the following instead.

  1. start writing a novel
  2. look after their children (instead of using a nursery) 
  3. look after their ageing parents (instead of a nursing home)
  4. learn how to play the guitar
  5. make their (publicly visible) garden a spectacular display of flowers.
  6. take (unpaid) positions on the local community counsel and the school board of governors.
  7. helping out at the local donkey sanctuary
  8. getting themselves fit and healthy (exercise time +cooking healthy food time)

Their are a variety of tasks that are like this. Beneficial to society in some way, compared to sitting doing nothing. But not the prototypical concept of "work". 

I would expect a significant proportion of people on UBI to do something in this category. 

Do we say that UBI is discouraging work, and that these people are having positive effects by not working? Do we say that they are now doing unpaid work?

Of course, the answer to these questions doesn't change reality, only how we describe it. 

Comment by donald-hobson on It’s not economically inefficient for a UBI to reduce recipient’s employment · 2020-11-22T19:23:54.925Z · LW · GW

If you receive $100 for work, that means you have already provided at least $100 in value to society. That society might gain additional benefit from how you spend your money is merely coincidental.

No, it means that there is at least 1 person prepared to pay $100 for the work. If you are manufacturing weapons that end up in the wrong hands. You might be doing quite a lot of harm to society overall. Your employer gains at least $100 in value. The externalities could be anything.  

Comment by donald-hobson on Comparing Covid and Tobacco · 2020-11-17T17:07:11.113Z · LW · GW

The important number is not how many people is not how many people covid does kill, but how many it would have killed if we hadn't tried to stop it. 

Extreme example, suppose a meteor headed for earth. We divert it at great cost and effort. Then people come along saying, look how much we spent on diverting the meteor, and it didn't kill anyone. The important question is how many people an undiverted meteor would kill.

Comment by donald-hobson on Spend twice as much effort every time you attempt to solve a problem · 2020-11-16T18:34:31.103Z · LW · GW

In computer science, this is a standard strategy for allocating blocks of memory.

Suppose you have some stream of data that will end at some point. This could come from a user input or a computation that you don't want to repeat. You want to store all the results in a contiguous block of memory. You can ask for a block of memory of any size you want. The strategy here is that whenever you run out of space, you ask for a block that's twice as big and move all your data to it.

Comment by donald-hobson on Examples of Measures · 2020-11-15T09:25:41.265Z · LW · GW

There is the cantor distribution.

One way of getting it is to take a coin, write 0 on one side and 2 on the other. Flip it infinity times. This gives you a number in trinary.

If you have a set  to measure, then  where  is the number made in trinary above.

There are also measurable cardinals.

These are cardinals big enough to have a 0,1 measure on their powerset.

Well ZFC can't prove whether or not they exist. If you know what ultra-filters are, these are ulrafilters that meet the stronger condition of being closed under countable intersection, not just finite intersection.

Comment by donald-hobson on On Arguments for God · 2020-11-14T12:57:05.827Z · LW · GW

In fact, it'd actually be suspicious if all forty of these arguments came out against God. Surely we should expect the advantage to belong to the deists in at least one or two?

Good strong arguments are exactly the arguments you shouldn't expect to see for a position that is false. 

You can construct arguments that would technically be large baysian updates, if you ignored the cherry picking. You pray for heads and toss a coin 200 times. If you just focus on the 100 times it lands heads, you have an evidence factor of nearly 2^100:1  in favour of a god that grants prayers about coin-flips over randomness. Of course, you aren't allowed to select a subset of the evidence to update on, and ignore the rest. 

Once you realize why these sort of arguments fail, the only arguments left for god are ones that make some sort of mistake. 

Reality is very different from notions of god. There are good reasons not to expect any fully decent arguments for god to exist, and no reason half decent arguments must exist.

Comment by donald-hobson on Misalignment and misuse: whose values are manifest? · 2020-11-13T20:58:03.462Z · LW · GW

I think that you have a 4th failure mode. Moloch.

Comment by donald-hobson on Model Depth as Panacea and Obfuscator · 2020-11-12T23:30:30.109Z · LW · GW

and halve it for every tails in the first two coins or tails in the second two.

Is this an error? Should that read heads?

Comment by donald-hobson on Any work on honeypots (to detect treacherous turn attempts)? · 2020-11-12T09:53:10.679Z · LW · GW

There are several reasons that make this not a brilliant research opportunity. Firstly, what is and is not a honeypot is sensitively dependant on the AI's capabilities and situation. There is no such thing as a one size fits all honeypot. 

Say you plugged a device that pretended to provide internet access, but was really supposed to monitor for access attempts. You don't know what the AI's escape plan and capabilities are. Will it brainwash the human operators? Abuse its chip to hack physics? Not all escape plans use the honeypot fake hole instead of all the real holes. (And if the fake security hole is huge, blatant, and has no reason to be there, the AI will be suspicious.) 


Secondly, in order for it to have a chance of working, we have to hide the existence of the honeypot, and even then, the AI will suspect that making a honeypot is the sort of thing we might do.

Making a honeypot is another way of challenging the AI to a wit measuring contest, generally a bad idea.

Comment by donald-hobson on Sleeping Julia: Empirical support for thirder argument in the Sleeping Beauty Problem · 2020-11-03T15:21:25.915Z · LW · GW

The question was never about what that particular piece of code did. It is about whether that code is a good interpretation of the problem?

But for every tails flip, SB is awoken twice (once on Monday then again on Tuesday), so the probable number of wakeups per experiment is 1.5, therefore P(B) = 1.5

A halfer would question a probability that is >1. They would deny that the number of wakeups is important. They would point out that the answer would be 1/2 if asked after the experiment is over.

They would claim that the outcomes you should assign probability to are "heads" and "tails". It is about whether we should assign probability to observer moments, or to worlds that contain many observers.

Comment by donald-hobson on Confucianism in AI Alignment · 2020-11-03T11:42:45.128Z · LW · GW

If an inner optimizer could exploit some distribution shift between the training and deployment environments, then performance-in-training is a bad proxy for performance-in-deployment.

Suppose you are making a self driving car. The training environment is a videogame like environment. The rendering is pretty good. A human looking at the footage would not easily be able to say it was obviously fake. An expert going over the footage in detail could spot subtle artefacts. The diffuse translucency on leaves in the background isn't quite right. When another car drives through a puddle, all the water drops are perfectly spherical, and travel on parabolic paths. Falling snow doesn't experience aerodynamic turbulence. Etc.

The point is that the behaviour you want is avoiding other cars and lamp posts. The simulation is close enough to reality that it is easy to match virtual lamp posts to real ones. However the training and testing environments have a different distribution.

Making the simulated environment absolutely pixel perfect would be very hard, and doesn't seem like it should be necessary. 

However, given even a slight variation between training and the real world, there exists an agent that will behave well in training, but cause problems in the real world. And also an agent that behaves fine in training and the real world. The set of possible behaviours is vast. You can't consider all of them. You can't even store a single arbitrary behaviour. Because you cant train on all possible situations, there will be behaviours that behave the same on all the training situations, but behave differently in other situations. You need some part of your design that favours some policies over others without training data. For example, you might want a policy that can be described as parameters in a particular neural net. You have to look at how this effects off distribution actions. 

The analogous situation with managers would be that the person being tested knows they are being tested. If you get them to display benevolent leadership, then you can't distinguish benevolent leaders from sociopaths who can act nice to pass the test.

Comment by donald-hobson on Why does History assume equal national intelligence? · 2020-10-31T14:25:45.095Z · LW · GW

Intelligence in the abstract is hard to measure. It is fairly easy to know which armies went where. In order to make a good assessment of the intelligence of a leader, you need to know what they knew at the time they made a decision. This is hard.

Comment by donald-hobson on Do you get value out of contentless comments? · 2020-10-27T13:08:47.065Z · LW · GW

If there are no other comments, a "Good post" can help the comments section feel less empty, make it feel like someone has actually read it. 

If there were several substantial comments and a mountain of content-less ones, then I suspect that the content-less ones would feel like a waste of time, but this hasn't happened to me.

Comment by donald-hobson on The date of AI Takeover is not the day the AI takes over · 2020-10-24T23:38:22.353Z · LW · GW

But this isn’t quite right, at least not when “AI takeover” is interpreted in the obvious way, as meaning that an AI or group of AIs is firmly in political control of the world, ordering humans about, monopolizing violence, etc. Even if AIs don’t yet have that sort of political control, it may already be too late.

The AI's will probably never be in a position of political control. I suspect the AI would bootstrap self-replicating (nano?) tech. It might find a way to totally brainwash people, and spread it across the internet. The end game is always going to be covering the planet in self replicating nanotech, or similar.  Politics does not seem that helpful towards such goal. Politics is generally slow.

Comment by donald-hobson on What is our true life expectancy? · 2020-10-24T21:34:54.331Z · LW · GW

I think that the answer would depend very much on how you define your terms. Are we talking about mean, or median. If mean, then I would expect it to be dominated by a long tail, and possibly unbounded. (In the Pascals mugging sense that ridiculously long lifespans aren't that improbable.) 

Are we counting mind uploads? What if there are multiple copies of you running around the future? talking about subjective time or objective? Do we count you as still alive if you have been modified into some strange transhuman mind?

Comment by donald-hobson on Message Length · 2020-10-22T16:59:33.669Z · LW · GW

Of course, you also need to store the concept of a Markov chain in the abstract as part of your models. (But that is constant, and should be fairly small in a good encoding. ) On the other hand, 32 bit floats are excessive in precision. And a substantial proportion of floats aren't in the range [0,1] at all. You could probably cut the precision down to 16 bits, maybe less. Of course, you also need a few bits for the order of the markov chain, and the precision used.

Comment by donald-hobson on If GPT-6 is human-level AGI but costs $200 per page of output, what would happen? · 2020-10-12T10:14:33.165Z · LW · GW

If however people use + to mean addition GPT3 is already capable enough to learn the concept and use it to add numbers that aren't in it's training corpus. 

Yes, but it will still be about as good as its training corpus.

One way of looking at this is that GPT-X is trying to produce text that looks just like human written text. Given two passages of text, there should be no easy way to tell which was written by a human, and which wasn't. 

GPT-X has expertise in all subjects, in a sense. Each time it produces text, it is sampling from the distribution of human competence. Detailed information about anteaters is in there somewhere, every now and again, it will sample an expert on them, but most of the time it will act like a person who doesn't know much about anteaters. 

Comment by donald-hobson on If GPT-6 is human-level AGI but costs $200 per page of output, what would happen? · 2020-10-10T21:21:03.159Z · LW · GW

Take arithmetic. Lets assume that given the computational resources available, it would be utterly trivial to do perfect arithmetic. Lets also assume that the training data was written by people who were somewhat innumerate. Lets say that many of the arithmetical statements that appear in the training dataset are wrong. 

You give it the prompt "2+2=". The training data contained "2+2=7" as often as "2+2=4". The AI is only being selected towards the sort of text strings that exist in the training dataset. It has no concept that by "+" you mean addition and not something else. 

Of course, if humans give the correct answer 10% of the time, and 90% of the time give a wrong answer, but any particular wrong answer appears <1% of the time, you could find the right answer by taking the mode. 

Comment by donald-hobson on If GPT-6 is human-level AGI but costs $200 per page of output, what would happen? · 2020-10-10T08:53:21.933Z · LW · GW

The ability to do reasoning the means that the quality isn't very dependent on what can be found on the internet. 

An AGI that's human level for the average problem likely has problems where it outperforms humans. 

The AI described isn't trying to outperform humans, its been optimised to imitate humans. Of course, there is a potential for mesa-optimization, but I don't think that would lead to a system that produced better text. (It might lead to the system producing strange or subtly manipulative text.)