Would Covid19 patients benefit from blood transfusions from people who have recovered? 2020-03-29T22:27:58.373Z · score: 13 (7 votes)
Programming: Cascading Failure chains 2020-03-28T19:22:50.067Z · score: 8 (5 votes)
Bogus Exam Questions 2020-03-28T12:56:40.407Z · score: 7 (2 votes)
How hard would it be to attack coronavirus with CRISPR? 2020-03-06T23:18:09.133Z · score: 8 (4 votes)
Intelligence without causality 2020-02-11T00:34:28.740Z · score: 9 (3 votes)
Donald Hobson's Shortform 2020-01-24T14:39:43.523Z · score: 5 (1 votes)
What long term good futures are possible. (Other than FAI)? 2020-01-12T18:04:52.803Z · score: 9 (2 votes)
Logical Counterfactuals and Proposition graphs, Part 3 2019-09-05T15:03:53.262Z · score: 6 (2 votes)
Logical Counterfactuals and Proposition graphs, Part 2 2019-08-31T20:58:12.851Z · score: 15 (4 votes)
Logical Optimizers 2019-08-22T23:54:35.773Z · score: 12 (9 votes)
Logical Counterfactuals and Proposition graphs, Part 1 2019-08-22T22:06:01.764Z · score: 23 (8 votes)
Programming Languages For AI 2019-05-11T17:50:22.899Z · score: 3 (2 votes)
Propositional Logic, Syntactic Implication 2019-02-10T18:12:16.748Z · score: 6 (5 votes)
Probability space has 2 metrics 2019-02-10T00:28:34.859Z · score: 90 (38 votes)
Allowing a formal proof system to self improve while avoiding Lobian obstacles. 2019-01-23T23:04:43.524Z · score: 6 (3 votes)
Logical inductors in multistable situations. 2019-01-03T23:56:54.671Z · score: 8 (5 votes)
Boltzmann Brains, Simulations and self refuting hypothesis 2018-11-26T19:09:42.641Z · score: 0 (2 votes)
Quantum Mechanics, Nothing to do with Consciousness 2018-11-26T18:59:19.220Z · score: 13 (13 votes)
Clickbait might not be destroying our general Intelligence 2018-11-19T00:13:12.674Z · score: 26 (10 votes)
Stop buttons and causal graphs 2018-10-08T18:28:01.254Z · score: 6 (4 votes)
The potential exploitability of infinite options 2018-05-18T18:25:39.244Z · score: 3 (4 votes)


Comment by donald-hobson on Implications of the Doomsday Argument for x-risk reduction · 2020-04-03T14:27:11.633Z · score: 6 (3 votes) · LW · GW

Suppose we ignore the simulation argument and take the evidence of history and astronomy at face value. The doomsday argument provides a good prior. However, the evidence that shows we are on early earth is really strong, and the prior is updated away. If we take the simulation hypothesis into account, then there could be a version of us in reality, and many in simulations. The relative balance of preventing X risk vs having a good time is swung, but still strongly cares about X risk. Actually, the doomsday argument puts the probability that infinitely many people will exist, but only finitely many have existed so far at 0, so I'm don't think I believe it.

Comment by donald-hobson on Solipsism is Underrated · 2020-03-29T11:32:15.839Z · score: 1 (1 votes) · LW · GW

Someone that knows quantum physics but almost no computing looks at a phone. They don't know how it works inside. They are uncertain about how apps result from material phenomenon. This is just normal uncertainty over a set of hypothesis. One of those hypotheses is the actual answer, many others will look like alternate choices of circuit board layout or programming language. They still need to find out how the phone works, but that is because they have many hypothesis that involve atoms. They have no reason to doubt that the phone is made of atoms.

I don't know how your brain works either, but I am equally sure it is made of (atoms, quantum waves, strings or whatever). I apply the same to my own brain.

In the materialist paradigm I can understand Newtonian gravity as at least an approximation of whatever the real rules are. How does a solipsist consider it?

Comment by donald-hobson on Solipsism is Underrated · 2020-03-28T23:53:27.717Z · score: 4 (4 votes) · LW · GW
Suppose that you update on the evidence that you experience conscious qualia

What exactly would it mean to perform a baysian update on you not experiencing qualia?

The only ontological primitive is my own mind.

The primitives of materialism are described in equations. Does a solipsist seek an equation to tell them how angry they will be next Tuesday? If not, what is the substance of a solipsistic model of the world?

This belief in some mysterious ability for the mental to supervene on the physical

I am not sure what you mean my that, I consider my mind to be just an arrangement of atoms. An arrangement governed by the same laws as the rest of the universe.

how puzzling is the view, that the activity of these little material things somehow is responsible for conscious qualia?

I am not sure where the instinct that consciousness can't be materialistic comes from, although I would suspect that it might come from a large amount of uncertainty, and an inability to imagine any specific answer that you would consider a good explanation. Wherever this instinct comes from, I don't think it is reliable.

You know that "if a tree falls in a forest, and there is no one there to hear it, does it make a sound?" thing. Even after all the factual questions, like if audio equipment would record something, have been answered, there is a feeling of a question remaining. I expect any explanation of qualia to look somewhat similar, a description of how mental imperfections produce a sensation of something.

Consider the limiting case of describing minds in terms of algorithms, you scan a philosophers brain, put the data into a computer, and predict exactly their discussion on qualia. Once you have a complete understanding of why the philosopher talks about qualia, if the philosopher has any info about qualia at all, the process by which they gained that info should be part of the model.

Pick something up, drop it, watch it fall. Can solipsism consider this observation to be more likely than some max entropy observation? How does a solipsist predict the experience of watching the object fall.

Comment by donald-hobson on What are the most plausible "AI Safety warning shot" scenarios? · 2020-03-27T11:55:29.740Z · score: 3 (2 votes) · LW · GW

I agree that these aren't very likely options. However, given two examples of an AI suddenly stopping when it discovers something, there are probably more for things that are harder to discover. In the pascel mugging example, the agent would stop working, only when it can deduce what potential muggers might want it to do, something much harder than noticing the phenomenon. The myopic agent has little incentive to make a non myopic version of itself. If dedicating a fraction of resources into making a copy of itself reduced the chance of the missile hacking working from 94%, to 93%, we get a near miss.

One book, probably not. A bunch of books and articles over years, maybe.

Comment by donald-hobson on AGI in a vulnerable world · 2020-03-26T23:46:36.572Z · score: 2 (4 votes) · LW · GW

I put non-trivial probability mass (>10%) on a relitivisticly expanding bubble of Xonium (computronium, hedonium ect) within 1 second of AGI.

While big jumps are rarer than small jumps, they cover more distance, so it is quite possible we go from a world like this one, except with self driving cars, and a few other narrow AI applications to something smart enough to bootstrap very fast.

Comment by donald-hobson on What are the most plausible "AI Safety warning shot" scenarios? · 2020-03-26T23:24:21.283Z · score: 5 (3 votes) · LW · GW
A "AI safety warning shot" is some event that causes a substantial fraction of the relevant human actors (governments, AI researchers, etc.) to become substantially more supportive of AI research and worried about existential risks posed by AI.

A really well written book on AI safety, or other public outreach campaign could have this effect.

For many events, such as a self driving car crashing, it might be used as evidence for an argument about AI risk.

On to powerful AI systems causing harm, I agree that your reasoning applies to most AI's. There are a few designs that would do something differently. Myopic agents are ones with lots of time discounting within their utility function. If you have a full super-intelligence that wants to do X as quickly as possible, such that the fastest way to do X will also destroy itself, that might be survivable. Consider an AI set to maximize the probability that its own computer case is damaged within the next hour. The AI could bootstrap molecular nanotech, but that would take several hours. The AI thinks that time travel is likely impossible, so by that point, all the mass in the universe can't help it. The AI can hack a nuke and target itself. Much better by its utility function. Nearly max utility. If it can, it might upload a copy of its code to some random computer. (There is some tiny chance that time travel is possible, or that its clock is wrong) So we only get a near miss, if the AI doesn't have enough spare bandwidth or compute to do both. This is assuming that it can't hack reality in a microsecond.

There are a few other scenarios, for instance impact minimising agents. There are some designs of agents that are restricted to have a "small" effect on the future, as a safety measure. This is measured by the difference between what actually happens, and what would happen if it did nothing. When this design understands chaos theory, it will find that all other actions result in too large an effect, and do nothing. It might do a lot of damage before this somehow, depending on circumstances. I think that the AI discovering some fact about the universe that causes the AI to stop optimising effectively is a possible behaviour mode. Another example of this would be pascals mugging. The agent acts dangerously, and then starts outputting gibberish as it capitulates to a parade of fanciful pascals muggers.

Comment by donald-hobson on The questions one needs not address · 2020-03-22T17:59:43.468Z · score: 2 (2 votes) · LW · GW

The problem is that questions don't come with little labels saying whether or not they are answerable. Ban all deep philosophy and you don't get Francis Bacon or Isaac Newton. We can now say that trying to answer questions like "what is the true nature of god" isn't going to work. We now know that an alchemist can't turn lead into gold by rubbing lemons on it. However, it was a reasonable thing to try, given the knowledge of the time, and other alchemical experiments produced useful results like phosphorus.

Celebrating the people who dedicated their lives to building the first steam engine, while mocking people who tried to build perpetual motion machines before conservation of energy was understood, is just pure hindsight, and so can't be used as a lesson for the future.

Go ahead and mock those who aim for perpetual motion in the modern day.

people wasting their time thinking about ill-phrased questions, just replacing “God” with “a simulation” or replacing “repenting for the end times” with “handling AI risk”.

Given current evidence, I suspect that this field is a steam engine not a perpetual motion machine. I suspect that good answers are possible. We might not be skilled enough to reach them, but we know little enough about how much skill is needed that we can't be confident of failure. At least a few results, like mesa optimisers, look like successes.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-21T22:32:17.748Z · score: 2 (2 votes) · LW · GW

Soap and water or hand sanitiser are apparently fine to get covid19 off your skin. Suppose I rub X on my hands, then I touch an infected surface, then I touch my food or face. What X will kill the virus, without harming my hands?

I was thinking zinc salts, given zincs antiviral properties. Given soaps tendency to attach to the virus, maybe zinc soaps? Like a zinc atom in a salt with a fatty acid? This is babbling by someone who doesn't know enough biology to prune.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-21T20:11:32.004Z · score: 1 (1 votes) · LW · GW

From Star Slate Codex "I myself am a Scientismist"

Antipredictions do not always sound like antipredictions. Consider the claim “once we start traveling the stars, I am 99% sure that the first alien civilization we meet will not be our technological equals”. This sounds rather bold – how should I know to two decimal places about aliens, never having met any?
But human civilization has existed for 10,000 years, and may go on for much longer. If “technological equals” are people within about 50 years of our tech level either way, then all I’m claiming is that out of 10,000 years of alien civilization, we won’t hit the 100 where they are about equivalent to us. 99% is the exact right probability to use there, so this is an antiprediction and requires no special knowledge about aliens to make.

I disagree. I think that it is likely that a society can get to a point where they have all the tech. I think that we will probably do this within a million years (and possibly within 5 minutes of ASI) Any aliens we meet will be technological equals, or dinosaurs with no tech whatsoever.

Comment by donald-hobson on What's the present value of the Future? · 2020-03-21T15:09:18.193Z · score: 6 (3 votes) · LW · GW

How much the AI benifits you would depend on all sorts of features of your utility function. If you only care about yourself, and you are going to be dead by then, you don't care. If you are utilitarian, but think that the other shareholders are also utilitarian, you don't care. If you think that the amount of resources needed to give you a maximally pleasant life is small, you are selfish, and a few shareholders will give 1% of their resources to the rest of humanity, you don't care. You should only be buying stocks if you want something that other shareholders don't want and that takes a LOT of resources.

However, I think that you would need a bizzare sequence of events to get a shareholder value maximiser. In order to be able to make something like that, you need to have solved basically all of friendly AI theory.

Side note: "Shareholder value maximizers" that are reinforcement learning agents trained on the company's stock market data are easier to make, and they will dismantle the actual shareholders to convert their atoms into cash (or computers doing electronic fund transfers). This is something entirely different.

The AI's I am talking about are fully friendly CEV maximizers, just set on the list of shareholders, not the list of all humans. Anyone who can make such a thing can easily set the AI to maximize the CEV of just themselves, or all humanity. (Both easier groups to define than "shareholders") A moral programmer that didn't want to sieze the reigns of destiny might put all humanity into the CEV, a selfish programmer might put themselves. A team of selfish programmers might put all of themselves. ( Moral programmers might put themselves in the CEV on the grounds that it is their morality, and their desire to not take over the world. This might be a good idea if the CEV procedure would be seriously screwed up by nutcases. ) But in order for a shareholder value maximizer to be created, the shareholders have to exert meaningful power through several layers of management and on to the programmers. When each layer has a huge incentive to cheat.

Actually, I am not sure what a "libertarian" future means in this context. I expect that the world will be sufficiently different that political philosophies just won't apply. Any friendly AI can take in a utility function and decide what policies maximise it. Worry about what utility function is being maximized, and leave the AI to figure out how to maximize it. (The answer will probably not be communism, libertarianism or whatever your favourite political philosophy is. Within the context of medevel farms, an argument about whether oxen or donkeys are better at ploughing fields is entirely meaningful. Don't expect the nanotech wielding super intelligence to be on the pro oxen or pro donkey side. Obviously one side must be better, and the AI could figure it out, but the AI will figure out some third alternative that makes the other two options look very similar. )

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-20T12:22:12.608Z · score: 1 (1 votes) · LW · GW

This seems like responding to a trolley problem with a discussion of how to activate the emergency breaks. In the real world, it would be good advice, but it totally misses the point. The point is to investigate morality on toy problems before bringing in real world complications.

Comment by donald-hobson on Bucky's Shortform · 2020-03-17T23:09:17.615Z · score: 1 (1 votes) · LW · GW

Isn't that what the vote up button is for?

Comment by donald-hobson on Optimization Process's Shortform · 2020-03-17T21:44:54.559Z · score: 1 (1 votes) · LW · GW

This is exactly conditional to a bond that pays out in one year "unconditionally". Ie this is a loan with interest. (There are a few contrived scenarios where humans are extinct and money isn't worthless, depending on the definitions of those words. Would this bond pay out in a society of uploaded minds?)

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-17T21:36:38.685Z · score: 5 (4 votes) · LW · GW

Here is a moral dilemma.

Alice has a quite nice life, and believes in heaven. Alice thinks that when she dies, she will go to heaven (Which is really nice) and so wants to kill herself. You know that heaven doesn't exist. You have a choice of

1) Let Alice choose life or death, based on her own preferences and beliefs.(death)

2) Choose what Alice would choose if she had the same preferences but your more accurate beliefs. (life)

Bob has a nasty life, (and its going to stay that way). Bob would choose oblivion if he thought it was an option, but Bob believes that when he dies, he goes to hell. You cal

1) Let Bob choose based on his own preferances and beliefs (life)

2) Choose for Bob based on your beliefs and his preferences. (death)

These situations feel like they should be analogous, but my moral intuitions say 2 for Alice, and 1 for Bob.

Comment by donald-hobson on Positive Feedback -> Optimization? · 2020-03-17T20:31:20.366Z · score: 3 (2 votes) · LW · GW

Consider a pencil balanced on its point. It has multiple positive feedback loops, (different directions to fall in) and falling far in one direction prevents falling in others. But once it has fallen, it just sits there. That said, evolution can settle into a strong local minimum, and just sit there.

Comment by donald-hobson on Positive Feedback -> Optimization? · 2020-03-17T00:07:13.583Z · score: 5 (3 votes) · LW · GW

Consider the differential equation where has many positive eigenvalues. This is the simplest case of

a dynamical system containing multiple instabilities (i.e. positive feedback loops),

Where is the selection? It isn't there. You have multiple independent exponential growth rates.

Consider a chaotic system like a double pendulum. Fix to a particular typical solution.

consider as a differential equation in . Here represents the difference between and some other solution to . If you start at then stays at . However, small variations will grow exponentially. After a while, you just get a difference between 2 arbitrary chaotic paths.

I can't see a way of meaningfully describing these as optimizing processes with competing subagents. Arguably could be optimising . However, this doesn't seem canonical, as for any invertable . and describes an exactly isomorphic system, but dosen't preserve modulus. This isomorphism does preserve . That could be the thing being optimised.

Comment by donald-hobson on Coronavirus: Justified Practical Advice Thread · 2020-03-13T11:17:13.161Z · score: 1 (1 votes) · LW · GW

I no longer stand by this comment.

Comment by donald-hobson on Puzzles for Physicalists · 2020-03-12T21:03:29.974Z · score: 1 (1 votes) · LW · GW

I consider references to be about agents, not about the world. You could have humans using references like "this planet" while aliens use "planet code 1538906..." to refer to the planet they are standing on. (Maybe the aliens travel interplanetary, and need the recordings to make sense later.)

Suppose there was a big scientific debate between the H2O model and the XYZ model, we both know that one is true, but don't know which. It would be meaningful to say, "If the H2O model is true, the H2O will be in the test tube, but if the XYZ model is true, the XYZ will have boiled off." It is conveying partial information, allowing several possibilities, but excluding others. It is equally meaningful to say "If the XYZ model is true, that bottle contains XYZ, if the H2O model is true, that bottle contains H20." but it is quicker to say "that bottle contains water"

Comment by donald-hobson on Why don't singularitarians bet on the creation of AGI by buying stocks? · 2020-03-12T19:06:52.391Z · score: 10 (4 votes) · LW · GW
If your singularity stocks rapidly increase in value,

Why would they do that? anyone that believes the singularity is coming will want to sell them off, those that don't have little reason to buy. Stock markets can't price in their own non-existence.

Second, being a shareholder in the singularity could help you affect it.

Suppose google has almost reached AGI. $1000 of shares isn't going to buy meaningful influence on the details of a particular (probably secretive) project. I would be better off arranging to frequent the same social clubs as the programmers, and getting into discussions about AI, or mailing a copy of "Superintelligence" by Bostrum to all the team.

Comment by donald-hobson on Zoom In: An Introduction to Circuits · 2020-03-11T13:01:56.534Z · score: 5 (3 votes) · LW · GW

Is conway's life, with random starting state, interpretable? If you zoom in on any single square, it is trivial to predict what it will do. Zoom out and you need a lot of compute. There is no obvious way to predict if a cell will be on in 1000000 timesteps without brute force simulating the whole thing (at least the past light cone). What would an interpretability tool for conway's life look like?

Comment by donald-hobson on Quadratic models and (un)falsified data · 2020-03-09T13:15:23.115Z · score: 3 (2 votes) · LW · GW

If we assume that china is an infinite plain, with Chinese people distributed uniformly across it, and that individual Chinese people don't travel much, then could be accurate. A circle of disease spreading at a constant distance per day.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-08T10:54:21.154Z · score: 1 (1 votes) · LW · GW

Sodium Clorate decomposes into salt and oxygen at 600C, it is mixed with iron powder for heat to make the oxygen generators on planes. To supply oxygen, you would need 1.7kg per day. (plus a bit more to burn the iron) And it's bulk price <$1 /kg. However, 600C would make it harder to jerry rig a generator, although maybe wrapping a saucepan in fiberglass...

Comment by donald-hobson on ChristianKl's Shortform · 2020-03-08T10:32:15.819Z · score: 1 (1 votes) · LW · GW

If a starship full of fuel is in orbit, and gets nudged downward, hitting the earths atmosphere, it gets very hot. If it doesn't have a giant heatshield, it will vaporize the fuel, leading to an explosion in the upper atmosphere. If you used the fuel to slow down, you could reach earth with mostly empty tanks, but still cause some damage if you hit a city.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-08T10:26:45.169Z · score: 1 (1 votes) · LW · GW

Given bulk prices of conc hydrogen peroxide, and human oxygen use, breating pure oxygen could cost around $3 per day for 5l 35% h2o2 (Order of magnitude numbers) However, this conc of h202 is quite dangerous stuff.

Powdered baking yeast will catalytically decompose hydrogen peroxide, and it shouldn't be hard to tape a bin bag to a bucket to a plastic bottle with a drip hole to a vacuum cleaner tube to make an apollo 13 style oxygen generator ... (I think)

(I am trying to figure out a cheap and easy oxygen source, does breathing oxygen help with coronavirus?)

Comment by donald-hobson on How hard would it be to attack coronavirus with CRISPR? · 2020-03-07T21:48:19.380Z · score: 2 (2 votes) · LW · GW

Non paywalled link to same content (PDF)

Comment by donald-hobson on How hard would it be to attack coronavirus with CRISPR? · 2020-03-07T10:46:18.583Z · score: 2 (2 votes) · LW · GW

Cas 9 cuts rna too.

Comment by donald-hobson on How hard would it be to attack coronavirus with CRISPR? · 2020-03-07T00:09:14.326Z · score: 1 (3 votes) · LW · GW

cas9 can be set to target arbitrary DNA sequences, it is part of a bacterial virus defence mechanism. If you have your cas9, in a cell, and then a virus tries to infect it, the viral genome will get chopped up. I don't think viral DNA works when chopped up, especially as the researchers can cut somewhere that will do lots of damage, like in the middle of a gene that makes a key protein.

This is assuming that you can get cas9 into cells.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-03-06T20:20:30.551Z · score: 3 (2 votes) · LW · GW

Just realized that a mental move of "trying to solve AI alignment" was actually a search for a pre-cached value for "solution to AI alignment", realized that this was a useless way of thinking, although it might make a good context shift.

Comment by donald-hobson on What does Solomonoff induction say about brain duplication/consciousness? · 2020-03-06T15:07:28.453Z · score: 1 (1 votes) · LW · GW

We can make Solomonoff induction believe all sorts of screwy things about consciousness. Take a few trillion identical computers running similar computations. Put something really special and unique next to one of the cases, say a micro black hole. Run solomonoff induction on all the computers, each with different input. Each inductor simulates the universe and has to know its own position in order to predict its input. The one next to the black hole can most easily locate itself as the one next to the black hole, if the black hole is moved, it will believe its consciousness resides in "the computer next to the black hole" and predict accordingly.

Comment by donald-hobson on Robustness to fundamental uncertainty in AGI alignment · 2020-03-04T21:19:37.153Z · score: 4 (2 votes) · LW · GW

When it is all over, we will either have succeed or failed. (The pay-off set is strongly bimodal.)

The magnitude of the pay-off is irrelevant to the optimal strategy. Suppose research program X has a 1% chance of FAI in 10 years, a 1% chance of UFAI in 10 years, and 98% chance of nothing. Is it a good option? That depends on our P(FAI | no AI in 10 years). If FAI will probably arive in 15 years, X is bad. UFAI in 15 years and X may be good. Endorse only those research programms such that you think P(FAI | that research program makes AI) > P(FAI | no one else makes AI before the research program has a chance to). Actually, this assumes unlimited research talent.

Avoiding avenues with small chances of UFAI corresponds to optimism about the outcome.

I think that it should be fairly clear whether an AI project would produce UFAI before it is run. P(friendly) <5% or >80% usually. Probability on serious assesment by competent researchers. So say that future Miri can tell if any given design is friendly in a few months. If some serious safety people study the design and become confident that its FAI then run it. If they think its UFAI, they won't run it. If someone with limited understanding and lack of knowledge of their ignorance manages to build an AI, its UFAI, and they don't know that, so they run it. I am assuming that there are people who don't realise the problem is hard, those that know they can't solve it, and those who can solve it, in order of increasing expertise. Most of the people reading this will be in the middle category. (not counting 10^30 posthuman historians ;-) People in the middle category won't build any ASI. Those in the first category will usually produce nothing, but might produce a UFAI, those in the third might produce a FAI.

Comment by donald-hobson on Coronavirus is Here · 2020-03-03T17:48:33.272Z · score: 2 (3 votes) · LW · GW

Coronavirus is where? The west coast of America? The UK government is being moderately competent as far as I can tell. (They haven't banned the test)

Comment by donald-hobson on Is the framing problem crucial in creating an AGI? · 2020-03-03T14:48:53.778Z · score: 1 (1 votes) · LW · GW

An AGI is software that can take arbitrary input and output, and figure out what is going on and how to use it to its advantage. One you have an AGI, you can hook it up to a robot, and it will figure out that its controling a robot, and be able to do stuff. Hook it up to the stock market, and it will figure out how to trade or whatever. Give it any input data from the rest of the world, and any output, and it can use them. A robot is just a conceptually simple special case. The input can be a keyboard and mouse, touch sensors on robot hand, or all the text on some website. The output can be display to a screen, motor movements, rocket firings to navigate a space probe, speakers, comments on websites or whatever. In every case, your AI takes some input data, processes it in some "intelligent" way and then outputs data. A machine that always adds 3 to the input isn't intelligent. We have some intuitions about what an "intelligence" would do. We can set up toy problems, often involving robots, consult our intuitions about intelligence and then try to find a mathematical pattern.

I am not sure what you mean by framing problem.

Comment by donald-hobson on Defense Against The Dark Arts: Case Study #1 · 2020-03-03T14:36:44.924Z · score: 1 (1 votes) · LW · GW

Suppose "spiritual enlightenment" = "memetic wireheading". You would expect it to be more prevalent in places with worse material living standards, both because people who are chanting mantras aren't building stuff, and because people who are already comfy have less incentive to try.

Comment by donald-hobson on Is the framing problem crucial in creating an AGI? · 2020-03-03T14:08:06.852Z · score: 3 (2 votes) · LW · GW

Suppose that you are in a project trying to build a time machine. You correctly reason that a time machine will require buttons and dials for you to control it. You go off and make a load of nice buttons, dials, date counters and clock faces. You have correctly identified that X needs done, and are doing X, but X isn't the hard part or limiting factor, and isn't very useful.
You are working on an AI project, you correctly deduce that your AI will run on a computer, and computers need to be plugged in, so you go and practice wiring up plugs. I feel that you are looking in much the same direction when you talk about operating systems, cache ect. These are not the difficult part of the problem.

Comment by donald-hobson on Is the framing problem crucial in creating an AGI? · 2020-03-03T13:23:08.772Z · score: 1 (1 votes) · LW · GW

I think you are fundamentally confused. The only difference between RAM and hard drive is that hard drive is bigger and stores data while the computer is off, and RAM is faster.

AI design process.

1) Come up with a formal mathematical definition of what you want the AI to do.

2) Come up with an algorithm that would approximate it, given vast but finite amounts of compute.

3) Find an algorithm that would work given real world amounts of compute.

4) If your algorithm is a bit slow, tweak it to fit the details of modern computers. (This is the place to worry about cpu vs gpu, RAM vs hard disk ect.)

5) Run the code.

As far as I know, noone has done stage 3 for any kind of AGI.

once we create a huge dataset that doesnt overfit to one domain of knowledge we will be able to create an AGI much easier.

I disagree, it would be easy to make such a dataset, downloading wikipedia would be a good start, but we don't know how to make an AI that can learn from it well.

Comment by donald-hobson on What does Solomonoff induction say about brain duplication/consciousness? · 2020-03-03T12:30:00.930Z · score: 1 (1 votes) · LW · GW

I consider this to be a flaw in AIXI type designs. To actually make sense, these designs need hypercompute, and so have to guess at what rules allow the hypercompute to interact with the normal universe. I have a rough idea of some kind of FDTish agent that can solve this, but can't formalize it.

Comment by donald-hobson on I don't understand Rice's Theorem and it's killing me · 2020-03-02T14:58:46.135Z · score: 1 (1 votes) · LW · GW

The abstract formalism of a turing machine has an infinite amount of memory. Any process that exists in the real physical world (as far as I know) has only a finite amount of memory. My computer contains around bits. So it has distinct computational states. To see if a program will hault, just simulate for timesteps, and if it hasn't halted yet, it never will. (Actually, if you really zoom in on the physical details, the hard drive will wear out before then)

If you ran a programming language on some magic computer with infinite memory, there is no procedure takes in an arbitrary piece of programming code and returns "halt" or "no halt" correctly.

You can run arbitrary code and see if it has halted yet. So "does arbitrary code halt in a googleplex timesteps?" is computable. But if someone hands you a piece of code that runs forever, you can't always tell if it runs forever or just runs for longer than you have simulated it.

Whenever something like a card game turns out to be turing complete, there is some score or number of points that is an arbitrary natural number. In other words, an unboundedly large amount of info can be encoded in the digits of the score. In a real game, the score is small, but in principle you could get a score so big you couldn't write it down and have to stop playing.

Comment by donald-hobson on Coronavirus: Justified Practical Advice Thread · 2020-03-01T11:17:45.073Z · score: 3 (2 votes) · LW · GW

Infections tend to have exponential growth for a bit, and then stop. Current rates of growth outside china are 10X every 2 weeks. So if you need 2 weeks prep time on your practical advice, decide how much of a prob you are worried about, and divide by 10. Wait to see if the rate gets that high.

Comment by donald-hobson on Coronavirus: Justified Practical Advice Thread · 2020-03-01T11:06:37.179Z · score: -2 (4 votes) · LW · GW

I suspect that the rate of spread is <1 in firstworld countries. I predict that we will see a few people who have travelled abroad catch the virus. If the virus does start spreading out of control, there will be time to stock up when the numbers are in the thousands. (In the UK) I would be more worried if I was in china or south korea.

Edit: I no longer believe this

Comment by donald-hobson on Trace: Goals and Principles · 2020-02-29T17:08:05.749Z · score: 3 (2 votes) · LW · GW

Pick an arbitrary set , with a (transitive) group action from a group .

For each element of , you have a copy of all the nodes.

Each symbol now has multiple pointers.

A=Symbol(func=const True)


C=Symbol(func=xor, parents=[pa,pb])

In your example, and the g being used in the pointers are -1,0,1. As you are only using a finite subset of G, you have the structure of a Caley graph.

The graphs you might want include various latices and infinite trees.

Your evaluation is a function from pairs (s,x) where s is a Symbol and x to values.

Comment by donald-hobson on Donald Hobson's Shortform · 2020-02-28T16:22:27.201Z · score: 1 (1 votes) · LW · GW

Looking at formalism for AIXI and other similar agent designs. Big mess of and with indicies. Would there be a better notation?

Comment by donald-hobson on Harry Potter and the Methods of Rationality discussion thread, March 2015, chapter 116 · 2020-02-28T00:03:46.477Z · score: 1 (1 votes) · LW · GW

The niave protocol, any time you get a note, you come up with a random password. That password has to be on the bottom of the note. If the password has even a few bits of entropy, relative to outsiders, this will work. (Memory charms or time turners can get around this, but its still a good precaution)

Comment by donald-hobson on New article from Oren Etzioni · 2020-02-26T17:26:21.678Z · score: 4 (3 votes) · LW · GW
Preparing anyway even if it's very low probability because of extreme consequences is Pascal's Wager

ASI is probably coming sooner or later. Someone has to prepare at some point, the question is when.

I consider AI development to be a field that I have little definite info about. Its hard to assign less than 1% prob to statements about ASI. (Excepting the highly conjoined ones.) I don't consider things like dinosaur killing asteroids with 1 in 100 million probs to be pascal muggings.

If and when a canary "collapses" we will have ample time to design off switches and identify red lines we don't want AI to cross

We have a tricky task, and we don't know how long it will take. Having hit one of these switches doesn't help us to do the task much. A student is given an assignment in August, the due date is March next year. They decide to put it off until it snows. Snowfall is an indicator that the due date is coming soon, but not a good one. But either way, it doesn't help you do the assignment.

What is a "fully self driving" car, we have had algorithms that kind of usually work for years, and a substantial part of modern progress in the field looks like gathering more data, and developing driving specific tricks. Suppose that you needed 100 million hours of driving data to train current AI systems. A company pays drivers to put a little recording box in their car. It will take 5 years to gather enough data, and after that we will have self driving cars. What are you going to do in 5 years time that you can't do now. In reality, we aren't sure if you need 50, 100 or 500 million hours of driving data with current algorithms, and aren't sure how many people will want the boxes installed. (These boxes are usually built into satnavs or lane control systems in modern cars)

Limited versions of the Turing test (like Winograd Schemas)

What percentage do you want, and what will you do when gpt-5 hits it?

We are decades away from the versatile abilities of a 5 year old

A "result" got by focussing on the things that 5 year olds are good at,

Sometimes you have a problem like looking at an image of some everyday scene and saying whats happening in it that 5 year olds are (or at least were a few years ago) much better at. Looking at a load of stock data and using linear regression to find correlations between prices, nothing like that existed in the environment of evolutionary adaptedness, human brains aren't built to do that.

If and when a canary "collapses" we will have ample time to design off switches and identify red lines we don't want AI to cross

Even if that was true, how would you know that? Technological progress is hard to predict. Designing off switches is utterly trivial if the system isn't trying to avoid the off switch being pressed, and actually quite hard if the AI is smart enough to know about the off switch and remove it.

Comment by donald-hobson on Thoughts about Dr Stone and Mythology · 2020-02-26T16:46:17.406Z · score: 3 (3 votes) · LW · GW

I agree that getting chinese whispered into nonsense is a real failure mode, pop sci quantum mechanics shows this well enough. I think it unlikely that there was that much of a real, true and important point behind most myths. Repeatedly rerecord a sound on analogue media enough and you get static. Repeatedly retell a story and you get mythology. We can't tell much about what the signal started as, but I doubt it was all brilliant rationality, because most of our really old documents aren't that rational.

Comment by donald-hobson on How Low Should Fruit Hang Before We Pick It? · 2020-02-26T13:13:30.673Z · score: 1 (1 votes) · LW · GW

Suppose the AI finds a plan with 10^50 impact and 10^1000 utility. I don't want that plan to be run. Its probably a plan that involves taking over the universe and then doing something really high utility. I think a constraint is better than a scaling factor.

Comment by donald-hobson on Will AI undergo discontinuous progress? · 2020-02-23T20:24:14.516Z · score: 2 (2 votes) · LW · GW

I am not convinced that the distinction between continuous and discontinuous approaches is a feature of the territory. Zoom in in sufficient detail, and you see a continuous wavefunction of electron interacting with the continuous wavefunction of a silicon atom. Zoom out to evolutionary timescales, and the jump from hominids with pointy sticks to ASI is almost instant. The mathematical definition of continuity relies on your function being mathematically formal. Is distance from earth to the moon an even number of plank-lengths? Well there are a huge number of slightly different measurements you could make, depending on when you measure and exactly what points you measure between, and how you deal with relitivistic length contraction, the answer will be different. In a microsecond in which the code that is a fooming AI is doing garbage collection, is AI progress happening? You have identified an empirical variable called AI progress, but whether or not it is continuous depends on exactly how you fill in the details.

Imagine superintelligence has happened and we are discussing the details afterwords. We were in a world basically like this one, and then someone ran some code, which gained total cosmic power within a millisecond. Someone tries to argue that this was a continuous improvement, just a very fast one. What evidence would convince you one way or the other on this?

Comment by donald-hobson on Theory and Data as Constraints · 2020-02-23T16:21:32.505Z · score: 3 (2 votes) · LW · GW
I would bet that all the data needed in principle to, say, find a cure for Alzheimers is already available online - if only we knew how to effectively leverage it.

I agree. If "effectively leverage it" means a superintelligence with unlimited compute, then this is a somewhat weak statement. I would expect that a superintelligence given the human genome would figure out how to cure all diseases. I would expect it to be able to figure out a lot from any book on biology. I would expect it to be able to look at a few holliday photos, figure out the fundamental equations of reality, and that evolution happened on a planet ,that it was created by evolved intelligences with tech ect. From this, it could design nanobots programmed to find humans and cure them, even if it had no idea what humans look like, it just programs the nanobots to find the most intelligent life forms around.

Comment by donald-hobson on Will AI undergo discontinuous progress? · 2020-02-23T14:37:21.099Z · score: 3 (3 votes) · LW · GW

Suppose that different tasks take different levels of AI to do better than humans.

Firstly AI can do arithmatic, then play chess, then drive cars ect. Lets also assume that AI is much faster than humans. So imagine that AI research ability was rising from almost nothing to superhuman over the course of a year. A few months in and its inventing stuff like linear regression, impressive, but not as good as current human work on AI. There are a few months where the AI is worse than a serious team of top researchers, but better than an intern. So if you have a nieche use for AI, that can be automatically automated. The AI research AI designs a widget building AI. The humans could have made a widget building AI themselves, but so few widgets are produced, that it wasn't worth it.

Then the AI becomes as good as a top human research team and FOOM. How crazy the world gets before foom depends on how much other stuff is automated first. Is it easier to make an AI teacher, or an AI AI researcher? Also remember that bearocratic delays are a thing, there is a difference between having an AI that does medical diagnosis in a lab, and it being used in every hospital.

Comment by donald-hobson on Does rationalism affect your dreams? · 2020-02-22T23:44:36.410Z · score: 1 (1 votes) · LW · GW

I had a dream where I was flying by incrementing my own x and y coordiates. Somewhat related to simulated worlds, but also to straight programming.

Comment by donald-hobson on What would you do with an Evil AI? · 2020-02-22T22:27:13.705Z · score: 1 (1 votes) · LW · GW

If I am confidant that I have the original source code, as written by humans, I read that. I am looking for deep abstract principles. I am looking only for abstract ideas that are general to the field of AI.

If I can encrypt the code in a way that only a future superintelligence can crack, and I feel hopeful about FAI, I do that. Otherwise, secure erase, possibly involving anything that can slag the hard drives that is lying around.