## Posts

## Comments

**daozaich**on Math: Textbooks and the DTP pipeline · 2018-07-10T14:26:38.824Z · LW · GW

The Definition-Theorem-Proof style is just a way of compressing communication. In reality, heuristic / proof-outline comes first; then, you do some work to fill the technical gaps and match to the existing canon, in order to improve readability and conform to academic standards.

Imho, this is also the proper way of reading maths papers / books: Zoom in on the meat. Once you understood the core argument, it is often unnecessary too read definitions or theorems at all (Definition: Whatever is needed for the core argument to work. Theorem: Whatever the core argument shows). Due to the perennial mismatch between historic definitions and theorems and the specific core arguments this also leaves you with stronger results than are stated in the paper / book, which is quite important: You are standing on the shoulders of giants, but the giants could not foresee where you want to go.

**daozaich**on RFC: Mental phenomena in AGI alignment · 2018-07-08T00:33:59.847Z · LW · GW

This paints a bleak picture for the possibility of aligning mindless AGI since behavioral methods of alignment are likely to result in divergence from human values and algorithmic methods are too complex for us to succeed at implementing.

To me it appears like the terms cancel out: Assuming we are able to overcome the difficulties of more symbolic AI design, the prospect of aligning such an AI seem less hard.

In other words, the main risk is wasting effort on alignment strategies that turn out to be mismatched to the eventually implemented AI.

**daozaich**on What will we do with the free energy? · 2018-07-03T18:58:54.327Z · LW · GW

The negative prices are a failure of the market / regulation, they don't actually mean that you have free energy.

That being said, the question for the most economical opportunistic use of intermittent energy makes sense.

**daozaich**on Why it took so long to do the Fermi calculation right? · 2018-07-03T18:52:35.667Z · LW · GW

No. It boils down to the following fact: If you take given estimates on the distribution of parameter values at face value, then:

(1) The expected number of observable alien civilizations is medium-large (2) If you consider the distribution of the number of alien civs, you get a large probability of zero, and a small probability of "very very many aliens", that integrates up to the medium-large expectation value.

Previous discussions computed (1) and falsely observed a conflict with astronomical observations, and totally failed to compute (2) *from their own input data*. This is unquestionably an embarrassing failure of the field.

**daozaich**on Logical uncertainty and Mathematical uncertainty · 2018-06-27T18:03:54.660Z · LW · GW

What is logical induction's take on probabilistic algorithms? That should be the easiest test-case.

Say, before "PRIME is in P", we had perfectly fine probabilistic algorithms for checking primality. A good theory of mathematical logic with uncertainty should permit us to use such an algorithm, without random oracle, for things you place as "logical uncertainty". As far as I understood, the typical mathematician's take is to just ignore this foundational issue and do what's right (channeling Thurston: Mathematicians are in the business of producing human understanding, not formal proofs).

**daozaich**on Monty Hall in the Wild · 2018-06-08T22:16:51.626Z · LW · GW

It’s excellent news! Your boss is a lot more likely to complain about some minor detail if you’re doing great on everything else, like actually getting the work done with your team.

Unfortunately this way of thinking has a huge, giant failure mode: It allows you to rationalize away critique about points you consider irrelevant, but that are important to your interlocutor. Sometimes people / institutions consider it really important that you hand in your expense sheets correctly or turn up in time for work, and finishing your project in time with brilliant results is *not* a replacement for "professional demeanor". This was not a cheap lesson for me; people did tell me, but I kinda shrugged it off with this kind of glib attitude.

**daozaich**on Editor Mini-Guide · 2018-06-08T20:29:04.664Z · LW · GW

Is there a way of getting "pure markdown" (no wysiwyg at all) including Latex? Alternatively, a hotkey-less version of the editor (give me buttons/menus for all functionality)?

I'm asking because my browser (chromium) eats the hotkeys, and latex (testing: $\Sigma$ ) appears not to be parsed from markdown. I would be happy with any syntax you choose. For example \Sigma; alternatively the github classic of `using backticks`

appears still unused here.

edit: huh, backticks are in use and html-tags gets eaten.

**daozaich**on Beyond Astronomical Waste · 2018-06-08T19:45:40.743Z · LW · GW

Isn't all this massively dependent on how your utility $U$ scales with the total number $N$ of well-spent computations (e.g. one-bit computes)?

That is, I'm asking for a gut feeling here: What are your relative utilities for $10^{100}$, $10^{110}$, $10^{120}$, $10^{130}$ universes?

Say, $U(0)=0$, $U(10^100)=1$ (gauge fixing); instant pain-free end-of-universe is zero utility, and a successful colonization of the entire universe with a suboptimal black hole-farming near heat-death is unit utility.

Now, per definitionem, the utility $U(N)$ of a $N$-computation outcome is the inverse of the probability $p$ at which you become indifferent to the following gamble: Immediate end-of-the-world at probability $(1-p)$ vs an upgrade of computational world-size to $N$ at propability $p$.

I would personally guess that $U(10^{130})< 2 $; i.e. this upgrade would probably not be worth a 50% risk of extinction. This is *massively* sublinear scaling.

**daozaich**on Into the Kiln: Insights from Tao's 'Analysis I' · 2018-06-02T13:41:30.245Z · LW · GW

What was initially counterintuitive is that even though , the series doesn't converge.

This becomes much less counterintuitive if you instead ask: How would you construct a sequence with divergent series?

Obviously, take a divergent series, e.g. , and then split the th term into .

**daozaich**on Understanding is translation · 2018-06-01T21:04:50.077Z · LW · GW

FWIW, looking at an actual compiler, we see zero jumps (using a conditional move instead):

```
julia> function test(n)
i=0
while i<n
i += 1
end
return i
end
test (generic function with 1 method)
julia> @code_native test(10)
.text
Filename: REPL\[26\]
pushq %rbp
movq %rsp, %rbp
Source line: 3
xorl %eax, %eax
testq %rdi, %rdi
cmovnsq %rdi, %rax
Source line: 6
popq %rbp
retq
nop
```

edit: Sorry for the formatting. I don't understand how source-code markup is supposed to work now?

edit2: Thanks, the markup works now!

edit3: So, to tie this into your greater point:

If you don't ask "how would I code this in assembly" but rather "how should my compiler reason about this code", then it is clear that the loop can be obviously eliminated: You place a phi-node at the end of the loop, and a tiny bit of inductive reasoning makes the loop body obviously dead code if n is an integer type. Slightly more magical (meaning I'm not a compiler expert) is the fact that the compiler (LLVM) can completely eliminate the following loop (replacing it with an explicit formula):

```
julia> function sumN6(lim)
s=0
i=0
while i<lim
i+=1
s+= i*i*i*i*i*i
end
return s
end
```

**daozaich**on Decision theory and zero-sum game theory, NP and PSPACE · 2018-05-25T22:37:05.226Z · LW · GW

"what move should open with in reversi" would be considered as an admissible decision-theory problem by many people. Or in other words: Your argument that EU maximization is in NP only holds for utility functions that permit computation in P of expected utility given your actions. That's not quite true in the real world.

**daozaich**on Moral frameworks and the Harris/Klein debate · 2018-05-05T17:26:13.004Z · LW · GW

This, so much.

So, in the spirit of learning from other's mistakes (even better than learning from my own): I thought Ezra made his point very clear.

So, all of you people who missed Ezra's point (confounded data, outside view) on first reading:

How could Ezra have made clearer what he was arguing, short of adopting LW jargon? What can we learn from this debacle of a discussion?

Edit: tried to make my comment less inflammatory.

**daozaich**on Weird question: could we see distant aliens? · 2018-04-24T16:30:53.685Z · LW · GW

>I was imagining a sort of staged rocket, where you ejected the casing of the previous rockets as you slow, so that the mass of the rocket was always a small fraction of the mass of the fuel.

Of course, but your very last stage is still a rocket with a reactor. And if you cannot build a rocket with 30g motor+reactor weight, then you cannot go to such small stages and your final mass on arrival includes the smallest efficient rocket motor / reactor you can build, zero fuel, and a velocity that is below escape velocity of your target solar system (once you are below escape velocity I'll grant you maneuvers with zero mass cost, using solar sails; regardless, tiny solar-powered ion-drives appear reasonable, but generate not enough thrust to slow down from relativistic to below-escape in the time-frame before you have passed though your target system).

>But Eric Drexler is making some strong arguments that if you eject the payload and then decelerate the payload with a laser fired from the rest of the "ship", then this doesn't obey the rocket equation. The argument seems very plausible (the deceleration of the payload is *not* akin to ejecting a continuous stream of small particles - though the (tiny) acceleration of the laser/ship is). I'll have to crunch the number on it.

That does solve the "cannot build a small motor" argument, potentially at the cost of some inefficiency.

It still obeys the rocket equation. The rocket equation is like the 2nd law of thermodynamics: It is not something you can trick by clever calculations. It applies for all propulsion systems that work in a vacuum.

You can only evade the rocket equation by finding (non-vacuum) stuff to push against; whether it be the air in the atmosphere (airplane or ramjet is more efficient than rocket!), the solar system (gigantic launch contraption), various planets (gravitational slingshot), the cosmic microwave background, the solar wind, or the interstellar medium. Once you have found something, you have three choices: Either you want to increase relative velocity and expend energy (airplane, ramjet), or you want to decrease relative velocities (air-braking, use of drag/friction, solar sails when trying to go with the solar wind, braking against the interstellar medium, etc), or you want an elastic collision, e.g. keep absolute relative velocity the same but reverse direction (gravitational slingshot).

Slingshots are cool because you extract energy from the fact that the planets have different velocities: Having multiple planets is not thermodynamic ground state, so you steal from the potential energy / negative entropy left over from the formation of the solar system. Alas, slingshots can't bring you too much above escape velocity, nor slow you down to below escape if you are significantly faster.

Edit: probably stupid idea, wasn't thinking straight <strike> Someone should tell me whether you can reach relativistic speeds by slingshotting in a binary or trinary of black holes. That would be quite elegant (unbounded escape velocity, yay! But you have time dilation when close to the horizon, so unclear whether this takes too long from the viewpoint of outside observers; also, too large shear will pull you apart).</strike>

edit2: You can afaik also push against a curved background space-time, if you have one. Gravity waves technically count as vacuum, but not for the purpose of the rocket equation. Doesn't help, though, because space-time is pretty flat out there, not just Ricci-flat (=vacuum).

**daozaich**on Weird question: could we see distant aliens? · 2018-04-24T10:45:57.763Z · LW · GW

If you have to use the rocket equation twice, then you effectively double delta-v requirements and square the launch-mass / payload-mass factor.

Using Stuart's numbers, this makes colonization more expensive by the following factors:

0.5 c: Antimatter 2.6 / fusion 660 / fission 1e6

0.8 c: Antimatter 7 / fusion 4.5e5 / fission 1e12

0.99c Antimatter 100 / fusion 4.3e12 / fission 1e29

If you disbelieve in 30g fusion reactors and set a minimum viable weight of 500t for an efficient propulsion system (plus negligible weight for replicators) then you get an additional factor of 1e7.

Combining both for fusion at 0.8c would give you a factor of 5e12, which is significantly larger than the factor between "single solar system" and "entire galaxy". These are totally pessimistic assumptions, though: Deceleration probably can be done cheaper, and with lower minimal mass for efficient propulsion systems. And you almost surely can cut off quite a bit of rocket-delta-v on acceleration (Stuart assumed you can cut 100% on acceleration and 0% on deceleration; the above numbers assumed you can cut 0% on acceleration and 0% on deceleration).

Also, as Stuart noted, you don't need to aim at every reachable galaxy, you can aim at every cluster and spread from there.

So, I'm not arguing with Stuart's greater claim (which is a really nice point!), I'm just arguing about local validity of his arguments and assumptions.

**daozaich**on On exact mathematical formulae · 2018-04-23T22:18:25.349Z · LW · GW

You're right, I should have made that clearer, thanks!

**daozaich**on Weird question: could we see distant aliens? · 2018-04-23T20:54:44.468Z · LW · GW

I would not fret too much about slight overheating of the payload; most of the launch mass is propulsion fuel anyway, and in worst-case the payload can rendezvous with the fuel in-flight, after the fuel has cooled down.

I would be very afraid of the launch mass, including solar sail / reflector loosing (1) reflectivity (you need a very good mirror that continues to be a good mirror when hot; imperfections will heat it) and (2) structural integrity.

I would guess that, even assuming technological maturity (can do anything that physics permits), you cannot keep structural integrity above, say, 3000K, for a launch mass that is mostly hydrogen. I think that this is still icy cold, compared to the power output you want.

So someone would need to come up with either

1. amazing schemes for radiative heat-dissipation and heat pumping (cannot use evaporative cooling, would cost mass),

2. something weird like a plasma mirror (very hot plasma contained by magnetic fields; this would be hit by the laser, which pushes it via radiation pressure and heats it; momentum is transferred from plasma to launch probe via magnetic field; must not loose too many particles, and might need to maintain a temperature gradient so that most radiation is emitted away from the probe; not sure whether you can use dynamo flow to extract energy from the plasma in order to run heat pumps, because the plasma will radiate a lot of energy in direction of the probe),

3. show that limiting the power so that the sail has relatively low equilibrium temperature allows for enough transmission of momentum.

No 3 would be the simplest and most convincing answer.

I am not sure whether a plasma mirror is even thermo-dynamically possible. I am not sure whether sufficient heat-pumps plus radiators are "speculative engineering"-possible, if you have a contraption where your laser pushes against a shiny surface (necessitating very good focus of the laser). If you have a large solar sail (high surface, low mass) connected by tethers, then you probably cannot use active cooling on the sail; therefore there is limited room for fancy future-tech engineering, and we should be able to compute some limits now.

---------------------

Since I already started raising objections to your paper, I'll raise a second point: You compute the required launch mass from rocket-equation times final payload, with the final payload having very low weight. This assumes that you can actually build such a tiny rocket! While I am willing to suspend disbelieve and assume that a super efficient fusion-powered rocket of 500 tons might be built, I am more skeptical if your rocket, including fusion reactor but excluding fuel, is limited to 30 gram of weight.

Or did I miss something? While this would affect your argument, my heart is not really in it: Braking against the interstellar medium appears, to me, to circumvent a lot of problems.

---------------------

Because I forgot and you know your paper better than me: Do any implicit or explicit assumptions break if we lose access to most of the fuel mass for shielding during the long voyage?

If you could answer with a confident "no, our assumptions do not beak when cannot use the deceleration fuel as shielding", then we can really trade-off acceleration delta-v against deceleration delta-v, and I stay much more convinced about your greater point about the Fermi paradox.

**daozaich**on The many ways AIs behave badly · 2018-04-23T13:25:38.390Z · LW · GW

This was a very fun article. Notably absent from the list, even though I would absolutely have expected it (since the focus was on evolutionary algorithms, even though many observations also apply to gradient-descent):

Driving genes. Biologically, a "driving gene" is one that cheats in (sexual) evolution, by ensuring that it is present in >50% of offspring, usually by weirdly interacting with the machinery that does meiosis.

In artificial evolution that uses "combination", "mutation" and "selection", these would be regions of parameter-space that are attracting under "combination"-dynamics, and use that to beat selection pressure.

**daozaich**on Weird question: could we see distant aliens? · 2018-04-23T12:41:41.639Z · LW · GW

If you assume that Dysoning and re-launch take 500 years, this barely changes the speed either, so you are very robust.

I'd be interested in more exploration of deceleration strategies. It seems obvious that braking against the interstellar medium (either dust or magnetic field) is viable to some large degree; at the very least if you are willing to eat a 10k year deceleration phase. I have taken a look at the two papers you linked in your bibliography, but would prefer a more systematic study. Important is: Do we know ways that are definitely not harder than building a dyson swarm, and is one galaxy's width (along smallest dimension) enough to decelerate? Or is the intergalactic medium dense enough for meaningful deceleration?

I would also be interested in a more systematic study of acceleration strategies. Your arguments absolutely rely on circumventing the rocket equation for acceleration; break this assumption, and your argument dies.

It does not appear obvious to me that this is possible: Say, coil guns would need a ridiculously long barrel and mass, or would be difficult to maneuver (you want to point the coil gun at all parts of the sky). Or, say, laser acceleration turns out to be very hard because of (1) lasers are fundamentally inefficient (high thermal losses), and cannot be made efficient if you want very tight beams and (2) cooling requirement for the probes during acceleration turn out to be unreasonable. [*]

I could imagine a world where you need to fall back to the rocket equation for a large part of the acceleration delta-v, even if you are a technologically mature superintelligence with dyson swarm. Your paper does not convince me that such a world is impossble (and it tries to convince me that hypothetical worlds are impossible, where it would be hard to rapidly colonize the entire universe if you have reasonably-general AI).

Obviously both points are running counter to each other: If braking against the interstellar medium allows you to get the delta-v for deceleration down to 0.05 c from, say 0.9 c, but acceleration turns out to be so hard that you need to get 0.8 c with rockets (you can only do 0.1c with coil guns / lasers, instead of 0.9 c), then we have not really changed the delta-v calculus; but we have significantly changed the amount of available matter for shielding during the voyage (we now need to burn most of the mass during acceleration instead of deceleration, which means that we are lighter during the long voyage).

[*] Superconductors can only support a limited amount of current / field-strength. This limits the acceleration. Hence, if you want larger delta-v, you need a longer barrel. How long, if you take the best known superconductors? At which fraction of your launch probe consisting of superconducting coils, instead of fusion fuel? Someone must do all these calculations, and then discuss how the resulting coil gun is still low-enough mass compared to the mass of a dyson swarm, and how to stabilize, power, cool and maneuver this gun. Otherwise, the argument is not convincing.

edit: If someone proposes a rigid barrel that is one light-hour long then I will call BS.

**daozaich**on On exact mathematical formulae · 2018-04-23T09:21:52.213Z · LW · GW

Computability does not express the same thing we mean with "explicit". The vague term "explicit" crystallizes an important concept, which is dependent on social and historical context that I tried to elucidate. It is useful to give a name to this concept, but you cannot really prove theorems about it (there should be no technical definition of "explicit").

That being said, computability is of course important, but slightly too counter-intuitive in practice. Say, you have two polynomial vectorfields. Are solutions (to the differential equation) computable? Sure. Can you say whether the two solutions, at time t=1 and starting in the origin, coincide? I think not. Equality of computable reals is not decidable after all (literally the halting problem).

**daozaich**on On exact mathematical formulae · 2018-04-23T09:08:20.212Z · LW · GW

It depends on context. Is the exponential explicit? For the last 200 years, the answer is "hell yeah". Exponential, logarithm and trigonometry (complex exponential) appear very often in life, and people can be expected to have a working knowledge of how to manipulate them. Expressing a solution in terms of exponentials is like meeting an old friend.

120 years ago, knowing elliptic integrals, their theory and how to manipulate them was considered basic knowledge that every working mathematician or engineer was expected to have. Back then, these were explicit / basic / closed form.

If you are writing a computer algebra system of similar ambition to maple / mathematica / wolfram alpha, then you better consider them explicit in your internal simplification routines, and write code for manipulating them. Otherwise, users will complain and send you feature requests. If you work as editor at the "Bronstein mathematical handbook", then the answer is yes for the longer versions of the book, and a very hard judgement call for shorter editions.

Today, elliptic integrals are not routinely taught anymore. It is a tiny minority of mathematicians that has working knowledge on these guys. Expressing a solution in terms of elliptic integrals is not like meeting an old friend, it is like meeting a stranger who was famous a century ago, a grainy photo of whom you might have once seen in an old book.

I personally would not consider the circumference of an ellipse "closed form". Just call it the "circumference of the ellipsis", or write it as an integral, depending on how to better make apparent which properties you want.

Of course this is a trade-off, how much time to spend developing an intuition and working knowledge of "general integrals" (likely from a functional analysis perspective, as an operator) and how much time to spend understanding specific special integrals. The specific will always be more effective and impart deeper knowledge when dealing with the specifics, but the general theory is more applicable and "geometric"; you might say that it extrapolates very well from the training set. Some specific special functions are worth it, eg exp/log, and some used to be considered worthy but are today not considered worthy, evidenced by revealed preference (what do people put into course syllabi).

So, in some sense you have a large edifice of "forgotten knowledge" in mathematics. This knowledge is archived, of course, but the unbroken master-apprentice chains of transmission have mostly died out. I think this is sad; we, as a society, should be rich enough to sponsor a handful of people to keep this alive, even if I'd say "good riddance" for removing it from the "standard canon".

Anecdote: Elliptic integrals sometimes appear in averaging: You have a differential equation (dynamical system) and want to average over fast oscillations in order to get an effective (ie leading order / approximate) system with reduced dimension and uniform time-scales. Now, what is your effective equation? You can express it as "the effective equation coming out of Theorem XYZ", or write it down as an integral, which makes apparent both the procedure encoded in Theorem XYZ and an integral expression that is helpful for intuition and calculations. And sometimes, if you type it into Wolfram alpha, it transforms into some extremely lenghty expression containing elliptic integrals. Do you gain understanding from this? I certainly don't, and decided not to use the explicit expressions when I met them in my research (99% of the time, mathematica is not helpful; the 1% pays for the trivial inconvenience of always trying whether there maybe is some amazing transformation that simplifies your problem).

**daozaich**on The First Rung: Insights from 'Linear Algebra Done Right' · 2018-04-22T19:44:13.457Z · LW · GW

Regarding insolubility of the quintic, I made a top level post with essentially the same point, because it deserves to be common knowledge, in full generality.

**daozaich**on Multi-winner Voting: a question of Alignment · 2018-04-22T16:18:38.074Z · LW · GW

I guess that this is due to the historical fact that candidates in the US are supposed to be district-local, not state-local, and districts are supposed to be as small as possible. I'm not an American, so I cannot say how strong this is as a constraint for modified electoral systems.

If you had a small party/faction, with say 10% of popular vote, reaching up to maybe 30% in their strongest districts, then I would definitely see a problem: Such a party simply does not fit purely district-local representation (one-to-one mapping between districts and representatives). Think e.g. an environmentalist party.

If your representative chamber is more about opposing ideologies and national governance than about opposing local interests, then why not ditch this one-to-one mapping?

I mean, this works even in the EU parliament, and you can't tell me that opposing local interests across US districts are harder than across states within EU countries? And you have a second chamber (the senate) that is explicitly about conflicting local interests.

I presume that you know how German federal elections, or European parliament elections are run; easy to google, far from perfect, but gets at least this right. I would definitely be opposed to a change to PLACE at home, for these reasons, but agree that PLACE beats FPTP by lengths.

And something that PLACE gets extremely right is to involve the general populace in the within-party selection. In Germany, only party members are allowed to vote in primaries (and members pay contributions and are expected to be activist, and parties can expel or reject prospective members). This slightly sucks; "too radical" candidates / factions are routinely squashed by the high party functionaries.

Of course this gives you an amusing game-theoretical problem in e.g. EU parliament elections: If countries were entirely free to select their voting system, they would have incentive to move to a winner-takes-all system, which strengthens the representation of their national interest. Same way you cannot get rid of your accursed winner-takes-all within states in US presidential elections, as long as states are free to design their voting systems. And, on the same lines, if you must have a winner-takes-all system, then each state should have votes roughly based on square root of population (I am yet again ashamed to be German for the way we treated the Polish, this time after they proposed the square-root for the EU council).

**daozaich**on Multi-winner Voting: a question of Alignment · 2018-04-22T01:03:57.327Z · LW · GW

Re PLACE: Interesting proposal. Have you considered the following problem (I'd guess you have; a link would be appreciated):

Candidates are not exchangeable. Candidate A has done a very good job in the legislature. An opposing faction may decide to coordinate to support his local opposing candidate B, in order to keep person A out of parliament.

Or, in other words: Two candidates running in the same district cannot both become part of parliament. This opens a huge amount of gaming, in order to squash small parties / factions that do not have a deep bench of good candidates. Party A and its voters have large influence on the composition of Party B's parliamentary faction, and can strategically plan this.

The standard solution, e.g. in German federal elections, is to have pools: Candidates can be elected locally (in-district) or statewide. Candidates who won statewide are expected to still try to represent their district (where they lost), if they ran locally at all.

**daozaich**on Weird question: could we see distant aliens? · 2018-04-21T23:36:20.013Z · LW · GW

One guess for cheap signaling would be to seed stellar atmospheres with stuff that should not belong. Stellar spectra are really good to measure, and very low concentration of are visible (create a spectral line). If you own the galaxy, you can do this at sufficiently many stars to create a spectral line that should not belong. If we observed a galaxy with "impossible" spectrum, we would not immediately know that it's aliens; but we would sure point everything we have at it. And spectral data is routinely collected.

I am not an astronomer, though. So this is not meant as an answer, but rather as a starting point for others to do more literature research. I think I have seen this discussed somewhere, using technetium; but googling revealed that stars with technetium actually exist!

**daozaich**on Weird question: could we see distant aliens? · 2018-04-21T23:14:49.898Z · LW · GW

I think communicating without essentially conquering the Hubble volume is still an interesting question. I would not rule out a future human ethical system that restricts expansion to some limited volume, but does not restrict this kind of omnidirectional communication. Aliens being alien, we should not rule out them having such a value system either.

That being said, your article was really nice. Send multiplying probes everywhere, watch the solar system form and wait for humans to evolve in order to say "hi" is likely to be amazingly cheap.

**daozaich**on A voting theory primer for rationalists · 2018-04-16T11:32:26.306Z · LW · GW

Re SODA: The setup appears to actively encourage candidates to commit to a preference order. Naively, I would prefer a modification along the following lines; could you comment?

(1) Candidates may make promises about their preference order among other candidates; but this is not enforced (just like ordinary pre-election promises). (2) The elimination phase runs over several weeks. In this time, candidates may choose to drop out and redistribute their delegated votes. But mainly, the expected drop-outs will negotiate with expected survivors, in order to get at least some of their policies implemented (with the same kind of enforcement as regular pre-election promises). Hence, this phase is "coalition building".

An interesting final phase (3), in order to encourage compromise / beneficial trade would be something like: Once we are down to candidates with > 25% approval, we randomize the result. The probability of a candidate to win could be something like the square, or third power, of his approval. The threshold of 25% is in order to prevent complete crackpots from winning the election, and might need to be even more brutal. The randomization serves to allow the two remaining highest approval platforms to negotiate a compromise, weighted by their chance of winning the final lottery. In practice, one would hope that the randomization is never applied: That is, the highest candidate makes enough concessions in order to make the second candidate agree to drop out.

This way, we preserve minority rights, and get a pretty good outcome if the candidates are actually reasonable and trustworthy parties (instead of persons) that are capable of committing to modify the values they will espouse once in power.

Obvious disadvantages are (1) negotiation skill and integrity become essential (integrity in the christiano sense: You need to be predictable in which promises you will break under which contingencies), because the winner is decided though coalition building, rather than voting; and (2) some voters might object to shady backroom deals being the explicit procedure, and (3) randomization may make losers very pissed if the dice are ever rolled (which hopefully is never, because of positive-sum trades).

You can obviously combine this with a 3-2-1 system (where only candidates reaching at least 50% "OK" votes are eligible, and otherwise "Good" votes are counted; if no candidate receives 50% OK, then the election is repeated, under the assumption that Belgium is better than Trump, i.e. better a constitutional crisis and government by interim civil servants than government by crazy elected officials).

edit: I forgot to mention the other big advantage of the probabilistic phase (3): Enforcing continuity, i.e. preventing small factions from gaining oversized influence by playing kingmaker.

**daozaich**on Kaj's shortform feed · 2018-04-06T16:28:34.149Z · LW · GW

Regarding measurement of pain:suffering ratio

A possible approach would be to use self-reports (the thing that doctor's always ask about, pain scale 1-10) vs revealed preferences (how much painkillers were requested? What trade-offs for pain relief do patients choose?).

Obviously this kind of relation is flawed on several levels: Reported pain scale depends a lot on personal experience (very painful events permanently change the scale, ala "I am in so much pain that I cannot walk or concentrate, but compared to my worst experience... let's say 3?"). Revealed preferences depend a lot on how much people care about the alternatives (e.g. if people have bad health insurance or really important stuff to do they might accept a lot of subjective suffering in order to get out of hospital one day early). Likewise, time preference might enter a lot into revealed preference.

Despite these shortcomings, that's where I would start thinking about what such a ratio would mean. If one actually did a study with new questionaires, one should definitely ask patients for some examples in order to gauge their personal pain-scale, and combine actual revealed preferences with answers to hypothetical questions "how much money would pain relief be worth to you? How much risk of death? How many days of early hospital release? etc", even if the offer is not actually on the table.

**daozaich**on Against Occam's Razor · 2018-04-06T15:03:33.669Z · LW · GW

>But the greatest merit of Occamian prior is that it vaguely resembles the Lazy prior.

...

>With that in mind, I asked what prior would serve this purpose even better and arrived at Lazy prior. The idea of encoding these considerations in a prior may seem like an error of some kind, but the choice of a prior is subjective by definition, so it should be fine.

Encoding convenience * probability into some kind of pseudo-prior such that the expected-utility maximizer is the maximum likelihood model with respect to the pseudo-prior does seem like a really useful computational trick, and you are right that terminology should reflect this. And you are right that the Occam prior has the nice property that weight-by-bit-count is often close to convenience, and hence makes the wrong naive approach somewhat acceptable in practice: That is, just taking the max likelihood model with respect to bit-count should often be a good approx for weight-by-bitcount * convenience (which is the same as weight-by-bitcount for probability and maximize expected utility).

In cases where we know the utility we can regenerate probabilities afterwards. So I would now be really interested in some informal study of how well Occam actually performs in practice, after controlling for utility: You are right that the empirical success of Occam might be only due to the implicit inclusion of convenience (succinct-by-bit-count models are often convenient) when doing the (wrong!) max-likelihood inference. I had not considered this, so thanks also for your post; we both learned something today.

I'd also remark/reiterate the point in favor of the Lazy prior: The really terrible parts of working with Occam (short descriptions that are hard to reason about, aka halting problem) get cancelled out in the utility maximization anyway. Lazy avoids invoking the halting-problem oracle in your basement for computing these terms (where we have the main differences between Occam vs Lazy). So you are right after all: Outside of theoretical discussion we should all stop using probabilities and Occam and switch to some kind of Lazy pseudo-prior. Thanks!

That being said, we all appear to agree that Occam is quite nice as an abstract tool, even if somewhat naive in practice.

A different point in favor of Occam is "political objectivity": It is hard to fudge in motivated reasoning. Just like the "naive frequentist" viewpoint sometimes wins over Bayes with respect to avoiding politically charged discussions of priors, Occam defends against "witchcraft appears natural to my mind, and the historical record suggests that humans have evolved hardware acceleration for reasoning about witchcraft; so, considering Lazy-prior, we conclude that witches did it" (Occam + utility maximization rather suggests the more palatable formulation "hence it is useful to frame these natural processes in terms of Moloch, Azatoth and Cthulhu battling it out", which ends up with the same intuitions and models but imho better mental hygiene)

**daozaich**on Against Occam's Razor · 2018-04-06T12:57:58.613Z · LW · GW

I have a feeling that you mix probability and decision theory. Given some observations, there are two separate questions when considering possible explanations / models:

1. What probability to assign to each model?

2. Which model to use?

Now, our toy-model of perfect rationality would use some prior, e.g. the bit-counting universal/kolmogorov/occam one, and bayesian update to answer (1), i.e. compute the posterior distribution. Then, it would weight these models by "convenience of working with them", which goes into our expected utility maximization for answering (2), since we only have finite computational resources after all. In many cases we will be willing to work with known wrong-but-pretty-good models like Newtonian gravity, just because they are so much more convenient and good enough.

I have a feeling that you correctly intuit that convenience should enter the question which model to adopt, but misattribute this into the probability-- but which model to adopt should formally be bayesian update + utility maximization (taking convenience and bounded computational resources into account), and definitely not "Bayesian update only", which leads you to the (imho questionable) conclusion that the universal / kolmogorov / occam prior is flawed for computing probability.

On the other hand, you are right that the above toy model of perfect rationality is computationally bad: Computing the posterior distribution after some prior and then weighting by utility/convenience is of stupid if directly computing prior * convenience is cheaper than computing prior and convenience separately and then multiplying. More generally, probability is a nice concept for human minds to reason about reasoning, but we ultimately care about decision theory only.

Always combining probability and utility might be a more correct model, but it is often conceptually more complex to my mind, which is why I don't try to always adopt it ;)

**daozaich**on Brains and backprop: a key timeline crux · 2018-03-11T22:04:07.428Z · LW · GW

I think part of the assumption is that reflection can be bolted on trivially if the pattern matching is good enough. For example, consider guiding an SMT / automatic theorem prover by deep-learned heuristics, e.g. (https://arxiv.org/abs/1701.06972)[https://arxiv.org/abs/1701.06972] . We know how to express reflection in formal languages; we know how to train intuition for fuzzy stuff; me might learn how to train intuition for formal languages.

This is still borderline useless; but there is no reason, a priori, that such approached are doomed to fail. Especially since labels for training data are trivial (check the proof for correctness) and machine-discovered theorems / proofs can be added to the corpus.

**daozaich**on Why mathematics works · 2018-03-09T16:53:48.541Z · LW · GW

I strongly disagree that anthropics explains the unreasonable effectiveness of mathematics.

You can argue that a world, where people develop a mind and mathematical culture like ours (with its notion of "modular simplicity") should be a world where mathematics is effective in everyday phenomena like throwing a spear.

This tells us nothing about what happens if we extrapolate to scales that are not relevant to everyday phenomena.

For example, physics appears to have very simple (to our mind) equations and principles, even at scales that were irrelevant during our evolution. The same kind of thought-processes are useful both for throwing spears / shooting cannons and for describing atoms. This is the unreasonable effectiveness of mathematics.

On the other hand, there are many phenomena where mathematics is not unreasonably effectice; take biological systems. There, our brains / culture have evolved heuristics that are useful on a human scale, but are entirely bogus on a micro-scale or macro-scale. Our mathematics is also really bad at describing the small scales; reductionism is just not that useful for understanding, say, how a genome defines an organism, and our brains / culture are not adapted to understanding this.

I think a counterfactual world, where physics outside the scales of human experience were as incomprehensibly complex (to our minds) as biology outside human scales does sound realistic. It does seem like a remarkable and non-trivial observation that the dynamics of a galaxy, or the properties of a semiconductor, are easy to understand for a culture that learned how a cannonball flies; whereas learning how to cultivate wheat or sheep is not that helpful for understanding cancer.

**daozaich**on Prize for probable problems · 2018-03-08T23:49:26.744Z · LW · GW

[Meta: Even low-effort engagement, like "known + keyword" or "you misunderstood everything; read <link>" or "go on talking / thinking" is highly appreciated. Stacks grow from the bottom to the top today, unlike x86 or threads on the internet]

------------

Iterative amplification schemes work by having each version trained by previous iteration ; and, whenever version fails at finding a good answer (low confidence in the prediction), punting the question to , until it reaches the human overseer at , which is the ground truth for our purposes. At the same time, the distribution of relevant question widens at each iteration, as capabilities improve: For example, a very stupid clippy will need to deal with simple linear optimization problems of its supply chain; a very smart clippy will need to figure out a proper plan for world-domination.

Now, in order to correctly punt questions down the call-stack, each level when faced with a problem must decide whether it can solve it (providing training data for level ), or whether it must punt downwards. More precisely, level must recognize whether is inside the distribution it was trained with (resp. , the distribution of problems it can generate / handle post-training).

False positives (underconfidence) lead to increased load on the human overseers / capability loss. False negatives (overconfidence) lead to safety failures: If they happen during training, then the machine learns wrong values / ethics / whatever-it-should-learn, if they happen in production then they lead to bad actions (I am assuming that no one would run the AI unsupervised, but supervision will, by necessity, be pretty light).

This is a problem, even with a very shallow call-stack (few iterations), but it gets (possibly exponentially!) worse with deeper stacks: After all, if even a single false-negative mistake occurs along the chain, we have a safety incident; naively, the chance of mistakes grows exponentially in the depth, for fixed false-negative rate at each level. In other words, we would need to ensure that the infinite series converges, where is the false-negative rate at level .

Something that may makes this problem either simpler or harder is the fact that, once a question has been bumped down a couple of times, it can be very far out-of-distribution. This makes it much easier to get mediocre reliability (the questions bumped down from very high levels are very weird for the low levels), but, at the same time, the low levels are really, really not equipped for dealing with these weird questions. In the clippy example, Paul's scheme would totally allow the high capability clippy to convince low levels into accepting its proposal by exploiting its weaknesses (adversarial examples).

In different words: The entire scheme needs really good anomaly detection. Anomaly detection is a machine-learning problem, right? So learn this along the way?

The problem is that, while training the anomaly detection at level , we cannot sample from the distribution , because we simply don't know it yet. As we run amplification, we extend both the capability of answering questions and the range of questions that come up.

Note that this would not be a problem if the extensions of supports of the distributions were very mild, compared to the larger domain where the anomaly detection of level is reliable-by-training. This appears like a very dangerous assumption on the "mind-space" accessible to our training procedure, for the "problem-space" at hand and the specific training procedure. In other words: No universality of safety amplification, if it relies on this assumption. Please link me to relevant discussions if this assumption has been justified anywhere.

A different approach would be to not run the scheme in a hierarchical way; instead try to feed it into itself. This reintroduces the problem of "drift", i.e. instability: The system will acquire a specific culture, and we can only pray (i.e. must assume / ensure) that desireable states are an attractor, for the specific training procedure. I fear that this is extra-hard to check, because meta-stable systems look pretty stable, until you increase capability: Bad scaling.

A third approach to solving this problem would be to create, separately, a reliable anomaly detector. This may very well be an AI complete problem, but it may be simpler than alignment! Further, robust anomaly detection (recognizing that data points are outside of the training distribution) might be an avenue for differential progress that matters more for safety/alignment than capabilities.

Do you know other approaches for this problem?

------------

When is this harmless?

Well, this is entirely harmless if our problem domain contains only simple questions, where verification of ground truth is dirt cheap. For example, NP questions (like, e.g., all of mathematics! Proof verification is trivial after all). This observation should give us a bad feeling, because an AI that is really good at mathematics / programming and bad at everything else is one of the most dangerous situation we could imagine (magical function optimizers, aka mindless AIXI genies, being the only more hopeless case I can think of). On the other hand, typical NP questions don't scale down: It is currently entirely infeasible to use machine learning for theorem proving, simply because useful transformations are exponentially rare in the space of possible ones (I am aware of some papers using the mizar library; while the neural net + theorem prover beat the unaided prover, I was less than impressed by the results).

For problem domains that feel more like exptime, this is more likely to be a problem: Say, training to play games like Go. Then, we can play against our ancestors in order to judge performance, and gain access to some kind of ground truth. Unfortunately, (1) strength is not linearly ordered: You clearly can have situations where A beats B beats C beats A, and (2) if we wanted to optimize "strength against perfect play", aka min-max, then we don't have access to a perfect opponent during training. Afaik it is usual for training-through-amplification of Game AI to develop "fads", i.e. cheesy tactics, on the way; sometimes, these recur cyclically. This is also observed for the metagame in many multiplayer videogames. I have a feeling that the Go successes tell us a lot about how MCTS is amazingly stable against cheesy tactics; and who knows how much tweaking deepmind had to do until they got the amplification stable.

Now, safety amplification / value learning has a much, much harder problem: The ground truth is only accessible through examples / very expensive oracle queries (which might be fundamentally unsafe, at very high levels of capability: Don't let human operators talk to unaligned too-clever AI).

------------

Post-script: Writing this down in clear words made me slightly update against Paul's amplification schemes eventually growing into a solution. I still think that Paul's line of research is damn cool and promising, so I'm more playing devil's advocate here. The possible differential gain for capability in NP problems versus harder-than-NP alignment for this kind of amplification procedure made me slightly more pessimistic about our prospects in general. Moreover, it makes me rather skeptic whether amplification is a net win for safety / alignment in the differential progress view. I want to look more into anomaly detection now, for fun, my own short-term profit and long-term safety.

**daozaich**on Takeoff Speed: Simple Asymptotics in a Toy Model. · 2018-03-07T01:54:56.786Z · LW · GW

(1) As Paul noted, the question of the exponent alpha is just the question of diminishing returns vs returns-to-scale.

Especially if you believe that the rate is a product of multiple terms (like e.g. Paul's suggestion with one exponent for computer tech advances and another for algorithmic advances) then you get returns-to-scale type dynamics (over certain regimes, i.e. until all fruit are picked) with finite-time blow-up.

(2) Also, an imho crucial aspect is the separation of time-scales between human-driven research and computation done by machines (transistors are faster than neurons and buying more hardware scales better than training a new person up to the bleeding edge of research, especially considering Scott's amusing parable of the alchemists).

Let's add a little flourish to your model: You had the rate of research and the cumulative research ; let's give a name to the capability of the AI system. Then, we can model . This is your model, just splitting terms into , which tells us how hard AI progress is, and which tells us how good we are at producing research.

Now denote by the fraction of work that absolutely has to be done by humans, and by the speed-up factor for silicon over biology. Amdahl's law gives you , or somewhat simplified . This predicts a rate of progress that first looks like , as long as human researcher input is the limiting factor, then becomes when we have AIs designing AIs (recursive self-improvement, aka explosion), and then probably saturates at something (when the AI approaches optimality).

The crucial argument for fast take-off (as far as I understood it) is that we can expect to hit at some cross-over , and we can expect this to happen with a nonzero derivative . This is just the claim that human-level AI is possible, and that the intelligence of the human parts of the AI research project is not sitting at a magical point (aka: this is generic, you would need to fine-tune your model to get something else).

The change of the rate of research output from the regime to the regime sure looks like a hard-take-off singularity to me! And I would like to note that the function , i.e. the hardness AI research and the diminishing-returns vs returns-to-scale debate does not enter this discussion at any point.

In other words: If you model AI research as done by a team of humans and proto-AIs assisting the humans; and if you assert non-fungibility of humans vs proto-AI-assistents (even if you buy a thousand times more hardware, you still need the generally intelligent human researchers for some parts); and if you assert that better proto-AI-assistents can do a larger proportion of the work (at all); and if you assert that computers are faster than humans; then you get a possibly quite wild change at .

I'd like to note that the cross-over is not "human-level AI", but rather "" , i.e. an AI that needs (almost) no human assistence to progress the field of AI research.

On the opposing side (that's what Robin Hanson would probably say) you have the empirical argument that should decay like a power-law long before we ("the last 10% take 90% of the work" is a folk formulation for "percentile 90-99 take nine time as much work as percentile 0-89" aka power law, and is borne out quite well, empirically).

This does not have any impact on whether we cross with non-vanishing derivative, but would support Paul's view that the world will be unrecognizably crazy long before .

PS. I am currently agnostic about the hard vs soft take-off debate. Yeah, I know, cowardly cop-out.

edit: In the above, C kinda encodes how fast / good our AI is and q encodes how general it is compared to humans. All AI singularity stuff tacitly assumes that human intelligence (assisted by stupid proto-AI) is sufficiently general to design an AI that exceeds or matches the generality of human intelligence. I consider this likely. The counterfactual world would have our AI capabilities saturate at some subhuman level for a long time, using terribly bad randomized/evolutionary algorithms, until it either stumbles unto an AI design that has better generality or we suffer unrelated extinction/heat-death. I consider it likely that human intelligence (assisted by proto-AI) is sufficiently general for a take-off. Heat-death is not an exaggeration: Algorithms with exponentially bad run-time are effectively useless.

Conversely, I consider it very well possible that human intelligence is insufficiently general to understand how human intelligence works! (we are really, really bad at understanding evolution/gradient-descent optimized anything, an that's what we are)

**daozaich**on The abruptness of nuclear weapons · 2018-03-03T20:06:09.645Z · LW · GW

Just commenting that the progress to thermonuclear weapons represented another discontinuous jump (1-3 orders of magnitude).

Also, whether von Neumann was right depends on the probability for the cold war ending peacefully. If we retrospectively conclude that we had a 90% chance of total thermonuclear war (and just got very lucky in real life) then he was definitely right. If we instead argue from the observed outcome (or historical studies conclude that the eventual outcome was not due to luck but rather due to the inescapable logic of MAD), then he was totally nuts.

Near-misses are not necessarily a very strong guide to estimating retrospective risk. Both sides were incentivized to hide their fail-safes for escalation; to credibly commit to having a twitchy retaliation finger, and at the same time to not actually retaliate (the game is first chicken, then ultimatum, and never prisoner's dilemma). So I would be very wary of trusting the historical record on "what if Petrov had not kept a cool mind".

**daozaich**on Arguments about fast takeoff · 2018-03-03T19:16:42.549Z · LW · GW

Not sure. I encountered this once in my research, but the preprint is not out yet (alas, I'm pretty sure that this will still be not enough to reach commercial viability, so pretty niche and academic and not a very strong example).

Regarding "this is not common": Of course not for problems many people care about. Once you are in the almost-optimal class, there are no more giant-sized fruit to pick, so most problems will experience that large jumps never, once or twice over all of expected human history (sorting is