Posts
Comments
I doubt your optimism on the level of security that is realistically achievable. Don't get me wrong: The software industry has made huge progress (at large costs!) in terms of security. Where before, most stuff popped a shell if you looked at it funny, it is now a large effort for many targets.
Further progress will be made.
If we extrapolate this progress -- we will optimistically reach a point where impactful reliable 0day is out of reach for most hobbyists and criminals, and the domain of natsec of great powers.
But I don't see how raising this waterline will help for AI risk in particular?
As in: godlike superintelligence is game over anyway. AI that is comparably good at exploitation as the rest of humanity taken together, is beyond what is realistically defendable against, in terms of wide-spread deployed security level. An AI that doesn't reach that level without human assistance is probably not lethal anyways.
On the other hand, one could imagine pivotal acts by humans with limited-but-substantial AI assistance that rely on the lack of wide-spread security.
Pricing human + weakish AI collaborations out of the world-domination-via-hacking game might actually make matters worse, in so far as weakish non-independent AI might be easier to keep aligned.
A somewhat dystopian wholesale surveillance of almost every word written and said by humans, combined with AI that is good enough at text comprehension and energy efficient enough to pervasively and correctly identify scary-looking research and flag it to human operators for intervention is plausibly pivotal and alignable, and makes for much better cyberpunk novels than burning GPUs anyway (mentally paging cstross, I want my Gibson homage in form of a "Turing Police"/laundry-verse crossover).
Also, good that you mentioned rowhammer. Rowhammer and the DRAM industries half-baked pitiful response are humankinds capitulation in terms of "making at least some systems actually watertight".
The fixed point problem is worse than you think. Take the Hungarian astrology example, with an initial easy set with both a length limitation (e.g. < 100k characters) and simplicity limitation.
Now I propose a very simple improvement scheme: If the article ends in a whitespace character, then try to classify the shortened article with last character removed.
This gives you an infinite sequence of better and better decision boundaries (each time, a couple of new cases are solved -- the ones that are of lenth 100k + $N$, end in at least $N$ whitespace, and are in the easy set once the whitespace has been stripped). This nicely converges to the classifier that trims all trailing whitespace and then asks its initial classifier.
What I'm trying to say here is: The space of cases to consider can be large in many dimensions. The countable limit of a sequence of extensions needs not be a fixed point of the magical improvement oracle.
Generally, I'd go into a different direction: Instead of arguing about iterated improvement, argue that of course you cannot correctly extrapolate all decision problems from a limited amount of labeled easy cases and limited context. The style of counter-example is to construct two settings ("models" in the lingo of logic) A and B with same labeled easy set (and context made available to the classifier), where the correct answer for some datapoint x differs in both settings. Hence, safe extrapolation must always conservatively answer NO to x, and cannot be expected to answer all queries correctly from limited training data (typical YES / NO / MAYBE split).
I think the discussion about the fixed point or limit iterative improvement does not lead to the actually relevant argument that extrapolation cannot conjure information out of nowhere?
You could cut it out completely without weakening the argument against certain types of automated ontology identification being impossible.
The Definition-Theorem-Proof style is just a way of compressing communication. In reality, heuristic / proof-outline comes first; then, you do some work to fill the technical gaps and match to the existing canon, in order to improve readability and conform to academic standards.
Imho, this is also the proper way of reading maths papers / books: Zoom in on the meat. Once you understood the core argument, it is often unnecessary too read definitions or theorems at all (Definition: Whatever is needed for the core argument to work. Theorem: Whatever the core argument shows). Due to the perennial mismatch between historic definitions and theorems and the specific core arguments this also leaves you with stronger results than are stated in the paper / book, which is quite important: You are standing on the shoulders of giants, but the giants could not foresee where you want to go.
This paints a bleak picture for the possibility of aligning mindless AGI since behavioral methods of alignment are likely to result in divergence from human values and algorithmic methods are too complex for us to succeed at implementing.
To me it appears like the terms cancel out: Assuming we are able to overcome the difficulties of more symbolic AI design, the prospect of aligning such an AI seem less hard.
In other words, the main risk is wasting effort on alignment strategies that turn out to be mismatched to the eventually implemented AI.
The negative prices are a failure of the market / regulation, they don't actually mean that you have free energy.
That being said, the question for the most economical opportunistic use of intermittent energy makes sense.
No. It boils down to the following fact: If you take given estimates on the distribution of parameter values at face value, then:
(1) The expected number of observable alien civilizations is medium-large (2) If you consider the distribution of the number of alien civs, you get a large probability of zero, and a small probability of "very very many aliens", that integrates up to the medium-large expectation value.
Previous discussions computed (1) and falsely observed a conflict with astronomical observations, and totally failed to compute (2) from their own input data. This is unquestionably an embarrassing failure of the field.
What is logical induction's take on probabilistic algorithms? That should be the easiest test-case.
Say, before "PRIME is in P", we had perfectly fine probabilistic algorithms for checking primality. A good theory of mathematical logic with uncertainty should permit us to use such an algorithm, without random oracle, for things you place as "logical uncertainty". As far as I understood, the typical mathematician's take is to just ignore this foundational issue and do what's right (channeling Thurston: Mathematicians are in the business of producing human understanding, not formal proofs).
It’s excellent news! Your boss is a lot more likely to complain about some minor detail if you’re doing great on everything else, like actually getting the work done with your team.
Unfortunately this way of thinking has a huge, giant failure mode: It allows you to rationalize away critique about points you consider irrelevant, but that are important to your interlocutor. Sometimes people / institutions consider it really important that you hand in your expense sheets correctly or turn up in time for work, and finishing your project in time with brilliant results is not a replacement for "professional demeanor". This was not a cheap lesson for me; people did tell me, but I kinda shrugged it off with this kind of glib attitude.
Is there a way of getting "pure markdown" (no wysiwyg at all) including Latex? Alternatively, a hotkey-less version of the editor (give me buttons/menus for all functionality)?
I'm asking because my browser (chromium) eats the hotkeys, and latex (testing: $\Sigma$ ) appears not to be parsed from markdown. I would be happy with any syntax you choose. For example \Sigma; alternatively the github classic of using backticks appears still unused here.
edit: huh, backticks are in use and html-tags gets eaten.
Isn't all this massively dependent on how your utility $U$ scales with the total number $N$ of well-spent computations (e.g. one-bit computes)?
That is, I'm asking for a gut feeling here: What are your relative utilities for $10^{100}$, $10^{110}$, $10^{120}$, $10^{130}$ universes?
Say, $U(0)=0$, $U(10^100)=1$ (gauge fixing); instant pain-free end-of-universe is zero utility, and a successful colonization of the entire universe with a suboptimal black hole-farming near heat-death is unit utility.
Now, per definitionem, the utility $U(N)$ of a $N$-computation outcome is the inverse of the probability $p$ at which you become indifferent to the following gamble: Immediate end-of-the-world at probability $(1-p)$ vs an upgrade of computational world-size to $N$ at propability $p$.
I would personally guess that $U(10^{130})< 2 $; i.e. this upgrade would probably not be worth a 50% risk of extinction. This is massively sublinear scaling.
What was initially counterintuitive is that even though , the series doesn't converge.
This becomes much less counterintuitive if you instead ask: How would you construct a sequence with divergent series?
Obviously, take a divergent series, e.g. , and then split the th term into .
FWIW, looking at an actual compiler, we see zero jumps (using a conditional move instead):
julia> function test(n)
i=0
while i<n
i += 1
end
return i
end
test (generic function with 1 method)
julia> @code_native test(10)
.text
Filename: REPL\[26\]
pushq %rbp
movq %rsp, %rbp
Source line: 3
xorl %eax, %eax
testq %rdi, %rdi
cmovnsq %rdi, %rax
Source line: 6
popq %rbp
retq
nop
edit: Sorry for the formatting. I don't understand how source-code markup is supposed to work now?
edit2: Thanks, the markup works now!
edit3: So, to tie this into your greater point:
If you don't ask "how would I code this in assembly" but rather "how should my compiler reason about this code", then it is clear that the loop can be obviously eliminated: You place a phi-node at the end of the loop, and a tiny bit of inductive reasoning makes the loop body obviously dead code if n is an integer type. Slightly more magical (meaning I'm not a compiler expert) is the fact that the compiler (LLVM) can completely eliminate the following loop (replacing it with an explicit formula):
julia> function sumN6(lim)
s=0
i=0
while i<lim
i+=1
s+= i*i*i*i*i*i
end
return s
end
"what move should open with in reversi" would be considered as an admissible decision-theory problem by many people. Or in other words: Your argument that EU maximization is in NP only holds for utility functions that permit computation in P of expected utility given your actions. That's not quite true in the real world.
This, so much.
So, in the spirit of learning from other's mistakes (even better than learning from my own): I thought Ezra made his point very clear.
So, all of you people who missed Ezra's point (confounded data, outside view) on first reading:
How could Ezra have made clearer what he was arguing, short of adopting LW jargon? What can we learn from this debacle of a discussion?
Edit: tried to make my comment less inflammatory.
>I was imagining a sort of staged rocket, where you ejected the casing of the previous rockets as you slow, so that the mass of the rocket was always a small fraction of the mass of the fuel.
Of course, but your very last stage is still a rocket with a reactor. And if you cannot build a rocket with 30g motor+reactor weight, then you cannot go to such small stages and your final mass on arrival includes the smallest efficient rocket motor / reactor you can build, zero fuel, and a velocity that is below escape velocity of your target solar system (once you are below escape velocity I'll grant you maneuvers with zero mass cost, using solar sails; regardless, tiny solar-powered ion-drives appear reasonable, but generate not enough thrust to slow down from relativistic to below-escape in the time-frame before you have passed though your target system).
>But Eric Drexler is making some strong arguments that if you eject the payload and then decelerate the payload with a laser fired from the rest of the "ship", then this doesn't obey the rocket equation. The argument seems very plausible (the deceleration of the payload is *not* akin to ejecting a continuous stream of small particles - though the (tiny) acceleration of the laser/ship is). I'll have to crunch the number on it.
That does solve the "cannot build a small motor" argument, potentially at the cost of some inefficiency.
It still obeys the rocket equation. The rocket equation is like the 2nd law of thermodynamics: It is not something you can trick by clever calculations. It applies for all propulsion systems that work in a vacuum.
You can only evade the rocket equation by finding (non-vacuum) stuff to push against; whether it be the air in the atmosphere (airplane or ramjet is more efficient than rocket!), the solar system (gigantic launch contraption), various planets (gravitational slingshot), the cosmic microwave background, the solar wind, or the interstellar medium. Once you have found something, you have three choices: Either you want to increase relative velocity and expend energy (airplane, ramjet), or you want to decrease relative velocities (air-braking, use of drag/friction, solar sails when trying to go with the solar wind, braking against the interstellar medium, etc), or you want an elastic collision, e.g. keep absolute relative velocity the same but reverse direction (gravitational slingshot).
Slingshots are cool because you extract energy from the fact that the planets have different velocities: Having multiple planets is not thermodynamic ground state, so you steal from the potential energy / negative entropy left over from the formation of the solar system. Alas, slingshots can't bring you too much above escape velocity, nor slow you down to below escape if you are significantly faster.
Edit: probably stupid idea, wasn't thinking straight <strike> Someone should tell me whether you can reach relativistic speeds by slingshotting in a binary or trinary of black holes. That would be quite elegant (unbounded escape velocity, yay! But you have time dilation when close to the horizon, so unclear whether this takes too long from the viewpoint of outside observers; also, too large shear will pull you apart).</strike>
edit2: You can afaik also push against a curved background space-time, if you have one. Gravity waves technically count as vacuum, but not for the purpose of the rocket equation. Doesn't help, though, because space-time is pretty flat out there, not just Ricci-flat (=vacuum).
If you have to use the rocket equation twice, then you effectively double delta-v requirements and square the launch-mass / payload-mass factor.
Using Stuart's numbers, this makes colonization more expensive by the following factors:
0.5 c: Antimatter 2.6 / fusion 660 / fission 1e6
0.8 c: Antimatter 7 / fusion 4.5e5 / fission 1e12
0.99c Antimatter 100 / fusion 4.3e12 / fission 1e29
If you disbelieve in 30g fusion reactors and set a minimum viable weight of 500t for an efficient propulsion system (plus negligible weight for replicators) then you get an additional factor of 1e7.
Combining both for fusion at 0.8c would give you a factor of 5e12, which is significantly larger than the factor between "single solar system" and "entire galaxy". These are totally pessimistic assumptions, though: Deceleration probably can be done cheaper, and with lower minimal mass for efficient propulsion systems. And you almost surely can cut off quite a bit of rocket-delta-v on acceleration (Stuart assumed you can cut 100% on acceleration and 0% on deceleration; the above numbers assumed you can cut 0% on acceleration and 0% on deceleration).
Also, as Stuart noted, you don't need to aim at every reachable galaxy, you can aim at every cluster and spread from there.
So, I'm not arguing with Stuart's greater claim (which is a really nice point!), I'm just arguing about local validity of his arguments and assumptions.
You're right, I should have made that clearer, thanks!
I would not fret too much about slight overheating of the payload; most of the launch mass is propulsion fuel anyway, and in worst-case the payload can rendezvous with the fuel in-flight, after the fuel has cooled down.
I would be very afraid of the launch mass, including solar sail / reflector loosing (1) reflectivity (you need a very good mirror that continues to be a good mirror when hot; imperfections will heat it) and (2) structural integrity.
I would guess that, even assuming technological maturity (can do anything that physics permits), you cannot keep structural integrity above, say, 3000K, for a launch mass that is mostly hydrogen. I think that this is still icy cold, compared to the power output you want.
So someone would need to come up with either
1. amazing schemes for radiative heat-dissipation and heat pumping (cannot use evaporative cooling, would cost mass),
2. something weird like a plasma mirror (very hot plasma contained by magnetic fields; this would be hit by the laser, which pushes it via radiation pressure and heats it; momentum is transferred from plasma to launch probe via magnetic field; must not loose too many particles, and might need to maintain a temperature gradient so that most radiation is emitted away from the probe; not sure whether you can use dynamo flow to extract energy from the plasma in order to run heat pumps, because the plasma will radiate a lot of energy in direction of the probe),
3. show that limiting the power so that the sail has relatively low equilibrium temperature allows for enough transmission of momentum.
No 3 would be the simplest and most convincing answer.
I am not sure whether a plasma mirror is even thermo-dynamically possible. I am not sure whether sufficient heat-pumps plus radiators are "speculative engineering"-possible, if you have a contraption where your laser pushes against a shiny surface (necessitating very good focus of the laser). If you have a large solar sail (high surface, low mass) connected by tethers, then you probably cannot use active cooling on the sail; therefore there is limited room for fancy future-tech engineering, and we should be able to compute some limits now.
---------------------
Since I already started raising objections to your paper, I'll raise a second point: You compute the required launch mass from rocket-equation times final payload, with the final payload having very low weight. This assumes that you can actually build such a tiny rocket! While I am willing to suspend disbelieve and assume that a super efficient fusion-powered rocket of 500 tons might be built, I am more skeptical if your rocket, including fusion reactor but excluding fuel, is limited to 30 gram of weight.
Or did I miss something? While this would affect your argument, my heart is not really in it: Braking against the interstellar medium appears, to me, to circumvent a lot of problems.
---------------------
Because I forgot and you know your paper better than me: Do any implicit or explicit assumptions break if we lose access to most of the fuel mass for shielding during the long voyage?
If you could answer with a confident "no, our assumptions do not beak when cannot use the deceleration fuel as shielding", then we can really trade-off acceleration delta-v against deceleration delta-v, and I stay much more convinced about your greater point about the Fermi paradox.
This was a very fun article. Notably absent from the list, even though I would absolutely have expected it (since the focus was on evolutionary algorithms, even though many observations also apply to gradient-descent):
Driving genes. Biologically, a "driving gene" is one that cheats in (sexual) evolution, by ensuring that it is present in >50% of offspring, usually by weirdly interacting with the machinery that does meiosis.
In artificial evolution that uses "combination", "mutation" and "selection", these would be regions of parameter-space that are attracting under "combination"-dynamics, and use that to beat selection pressure.
If you assume that Dysoning and re-launch take 500 years, this barely changes the speed either, so you are very robust.
I'd be interested in more exploration of deceleration strategies. It seems obvious that braking against the interstellar medium (either dust or magnetic field) is viable to some large degree; at the very least if you are willing to eat a 10k year deceleration phase. I have taken a look at the two papers you linked in your bibliography, but would prefer a more systematic study. Important is: Do we know ways that are definitely not harder than building a dyson swarm, and is one galaxy's width (along smallest dimension) enough to decelerate? Or is the intergalactic medium dense enough for meaningful deceleration?
I would also be interested in a more systematic study of acceleration strategies. Your arguments absolutely rely on circumventing the rocket equation for acceleration; break this assumption, and your argument dies.
It does not appear obvious to me that this is possible: Say, coil guns would need a ridiculously long barrel and mass, or would be difficult to maneuver (you want to point the coil gun at all parts of the sky). Or, say, laser acceleration turns out to be very hard because of (1) lasers are fundamentally inefficient (high thermal losses), and cannot be made efficient if you want very tight beams and (2) cooling requirement for the probes during acceleration turn out to be unreasonable. [*]
I could imagine a world where you need to fall back to the rocket equation for a large part of the acceleration delta-v, even if you are a technologically mature superintelligence with dyson swarm. Your paper does not convince me that such a world is impossble (and it tries to convince me that hypothetical worlds are impossible, where it would be hard to rapidly colonize the entire universe if you have reasonably-general AI).
Obviously both points are running counter to each other: If braking against the interstellar medium allows you to get the delta-v for deceleration down to 0.05 c from, say 0.9 c, but acceleration turns out to be so hard that you need to get 0.8 c with rockets (you can only do 0.1c with coil guns / lasers, instead of 0.9 c), then we have not really changed the delta-v calculus; but we have significantly changed the amount of available matter for shielding during the voyage (we now need to burn most of the mass during acceleration instead of deceleration, which means that we are lighter during the long voyage).
[*] Superconductors can only support a limited amount of current / field-strength. This limits the acceleration. Hence, if you want larger delta-v, you need a longer barrel. How long, if you take the best known superconductors? At which fraction of your launch probe consisting of superconducting coils, instead of fusion fuel? Someone must do all these calculations, and then discuss how the resulting coil gun is still low-enough mass compared to the mass of a dyson swarm, and how to stabilize, power, cool and maneuver this gun. Otherwise, the argument is not convincing.
edit: If someone proposes a rigid barrel that is one light-hour long then I will call BS.
Computability does not express the same thing we mean with "explicit". The vague term "explicit" crystallizes an important concept, which is dependent on social and historical context that I tried to elucidate. It is useful to give a name to this concept, but you cannot really prove theorems about it (there should be no technical definition of "explicit").
That being said, computability is of course important, but slightly too counter-intuitive in practice. Say, you have two polynomial vectorfields. Are solutions (to the differential equation) computable? Sure. Can you say whether the two solutions, at time t=1 and starting in the origin, coincide? I think not. Equality of computable reals is not decidable after all (literally the halting problem).
It depends on context. Is the exponential explicit? For the last 200 years, the answer is "hell yeah". Exponential, logarithm and trigonometry (complex exponential) appear very often in life, and people can be expected to have a working knowledge of how to manipulate them. Expressing a solution in terms of exponentials is like meeting an old friend.
120 years ago, knowing elliptic integrals, their theory and how to manipulate them was considered basic knowledge that every working mathematician or engineer was expected to have. Back then, these were explicit / basic / closed form.
If you are writing a computer algebra system of similar ambition to maple / mathematica / wolfram alpha, then you better consider them explicit in your internal simplification routines, and write code for manipulating them. Otherwise, users will complain and send you feature requests. If you work as editor at the "Bronstein mathematical handbook", then the answer is yes for the longer versions of the book, and a very hard judgement call for shorter editions.
Today, elliptic integrals are not routinely taught anymore. It is a tiny minority of mathematicians that has working knowledge on these guys. Expressing a solution in terms of elliptic integrals is not like meeting an old friend, it is like meeting a stranger who was famous a century ago, a grainy photo of whom you might have once seen in an old book.
I personally would not consider the circumference of an ellipse "closed form". Just call it the "circumference of the ellipsis", or write it as an integral, depending on how to better make apparent which properties you want.
Of course this is a trade-off, how much time to spend developing an intuition and working knowledge of "general integrals" (likely from a functional analysis perspective, as an operator) and how much time to spend understanding specific special integrals. The specific will always be more effective and impart deeper knowledge when dealing with the specifics, but the general theory is more applicable and "geometric"; you might say that it extrapolates very well from the training set. Some specific special functions are worth it, eg exp/log, and some used to be considered worthy but are today not considered worthy, evidenced by revealed preference (what do people put into course syllabi).
So, in some sense you have a large edifice of "forgotten knowledge" in mathematics. This knowledge is archived, of course, but the unbroken master-apprentice chains of transmission have mostly died out. I think this is sad; we, as a society, should be rich enough to sponsor a handful of people to keep this alive, even if I'd say "good riddance" for removing it from the "standard canon".
Anecdote: Elliptic integrals sometimes appear in averaging: You have a differential equation (dynamical system) and want to average over fast oscillations in order to get an effective (ie leading order / approximate) system with reduced dimension and uniform time-scales. Now, what is your effective equation? You can express it as "the effective equation coming out of Theorem XYZ", or write it down as an integral, which makes apparent both the procedure encoded in Theorem XYZ and an integral expression that is helpful for intuition and calculations. And sometimes, if you type it into Wolfram alpha, it transforms into some extremely lenghty expression containing elliptic integrals. Do you gain understanding from this? I certainly don't, and decided not to use the explicit expressions when I met them in my research (99% of the time, mathematica is not helpful; the 1% pays for the trivial inconvenience of always trying whether there maybe is some amazing transformation that simplifies your problem).
Regarding insolubility of the quintic, I made a top level post with essentially the same point, because it deserves to be common knowledge, in full generality.
I guess that this is due to the historical fact that candidates in the US are supposed to be district-local, not state-local, and districts are supposed to be as small as possible. I'm not an American, so I cannot say how strong this is as a constraint for modified electoral systems.
If you had a small party/faction, with say 10% of popular vote, reaching up to maybe 30% in their strongest districts, then I would definitely see a problem: Such a party simply does not fit purely district-local representation (one-to-one mapping between districts and representatives). Think e.g. an environmentalist party.
If your representative chamber is more about opposing ideologies and national governance than about opposing local interests, then why not ditch this one-to-one mapping?
I mean, this works even in the EU parliament, and you can't tell me that opposing local interests across US districts are harder than across states within EU countries? And you have a second chamber (the senate) that is explicitly about conflicting local interests.
I presume that you know how German federal elections, or European parliament elections are run; easy to google, far from perfect, but gets at least this right. I would definitely be opposed to a change to PLACE at home, for these reasons, but agree that PLACE beats FPTP by lengths.
And something that PLACE gets extremely right is to involve the general populace in the within-party selection. In Germany, only party members are allowed to vote in primaries (and members pay contributions and are expected to be activist, and parties can expel or reject prospective members). This slightly sucks; "too radical" candidates / factions are routinely squashed by the high party functionaries.
Of course this gives you an amusing game-theoretical problem in e.g. EU parliament elections: If countries were entirely free to select their voting system, they would have incentive to move to a winner-takes-all system, which strengthens the representation of their national interest. Same way you cannot get rid of your accursed winner-takes-all within states in US presidential elections, as long as states are free to design their voting systems. And, on the same lines, if you must have a winner-takes-all system, then each state should have votes roughly based on square root of population (I am yet again ashamed to be German for the way we treated the Polish, this time after they proposed the square-root for the EU council).
Re PLACE: Interesting proposal. Have you considered the following problem (I'd guess you have; a link would be appreciated):
Candidates are not exchangeable. Candidate A has done a very good job in the legislature. An opposing faction may decide to coordinate to support his local opposing candidate B, in order to keep person A out of parliament.
Or, in other words: Two candidates running in the same district cannot both become part of parliament. This opens a huge amount of gaming, in order to squash small parties / factions that do not have a deep bench of good candidates. Party A and its voters have large influence on the composition of Party B's parliamentary faction, and can strategically plan this.
The standard solution, e.g. in German federal elections, is to have pools: Candidates can be elected locally (in-district) or statewide. Candidates who won statewide are expected to still try to represent their district (where they lost), if they ran locally at all.
One guess for cheap signaling would be to seed stellar atmospheres with stuff that should not belong. Stellar spectra are really good to measure, and very low concentration of are visible (create a spectral line). If you own the galaxy, you can do this at sufficiently many stars to create a spectral line that should not belong. If we observed a galaxy with "impossible" spectrum, we would not immediately know that it's aliens; but we would sure point everything we have at it. And spectral data is routinely collected.
I am not an astronomer, though. So this is not meant as an answer, but rather as a starting point for others to do more literature research. I think I have seen this discussed somewhere, using technetium; but googling revealed that stars with technetium actually exist!
I think communicating without essentially conquering the Hubble volume is still an interesting question. I would not rule out a future human ethical system that restricts expansion to some limited volume, but does not restrict this kind of omnidirectional communication. Aliens being alien, we should not rule out them having such a value system either.
That being said, your article was really nice. Send multiplying probes everywhere, watch the solar system form and wait for humans to evolve in order to say "hi" is likely to be amazingly cheap.
Re SODA: The setup appears to actively encourage candidates to commit to a preference order. Naively, I would prefer a modification along the following lines; could you comment?
(1) Candidates may make promises about their preference order among other candidates; but this is not enforced (just like ordinary pre-election promises). (2) The elimination phase runs over several weeks. In this time, candidates may choose to drop out and redistribute their delegated votes. But mainly, the expected drop-outs will negotiate with expected survivors, in order to get at least some of their policies implemented (with the same kind of enforcement as regular pre-election promises). Hence, this phase is "coalition building".
An interesting final phase (3), in order to encourage compromise / beneficial trade would be something like: Once we are down to candidates with > 25% approval, we randomize the result. The probability of a candidate to win could be something like the square, or third power, of his approval. The threshold of 25% is in order to prevent complete crackpots from winning the election, and might need to be even more brutal. The randomization serves to allow the two remaining highest approval platforms to negotiate a compromise, weighted by their chance of winning the final lottery. In practice, one would hope that the randomization is never applied: That is, the highest candidate makes enough concessions in order to make the second candidate agree to drop out.
This way, we preserve minority rights, and get a pretty good outcome if the candidates are actually reasonable and trustworthy parties (instead of persons) that are capable of committing to modify the values they will espouse once in power.
Obvious disadvantages are (1) negotiation skill and integrity become essential (integrity in the christiano sense: You need to be predictable in which promises you will break under which contingencies), because the winner is decided though coalition building, rather than voting; and (2) some voters might object to shady backroom deals being the explicit procedure, and (3) randomization may make losers very pissed if the dice are ever rolled (which hopefully is never, because of positive-sum trades).
You can obviously combine this with a 3-2-1 system (where only candidates reaching at least 50% "OK" votes are eligible, and otherwise "Good" votes are counted; if no candidate receives 50% OK, then the election is repeated, under the assumption that Belgium is better than Trump, i.e. better a constitutional crisis and government by interim civil servants than government by crazy elected officials).
edit: I forgot to mention the other big advantage of the probabilistic phase (3): Enforcing continuity, i.e. preventing small factions from gaining oversized influence by playing kingmaker.
Regarding measurement of pain:suffering ratio
A possible approach would be to use self-reports (the thing that doctor's always ask about, pain scale 1-10) vs revealed preferences (how much painkillers were requested? What trade-offs for pain relief do patients choose?).
Obviously this kind of relation is flawed on several levels: Reported pain scale depends a lot on personal experience (very painful events permanently change the scale, ala "I am in so much pain that I cannot walk or concentrate, but compared to my worst experience... let's say 3?"). Revealed preferences depend a lot on how much people care about the alternatives (e.g. if people have bad health insurance or really important stuff to do they might accept a lot of subjective suffering in order to get out of hospital one day early). Likewise, time preference might enter a lot into revealed preference.
Despite these shortcomings, that's where I would start thinking about what such a ratio would mean. If one actually did a study with new questionaires, one should definitely ask patients for some examples in order to gauge their personal pain-scale, and combine actual revealed preferences with answers to hypothetical questions "how much money would pain relief be worth to you? How much risk of death? How many days of early hospital release? etc", even if the offer is not actually on the table.
>But the greatest merit of Occamian prior is that it vaguely resembles the Lazy prior.
...
>With that in mind, I asked what prior would serve this purpose even better and arrived at Lazy prior. The idea of encoding these considerations in a prior may seem like an error of some kind, but the choice of a prior is subjective by definition, so it should be fine.
Encoding convenience * probability into some kind of pseudo-prior such that the expected-utility maximizer is the maximum likelihood model with respect to the pseudo-prior does seem like a really useful computational trick, and you are right that terminology should reflect this. And you are right that the Occam prior has the nice property that weight-by-bit-count is often close to convenience, and hence makes the wrong naive approach somewhat acceptable in practice: That is, just taking the max likelihood model with respect to bit-count should often be a good approx for weight-by-bitcount * convenience (which is the same as weight-by-bitcount for probability and maximize expected utility).
In cases where we know the utility we can regenerate probabilities afterwards. So I would now be really interested in some informal study of how well Occam actually performs in practice, after controlling for utility: You are right that the empirical success of Occam might be only due to the implicit inclusion of convenience (succinct-by-bit-count models are often convenient) when doing the (wrong!) max-likelihood inference. I had not considered this, so thanks also for your post; we both learned something today.
I'd also remark/reiterate the point in favor of the Lazy prior: The really terrible parts of working with Occam (short descriptions that are hard to reason about, aka halting problem) get cancelled out in the utility maximization anyway. Lazy avoids invoking the halting-problem oracle in your basement for computing these terms (where we have the main differences between Occam vs Lazy). So you are right after all: Outside of theoretical discussion we should all stop using probabilities and Occam and switch to some kind of Lazy pseudo-prior. Thanks!
That being said, we all appear to agree that Occam is quite nice as an abstract tool, even if somewhat naive in practice.
A different point in favor of Occam is "political objectivity": It is hard to fudge in motivated reasoning. Just like the "naive frequentist" viewpoint sometimes wins over Bayes with respect to avoiding politically charged discussions of priors, Occam defends against "witchcraft appears natural to my mind, and the historical record suggests that humans have evolved hardware acceleration for reasoning about witchcraft; so, considering Lazy-prior, we conclude that witches did it" (Occam + utility maximization rather suggests the more palatable formulation "hence it is useful to frame these natural processes in terms of Moloch, Azatoth and Cthulhu battling it out", which ends up with the same intuitions and models but imho better mental hygiene)
I have a feeling that you mix probability and decision theory. Given some observations, there are two separate questions when considering possible explanations / models:
1. What probability to assign to each model?
2. Which model to use?
Now, our toy-model of perfect rationality would use some prior, e.g. the bit-counting universal/kolmogorov/occam one, and bayesian update to answer (1), i.e. compute the posterior distribution. Then, it would weight these models by "convenience of working with them", which goes into our expected utility maximization for answering (2), since we only have finite computational resources after all. In many cases we will be willing to work with known wrong-but-pretty-good models like Newtonian gravity, just because they are so much more convenient and good enough.
I have a feeling that you correctly intuit that convenience should enter the question which model to adopt, but misattribute this into the probability-- but which model to adopt should formally be bayesian update + utility maximization (taking convenience and bounded computational resources into account), and definitely not "Bayesian update only", which leads you to the (imho questionable) conclusion that the universal / kolmogorov / occam prior is flawed for computing probability.
On the other hand, you are right that the above toy model of perfect rationality is computationally bad: Computing the posterior distribution after some prior and then weighting by utility/convenience is of stupid if directly computing prior * convenience is cheaper than computing prior and convenience separately and then multiplying. More generally, probability is a nice concept for human minds to reason about reasoning, but we ultimately care about decision theory only.
Always combining probability and utility might be a more correct model, but it is often conceptually more complex to my mind, which is why I don't try to always adopt it ;)
I think part of the assumption is that reflection can be bolted on trivially if the pattern matching is good enough. For example, consider guiding an SMT / automatic theorem prover by deep-learned heuristics, e.g. (https://arxiv.org/abs/1701.06972)[https://arxiv.org/abs/1701.06972] . We know how to express reflection in formal languages; we know how to train intuition for fuzzy stuff; me might learn how to train intuition for formal languages.
This is still borderline useless; but there is no reason, a priori, that such approached are doomed to fail. Especially since labels for training data are trivial (check the proof for correctness) and machine-discovered theorems / proofs can be added to the corpus.
I strongly disagree that anthropics explains the unreasonable effectiveness of mathematics.
You can argue that a world, where people develop a mind and mathematical culture like ours (with its notion of "modular simplicity") should be a world where mathematics is effective in everyday phenomena like throwing a spear.
This tells us nothing about what happens if we extrapolate to scales that are not relevant to everyday phenomena.
For example, physics appears to have very simple (to our mind) equations and principles, even at scales that were irrelevant during our evolution. The same kind of thought-processes are useful both for throwing spears / shooting cannons and for describing atoms. This is the unreasonable effectiveness of mathematics.
On the other hand, there are many phenomena where mathematics is not unreasonably effectice; take biological systems. There, our brains / culture have evolved heuristics that are useful on a human scale, but are entirely bogus on a micro-scale or macro-scale. Our mathematics is also really bad at describing the small scales; reductionism is just not that useful for understanding, say, how a genome defines an organism, and our brains / culture are not adapted to understanding this.
I think a counterfactual world, where physics outside the scales of human experience were as incomprehensibly complex (to our minds) as biology outside human scales does sound realistic. It does seem like a remarkable and non-trivial observation that the dynamics of a galaxy, or the properties of a semiconductor, are easy to understand for a culture that learned how a cannonball flies; whereas learning how to cultivate wheat or sheep is not that helpful for understanding cancer.
[Meta: Even low-effort engagement, like "known + keyword" or "you misunderstood everything; read <link>" or "go on talking / thinking" is highly appreciated. Stacks grow from the bottom to the top today, unlike x86 or threads on the internet]
------------
Iterative amplification schemes work by having each version trained by previous iteration ; and, whenever version fails at finding a good answer (low confidence in the prediction), punting the question to , until it reaches the human overseer at , which is the ground truth for our purposes. At the same time, the distribution of relevant question widens at each iteration, as capabilities improve: For example, a very stupid clippy will need to deal with simple linear optimization problems of its supply chain; a very smart clippy will need to figure out a proper plan for world-domination.
Now, in order to correctly punt questions down the call-stack, each level when faced with a problem must decide whether it can solve it (providing training data for level ), or whether it must punt downwards. More precisely, level must recognize whether is inside the distribution it was trained with (resp. , the distribution of problems it can generate / handle post-training).
False positives (underconfidence) lead to increased load on the human overseers / capability loss. False negatives (overconfidence) lead to safety failures: If they happen during training, then the machine learns wrong values / ethics / whatever-it-should-learn, if they happen in production then they lead to bad actions (I am assuming that no one would run the AI unsupervised, but supervision will, by necessity, be pretty light).
This is a problem, even with a very shallow call-stack (few iterations), but it gets (possibly exponentially!) worse with deeper stacks: After all, if even a single false-negative mistake occurs along the chain, we have a safety incident; naively, the chance of mistakes grows exponentially in the depth, for fixed false-negative rate at each level. In other words, we would need to ensure that the infinite series converges, where is the false-negative rate at level .
Something that may makes this problem either simpler or harder is the fact that, once a question has been bumped down a couple of times, it can be very far out-of-distribution. This makes it much easier to get mediocre reliability (the questions bumped down from very high levels are very weird for the low levels), but, at the same time, the low levels are really, really not equipped for dealing with these weird questions. In the clippy example, Paul's scheme would totally allow the high capability clippy to convince low levels into accepting its proposal by exploiting its weaknesses (adversarial examples).
In different words: The entire scheme needs really good anomaly detection. Anomaly detection is a machine-learning problem, right? So learn this along the way?
The problem is that, while training the anomaly detection at level , we cannot sample from the distribution , because we simply don't know it yet. As we run amplification, we extend both the capability of answering questions and the range of questions that come up.
Note that this would not be a problem if the extensions of supports of the distributions were very mild, compared to the larger domain where the anomaly detection of level is reliable-by-training. This appears like a very dangerous assumption on the "mind-space" accessible to our training procedure, for the "problem-space" at hand and the specific training procedure. In other words: No universality of safety amplification, if it relies on this assumption. Please link me to relevant discussions if this assumption has been justified anywhere.
A different approach would be to not run the scheme in a hierarchical way; instead try to feed it into itself. This reintroduces the problem of "drift", i.e. instability: The system will acquire a specific culture, and we can only pray (i.e. must assume / ensure) that desireable states are an attractor, for the specific training procedure. I fear that this is extra-hard to check, because meta-stable systems look pretty stable, until you increase capability: Bad scaling.
A third approach to solving this problem would be to create, separately, a reliable anomaly detector. This may very well be an AI complete problem, but it may be simpler than alignment! Further, robust anomaly detection (recognizing that data points are outside of the training distribution) might be an avenue for differential progress that matters more for safety/alignment than capabilities.
Do you know other approaches for this problem?
------------
When is this harmless?
Well, this is entirely harmless if our problem domain contains only simple questions, where verification of ground truth is dirt cheap. For example, NP questions (like, e.g., all of mathematics! Proof verification is trivial after all). This observation should give us a bad feeling, because an AI that is really good at mathematics / programming and bad at everything else is one of the most dangerous situation we could imagine (magical function optimizers, aka mindless AIXI genies, being the only more hopeless case I can think of). On the other hand, typical NP questions don't scale down: It is currently entirely infeasible to use machine learning for theorem proving, simply because useful transformations are exponentially rare in the space of possible ones (I am aware of some papers using the mizar library; while the neural net + theorem prover beat the unaided prover, I was less than impressed by the results).
For problem domains that feel more like exptime, this is more likely to be a problem: Say, training to play games like Go. Then, we can play against our ancestors in order to judge performance, and gain access to some kind of ground truth. Unfortunately, (1) strength is not linearly ordered: You clearly can have situations where A beats B beats C beats A, and (2) if we wanted to optimize "strength against perfect play", aka min-max, then we don't have access to a perfect opponent during training. Afaik it is usual for training-through-amplification of Game AI to develop "fads", i.e. cheesy tactics, on the way; sometimes, these recur cyclically. This is also observed for the metagame in many multiplayer videogames. I have a feeling that the Go successes tell us a lot about how MCTS is amazingly stable against cheesy tactics; and who knows how much tweaking deepmind had to do until they got the amplification stable.
Now, safety amplification / value learning has a much, much harder problem: The ground truth is only accessible through examples / very expensive oracle queries (which might be fundamentally unsafe, at very high levels of capability: Don't let human operators talk to unaligned too-clever AI).
------------
Post-script: Writing this down in clear words made me slightly update against Paul's amplification schemes eventually growing into a solution. I still think that Paul's line of research is damn cool and promising, so I'm more playing devil's advocate here. The possible differential gain for capability in NP problems versus harder-than-NP alignment for this kind of amplification procedure made me slightly more pessimistic about our prospects in general. Moreover, it makes me rather skeptic whether amplification is a net win for safety / alignment in the differential progress view. I want to look more into anomaly detection now, for fun, my own short-term profit and long-term safety.
(1) As Paul noted, the question of the exponent alpha is just the question of diminishing returns vs returns-to-scale.
Especially if you believe that the rate is a product of multiple terms (like e.g. Paul's suggestion with one exponent for computer tech advances and another for algorithmic advances) then you get returns-to-scale type dynamics (over certain regimes, i.e. until all fruit are picked) with finite-time blow-up.
(2) Also, an imho crucial aspect is the separation of time-scales between human-driven research and computation done by machines (transistors are faster than neurons and buying more hardware scales better than training a new person up to the bleeding edge of research, especially considering Scott's amusing parable of the alchemists).
Let's add a little flourish to your model: You had the rate of research and the cumulative research ; let's give a name to the capability of the AI system. Then, we can model . This is your model, just splitting terms into , which tells us how hard AI progress is, and which tells us how good we are at producing research.
Now denote by the fraction of work that absolutely has to be done by humans, and by the speed-up factor for silicon over biology. Amdahl's law gives you , or somewhat simplified . This predicts a rate of progress that first looks like , as long as human researcher input is the limiting factor, then becomes when we have AIs designing AIs (recursive self-improvement, aka explosion), and then probably saturates at something (when the AI approaches optimality).
The crucial argument for fast take-off (as far as I understood it) is that we can expect to hit at some cross-over , and we can expect this to happen with a nonzero derivative . This is just the claim that human-level AI is possible, and that the intelligence of the human parts of the AI research project is not sitting at a magical point (aka: this is generic, you would need to fine-tune your model to get something else).
The change of the rate of research output from the regime to the regime sure looks like a hard-take-off singularity to me! And I would like to note that the function , i.e. the hardness AI research and the diminishing-returns vs returns-to-scale debate does not enter this discussion at any point.
In other words: If you model AI research as done by a team of humans and proto-AIs assisting the humans; and if you assert non-fungibility of humans vs proto-AI-assistents (even if you buy a thousand times more hardware, you still need the generally intelligent human researchers for some parts); and if you assert that better proto-AI-assistents can do a larger proportion of the work (at all); and if you assert that computers are faster than humans; then you get a possibly quite wild change at .
I'd like to note that the cross-over is not "human-level AI", but rather "" , i.e. an AI that needs (almost) no human assistence to progress the field of AI research.
On the opposing side (that's what Robin Hanson would probably say) you have the empirical argument that should decay like a power-law long before we ("the last 10% take 90% of the work" is a folk formulation for "percentile 90-99 take nine time as much work as percentile 0-89" aka power law, and is borne out quite well, empirically).
This does not have any impact on whether we cross with non-vanishing derivative, but would support Paul's view that the world will be unrecognizably crazy long before .
PS. I am currently agnostic about the hard vs soft take-off debate. Yeah, I know, cowardly cop-out.
edit: In the above, C kinda encodes how fast / good our AI is and q encodes how general it is compared to humans. All AI singularity stuff tacitly assumes that human intelligence (assisted by stupid proto-AI) is sufficiently general to design an AI that exceeds or matches the generality of human intelligence. I consider this likely. The counterfactual world would have our AI capabilities saturate at some subhuman level for a long time, using terribly bad randomized/evolutionary algorithms, until it either stumbles unto an AI design that has better generality or we suffer unrelated extinction/heat-death. I consider it likely that human intelligence (assisted by proto-AI) is sufficiently general for a take-off. Heat-death is not an exaggeration: Algorithms with exponentially bad run-time are effectively useless.
Conversely, I consider it very well possible that human intelligence is insufficiently general to understand how human intelligence works! (we are really, really bad at understanding evolution/gradient-descent optimized anything, an that's what we are)
Just commenting that the progress to thermonuclear weapons represented another discontinuous jump (1-3 orders of magnitude).
Also, whether von Neumann was right depends on the probability for the cold war ending peacefully. If we retrospectively conclude that we had a 90% chance of total thermonuclear war (and just got very lucky in real life) then he was definitely right. If we instead argue from the observed outcome (or historical studies conclude that the eventual outcome was not due to luck but rather due to the inescapable logic of MAD), then he was totally nuts.
Near-misses are not necessarily a very strong guide to estimating retrospective risk. Both sides were incentivized to hide their fail-safes for escalation; to credibly commit to having a twitchy retaliation finger, and at the same time to not actually retaliate (the game is first chicken, then ultimatum, and never prisoner's dilemma). So I would be very wary of trusting the historical record on "what if Petrov had not kept a cool mind".
Not sure. I encountered this once in my research, but the preprint is not out yet (alas, I'm pretty sure that this will still be not enough to reach commercial viability, so pretty niche and academic and not a very strong example).
Regarding "this is not common": Of course not for problems many people care about. Once you are in the almost-optimal class, there are no more giant-sized fruit to pick, so most problems will experience that large jumps never, once or twice over all of expected human history (sorting is even if you are a super-intelligence) (pulling the numbers 0,1,2 out of my ass; feel free to do better). On the other hand, there is a long tail of problems very few people care about (e.g. because we have no fast solution and hence cannot incorporate a solver into a bigger construction). There, complexity-class jumps do not appear to be so uncommon.
Cheap prediction: Many visual machine-learning algos will gain in complexity class once they can handle compressed data (under the usual assumption that, after ordering by magnitude, coefficients will decay like a power-law for suitable wavelet-transforms of real-world input data; the "compression" introduces a cut-off / discretization, and a cool ML algo would only look at the coefficients it cares about and hence be able to e.g. output a classification in finite time for infinite input data). Also cheap prediction: This will turn out to be not as hot as it sounds, since GPUs (and to a lesser extent CPUs) are just too damn good at dealing with dense matrices and FFT is just too damn fast.
-----
Since I'm bad at computer history, some maybe-examples that spring to mind:
Probable example: Asymmetric crypto.
Very-maybe examples:
Fast fourier transform (except if you want to attribute it to Gauss). Linear programming (simplex algorithm). Multi-grid methods for PDE / the general shift from matrix-based direct solvers to iterative operator-based solvers. Probabilistic polynomial identity testing. Bitcoin (not the currency, rather the solution to trust-less sybil-safe global consensus that made the currency viable). Maybe Zk-snark (to my eternal shame I must admit that I don't understand the details of how they work).
For large economic impact, possibly discontinuous: MP3 / the moment where good audio compression suddenly became viable and made many applications viable. I think this was more of an engineering advance (fast software decoding, with large hardware overhang on general-purpose CPUs), but others here probably know more about the history.
Everyone, feel free to list other historic examples of kinda-discontinuous algorithmic advances and use your superior historical knowledge to tear down my probably very bad examples.
-----
Separate point: Many processes require "critical mass" (which I called "viability"), and such processes amplify discontinuities / should be modeled as discontinuities.
Physics / maths intuition pump would be phase transitions or bifurcations in dynamical systems; e.g. falling off a saddle-node looks quite discontinuous if you have a separation of time-scales, even if it is not discontinuous once you zoom in far enough. Recursive self-improvement / general learning does have some superficial resemblance to such processes, and does have a separation of time-scales. Tell me if you want me to talk about the analogies more, but I was under the impression that they have been discussed ad-nauseam.
-----
I'll continue to think about historic examples of complexity-class-like algorithmic advancements with economic impact and post if anything substantial comes to mind.
I imagine the "secret sauce" line of thinking as "we are solving certain problems in the wrong complexity class". Changing complexity class of an algorithm introduces a discontinuity; when near a take-off, then this discontinuity can get amplified into a fast take-off. The take-off can be especially fast if the compute hardware is already sufficient at the time of the break-through.
In other words: In order to expect a fast take-off, you only need to assume that the last crucial sub-problem for recursive self-improvement / explosion is done in the wrong complexity class prior to the discovery of a good algorithm.
For strong historical precedents, I would look for algorithmic advances that improved empirical average complexity class, and at the same time got a speed-up of e.g. 100 x on problem instances that were typical prior to the algorithmic discovery (so Strassen matrix-multiply is out).
For weaker historical precedent, I would look for advances that single-handedly made the entire field viable -- that is, prior to the advance one had a tiny community caring about the problem; post the advance, the field (e.g. type of data analysis) became viable at all (hence, very limited commercial / academic interest in the subfield prior to its breakthrough). I think that this is meaningful precedent because people optimize for expected pay-off, and it is sometimes surprising that some small-but-crucial-if-possible subproblem can be solved at all (reasonable quickly)!
And I do believe that there are many parts of modern ML that are in the wrong complexity class (this does not mean that I could do better, nor that I necessarily expect an improvement or even discontinuous jump in usefulness).
Thanks, and sorry for presumably messing up the formatting.
The assumption I'm talking about is that the state of the rest of the universe (or multiverse) does not affect the marginal utility of there also being someone having certain experiences at some location in the uni-/multi-verse.
Now, I am not a friend of probabilities / utilities separately; instead, consider your decision function.
Linearity means that your decisions are independent of observations of far parts of the universe. In other words, you have one system over which your agent optimizes expected utility; and now compare it to the situation where you have two systems. Your utility function is linear iff you can make decisions locally, that is, without considering the state of the other system.
Clearly, almost nothing has a linear decision / utility function.
I think people mistake the following (amazing) heuristic for linear utility: If there are very many local systems, and you have a sufficiently smooth utility and probability distribution for all of them, then you can do mean-field: You don't need to look, the law of large numbers guarantees strong bounds. In this sense, you don't need to couple all the systems, they just all couple to the mean-field.
To be more practical: Someone might claim to have almost linear (altruistic) utility for QALYs over the 5 years (so time-discounting is irrelevant). Equivalently, whether some war in the middle east is terrible or not does not influence his/her malaria-focused charity work (say, he/she only decides on this specific topic).
Awesome, he/she does not need to read the news! And this is true to some extent, but becomes bullshit at the tails of the distribution. (the news become relevant if e.g. the nukes fly, because they bring you into a more nonlinear regime for utility; on the other hand, given an almost fixed background population, log-utility and linear-utility are indistinguishable by Taylor's rule)
Re pascal's muggle: Obviously your chance of getting a stroke and hallucinating weird stuff outweighs your chance of witnessing magic. I think that it is quite clear that you can forget about the marginal cost of giving 5 bucks to an imaginary mugger before the ambulance arrives to maybe save you; decision-theoretically, you win by precommitting to pay the mugger and call an ambulance once you observe something sufficiently weird.
Real-world anectdata how one big company (medical equipment) got OK at security:
At some time they decided that security was more important now. Their in-house guy (dev->dev management -> "congrats, you are now our chief security guy") got to hire more consultants for their projects, went to trainings and, crucially, went to cons (e.g. defcon). He was a pretty nice guy, and after some years he became fluent at hacker-culture. In short, he became capable of judging consultant's work and hiring real security people. And he made some friends on the way. I think this is the best path to aquire institutional knowledge: Take a smart person loyal to the company, immerse them into the knowledgable subculture (spending money on failures on the way), use the aquired knowledge to hire real professionals (really hire, or hire as consultants for projects, or hire to give trainings).
Different big company (not software related), same thing. After some years their security guys became fed up with their lack of internal political capital, quit and switched career to "real" security.
Yep. The counter-example would be Apple iOS.
I never expected it to become as secure as it did. And Apple security are clowns (institutionally, no offense inteded for the good people working there), and UI tends to beat security in tradeoffs.
Everything exposed to an attacker, and everything those subsystems interact with, and everything those parts interact with! You have to build all of it robustly!
seems false to me, if you have good isolation--which is what a project like Qubes tries to accomplish.
I agree with you here that Qubes is cool; but the fact that it is (performantly) possible was not obvious before it was cooked up. I certainly failed to come up with the idea of Qubes before hearing it (even after bluepill), and I am not ashamed of this: Qubes is brilliant (and IOMMU is cheating).
Also, in some sense Qubes is doing exactly what Carol says. Qubes only has a chance of working because the fundamental design for hardware-assisted security-by-isolation trumps all other considerations in their trade-offs. The UI is fundamentally constrained (to prevent window-redressing), as is performance (3d accelleration) and ease-of-use. All these constraints were known and documented before even a single line of code was written (afaik). Qubes can only work because it has security as one of its main goals, and has brilliant security people as project leads with infinite internal political capital.
That said, going on a tangent about qubes:
I really want to see painless live-migration of Qubes (migrate an application between different hosts, without interupting -- say, from a lousy netbook to a fat workstation and back), this would be a killer feature for non-security-nerds. Unfortunately xen cannot do x86 <-> arm (qemu?); live-migration
smartphone<->workstation would be awesome (just bring my smartphone, plug it in as a boot-drive and continue your work on a fat machine -- secure as long as there is no hardware implant).
Re Qubes security: You still have the bad problem of timing-sidechannels which cross VM borders; you should view Qubes as an awesome mitigation, not a solution (not to speak of the not-so-rare xen outbreaks), and you still need to secure your software. That is, Qubes attempts to prevent privelege escalation, not code exec; if the vulnerability is in the application which handles your valuable data, then Qubes cannot help you.
edit timeout over, but the flags for requesting a chain-of-trust from your recursive resolver/ cache should of course by (+CD +AD +RD).
*shrugs*
Yeah, ordinary paranoia requires that you have unbound listening on localhost for your DNS needs. Because there should be a mode to ask my ISP-run recursive resolver to deliver the entire cert-chain. Thisis a big fail of DNSSEC (my favorite would be -CD +AD +RD, this flag combination should still be free and means "please recurse; please use dnssec; please don't check key validity").
Yes, and DNSSEC over UDP breaks in some networks, then you need to run it via TCP (or do big a big debugging-session in order to figure out what broke).
And sure, DNSSEC implementations can have bugs, the protocol can have minor glitches, many servers suck at choosing good keys or doing key-rollover correctly.
But: Compared to X.509 browser CA and compared to DNS with transport-layer security (like DJB's horrible dnscurve), this is a sound global PKI and is "realexistierend". And it has so many good features -- for example, the keys to your kingdom reside on airgapped smartcard (offline-signing), instead of waiting in RAM for the next heartbleed (openssl is famous for its easy-to-verify beautiful bug-free code, after all) or sidechannel (hah! virtualization for the win! Do you believe that openssl manages to sign a message without leaking bits via L1cache access patterns? If you are on AWS then the enemy shares the same physical machine, after all) or ordinary break-in.
If you are interested I can write a long text why transport-layer security for DNS misses the fucking point (hint: how do you recover from a poisoned cache? What is the guaranteed time-of-exposure after Suzy ran away with your authorative server? Did these people even think about these questions??!!1 How could DJB of all people brain-fart this so hard?).
Hey,
fun that you now post about security. So, I used to work as itsec consultant/reasearcher for some time; let me give my obligatory 2 cents.
On the level of platitudes: my personal view of security mindset is to zero in on the failure modes and tradeoffs that are made. If you additionally have a good intuition on what's impossible, then you quickly discover either failure modes that were not known to the original designer -- or, also quite frequently, the system is broken even before you look at it ("and our system archieves this kind of security, and users are supposed to use it that way" -- "lol, you're fucked"). The following is based on aesthetics/ intuition, and all the attack scenarios are post-hoc rationalizations (still true).
So, now let's talk passwords. The standard procedure for implementing password-login on the internet is horrible. Absolutely horrible. Let me explain first why it is horrible on the object level, second some ways to do better (as an existence proof), and third why this hasn't gotten fixed yet, in the spirit of inadequacy analysis.
First: So, your standard approach is the following: Server stores salted and hashed password for each user (using a good key-derivation function ala scrypt, not a hash, you are not stupid). User wants to login, you serve him the password-prompt html, he enters password, his browser sends password via some POST form, you compare, either accept or reject user. Obviously you are using https everywhere.
Failure modes: (1) User has chosen extremely bad password. Well, you obviously rate-limit login attempts. Otherwise, there is no defense against stupid (you maybe check against john-the-ripper's top N passwords on creation (bloom filter lookup, hit SSD once). Users can often get away with 30 bit of entroy, because attackers can only use rate-limited online attacks.
(2) Someone steals your password database. Does not allow to log into your server, does not allow to log into other servers if user stupidly reused password. Nice! However, attacker can now do offline attempts at the passwords -- hence, anyone below 45 bits is fucked, and anyone below 80 bits should feel uncomfortable. This is however unpreventable.
(3) Someone MitMs you (user sits at starbucks). You are using SSL, right? And Eve did not get a certificate from some stupid CA in the standard browser CA list? Standard browser X.509 PKI is broken by design; any Certificate Authority on this list, and any governent housing one of the CAs on this list get a free peek at your SSL connection. Yeah, the world sucks, live with it (PS: DNSSEC is sound, but no one uses it for PKI-- because, face it, the world sucks)
(4) Your user gets phished. This does not necessarily mean that the user is stupid; there are lots of ways non-stupid users can get phished (say- some other tab changed the tab-location while the user was not watching, and he/she did not check the address bar again).
(5) You got XSSed. Password gone.
(6) Your user catched a keylogger. Game over.
(7) Your SSL termination point leaks memory (e.g. heartbleed, cloudfare cache bug, etc).
Ok, what could we possibly fix? Terrible password is always game over, semi-bad password + database stolen is always game over , keylogger is
always game over -- try two-factor for all of those.
The others?
First, there is no reason for the user to transmit the password in the SSL-encrypted plain. For example, you could send (nonce, salt) to the user, the user computes hash(nonce, scrypt(password, salt)) and sends this. Compare, be happy. Now, if eve reads the http plaintext she still does not know the password and also knows no valid login-token (that's why the nonce was needed!). This has one giant advantage: You cannot trust yourself to store the password on disk; you should not trust yourself to store it in RAM. There are a thousand things that can go wrong. Oh, btw, Ben could implement this now on lesserwrong!
But this system still stinks. Fundamentally: (1) you should not trust SSL to correctly verify your site to the user (mitm/bad CA), (2) you should not rely on the combination of ill-trained lusers and badly designed browser UI, and (3) you should not trust yourself to render a password box to the user, because your fancy secure development lifecycle-process will not prevent Joe WebDev from dropping XSS into your codebase.
But, we have assets: User really needs only to prove posession of some secret; server knows some other derived secret; we can make it so that phishing becomes mostly harmless.
How? First, the browser needs a magic special API for rendering password forms. This must be visually distinct from web forms, and inaccessible to JS. Second, we need a protocol. Let me design one over a glass of whisky.
Server has a really, really good way of authenticating itself: It knows a fucking hash of the users password. So, an easy way would be: server stores scrypt("SMP",username, domain, password). When user enters his password, his user agent (browser) regenerates this shared secret, and then both parties play their favorite zero-knowledge mutual authentication (no server-generated salt: username + domain is enough) -- e.g. socialist millionare [1]. Hence, the user cannot get phished. He can enter all his passwords for all his accounts anywhere in the net, as long as he sticks to the magic browser form.
Now, this still stinks: If Suzy Sysadmin decides to take ownership of your server's password database, then she can impersonate any user. Meh, use public key crypto: scrypt("curvexyz",username, domain, password) as the seed to generate a key-pair in your favorite curve. Server stores the public key, user-agent regenerates the private key, proves posession -- AFTER the server has authenticated itself. Now Suzy fails against high-entropy passwords (which resist offline cracking). And when user logs in to Suzy's rogue impersonation server, then he will still not leak valid credentials.
In other words, password authentication is cryptographically solved.
Now, something really depressing: No one is using a proper solution. First I wanted to tell a tale how browser-vendors and websites will fail forever to coordinate on one, especially since both webdevs (enemies of the people) want to render beautiful password forms, and the CA industry wants to live too (middlemen are never happy to be cut, both economical and cyptographic ones). But then I saw that openssh uses brain-dead password auth only. God, this depresses me, I need more whisky.
Small number of links:
[1] https://en.wikipedia.org/wiki/Socialist_millionaires
[2] https://en.wikipedia.org/wiki/Password-authenticated_key_agreement -- our goals have a name, we want "augmented PAKE"!
[3] https://tools.ietf.org/html/rfc8236 -- how a more sober protocol looks like.
[4] https://crypto.stackexchange.com/questions/25338/why-arent-zero-knowledge-proofs-used-for-authentication-in-practice