Book review: The Sleepwalkers

2019-04-23T00:10:00.972Z · score: 18 (4 votes)
Comment by ricraz on More realistic tales of doom · 2019-04-18T02:40:29.012Z · score: 2 (1 votes) · LW · GW

Mostly I am questioning whether things will turn out badly this way.

Do you not expect this threshold to be crossed sooner or later, assuming AI alignment remains unsolved?

Probably, but I'm pretty uncertain about this. It depends on a lot of messy details about reality, things like: how offense-defence balance scales; what proportion of powerful systems are mostly aligned; whether influence-seeking systems are risk-neutral; what self-governance structures they'll set up; the extent to which their preferences are compatible with ours; how human-comprehensible the most important upcoming scientific advances are.

Comment by ricraz on More realistic tales of doom · 2019-04-17T16:53:53.343Z · score: 2 (1 votes) · LW · GW
I think the idea is that once influence-seeking systems gain a certain amount of influence, it may become faster or more certain for them to gain more influence by causing a catastrophe than to continue to work within existing rules and institutions.

The key issue here is whether there will be coordination between a set of influence-seeking systems that can cause (and will benefit from) a catastrophe, even when other systems are opposing them. If we picture systems as having power comparable to what companies have now, that seems difficult. If we picture them as having power comparable to what countries have now, that seems fairly easy.

Comment by ricraz on More realistic tales of doom · 2019-04-09T16:41:01.080Z · score: 7 (3 votes) · LW · GW
Eventually we reach the point where we could not recover from a correlated automation failure. Under these conditions influence-seeking systems stop behaving in the intended way, since their incentives have changed---they are now more interested in controlling influence after the resulting catastrophe then continuing to play nice with existing institutions and incentives.

I'm not sure I understand this part. The influence-seeking systems which have the most influence also have the most to lose from a catastrophe. So they'll be incentivised to police each other and make catastrophe-avoidance mechanisms more robust.

As an analogy: we may already be past the point where we could recover from a correlated "world leader failure": every world leader simultaneously launching a coup. But this doesn't make such a failure very likely, unless world leaders also have strong coordination and commitment mechanisms between themselves (which are binding even after the catastrophe).

Comment by ricraz on What are CAIS' boldest near/medium-term predictions? · 2019-03-29T00:22:57.277Z · score: 2 (1 votes) · LW · GW

The 75% figure is from now until single agent AGI. I measure it proportionately because otherwise it says more about timeline estimates than about CAIS.

Comment by ricraz on What are CAIS' boldest near/medium-term predictions? · 2019-03-28T14:55:24.064Z · score: 7 (4 votes) · LW · GW

The operationalisation which feels most natural to me is something like:

  • Make a list of cognitively difficult jobs (lawyer, doctor, speechwriter, CEO, engineer, scientist, accountant, trader, consultant, venture capitalist, etc...)
  • A job is automatable when there exists a publicly accessible AI service which allows an equally skilled person to do just as well in less than 25% of the time that it used to take a specialist, OR which allows someone with little skill or training to do the job in about the same time that it used to take a specialist.
  • I claim that over 75% of the jobs on this list will be automatable within 75% of the time until a single superhuman AGI is developed.
  • (Note that there are three free parameters in this definition, which I've set to arbitrary numbers that seem intuitively reasonable).
Comment by ricraz on Disentangling arguments for the importance of AI safety · 2019-02-27T11:38:25.158Z · score: 4 (2 votes) · LW · GW

Thanks! I agree that more connection to past writings is always good, and I'm happy to update it appropriately - although, upon thinking about it, there's nothing which really comes to mind as an obvious omission (except perhaps citing sections of Superintelligence?) Of course I'm pretty biased, since I already put in the things which I thought were most important - so I'd be glad to hear any additional suggestions you have.

Comment by ricraz on How to get value learning and reference wrong · 2019-02-27T11:23:57.145Z · score: 2 (1 votes) · LW · GW

Kudos for writing about making mistakes and changing your mind. If I'm interpreting you correctly, your current perspective is quite similar to mine (which I've tried to explain here and here).

Comment by ricraz on Three Kinds of Research Documents: Clarification, Explanatory, Academic · 2019-02-15T17:43:42.288Z · score: 10 (3 votes) · LW · GW

Agreed that "clarification" is confusing. What about "exploration"?

Comment by ricraz on Arguments for moral indefinability · 2019-02-13T13:44:18.278Z · score: 5 (3 votes) · LW · GW

Thanks for the detailed comments! I only have time to engage with a few of them:

Most of this is underdefined, and that’s unsettling at least in some (but not necessarily all) cases, and if we want to make it less underdefined, the notion of 'one ethics' has to give.

I'm not that wedded to 'one ethics', more like 'one process for producing moral judgements'. But note that if we allow arbitrariness of scope, then 'one process' can be a piecewise function which uses one subprocess in some cases and another in others.

I find myself having similarly strong meta-level intuitions about wanting to do something that is "non-arbitrary" and in relevant ways "simple/elegant". ...motivationally it feels like this intuition is importantly connected to what makes it easy for me to go "all-in“ for my ethical/altruistic beliefs.

I agree that these intuitions are very strong, and they are closely connected to motivational systems. But so are some object-level intuitions like "suffering is bad", and so the relevant question is what you'd do if it were a choice between that and simplicity. I'm not sure your arguments distinguish one from the other in that context.

one can maybe avoid to feel this uncomfortable feeling of uncertainty by deferring to idealized reflection. But it’s not obvious that this lastingly solves the underlying problem

Another way of phrasing this point: reflection is almost always good for figuring out what's the best thing to do, but it's not a good way to define what's the best thing to do.

Comment by ricraz on Arguments for moral indefinability · 2019-02-13T13:29:22.824Z · score: 5 (3 votes) · LW · GW

For the record, this is probably my key objection to preference utilitarianism, but I didn't want to dive into the details in the post above (for a very long post about such things, see here).

Comment by ricraz on Coherent behaviour in the real world is an incoherent concept · 2019-02-13T12:01:02.458Z · score: 2 (1 votes) · LW · GW

From Rohin's post, a quote which I also endorse:

You could argue that while [building AIs with really weird utility functions] is possible in principle, no one would ever build such an agent. I wholeheartedly agree, but note that this is now an argument based on particular empirical facts about humans (or perhaps agent-building processes more generally).

And if you're going to argue based on particular empirical facts about what goals we expect, then I don't think that doing so via coherence arguments helps very much.

Comment by ricraz on Coherent behaviour in the real world is an incoherent concept · 2019-02-13T11:31:03.996Z · score: 2 (1 votes) · LW · GW
This seems pretty false to me.

I agree that this problem is not a particularly important one, and explicitly discard it a few sentences later. I hadn't considered your objection though, and will need to think more about it.

(Side note: I'm pretty annoyed with all the use of "there's no coherence theorem for X" in this post.)

Mind explaining why? Is this more a stylistic preference, or do you think most of them are wrong/irrelevant?

the "further out" your goal is and the more that your actions are for instrumental value, the more it should look like world 1 in which agents are valuing abstract properties of world states, and the less we should observe preferences over trajectories to reach said states.

Also true if you make world states temporally extended.

Comment by ricraz on Arguments for moral indefinability · 2019-02-12T13:39:10.645Z · score: 4 (2 votes) · LW · GW

If I had to define it using your taxonomy, then yes. However, it's also trying to do something broader. For example, it's intended to be persuasive to people who don't think of meta-ethics in terms of preferences and rationality at all. (The original intended audience was the EA forum, not LW).

Edit: on further reflection, your list is more comprehensive than I thought it was, and maybe the people I mentioned above actually would be on it even if they wouldn't describe themselves that way.

Another edit: maybe the people who are missing from your list are those who would agree that morality has normative force but deny that rationality does (except insofar as it makes you more moral), or at least are much more concerned with the former than the latter. E.g. you could say that morality is a categorical imperative but rationality is only a hypothetical imperative.

Arguments for moral indefinability

2019-02-12T10:40:01.226Z · score: 45 (14 votes)

Coherent behaviour in the real world is an incoherent concept

2019-02-11T17:00:25.665Z · score: 32 (12 votes)
Comment by ricraz on Book Trilogy Review: Remembrance of Earth’s Past (The Three Body Problem) · 2019-01-30T18:58:44.635Z · score: 1 (2 votes) · LW · GW

There are some interesting insights about the overall viewpoint behind this book, but gosh the tone of this post is vicious. I totally understand frustration with stupidity in fiction, and I've written such screeds in my time too. But I think it's well worth moderating the impulse to do so in cases like this where the characters whose absolute stupidity you're bemoaning map onto the outgroup in so many ways.

Comment by ricraz on Too Smart for My Own Good · 2019-01-24T02:00:44.030Z · score: 2 (1 votes) · LW · GW

Agreed, except that the behaviour described could also just be procrastination.

Comment by ricraz on Disentangling arguments for the importance of AI safety · 2019-01-24T01:43:33.998Z · score: 2 (1 votes) · LW · GW

I don't think it depends on how much A and B, because the "expected amount" is not a special point. In this context, the update that I made personally was "There are more shifts than I thought there were, therefore there's probably more of A and B than I thought there was, therefore I should weakly update against AI safety being important." Maybe (to make A and B more concrete) there being more shifts than I thought downgrades my opinion of the original arguments from "absolutely incredible" to "very very good", which slightly downgrades my confidence that AI safety is important.

As a separate issue, conditional on the field being very important, I might expect the original arguments to be very very good, or I might expect them to be very good, or something else. But I don't see how that expectation can prevent a change from "absolutely exceptional" to "very very good" from downgrading my confidence.

Vote counting bug?

2019-01-22T15:44:48.154Z · score: 7 (2 votes)
Comment by ricraz on Disentangling arguments for the importance of AI safety · 2019-01-22T11:43:15.378Z · score: 5 (3 votes) · LW · GW

Apologies if this felt like it was targeted specifically at you and other early AI safety advocates, I have nothing but the greatest respect for your work. I'll rewrite to clarify my intended meaning, which is more an attempt to evaluate the field as a whole. This is obviously a very vaguely-defined task, but let me take a stab at fleshing out some changes over the past decade:

1. There's now much more concern about argument 2, the target loading problem (as well as inner optimisers, insofar as they're distinct).

2. There's now less focus on recursive self-improvement as a key reason why AI will be dangerous, and more focus on what happens when hardware scales up. Relatedly, I think a greater percentage of safety researchers believe that there'll be a slow takeoff than used to be the case.

3. Argument 3 (prosaic AI alignment) is now considered more important and more tractable.

4. There's now been significant criticism of coherence arguments as a reason to believe that AGI will pursue long-term goals in an insatiable maximising fashion.

I may be wrong about these shifts - I'm speaking as a newcomer to the field who has a very limited perspective on how it's evolved over time. If so, I'd be happy to be corrected. If they have in fact occurred, here are some possible (non-exclusive) reasons why:

A. None of the proponents of the original arguments have changed their minds about the importance of those arguments, but new people came into the field because of those arguments, then disagreed with them and formulated new perspectives.

B. Some of the proponents of the original arguments have changed their minds significantly.

C. The proponents of the original arguments were misinterpreted, or overemphasised some of their beliefs at the expense of others, and actually these shifts are just a change in emphasis.

I think none of these options reflect badly on anyone involved (getting everything exactly right the first time is an absurdly high standard), but I think A and B would be weak evidence against the importance of AI safety (assuming you've already conditioned on the size of the field, etc). I also think that it's great when individual people change their minds about things, and definitely don't want to criticise that. But if the field as a whole does so (whatever that means), the dynamics of such a shift are worth examination.

I don't have strong beliefs about the relative importance of A, B and C, although I would be rather surprised if any one of them were primarily responsible for all the shifts I mentioned above.

Comment by ricraz on Disentangling arguments for the importance of AI safety · 2019-01-22T11:07:44.458Z · score: 4 (2 votes) · LW · GW

I endorse ESRogs' answer. If the world were a singleton under the control of a few particularly benevolent and wise humans, with an AGI that obeys the intention of practical commands (in a somewhat naive way, say, so it'd be unable to help them figure out ethics) then I think argument 5 would no longer apply, but argument 4 would. Or, more generally: argument 5 is about how humans might behave badly under current situations and governmental structures in the short term, but makes no claim that this will be a systemic problem in the long term (we could probably solve it using a singleton + mass surveillance); argument 4 is about how we don't know of any governmental(/psychological?) structures which are very likely to work well in the long term.

Having said that, your ideas were the main (but not sole) inspiration for argument 4, so if this isn't what you intended, then I may need to rethink its inclusion.

Disentangling arguments for the importance of AI safety

2019-01-21T12:41:43.615Z · score: 112 (37 votes)
Comment by ricraz on What AI Safety Researchers Have Written About the Nature of Human Values · 2019-01-16T15:52:21.307Z · score: 3 (2 votes) · LW · GW

Nice overview :) One point: the introductory sentences don't seem to match the content.

It is clear to most AI safety researchers that the idea of “human values” is underdefined, and this concept should be additionally formalized before it can be used in (mostly mathematical) models of AI alignment.

In particular, I don't interpret most of the researchers you listed as claiming that "[human values] should be formalized". I think that's a significantly stronger claim than, for example, the claim that we should try to understand human values better.

Comment by ricraz on Open Thread January 2019 · 2019-01-16T15:45:24.388Z · score: 5 (3 votes) · LW · GW

Are you claiming that price per computation would drop in absolute terms, or compared with the world in which Moore's law continued? The first one seems unobjectionable, the default state of everything is for prices to fall since there'll be innovation in other parts of the supply chain. The second one seems false. Basic counter-argument: if it were true, why don't people produce chips from a decade ago which are cheaper per amount of computation than the ones being produced today?

1. You wouldn't have to do R&D, you could just copy old chip designs.

2. You wouldn't have to keep upgrading your chip fabs, you could use old ones.

3. People could just keep collecting your old chips without getting rid of them.

4. Patents on old chip designs have already expired.

Comment by ricraz on Comments on CAIS · 2019-01-14T11:11:11.925Z · score: 7 (3 votes) · LW · GW
AI services can totally be (approximately) VNM rational -- for a bounded utility function.

Suppose an AI service realises that it is able to seize many more resources with which to fulfil its bounded utility function. Would it do so? If no, then it's not rational with respect to that utility function. If yes, then it seems rather unsafe, and I'm not sure how it fits Eric's criterion of using "bounded resources".

Note that CAIS is suggesting that we should use a different prior: the prior based on "how have previous advances in technology come about". I find this to be stronger evidence than how evolution got to general intelligence.

I agree with Eric's claim that R&D automation will speed up AI progress. The point of disagreement is more like: when we have AI technology that's able to do basically all human cognitive tasks (which for want of a better term I'll call AGI, as an umbrella term to include both CAIS and agent AGI), what will it look like? It's true that no past technologies have looked like unified agent AGIs - but no past technologies have also looked like distributed systems capable of accomplishing all human tasks either. So it seems like the evolution prior is still the most relevant one.

"Humans think in terms of individuals with goals, and so even if there's an equally good approach to AGI which doesn't conceive of it as a single goal-directed agent, researchers will be biased against it."
I'm curious how strong an objection you think this is. I find it weak; in practice most of the researchers I know think much more concretely about the systems they implement than "agent with a goal", and these are researchers who work on deep RL. And in the history of AI, there were many things to be done besides "agent with a goal"; expert systems/GOFAI seems like the canonical counterexample.

I think the whole paradigm of RL is an example of a bias towards thinking about agents with goals, and that as those agents become more powerful, it becomes easier to anthropomorphise them (OpenAI Five being one example where it's hard not to think of it as a group of agents with goals). I would withdraw my objection if, for example, most AI researchers took the prospect of AGI from supervised learning as seriously as AGI from RL.

A clear counterargument is that some companies will have AI CEOs, and they will outcompete the others, and so we'll quickly transition to the world where all companies have AI CEOs. I think this is not that important -- having a human in the loop need not slow down everything by a huge margin, since most of the cognitive work is done by the AI advisor, and the human just needs to check that it makes sense (perhaps assisted by other AI services).

I claim that this sense of "in the loop" is irrelevant, because it's equivalent to the AI doing its own thing while the human holds a finger over the stop button. I.e. the AI will be equivalent to current CEOs, the humans will be equivalent to current boards of directors.

To the extent that you are using this to argue that "the AI advisor will be much more like an agent optimising for an open-ended goal than Eric claims", I agree that the AI advisor will look like it is "being a very good CEO". I'm not sure I agree that it will look like an agent optimizing for an open-ended goal, though I'm confused about this.

I think of CEOs as basically the most maximiser-like humans. They have pretty clear metrics which they care about (even if it's not just share price, "company success" is a clear metric by human standards), they are able to take actions that are as broad in scope as basically any actions humans can take (expand to new countries, influence politics, totally change the lives of millions of employees), and almost all of the labour is cognitive, so "advising" is basically as hard as "doing" (modulo human interactions). To do well they need to think "outside the box" of stimulus and response, and deal with worldwide trends and arbitrarily unusual situations (has a hurricane just hit your factory? do you need to hire mercenaries to defend your supply chains?) Most of them have some moral constraints, but also there's a higher percentage of psychopaths than any other role, and it's plausible that we'd have no idea whether an AI doing well as a CEO actually "cares about" these sorts of bounds or is just (temporarily) constrained by public opinion in the same way as the psychopaths.

The main point of CAIS is that services aren't long-term goal-oriented; I agree that if services end up being long-term goal-oriented they become dangerous.

I then mentioned that to build systems which implement arbitrary tasks, you may need to be operating over arbitrarily long time horizons. But probably this also comes down to how decomposable such things are.

If you go via the CAIS route you definitely want to prevent unbounded AGI maximizers from being created until you are sure of their safety or that you can control them. (I know you addressed that in the previous point, but I'm pretty sure that no one is arguing to focus on CAIS conditional on AGI agents existing and being more powerful than CAIS, so it feels like you're attacking a strawman.)

People are arguing for a focus on CAIS without (to my mind) compelling arguments for why we won't have AGI agents eventually, so I don't think this is a strawman.

Given a sufficiently long delay, we could use CAIS to build global systems that can control any new AGIs, in the same way that government currently controls most people.

This depends on having pretty powerful CAIS and very good global coordination, both of which I think of as unlikely (especially given that in a world where CAIS occurs and isn't very dangerous, people will probably think that AI safety advocates were wrong about there being existential risk). I'm curious how likely you think this is though? If agent AGIs are 10x as dangerous, and the probability that we eventually build them is more than 10%, then agent AGIs are the bigger threat.

I also am not sure why you think that AGI agents will optimize harder for self-improvement.

Because they have long-term convergent instrumental goals, and CAIS doesn't. CAIS only "cares" about self-improvement to the extent that humans are instructing it to do so, but humans are cautious and slow. Also because even if building AGI out of task-specific strongly-constrained modules is faster at first, it seems unlikely that it's anywhere near the optimal architecture for self-improvement.

Compared to what? If the alternative is "a vastly superintelligent AGI agent that is acting within what is effectively the society of 2019", then I think CAIS is a better model. I'm guessing that you have something else in mind though.

It's something like "the first half of CAIS comes true, but the services never get good enough to actually be comprehensive/general. Meanwhile fundamental research on agent AGI occurs roughly in parallel, and eventually overtakes CAIS." As a vague picture, imagine a world in which we've applied powerful supervised learning to all industries, and applied RL to all tasks which are either as constrained and well-defined as games, or as cognitively easy as most physical labour, but still don't have AI which can independently do the most complex cognitive tasks (Turing tests, fundamental research, etc).

Comment by ricraz on Comments on CAIS · 2019-01-14T09:12:54.460Z · score: 11 (3 votes) · LW · GW

You're right, this is a rather mealy-mouthed claim. I've edited it to read as follows:

the empirical claim that we'll develop AI services which can replace humans at most cognitively difficult jobs significantly before we develop any single strongly superhuman AGI

This would be false if doing well at human jobs requires capabilities that are near AGI. I do expect a phase transition - roughly speaking I expect progress in automation to mostly require more data and engineering, and progress towards AGI to require algorithmic advances and a cognition-first approach. But the thing I'm trying to endorse in the post is a weaker claim which I think Eric would agree with.

Comment by ricraz on Comments on CAIS · 2019-01-12T20:30:40.539Z · score: 9 (4 votes) · LW · GW
AGI is ... something that approximates an expected utility maximizer.

This seems like a trait which AGIs might have, but not a part of how they should be defined. I think Eric would say that the first AI system which can carry out all the tasks we would expect an AGI to be capable of won't actually approximate an expected utility maximiser, and I consider it an open empirical question whether or not he's right.

Many risk-reducing services (especially ones that can address human safety problems) seem to require high-level general reasoning abilities, whereas many risk-increasing services can just be technical problem solvers or other kinds of narrow intelligences or optimizers, so CAIS is actually quite unsafe, and hard to make safe, whereas AGI / goal-directed agents are by default highly unsafe, but with appropriate advances in safety research can perhaps be made safe.

Yeah, good point. I guess that my last couple of sentences were pretty shallowly-analysed, and I'll retract them and add a more measured conclusion.

Comment by ricraz on Why is so much discussion happening in private Google Docs? · 2019-01-12T18:57:39.285Z · score: 15 (6 votes) · LW · GW

I agree with both those points, and would add:

3. The fact that access to the doc is invite-only, and therefore people feel like they've been specifically asked to participate.

Comments on CAIS

2019-01-12T15:20:22.133Z · score: 69 (18 votes)
Comment by ricraz on You can be wrong about what you like, and you often are · 2018-12-20T18:49:41.172Z · score: 5 (2 votes) · LW · GW
I'm just not one of those people who enjoys "deeper" activities like reading a novel. I like watching TV and playing video games.
I'm just not one of those people who likes healthy foods. You may like salads and swear by them, but I am different. I like pizza and french fries.
I'm just not an intellectual person. I don't enjoy learning.
I'm just not into that early retirement stuff. I need to maintain my current lifestyle in order to be happy.
I'm just not into "good" movies/music/art. I like the Top 50 stuff.

I'm curious why you chose these particular examples. I think they're mostly quite bad and detract from the reasonable point of the overall post. The first three, and the fifth, I'd characterise as "acquired tastes": they're things that people may come to enjoy over time, but often don't currently enjoy. So even someone who would grow to like reading novels, and would have a better life if they read more novels, may be totally correct in stating that they don't enjoy reading novels.

The fourth is a good example for many people, but many others find that retirement is boring. Also, predicting what your life will look like after a radical shift is a pretty hard problem, so if this is the sort of thing people are wrong about it doesn't seem so serious.

More generally, whether or not you enjoy something is different from whether that thing, in the future, will make you happier. At points in this post you conflate those two properties. The examples also give me elitist vibes: the implication seems to be that upper-class pursuits are just better, and people who say they don't like them are more likely to be wrong. (If anything, actually, I'd say that people are more likely to be miscalibrated about their enjoyment of an activity the more prestigious it is, since we're good at deceiving ourselves about status considerations).

Comment by ricraz on You can be wrong about what you like, and you often are · 2018-12-20T18:23:48.890Z · score: 2 (1 votes) · LW · GW

Why is the optimisation space convex?

Comment by ricraz on Double-Dipping in Dunning--Kruger · 2018-11-29T14:59:34.807Z · score: 2 (3 votes) · LW · GW

In general it's understandable not to consider that hypothesis. But when you are specifically making a pointed and insulting comment about another person, I think the bar should be higher.

Comment by ricraz on Double-Dipping in Dunning--Kruger · 2018-11-29T00:12:31.259Z · score: 6 (4 votes) · LW · GW

Some posts can focus on raising the sanity waterline. Other posts can be motivational and targeted at people's incorrect self-assessments. Note that successfully doing the latter is often quite a good way of making people achieve better outcomes.

Comment by ricraz on Double-Dipping in Dunning--Kruger · 2018-11-29T00:09:53.300Z · score: 7 (5 votes) · LW · GW

I downvoted this explanation because it's uncharitable to claim that someone is either lying or deluded when it seems plausible that they were instead making a joke. Perhaps you have other reasons to think that Alephywr isn't joking, but if so it's worth explaining that.

Comment by ricraz on Double-Dipping in Dunning--Kruger · 2018-11-28T17:29:02.548Z · score: 26 (14 votes) · LW · GW

I like and endorse the general theme of this post, but have some issues with the details.

The takeaway is this: if you're the kind of person who worries about statistics and Dunning--Kruger in the first place, you're already way above average and clearly have the necessary meta-cognition to not fall victim to such things.

I feel like this is good motivation but bad world-modelling. Two important ways in which it fails:

  • Social interactions. You gave the example of people not really knowing how funny they are. I don't think worrying about statistics in general helps with this, because this might just not be the type of thing you've considered as a failure mode, and also because it's very difficult to substitute deliberate analysis for buggy social intuitions.
  • People being bad at philosophy. There are very many smart people who confidently make ridiculous arguments - people smart enough to understand Dunning-Kruger, but who either think they're an exception, or else pay lip service to it and then don't actually process any change in beliefs.
The world is being run by people who are too incompetent to know it; people who are only in power because they're the ones who showed up, and because showing up is most of the battle.

I dislike lines of argument which point at people on top of a pile of utility and call them incompetent. I think it is plausibly very difficult to get to the top of the society, but that the skills required are things which are really difficult to measure or even understand properly, like "hustle" or "ambition" or "social skills" or "pays less attention to local incentive gradients" or "has no wasted mental motion in between deciding that x is a good idea and deciding to do x".

From now on, unless you have evidence that you're particularly bad at something, I want you to assume that you're 15 percentile points higher than you would otherwise estimate.

Nit: I prefer using standard deviations instead of percentile points when talking about high-level performance, because it better allows us to separate people with excellent skill from people with amazing skill. Also because "assume that you're 15 percentile points higher" leaves a lot of people above 100%.

Comment by ricraz on How democracy ends: a review and reevaluation · 2018-11-28T15:36:31.375Z · score: 4 (2 votes) · LW · GW

Insofar as we're talking about the collapse of democracy in America, millions of people will be adversely affected, many of them wealthy enough that they could already be taking important steps at fairly low cost.

Comment by ricraz on How democracy ends: a review and reevaluation · 2018-11-27T23:02:09.833Z · score: 6 (3 votes) · LW · GW


How democracy ends: a review and reevaluation

2018-11-27T10:50:01.130Z · score: 17 (9 votes)
Comment by ricraz on On first looking into Russell's History · 2018-11-09T17:50:15.259Z · score: 20 (4 votes) · LW · GW

I also don't have strong opinions on how accurate the book is, but that link really doesn't support the claim that the book is inaccurate. Its most scathing criticism of Russell: "As far as the omissions go, the grossest is the denial of any role to Eastern philosophy." Something I'm inclined to forgive in a "History of Western Philosophy". Then there are complaints about "inconsequential logical griping...from place to place" in the book, which again is not really a devastating blow.

Comment by ricraz on On first looking into Russell's History · 2018-11-09T17:42:58.549Z · score: 2 (1 votes) · LW · GW

Cheers, fixed. Just autocorrect being autocorrect as per.

On first looking into Russell's History

2018-11-08T11:20:00.935Z · score: 35 (11 votes)
Comment by ricraz on A compendium of conundrums · 2018-11-07T12:26:26.403Z · score: 2 (1 votes) · LW · GW

Pirate treasure: you're right that tiebreak information is needed. I've added it in now - assume the pirates are spiteful.

Blind maze: nope, it's a fairly ugly solution.

Prisoners and boxes: yes, you can save all of them. Is the solution to your variant the same as my variant?

Battleships: I've found an ugly solution, and I expect it's the one you intended, but is there anything nicer? Sbe rirel fdhner, pbafvqre rirel cbffvoyr zbirzrag irpgbe, juvpu tvirf hf n ghcyr bs yratgu 4. Gura jr vgrengr guebhtu rirel fhpu ghcyr hfvat qvntbanyvfngvba, naq ng rnpu gvzrfgrc jr pna pnyphyngr jurer n fuvc juvpu fgnegrq ba gung fdhner jbhyq unir raqrq hc ol gur pheerag gvzrfgrc, naq fubbg vg.

Comment by ricraz on Speculations on improving debating · 2018-11-06T14:22:45.741Z · score: 2 (1 votes) · LW · GW

I'm sympathetic with that view, but think it's far from clear-cut. For example, suppose you model rationality as the skill of identifying bad arguments plus the mental habit of applying that skill to your own ideas. When the former is the bottleneck, then debating probably has a positive effect on overall rationality; when the latter is the bottleneck, it is probably negative. Probably the latter is more common, but the effect of the former is bigger? I don't have a strong opinion on this though.

As an anecdotal point, I have been pleasantly surprised by how often you can win a debate by arguing primarily for things that you actually believe. The example that comes to mind is being assigned the pro-Brexit side in a debate, and focusing on the EU's pernicious effects on African development, and how trade liberalisation would benefit the bottom billion. In cases like these you don't so much rebut your opponents' points as reframe them to be irrelevant - and I do think that switching mental frameworks is an important skill.

Speculations on improving debating

2018-11-05T16:10:02.799Z · score: 26 (10 votes)
Comment by ricraz on When does rationality-as-search have nontrivial implications? · 2018-11-05T15:15:03.799Z · score: 6 (3 votes) · LW · GW

This was a very useful and well-explained idea. Strongly upvoted.

Comment by ricraz on Implementations of immortality · 2018-11-01T22:43:01.914Z · score: 5 (2 votes) · LW · GW

Hmm. Yeah, you're probably right. Although there is the common phenomenon of social subgroups stratifying into smaller and smaller cliques, which is some evidence that oppositional group identity matters even if it doesn't feel like it from the inside.

Comment by ricraz on Implementations of immortality · 2018-11-01T22:36:49.823Z · score: 2 (1 votes) · LW · GW
Age stratification in a world where people live arbitrarily long means you never have an opportunity to become a respected elder in your society; generations of more respected super-elders will be around no matter how old and wise you get.

For any x, you eventually get the opportunity to be in the top x% of society! And assuming that the size of a social circle/community stays roughly fixed, eventually you'll be at the very top of your community. Maybe the more respected super-elders will be in some other galaxy, but not on your world.

Also, in this world, are people youthful indefinitely? I think many of the age related changes in activity choices are driven by physical aging, not maturity, e.g., choosing cocktail parties over clubbing happens not because you realize one day that cocktail parties are a richer experience, but because one day you realize you get too tired by 10pm for the club scene.

Yes, people would be youthful indefinitely. I think there's a mix of reasons, but getting bored/moving on is definitely one of the main ones. Picture a jaded 40 year old in a nightclub - is the limiting factor really tiredness?

Comment by ricraz on Implementations of immortality · 2018-11-01T22:28:55.316Z · score: 2 (1 votes) · LW · GW
Obviously I would desire some amount of novelty, but it's mostly in the context of slotting into a roughly stable daily or weekly routine, rather than the routine itself varying much. (e.g. Thursday evening is for games, the games may vary and becoming increasingly complex, but they are still generally played on Thursday evenings).

The point about typical mind fallacy is well-taken but I don't really see how you can be confident in preferences like the one quoted above given that the timeframes we're talking about are much longer than your lifespan so far. I mean, many people have midlife crises after only a few decades of such routines. I have a strong intuition that after several centuries of games every Thursday evening, almost anyone would get very bored.

At the very least, I would want a mostly stable "island of calm" where things mostly remained the same, and where I would always return when I was tired of going on adventures.

This isn't ruled out by my proposal; note that "progress" doesn't mean discarding every aspect of the past. However, I am suspicious of this sort of island of calm, for roughly the same reason that I doubt it's valuable to most adults to regularly visit their childhood treehouse. (Also, if there are other people in this 'island of calm', then you can't force them to stay the same just for your sake...)

[This] reads to me like you're saying that it's a problem if people like me have the freedom to choose stability if that makes them happier than variety does.

People get stuck in local maxima, and often don't explore enough to find better options for themselves. The longer people live, the more valuable it is to have sufficient exploration to figure out the best option before choosing stability.

Implementations of immortality

2018-11-01T14:20:01.494Z · score: 21 (8 votes)

What will the long-term future of employment look like?

2018-10-24T19:58:09.320Z · score: 11 (4 votes)

Book review: 23 things they don't tell you about capitalism

2018-10-18T23:05:29.465Z · score: 19 (11 votes)
Comment by ricraz on Some cruxes on impactful alternatives to AI policy work · 2018-10-18T15:15:46.989Z · score: 5 (3 votes) · LW · GW

I generally agree with the idea of there being long tails to the left. Revolutions are a classic example - and, more generally, any small group of ideologically polarised people taking extreme actions. Environmentalists groups blocking genetically engineered crops might be one example; global warming skepticism another; perhaps also OpenAI.

I'm not sure about the "sleeping dragons", though, since I can't think of many cases where small groups created technologies that counterfactually wouldn't have happened (or even would have happened in safer ways).

Comment by ricraz on Some cruxes on impactful alternatives to AI policy work · 2018-10-14T09:41:48.575Z · score: 10 (3 votes) · LW · GW

There's something to this, but it's not the whole story, because increasing probability of survival is good no matter what the current level is. Perhaps if you model decreasing existential risk as becoming exponentially more difficult (e.g. going from 32% risk to 16% risk is as difficult as going from 16% to 8%) and with the possibility of accidental increases (e.g. you're trying to go from 32 to 16 but there's some probability you go to 64 instead) then the current expectation for the level of risk will affect whether you take high-variance actions or not.

Comment by ricraz on Some cruxes on impactful alternatives to AI policy work · 2018-10-14T09:35:15.388Z · score: 6 (2 votes) · LW · GW

I'm not sure how much to believe in this without concrete examples (the ones which come to mind are mostly pretty trivial, like Yahoo having a cluttered homepage and Google having a minimalist one, or MacOS being based on Unix).

Maybe Twitter is an example? I can easily picture it having a very different format. Still, I'm not particularly swayed by that.

Book review: The Complacent Class

2018-10-13T19:20:05.823Z · score: 21 (9 votes)
Comment by ricraz on non-judgemental awareness · 2018-10-11T13:06:40.691Z · score: 5 (3 votes) · LW · GW
When practicing to improve in a skill you want to get as much good quality information as possible (eg: Bob).

I just don't think this is true. The advice to always practice with good technique, to entrench good habits, is fairly common. And in my experience it's only once I've played a lot with good technique that I can even notice many of the subtleties.

Comment by ricraz on We can all be high status · 2018-10-10T18:02:14.162Z · score: 14 (9 votes) · LW · GW
So what I want to propose is that we define much more clearly what it takes to be taken seriously around here.

I mostly agree with your description of the problem, and I sympathise with your past self. However, I also think you understate the extent to which the EA and rationality communities are based around individual friendships. That makes things much messier than they might be in a corporation, and make definitions like the one you propose much harder.

On the other hand, it also means that there's another sense in which "we can all be high-status": within our respective local communities. I'm curious how you feel about that, because that was quite adequate for me for a long time, especially as a student.

On a broader level, one actionable idea I've been thinking about is to talk less about existential risk being "talent constrained", so that people who can't get full-time jobs in the field don't feel like they're not talented. A more accurate term in my eyes is "field-building constrained".

Some cruxes on impactful alternatives to AI policy work

2018-10-10T13:35:27.497Z · score: 146 (50 votes)
Comment by ricraz on A compendium of conundrums · 2018-10-10T13:24:42.285Z · score: 1 (1 votes) · LW · GW

Oh man, that was a stupid typo. I was very confused, mostly because I myself hadn't properly read the question. Edited now; yours is a clever solution though.

Comment by ricraz on A compendium of conundrums · 2018-10-10T08:11:32.102Z · score: 1 (1 votes) · LW · GW

Nope, some people get stuck on which links to cut. The standard answer is 3, which is the same as yours but with the assumption that you can detatch a link from its neighbours after 1 cut, which wasn't made explicit.

Comment by ricraz on Thinking of the days that are no more · 2018-10-09T13:06:19.357Z · score: 1 (1 votes) · LW · GW

I don't think there's any alternative. The reason that these contracts used to be hard to breach was mainly because of social norms - otherwise you could just leave and live in sin with someone else any time you wanted. But weaker contracts are only possible because the relevant social norms have changed. (Although there are probably some communities which take marriage much more seriously, and you could live there if you wanted to).

Then there are changes re who gets child custody, but it seems to me that having consistent legal judgements based on what's best for the kids is better than allowing some people to opt into more extreme contracts.

Another factor is laws around property ownership, but I think that even though the laws have weakened, opting in to prenups is a sufficient solution for anyone who wants stronger commitments. They have clauses changing property allocations depending on who's "at fault" for the divorce, right? (Although I guess I'm against prenups which specify custody arrangements, except insofar as they turn out to be good for kids).

Comment by ricraz on Thinking of the days that are no more · 2018-10-08T15:33:49.390Z · score: 2 (2 votes) · LW · GW

"Doesn't work well" by what metric - having children? I don't see why that should be the predominant consideration. I have many other goals when I go into relationships - enjoyment, companionship, self-improvement, security, signalling, etc. Now that people are much wealthier and have fewer children, the relative importance of hard-to-breach contracts has decreased, and it's plausible that for many people, moving even further towards flexible contracts is better for most of their goals.

A compendium of conundrums

2018-10-08T14:20:01.178Z · score: 12 (12 votes)

Thinking of the days that are no more

2018-10-06T17:00:01.208Z · score: 13 (6 votes)
Comment by ricraz on The Unreasonable Effectiveness of Deep Learning · 2018-10-06T15:43:01.667Z · score: 1 (1 votes) · LW · GW

I confess that I have a weakness for slightly fanciful titles. In my defence, though, I do actually think that "unreasonable" is a reasonable way of describing the success of neural networks. The argument in the original paper was something like "it could have been the case that math just wasn't that helpful in describing the universe, but actually it works really well on most things we try it on, and we don't have any principled explanation for why that is so". Similarly, it could have been the case that feedforward neural networks just weren't very good at learning useful functions, but actually they work really well on most things we try them on, and we don't have any principled explanation for why that is so.

The Unreasonable Effectiveness of Deep Learning

2018-09-30T15:48:46.861Z · score: 86 (26 votes)

Deep learning - deeper flaws?

2018-09-24T18:40:00.705Z · score: 42 (17 votes)

Book review: Happiness by Design

2018-09-23T04:30:00.939Z · score: 14 (6 votes)

Book review: Why we sleep

2018-09-19T22:36:19.608Z · score: 51 (24 votes)

Realism about rationality

2018-09-16T10:46:29.239Z · score: 141 (57 votes)

Is epistemic logic useful for agent foundations?

2018-05-08T23:33:44.266Z · score: 19 (6 votes)

What we talk about when we talk about maximising utility

2018-02-24T22:33:28.390Z · score: 27 (8 votes)

In Defence of Conflict Theory

2018-02-17T03:33:01.970Z · score: 6 (6 votes)

Is death bad?

2018-01-13T04:55:25.788Z · score: 8 (4 votes)