For moderately well-resourced people in the US, how quickly will things go from "uncomfortable" to "too late to exit"? 2020-06-12T16:59:56.845Z
Money isn't real. When you donate money to a charity, how does it actually help? 2020-02-02T17:03:04.426Z
Dagon's Shortform 2019-07-31T18:21:43.072Z
Did the recent blackmail discussion change your beliefs? 2019-03-24T16:06:52.811Z


Comment by Dagon on Training Better Rationalists? · 2021-08-05T16:04:32.919Z · LW · GW

Maybe if we can identify an enemy who's going to shoot at us, we can select and instill that level of commitment.  I suspect it comes from a pre-rational part of human motivation, and is not available to the vast majority of rationalists.

Comment by Dagon on Lifepool · 2021-08-05T15:43:24.963Z · LW · GW

I'd expect the expected rewards to such a thing are already FAR larger than any prize is likely to be.  You might be better off organizing systems or motivations for small components of life extension that aren't valuable enough on their own for the market to make the inventors and delivery participants rich.

Comment by Dagon on ozziegooen's Shortform · 2021-08-05T03:27:47.915Z · LW · GW

This is one of the most believable misunderstood-supervillian plots I could get sucked into.

Comment by Dagon on Decision-Making Training for Software Engineering · 2021-08-04T16:54:52.326Z · LW · GW

How would you feel about someone being given a pile of code, and having to add a feature that requires modifications throughout the codebase? 

I think this describes many internships or onboarding projects for developers.  My general opinion is that, when nobody's shooting at you, it's best to do this on real software, rather than training simulations. The best simulation is reality itself.

Comment by Dagon on Decision-Making Training for Software Engineering · 2021-08-04T16:52:20.131Z · LW · GW

I should have been more specific that the post-mortem is a critical part of the incident handling.  I see a lot of similarity in tactical decision-making, both in the incident handling (the decisions made) and in the post-mortem (the analysis and rationale).  

Strategic decision-making, tradeoffs about solving a narrow problem simply or leaving room for a class of problems, with more complexity (and structures to handle that complexity), is a related, but different set of skills.

Comment by Dagon on MikkW's Shortform · 2021-08-04T16:47:28.469Z · LW · GW

There's a critical modeling question to ask before this one - what is your ACTUAL preferece-aggregation function, in a mind-reading world where strategy doesn't come into play? is about aggregation of choice, not just voting errors.

Comment by Dagon on Decision-Making Training for Software Engineering · 2021-08-04T14:54:53.352Z · LW · GW

This describes a high-severity operational event (network outage, crash bug, etc.) pretty well.  The call leader is organizing teams and devs to focus on different parts of diagnosis and mitigation, and there's tons of unknowns and both relevant and irrelevant information coming in all the time.

Many companies/teams do dry-runs of such things, but on-the-job training is most effective.  The senior people guide the junior people through it as it happens, and after a few repetitions, the junior people (or the ones who display maturity and aptitude) become the leads.

For "regular" software engineering, I'd rather not encourage the idea that short, distinct scenarios are representative of the important parts.  Making something that's maintainable and extensible over many years isn't something that can be trained in small bites.

Comment by Dagon on Very Unnatural Tasks? · 2021-08-03T16:22:51.021Z · LW · GW

I suspect that the observed behaviors will feel normal - even if the goals and motives are inscrutable, humans will infer more pedestrian goals from the actions they see.  In other words, the motivations may be unnatural or weird, but the actions will still be actions, and we'll make up reasons that we can believe.

For instance, the AGI may cure cancer by making our DNA more resistant to mutation, because they fear uncontrolled variance in their pets.  We'll only notice the cure, not the motive.

Comment by Dagon on [deleted post] 2021-08-01T17:26:22.259Z

There will be no clear signs - ambiguity is required to keep humans under control (or rather, so the humans control themselves by incorrect beliefs).

Comment by Dagon on The World is Continuous, Not Discrete · 2021-08-01T16:53:14.738Z · LW · GW

I fear you're mixing ambiguity with uncertainty with un-quantized states.  A sensible definition of dead would be "never again going to take action".  There is a bright line where this is the case, though there's uncertainty whether it's been crossed.  Hearts can start beating again, though I don't know of any cases of post-cremation resurrection.

Healthy vs sick is similar regarding uncertainty and ambiguity.  Outside of movies, doctors don't pronounce you absolutely healthy.  They can say relatively healthy - better shape or less risk than some comparison group.  

Comment by Dagon on Brief thoughts on inefficient writing systems · 2021-07-29T22:14:00.283Z · LW · GW

Ease/speed of learning is the wrong metric to evaluate a language or writing system.  How effective it is for communication among the literate is far more important.   And that communication includes interesting hard-to-measure features such as social signaling, intentional ambiguity, and extensibility to new concepts.

Comment by Dagon on paulfchristiano's Shortform · 2021-07-27T23:13:54.444Z · LW · GW

I think multi-level search may help here.  To the extent that you can get a lower-confidence estimate of P much more quickly, you can budget your total search time such that you examine many programs and then re-examine only the good candidates.   If your confidence is linear with time/complexity of evaluation, this probably doesn't help.

Comment by Dagon on Josephine's Shortform · 2021-07-27T23:07:43.106Z · LW · GW

Good observation!  The converse holds too - we should change context for those things we want to do better than the current mechanisms can measure.

Comment by Dagon on Black ravens and red herrings · 2021-07-27T21:44:52.251Z · LW · GW

Unrelated to the paradox, there is a distinction between a definition/category statement, and a prediction about the state of the universe.  There are different uncertainties involved, and different evidence that would move those probabilities for you.  

If it means "by definition, any bird of this form which isn't black isn't a crow, it's something else BECAUSE it's non-black", then the uncertainty is about whether whoever you're talking with is using the same definition.

If it means "we never see this biological cluster of features without also having black pigment", that's much more empirical, and subject to observational evidence.   

Comment by Dagon on Black ravens and red herrings · 2021-07-27T20:39:52.585Z · LW · GW

Yup, the "update based on the magnitude of your surprise" heuristic matches this pretty well.  

Comment by Dagon on Future Of Work · 2021-07-24T01:17:54.060Z · LW · GW

I think you're undervaluing the cultural expectations of availability and cooperation during core work hours.  And the value to workers of contractual pay (for which employers demand contractual hours).  You're also forgetting the hidden-value alignment brought on by the expectation of a recurring long-term relationship.  It's hard to monitor most work in the short term, so having the engagements be longer-term makes it possible to adjust job and compensation based on years' of output rather than the latest delivery.

There certainly is more work than many think which can effectively be done piecemeal.  But there's lots more than you seem to acknowledge that is pretty well optimized by current norms.

Comment by Dagon on Fire Law Incentives · 2021-07-24T00:59:21.853Z · LW · GW

Watchmen was pretty good on this front.  Worm ( is LONG, but great.

Comment by Dagon on Punishing the good · 2021-07-23T01:57:34.893Z · LW · GW

Should you punish people for wronging others, or for making the wrong call about wronging others?

This is a topic where the answer depends a whole lot on how generally you're asking, and what your moral and decision framework is.  The key ambiguity is "should".  "what should you do" has been an open question for millennia.

The obvious consequentialist answer is that you should do either, both, or neither, depending on circumstance and your expected net impact of your action.  Other moral frameworks likely have different answers.

The signal I prefer to send on the topic, intended to encourage a mechanical, reductionist view of the universe, is that punishment should be automatic and non-judgemental, with as little as possible speculation as to motive or reasoning.  If it caused harm, the punishment should be proportional to the harm.  Yes, this lets luck and good-intentioned mistakes control more than an omniscient god might prefer.  But I don't have one of those, so I prefer to eliminate the other human failings of bad judgement that come with humans making a punishment call putatively based on inference about reasoning or motivation, but actually mostly based on biases and guesses.

I'm OK with adding punishment for very high-risk behaviors that don't happen to cause harm in the observed instance.  I don't have a theory for how to remove human bias from that part of judgement, but I also don't trust people enough to do without it.

Comment by Dagon on Fire Law Incentives · 2021-07-22T17:43:20.993Z · LW · GW

Beautiful.  I love supervillain storylines where I root against the heroes because the writers haven't done the math.

Comment by Dagon on The Utility Function of a Prepper · 2021-07-22T15:02:18.080Z · LW · GW

Realistic threat modeling takes into account severity and duration, not just probability distribution. "had to request outside supplies" describes normal life for most of us, not a situation to prep for.  

The thing that's hard to model is the wide-scale systemic fragility in the modern world.  Collapse could easily go deeper than expected, and then there's no "outside supplies" to be had.  It's very (very!) hard to predict the specific edges of that scenario that would let your individual preparation be effective.

Comment by Dagon on Matthew Barnett's Shortform · 2021-07-22T14:42:42.096Z · LW · GW

unless there is a chance of punishment or being hurt by victim's self-defense or a chance of better alternative interaction with given person.

There always is that chance.  That's mostly our disagreement.  Using real-world illustrations (murder) for motivational models (utility) really needs to acknowledge the uncertainty and variability, which the vast majority of the time "adds up to normal".  There really aren't that many murders among strangers.  And there are a fair number of people who don't value others' very highly.  

Comment by Dagon on Jaynesian interpretation - How does “estimating probabilities” make sense? · 2021-07-21T22:27:54.985Z · LW · GW

That's my take as well.  "estimating the probability" really means "calculating the plausibility based on this knowledge".

Comment by Dagon on The Utility Function of a Prepper · 2021-07-21T20:56:47.481Z · LW · GW

I doubt it's a difference in utility function (the value placed on a given state of the world), but in the distribution of predicted event.  If you're worried about a sudden event that disrupts everything for a few weeks to months, then surviving that with stored food and defense is what you plan for.  If you're worried about a complete collapse to pre-industrial capacity, then you think about seeds and skills to make rebuilding slightly faster (even if not in your lifetime, perhaps your grandkids).  

What the second group forgets is that even if it's a long-term collapse, there's still 1-10 years of short-term violence and starving people trying to eat your seed stock.  You may not need lots of guns, but you need lots of people you trust who have guns.

Having lots of potable water around is definitely a good start.  That's one of the reasons I argue against tankless water heaters - having 40-60 gallons of stored water is rarely a bad thing.  If you don't have a few weeks' worth of food and medicine that doesn't require electricity to eat, that's also valuable in a very wide range of scenarios.

Comment by Dagon on Why is Kleros valued so low? · 2021-07-21T20:46:56.992Z · LW · GW

Haven't looked, so maybe they solved this, but my primary concern with putting any of my money behind that kind of contract-validation is that I don't understand why we'd believe the predicted evaluation matches the actual evaluation of the contract.  Either the contract is objective and you don't want juries at all, or the contract is subjective but the distribution of test juries is likely very different from the actual jury.

Comment by Dagon on Ask Not "How Are You Doing?" · 2021-07-21T20:42:14.287Z · LW · GW

I often do stop at "hello". I also sometime use "what ho!" or other archaic/amusing variations.  But most of the time some form of "howzit / how are you / how's it going" is a lighweight conventional conversational handoff to let them know it's their turn to speak.  

In many conversations, a more direct/specific question would be offputting and unhelpful.  It adds pressure to be clever or informative, and takes away options to remain brief and lightweight ("fine.  you?").  I'd prefer most social contacts to start out with this negotiation - each participant has an opportunity to inject important/urgent topics, and only if both pass do you push harder for a non-urgent discussion topic.

Note, this is all rationalization, not rationality - I've had bad luck with trying to start conversations more quickly, and this is my reasoning for why that is.  I'm sticking with simple, short, ambiguous/meaningless introductory phrases (mostly - there are LOTS of exceptions.  Don't be boring on an early date, for instance - your goal there is to seem interesting and find topics of mutual conversation quickly.  

Comment by Dagon on Matthew Barnett's Shortform · 2021-07-21T20:30:52.591Z · LW · GW

I'm not sure what our disagreement actually is - I agree with your summary of Ayn Rand, I agree that there are lots of ways to hurt people without stabbing.  I'm not sure you're claiming this, but I think that failure to help is selfish too, though I'm not sure it's comparable with active harm.  

It may be that I'm reacting badly to the use of "truly selfish" - I fear a motte-and-bailey argument is coming, where we define it loosely, and then categorize actions inconsistently as "truly selfish" only in extremes, but then try to define policy to cover far more things.  

I think we're agreed that the world contains a range of motivated behaviors, from sadistic psychopaths (who have NEGATIVE nonzero terms for others' happiness) to saints (whose utility functions weight very heavily toward other's happiness over their own).  I don't know if we agree that "second-order effect" very often dominate the observed behaviors over most of this range.  I hope we agree that almost everyone changes their behavior to some extent based on visible incentives.

I still disagree with your post that a coefficient of 0 for you in someone's mind implies murder for pocket change.  And I disagree with the implication that murder for pocket change is impossible even if the coefficient is above 0 - circumstances matter more than innate utility function.

To the OP's point, it's hard to know how to accomplish "make people less selfish", but "make the environment more conducive to positive-sum choices so selfish people take cooperative actions" is quite feasible.

Comment by Dagon on [deleted post] 2021-07-19T19:32:54.334Z

I was somewhat unimpressed.  It's hard to take discussion of "mainstream economics" very seriously - there are a LOT of different parts of economics study, and this author doesn't explain what theory or policy he's concerned about.

To the extent that economics overlaps with politics, it certainly is subject to many of the same failings - oversimplification, pandering, and selective choice of theory as truth.  I blame politics more than economics for those things.

Comment by Dagon on Matthew Barnett's Shortform · 2021-07-19T18:49:28.251Z · LW · GW

You can't hypothesize zeros and get anywhere.  MANY MANY psychopaths exist, and very few of them find it more effective to murder people for spare change than to further their ends in other ways.  They may not care about you, but your atoms are useful to them in their current configuration.

Comment by Dagon on Matthew Barnett's Shortform · 2021-07-19T18:46:42.206Z · LW · GW

Judgement of evil follows the same pressures as evil itself.  Selfishness feels different from sadism to you, at least in part because it's easier to find cooperative paths with selfishness.  And this question really does come down to "when should I cooperate vs defect".  

Comment by Dagon on Jimrandomh's Shortform · 2021-07-19T18:34:42.851Z · LW · GW

It doesn't need to be linear (both partial-correlation of desires, and declining marginal desire are well-known), but the only alternative to aggregation in incoherency.  

I think you'd be on solid ground if you argue that humans have incoherent values, and this is a fair step in that direction.

Comment by Dagon on Is the argument that AI is an xrisk valid? · 2021-07-19T18:28:26.714Z · LW · GW

MIght not even want to imply that it's the main or only argument.  Maybe "this particular argument is invalid".

Comment by Dagon on Charlie Steiner's Shortform · 2021-07-16T17:54:10.336Z · LW · GW

Do you have a link to that argument?  I think Bayesean updates include either reducing a prior or increasing it, and then renormalizing all related probabilities.  Many updatable observations take the form of replacing an estimate of future experience (I will observe sunshine tomorrow) by a 1 or zero (I did or did not observe that, possibly not quite 0 or 1 if you want to account for hallucinations and imperfect memory).

Anthropic updates are either bayesean or impossible.  The underlying question remains "how does this experience differ from my probability estimate"?   For Bayes or for Solomonoff, one has to answer "what has changed for my prediction?  In what way am I surprised and have to change my calculation?"

Comment by Dagon on Essentialness of Data · 2021-07-14T20:06:16.576Z · LW · GW

I mostly agree, but the underlying difficulty is not technical implementation, but social (and legal) acceptance.  It's almost impossible to explain the topic to a layperson who's worried about it but not very sophisticated.  And it's very hard, even for experts, to define "good" and "bad" uses of anonymized-but-segmented (by business/interest/demographics/etc) data.

Comment by Dagon on Analyzing Punishment as Preventation · 2021-07-14T19:18:36.346Z · LW · GW

I'm not sure you can separate reasons very cleanly, and I'm pretty sure you can't believe individual declarations of their reasons for punishment, let alone groups.  Punishment behaviors and mechanisms have evolved over a very very long time, and there is no simple causality to deconstruct.

also, very minor correction: "Prevention" is the correct word, without the extra "ta".

Comment by Dagon on [Link] Musk's non-missing mood · 2021-07-14T01:57:17.883Z · LW · GW

To an individual human, death by AI (or by climate catastrophe) is worse than old age "natural" death only to the extent that it comes sooner, and perhaps in being more violent.  To someone who cares about others, the large number of looming deaths is pretty bad.  To someone who cares about the species, or who cares about quantity of sentient individuals, AI is likely to reduce total utility by quite a bit.

To someone who loves only abstract intelligence and quantifies by some metric I don't quite get, AI may be just as good as (or better than) people.

Comment by Dagon on Reasons for Punishment · 2021-07-12T22:26:09.193Z · LW · GW

I think there are a number of non-rational (or pre-rational) motivations that drive many forms of punishment as well.

  • Revenge.  Evolved/Naive tit-for-tat strategy.
  • Envy.  Reduction in feeling the unfairness of someone getting away with something.
  • Spite.  Any chance to hurt someone, with some socially-acceptible justification.
  • Focus on "other".  Punishment is a good way to frame the target as out-group, and to be closer friends with the people helping to punish.
Comment by Dagon on A low-probability strategy to elminate suffering via hostile takeover of a publically traded corporation · 2021-07-12T02:09:02.199Z · LW · GW

what would you sacrifice to eliminate all suffering for all humans for all time, with, say, a 1% chance of success? What about 10%? 50%? 0.01%?

I've already invested more effort in this than it seems worth.  

Comment by Dagon on The More Power At Stake, The Stronger Instrumental Convergence Gets For Optimal Policies · 2021-07-11T18:06:38.483Z · LW · GW

I wonder if you can (or should) make "power-seeking" a multidimensional factor - seeking power over some aspects of action and not needing it for others.   To the extent that an agent is aligned with another, the power relationship is irrelevant - they're working together to seek the same states of the universe anyway.

In other words, "power" is really just the ability to enforce some amount of behavioral alignment on others.  Power-seeking is obviously useful in a sea of unaligned humans, as you can force them to act more like how they would if they were aligned with you.  

Edit: Thanks for pointing out my misunderstanding, Pattern.  I mistakenly took "power-seeking" to mean mostly social power, rather than general prediction/optimization power.  

Comment by Dagon on Agency and the unreliable autonomous car · 2021-07-10T15:28:01.379Z · LW · GW

Why are you talking about emulation?

That's an excellent question - I don't know if the connection between formal proof and emulation/reflection exists anywhere outside of my mind.  I believe my arguments hold for the impossibility of proving something without additional resources over just calculating it (possibly using a method that has proofs about it's correctness, which happened outside the computation itself).

Comment by Dagon on Improving capital gains taxes · 2021-07-10T15:18:03.264Z · LW · GW

Oh, I should note that progressive taxation is something I very much dislike.  The accounting needs of tracking everything to figure out marginal rates is horrific.  Linear consumption taxes are way simpler, and progressiveness (if needed, I'd argue it's not) should be done by rebates or payments, not by tax rate differential.

Comment by Dagon on A world in which the alignment problem seems lower-stakes · 2021-07-09T23:22:16.848Z · LW · GW

Still not on board with the value of this.  Why would you expect an AGI that does no harm (as far as you know) in an unpopulated and unobserved portion of the universe also does no harm on Earth (where you keep your stuff, and get the vast majority of your utility, but also with a radically different context - nearly unrelated terms in your utility function).

Comment by Dagon on Dagon's Shortform · 2021-07-09T23:07:40.268Z · LW · GW

There's a lot of similarity between EMH (efficient markets hypothesis) and Aumann's agreement theorem.

  1. Both are about updating on other agents' information.
  2. Both make pretty strong statements about common knowlege.
  3. Neither one applies in the interesting cases.
Comment by Dagon on Improving capital gains taxes · 2021-07-09T19:46:17.613Z · LW · GW

What's the next-best option?  I'm trying to understand why you want to improve this part of income taxation, rather than a more radical change to taxation mechanisms.  If major changes are possible, we should drop income taxation entirely, in favor of consumption taxes (value-added tax has some VERY nice features in terms of simplicity and spread of distortion), pigouvean taxes (for things where we seek distortion), and transfer taxes (for gifts and inheritance, any non-arms-length transaction).

Comment by Dagon on For reducing caffeine dependence, does daily maximum matter more, or does total daily intake matter more? · 2021-07-09T19:39:19.404Z · LW · GW

Ah, I see.  It's been a long time since I've tried to moderate my consumption, but when I did (and when friends have done so), the metric was "total consumption per day", so I don't have much evidence either way whether the distribution across time matters.  The preferred way to reduce was to cut out non-morning intake, then to reduce late-morning intake, then to taper and remove initial-morning intake.  This can be done by brewing a mixed decaf/caf blend, and making it weaker (more decaf) after your first. gives some data about metabolic clearing rate of caffeine (1.5 to 9.5 hour half-life, not very helpful, but if you're at the midpoint of that (5.5 hours to clear half of the caffeine in your bloodstream), that implies your afternoon dose is on top of significant remnants of the morning, and "total daily dose" matters more than "max ingestion at any one time".  

Comment by Dagon on For reducing caffeine dependence, does daily maximum matter more, or does total daily intake matter more? · 2021-07-09T17:04:24.425Z · LW · GW

If you're able to adjust by 20mg/3 days, it seems this question can just be tested directly.  I look forward to hearing how it goes, and how it compares with any advice you get.

Comment by Dagon on A world in which the alignment problem seems lower-stakes · 2021-07-08T22:39:34.679Z · LW · GW

Sure, but the stakes are embedded in U.  The whole post would be simpler to just say "an AGI that can't impact my utility is low stakes".   The partitioning of the universe only "works" if you don't attempt to claim that U(left) is significant.   To the extent that you CARE about what happens in that part of the universe, your utility is impacted by the AGI's alignment or lack thereof.

Comment by Dagon on A world in which the alignment problem seems lower-stakes · 2021-07-08T22:19:52.084Z · LW · GW

If "you and your civilization" has an extreme privilege in your utility function, that means the AI is safer when it's partitioned away from you, but it also means it doesn't generate much utility in the first place.  that's not changing the problem, it's just setting the coefficient of u(right) very low.  

I imagine you can reduce the stakes of any alignment problem by not caring about the outcome.

Comment by Dagon on A world in which the alignment problem seems lower-stakes · 2021-07-08T20:31:51.318Z · LW · GW

Thanks, so in this thought experiment, you influence an AI such that you justify (to yourself) imagining a nicer future for unknowable parts of the universe?  

I suspect for most of us, it's cleaner to model it as utility of your perceived/expected state of the universe(s) than a joint utility over multiple universes, but I think I understand what you're saying, at least.  

I'm torn about whether this seems lower-stakes or not.  There's a whole lot in my light cone which I theoretically can, but actually don't watch.  My utility-from-imagination is the same for those areas as for the truly inaccessible ones.  Thus, errors in specification or implementation of the AGI seem to have as big an impact in your imagined part of the universe as they would in your part of reality.  It may be lower-stakes if your imagination is flawed and you never believe there was an error, but then it seems wireheading would be easier.

Comment by Dagon on Agency and the unreliable autonomous car · 2021-07-08T16:33:18.387Z · LW · GW

for a given set of resources (cpu-time or instruction count, reads/writes/total storage, etc.), there are computations that can be done directly, which cannot be done in an emulator which takes some of those resources.  

There is some amount of underlying useful work that's being done (calculating expected value of hypothetical actions) which is feasible to directly calculate, and infeasible to calculate the calculation. 

When the useful work IS the emulation, then of course it's using it's full power.  But it can't emulate and verify the emulation/verification (without additional resources).

Comment by Dagon on A world in which the alignment problem seems lower-stakes · 2021-07-08T15:58:54.568Z · LW · GW

I don't follow the half-universe argument.  Are you somehow sending the AGI outside of your light-cone?  Or have you crafted the AGI utility function and altered your own to not care about the others' half?  I don't get the model of utility that works for 

The only information you have about the other half is your utility.

My conception of utility is that it's a synthetic calculation from observations about the state of the universe, not that it's a thing on it's own which can carry information.