Dagon's Shortform 2019-07-31T18:21:43.072Z · score: 3 (1 votes)
Did the recent blackmail discussion change your beliefs? 2019-03-24T16:06:52.811Z · score: 37 (14 votes)


Comment by dagon on The "Commitment Races" problem · 2019-08-23T14:20:34.194Z · score: 2 (1 votes) · LW · GW

I think you're missing at least one key element in your model: uncertainty about future predictions. Commitments have a very high cost in terms of future consequence-effecting decision space. Consequentialism does _not_ imply a very high discount rate, and we're allowed to recognize the limits of our prediction and to give up some power in the short term to reserve our flexibility for the future.

Also, one of the reasons that this kind of interaction is rare among humans is that commitment is impossible for humans. We can change our minds even after making an oath - often with some reputational consequences, but still possible if we deem it worthwhile. Even so, we're rightly reluctant to make serious committments. An agent who can actually enforce it's self-limitations is going to be orders of magnitude more hesitant to do so.

All that said, it's worth recognizing that an agent that's significantly better at predicting the consequences of potential commitments will pay a lower cost for the best of them, and has a material advantage over those who need flexibility because they don't have information. This isn't a race in time, it's a race in knowledge and understanding. I don't think there's any way out of that race - more powerful agents are going to beat weaker ones most of the time.

Comment by dagon on Time Travel, AI and Transparent Newcomb · 2019-08-22T23:01:57.650Z · score: 2 (1 votes) · LW · GW

Let us suppose <impossible thing>. Now <impossible result> remains impossible, how? Maybe the universe has a mysterious agency we can trick or bargain with!

I think you'll need to back up a bit further if you want to explore this. "time travel is possible" isn't well enough defined to be able to reason about, except in the human conceptual space with no physics attached. And if you're assuming away physics, you don't need to explain anything, just let the paradoxes happen.

Comment by dagon on Davis_Kingsley's Shortform · 2019-08-20T23:51:26.276Z · score: 2 (1 votes) · LW · GW

I can't tell if this is just another example that strategic choices tend to be valuable (guaranteed non-negative, but in practice usually positive). OF COURSE an opponent's choice is going to reduce your value in a zero-sum game.

I do want to warn against applying to other aspects of life that aren't purely zero-sum and aren't designed by a human to balance the power between both parties. See also

Comment by dagon on Goodhart's Curse and Limitations on AI Alignment · 2019-08-20T23:24:09.907Z · score: 4 (2 votes) · LW · GW

I don't understand why doesn't get more consideration. In a complex interconnected system, V can not only be much less than E, it can be much less than would be obtained with ~C. You may not get mere utopia, you may get serious dystopia.

Comment by dagon on What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause? · 2019-08-20T23:03:30.749Z · score: 5 (6 votes) · LW · GW

Other reasons that people may have (I have some of these reasons, but not all):

  • not a classical utilitarian
  • don't believe those timelines
  • too distant to feel an emotional tie to
  • unclear what to do even if it is a priority
  • very high discount rate for future humans
  • belief that moral value is relative with cognitive ability (an extremely smart AI may be worth a few quitillion humans in a moral/experiential sense)

Of these, I think the one that I'm personally least moved by while acknowleging it as one of the better arguments against utilitarianism is the last. It's clear that there's SOME difference in moral weight for different experiences of different experiencers. Which means there's some dimension on which a utility monster is conceivable. If it's a dimension that AGI will excel on, we can maximize utility by giving it whatever it wants.

Comment by dagon on Swimmer963's Shortform · 2019-08-20T17:08:13.540Z · score: 2 (1 votes) · LW · GW

(don't write fiction, but have run and playtested a lot of RPGs, which share many of the worldbuilding elements).

Among the hard parts is figuring out how much suspension of disbelief your audience will willingly bring, on what topics. This _is_ fiction, so we're not generally trying to truly predict a hypothetical "possible" outcome, we're trying to highlight similarities and differences from our own. This VERY OFTEN implies assuming a similarity (where the point of departure has less effect that is likely) and then justifying it or constraining the departure so it's less difficult to maintain that this element of society would still be recognizable.

Comment by dagon on Negative "eeny meeny miny moe" · 2019-08-20T16:14:48.591Z · score: 3 (3 votes) · LW · GW

Ehn. For kids who will EVER accept this as fair, you're putting too much thought into politics. If the kids are this manipulable, they'll probably accept your authority in the one-shot case as well.

Also, more iterations gives them more time to realize that you're cheating (by shifts in how to count syllables) or that the game is fully deterministic (and you're cheating by deciding who to start with).

This is only usable for such low-stakes cases where the participants don't mind that it's not fair. And in those cases, don't waste time on pointless complexity. Of course, if this is part of the entertainment, I reverse that advice - choose the single-elimination method to extend the duration of the tension of not knowing.

Comment by dagon on Do We Change Our Minds Less Often Than We Think? · 2019-08-19T23:37:19.621Z · score: 9 (5 votes) · LW · GW

This question is very sensitive to reference classes and definitions. I change my estimates of my future choices very very often, but the vast majority of my decisions are too trivial to notice that I'm doing so. Yesterday at breakfast I thought I'd probably have tacos for dinner. I didn't.

For decisions that seem more important, I spend more time on them, and ALSO probably change my mind less often than I intend to. The job-change example is a good one: I usually know what I want after the first few conversations, but I intentionally force myself to consider other alternatives and collect data to be more confident in the decision. Part of my tactics for doing that research and consideration is to understate the chance that I'll pick the leading option (semi-intentionally; it's motivated by wanting to make a more reasoned decision in the future, not an honest neutral prediction, but I'd admit that if pressed).

I don't change my overall values or relationship tenets very often at all, but I do change priorities and activities a whole lot, and am often surprised by what seems best at the time, compared to when planning.

Comment by dagon on Beliefs Are For True Things · 2019-08-17T18:51:16.610Z · score: 4 (2 votes) · LW · GW

I think this is an important point. There's a false dichotomy (and a lossy reduction in dimensionality) in "you can believe true things or you can believe useful things". You can and should strive for useful true beliefs. If you're not , you likely have some useless beliefs, and it's _really_ hard to define "true" for beliefs that don't actually do anything.

You shouldn't believe falsehoods full stop. Falsehoods are not useful beliefs, as they make incorrect predictions. You ALSO shouldn't spend a whole lot of effort on truth of beliefs that don't matter. You should have a LOT of topics where you don't have strong beliefs.

Comment by dagon on Matthew Barnett's Shortform · 2019-08-17T15:15:06.639Z · score: 2 (1 votes) · LW · GW

I like your phrasing better than mine. "only" is definitely too strong. "most likely path to"?

Comment by dagon on Matthew Barnett's Shortform · 2019-08-16T23:05:20.622Z · score: 2 (1 votes) · LW · GW

A lot depends on your model of progress, and whether you'll be able to predict/recognize what's important to understand, and how deeply one must understand it for the project at hand.

Perhaps you shouldn't frame it as "study early" vs "study late", but "study X" vs "study Y". If you don't go deep on math foundations behind ML and decision theory, what are you going deep on instead? It seems very unlikely for you to have significant research impact without being near-expert in at least some relevant topic.

I don't want to imply that this is the only route to impact, just the only route to impactful research.
You can have significant non-research impact by being good at almost anything - accounting, management, prototype construction, data handling, etc.

Comment by dagon on Matthew Barnett's Shortform · 2019-08-16T21:57:33.226Z · score: 3 (2 votes) · LW · GW

Beware motivated reasoning. There's a large risk that you have noticed that something is harder for you than it seems for others, and instead of taking that as evidence that you should find another avenue to contribute, you convince yourself that you can take the same path but do the hard part later ( and maybe never ).

But you may be on to something real - it's possible that the math approach is flawed, and some less-formal modeling (or other domain of formality) can make good progress. If your goal is to learn and try stuff for your own amusement, pursuing that seems promising. If your goals include getting respect (and/or payment) from current researchers, you're probably stuck doing things their way, at least until you establish yourself.

Comment by dagon on Matthew Barnett's Shortform · 2019-08-16T18:40:06.480Z · score: 2 (3 votes) · LW · GW

I think you might be goodhearting a bit (mistaking the measure for the goal) when you claim that final exam performance is productive. The actual product is the studying and prep for the exam, not the exam itself. The time limits and isolated environment is helpful in proctoring (it ensures the output is limited enough to be able to grade, and ensures that no outside sources are being used), not for productivity.

That's not to say that these elements (isolation, concentration, time awareness, expectation of a grading/scoring rubric) aren't important, just that they're not necessarily sufficient nor directly convertible from an exam setting.

Comment by dagon on Partial summary of debate with Benquo and Jessicata [pt 1] · 2019-08-15T23:36:01.518Z · score: 3 (4 votes) · LW · GW
The problem with having the conversation in public is precisely that other people will be asking "wait, what precious thing, exactly?" which derails the high context conversation.

I get that, but if the high-context extensive private conversation doesn't or can't) identify the precious thing, it seems somewhat likely that either you're both politely accepting that the other may be thinking about something else entirely, and/or it may not actually be a thing.

I very much like your idea that you should have the conversation with the default expectation of publishing at a later time. If you haven't been able to agree on what the thing is by then, I think the other people asking "wait, what precious thing exactly" are probably genuinely confused.

Note that I realize and have not resolved the tension between my worry that indescribable things aren't things, and my belief that much (and perhaps most) of human decision-making is based on illegible-but-valid beliefs. I wonder if at least some of this conversation is pointing to a tendency to leak illegible beliefs into intellectual discussions in ways that could be called "bias" or "deception" if you think the measurable world is the entirety of truth, but which could also be reasonably framed as "correction" or "debiasing" a limited partial view toward the holistic/invisible reality. I'm not sure I can make that argument, but I would respect it and take it seriously if someone did.

Comment by dagon on Partial summary of debate with Benquo and Jessicata [pt 1] · 2019-08-15T20:41:55.966Z · score: 5 (7 votes) · LW · GW

(note: this is more antagonistic than I feel - I agree with much of the direction of this, and appreciate the discussion. But I worry that you're ignoring a motivated blind spot in order to avoid biting some bullets).

So, there's something precious that dissolves when defined, and only seems to occur in low-stakes conversations with a small number of people. It's related to trust, ability to be wrong (and to point out wrongness). It feels like the ability to have rational discourse, but that feeling is not subject to rational discourse itself.

Is it possible that it's not truth-seeking (or more importantly, truth itself) you're worried about, but unstated friendly agreement to ignore some of the hard questions? In smaller, less important conversations, you let people get away with all sorts of simplifications, theoretical constructs, and superficial agreements, which results in a much more pleasant and confident feeling of epistemic harmony.

When it comes time to actually commit real resources, or take significant risks, however, you generally want more concrete and detailed agreement on what happens if you turn out to be incorrect in your stated, shared beliefs. Which indicates that you're less confident than you appear to be. This feels bad, and it's tempting for all participants to now accuse the other of bad faith. This happens very routinely in friends forming business partnerships, people getting married, etc.

Maybe it's not a loss in truth-seeking ability, it's a loss of the ILLUSION of truth-seeking ability. Humans vary widely in their levels of rationality, and in their capability to hold amounts of data and make predictions, and in their willingness to follow/override their illegible beliefs in favor of justifiable explicit ones. It's not the case that the rationalist community is no better than average: we're quite a bit better than average (and conversations like this may well improve it further). But average is TRULY abysmal.

I've long called it the "libertarian dilemma": agency and self-rule and rational decision-making is great for me, and for those I know well enough to respect, but the median human is pretty bad at it, and half of them are worse than that. When you're talking about influencing other people's spending decisions, it's a really tough call whether to nudge/manipulate them into making better decisions than they would if you neutrally present information in the way you (think you) prefer. Fundamentally, it may be a question of agency: do you respect people's right to make bad decisions with their money/lives?

Comment by dagon on Partial summary of debate with Benquo and Jessicata [pt 1] · 2019-08-15T18:41:29.279Z · score: 7 (6 votes) · LW · GW

This is awesome! I cannot sufficiently express my admiration for trying to make these kinds of discussions transparent and accessible.

There's a lot of surface area in this, even in the summary, so I don't think I can do justice in a comment. I'll instead just highlight a few things that resonated or confused me.

  • I don't know if the ambiguity is intentional, but I'm put off by statements like "billions of dollars at stake". If it's just a shorthand for "now it's real, and has real-world impact", fine, but different readers will have very different emotional reactions to that framing. I can read it as semi-literal "things we say have some amount of influence over people who will spend billions of dollars in the next few years", direct-literal "we have a budget in the billions that we're directly allocating", or hyperbolic "over the course of time, small improvements in thinking can shift billions of dollars in acting", and those three interpretations somewhat change my reaction to the rest of the discussion. Also, if it's the last one, you're thinking too small. Billions isn't that much in today's world - if you can solve politics, you're influencing at least trillions, and potentially beyond countable money.
  • It's not clear (but you don't list it as an important crux, so I'm confused), what that "precious thing" actually is. It may be the ability to be publicly admit epistemic uncertainty while still having public support (and money). It may be the ability to gracefully give up resources when it turns out you're wrong. It may be the ability to call out errors (motivated or not, systemic or not) without defensive counter-accusations. It may be an assumption of cooperation, so not much effort needs to be spent on skepticism of intent (only on cooperative skepticism of data and models). It may be something else I haven't identified. I suspect these are all related, but will need to be addressed separately.
  • The public vs private aspect is, I believe, far more critical than you seem to give it credit for. Especially if the $billions is other peoples' money, you'll need to deal with the fact that you're discussing other peoples' rationality (or lack thereof). If so, this is squarely in territory. And deeply related to questions of elitism vs populism, and how to accept the existence of power dynamics that are based on different criteria than you'd like.

Comment by dagon on Adjectives from the Future: The Dangers of Result-based Descriptions · 2019-08-14T17:00:55.013Z · score: 3 (2 votes) · LW · GW
Why not minimize the manipulation by describing both the intent and the means

I believe, in most cases, this actually happens when you read/discuss beyond the headline. Use more words, actually put effort into understanding rather than just assuming the the 2-4 word description is all there is.

In the examples you give, it would be somewhat misleading to describe both motive and method - "weight-loss program" doesn't specify mechanism because it applies to a lot of different mechanisms. The person describing it wants to convey the intent, not the mechanism - that detail is important for some things, and not for others, so it's left to you to decide if you want it. "Against Malaria" likewise. They believe that the right tactic is mosquito nets, but if things change and that stops being the case, they don't intend to change their mission or identity in order to use laser-guided toads or whatever.

Comment by dagon on Adjectives from the Future: The Dangers of Result-based Descriptions · 2019-08-13T23:44:53.200Z · score: 3 (2 votes) · LW · GW

I'd characterize these as "intent-description" as opposed to "activity-description". And I think the underlying problem is the compression inherent in short, catchy phrases to describe a complex endeavor that includes thousands or more people working on it. Famously and only somewhat tongue-in-cheek, one of the two unsolved computer science problems is "naming things".

Failure to look into the model and activity behind any intended consequence will leave you open to manipulation and incorrect expectations. Failure to look at the intent can lead you to ignore the possibility that tactics and methods might need to change, and how aware the org is of that.

Comment by dagon on Hazard's Shortform Feed · 2019-08-13T21:43:55.810Z · score: 4 (2 votes) · LW · GW

This happens on LW as well, fairly often. It's hard to really introduce a topic in a way that people BELIEVE you when you say you're exploring concept space and looking for ideas related to this, rather than trying to evaluate this actual statement. It's still worth trying to get that across when you can.

It's also important to know your audience/discussion partners. For many people, it's entirely predictable that when you say "I'm thinking about ... get everyone on patreon" they will react to the idea of getting their representation of "everyone" on their ideas of "patreon". In fact, I don't know what else you could possibly get.

It may be better to try to frame your uncertainty about the problem, and explore that for awhile, before you consider solutions, especially solutions to possibly-related-but-different problems. WHY are you thinking about funding and revenue? Do you need money? Do you want to give money to someone? Do you want some person C to create more content and you think person D will fund them? It's worth it to explore where Patreon succeeds and fails at whatever goals you have, but first you have to identify the goals.

Comment by dagon on Natural laws should be explicit constraints on strategy space · 2019-08-13T20:50:32.381Z · score: 10 (4 votes) · LW · GW

I'm not sure I understand your recommendation. You talk about pilot as a constraint and the obvious removal of the constraint (unmanned fighters). This is the opposite of a natural law: it's an assumed constraint or a constraint within a model, not a natural law.

I think " We have a good command of natural law at the scale where warmachines operate. " is exactly opposite of what I believe. We have some hints as to natural law in those scales, but we're nowhere near those constraints. There are a huge number of contingent constraints in our technology and modeling of the problem, which are very likely overcome-able with effort.

[edit after re-reading]

Do you mean "_only_ natural laws should be explicit constraints"? You're recommending that if we think we're constrained and can't identify the natural law that's binding, the constraint is probably imaginary or contingent on some other thing we should examine?

Comment by dagon on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T17:14:33.574Z · score: 10 (3 votes) · LW · GW

May I ask for a resolution comment (or follow-up questions if not resolved) when you've decided that this question has sufficient answers to make a decision or summarize a consensus?

It's not fair to pick on this one, and I apologize for that, but this is one of a number of recent topics that generate opinions and explore some models (some valuable, many interesting), but then kind of die out rather than actually concluding anything.

Comment by dagon on Dony's Shortform Feed · 2019-08-12T16:39:50.619Z · score: 2 (1 votes) · LW · GW

Beware over-generalization and There's a LOT of variation in human capabilities and preferences (including preferences about productivity vs rest). Some people do have 100-hour workweeks (I did for awhile, when I was self-employed. )

Try it, see how it works for you. If you're in a position of leadership over others, give them room to find what works best for them.

Comment by dagon on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T13:38:16.468Z · score: 4 (2 votes) · LW · GW

It doesn't require you to know the right people, it requires you to expend effort to determine the right people, and then to convince THOSE less-busy people that the referral is valuable.

For many, they have assistants or employees who perform this as an actual task - filter the volume of contacts and handle most things, escalating those that warrant it. That's great. For others, this is more informal - they have other communication channels like LessWrong, or twitter, or social network meshes, and you can get their attention by getting the attention of any of their friends or posting interesting stuff on those channels.

Either way (or ways in between and outside this), it uses the community to signal value of communication between individuals, rather than only discrete per-message signals that ignore any context.

Basically, there are two cases:

1) the recipient will want to talk with you, but doesn't know it. In this case, you need to show that you're interesting, not that you're interested. Spending money isn't interesting. Being interesting to people around me is interesting.

2) the recipient won't care, even after reading. In this case, money may compensate for their time, but probably not and it doesn't get you the attention you want anyway. A useless reply isn't worth their time nor your money.

Note that I'm assuming you're talking about trivial amounts of money (less than full-time equivalent pay for their time), and for more than a trivial form-letter response to collect the bounty. I'd be very interested in a SINGLE concrete example where any amount of money is a good value for both parties who wouldn't otherwise connect. Ideally, you'd give two examples: one of someone you wouldn't respond to without your $5, and one of someone who's not responding to you, who you'd pay $X to do so (including what X you'd pay and what kind of response would qualify).

After some more thought, I think my main objection is that adding small amounts of money to a communication is a pretty strong NEGATIVE signal that I want to read the communication. I want to read interesting things that lead to more interesting things. The fact that someone will pay to have me read it is an indication that I don't want to read it otherwise.

Comment by dagon on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T00:25:14.984Z · score: 2 (1 votes) · LW · GW

I will happily accept payment for reading and responding to e-mail. I will not pay to send one, and I don't know of any cases where I feel the need to pay for someone's initial reading of an e-mail (I may want to pay for their attention, but that will be a negotiation or fee for a thing, not for a mail).

What _might_ be valuable is a referral service - a way to have someone who (lightly) knows you and who (somewhat) knows the person you want to correspond with, who can vouch for the fact that there's some actual reason not to ignore your mail. No payment in money, some payment (and reinforcement) in reputation.

Basically, e-mail isn't the problem, the variance in quality of things for me to look at is the problem. Curation is the answer, not payment.

Comment by dagon on Matthew Barnett's Shortform · 2019-08-09T20:22:39.442Z · score: 2 (1 votes) · LW · GW

The problem is, if a conversational topic can be hurtful, the meta-topic can be too. "do you want to talk about the test" could be as bad or worse than talking about the test, if it's taken as a reference to a judgement-worthy sensitivity to the topic. And "Can I ask you if you want to talk about whether you want to talk about the test" is just silly.

Mr-hire's comment is spot-on - there are variant cultural expectations that may apply, and you can't really unilaterally decide another norm is better (though you can have opinions and default stances).

The only way through is to be somewhat aware of the conversational signals about what topics are welcome and what should be deferred until another time. You don't need prior agreement if you can take the hint when an unusually-brief non-response is given to your conversational bid. If you're routinely missing hints (or seeing hints that aren't), and the more direct discussions are ALSO uncomfortable for them or you, then you'll probably have to give up on that level of connection with that person.

Comment by dagon on Why do humans not have built-in neural i/o channels? · 2019-08-09T16:14:12.711Z · score: 2 (1 votes) · LW · GW

Fair enough - I underestimate the power of evolution at my epistemic peril. My point remains: more direct communication (unfiltered by many levels of decoding and processing) could easily be more harmful than helpful.

Aside from snow crash / basilisk scenarios (which are yet un-demonstrated), language and vision are pretty safe, as they're filtered through a lot of neural systems to find and pay special attention to surprising things. This is slow, but makes it way harder to trick than a more direct interface would.

Some drugs are an example of more direct impacts that are available today. If there were such a thing that's actually guided by a human specifically to alter your mind in ways desired by the communicator, it would be quickly abused and removed.

Comment by dagon on Why do humans not have built-in neural i/o channels? · 2019-08-08T20:42:20.887Z · score: 5 (5 votes) · LW · GW

I'm surprised nobody has mentioned risks of contagion and attack. Internal neural links are subject to viral and autoimmune problems, but there's at least a commonality of DNA on both ends of the link, so a shared evolutionary fate and at least a basis for trust.

Anyone who's done any infosec or network protocol work will laugh at the idea that trial and error (evolution) can make a safe high-bandwidth connection.

Comment by dagon on [Site Update] Behind the scenes data-layer and caching improvements · 2019-08-08T06:15:14.701Z · score: 6 (3 votes) · LW · GW

"something else weird" has my current vote. It's pretty snappy from my home machine, and was very slow from my work network. I'm going to assume it's their problem rather than yours until I see more evidence.

Comment by dagon on [Site Update] Behind the scenes data-layer and caching improvements · 2019-08-07T22:25:59.547Z · score: 5 (2 votes) · LW · GW

Congrats on continuing to make improvements! Unfortunately, this one seems like a performance regression - it's taking a long time to expand comments when I click a green count on the /daily page, and the notifications list seemed to be slower as well. Loading post pages (I ctrl-click to open in new tab very often) has always been a bit slow (2-5 seconds) and seems worse now (I just timed one at 7 seconds).

Comment by dagon on Weak foundation of determinism analysis · 2019-08-07T21:48:48.006Z · score: 1 (3 votes) · LW · GW
If someone goes so far as to alter their decisions based on their belief in determinism

You're mixing levels. If someone can alter their decisions, that implies there are multiple possible next states of the universe, and that strict determinism is wrong. If the universe is actually determined, nobody alters any decisions, they just experience the decisions they are calculated to make.

Comment by dagon on Weak foundation of determinism analysis · 2019-08-07T20:02:51.103Z · score: 2 (1 votes) · LW · GW

I mean to say that the idea of cause and effect is ... different, in a purely deterministic universe. Every state of the universe exists in a 4D (or more) volume, the idea that time is different from other dimensions is illusory. There is one state at a time and another state at another time. This may be what you mean by "timeless universe" - if every state is determined, then time is no different from distance.

Much like we can call configurations of atoms that don't have a lot of empty space between them "joined", we can call clusters earlier in time "causal" for those later in time. But this is map, not territory. There is likely some underlying complexity-reduction that does mean the whole shebang (universe from big bang to heat death) comprises fewer bits than a naive encoding, and this can be modeled as causality - patterns of time-adjacent configurations. You can draw the graph that models the events in a particular way, but that's just a modeling choice rather than a reality.

(and to reiterate, this isn't my preferred conceptualization of the universe, but I can't think how to disprove it. Same for simulation or Boltzmann brains or other acknowledgement that my perception and memory is darned limited - I don't prefer them as guiding models, but I can't disprove them. I'll even agree that determinism is prima facie absurd, much like the earth being round or the implications of special relativity are absurd. )

Comment by dagon on Dagon's Shortform · 2019-08-07T16:43:47.705Z · score: 5 (3 votes) · LW · GW

"One of the big lessons of market design is that participants have big strategy sets, so that many kinds of rules can be bent without being broken. That is, there are lots of unanticipated behaviors that may be undesirable, but it's hard to write rules that cover all of them. (It's not only contracts that are incomplete...) " -- Al Roth

I think this summarizes my concerns with some of the recent discussions of rules and norm enforcement. People are complicated and the decision space is much larger than usually envisioned when talking about any specific rule, or with rules in general. Almost all interactions have some amount of adversarial (or at least not-fully-aligned) beliefs or goals, which is why we need the rules in the first place.

Comment by dagon on Weak foundation of determinism analysis · 2019-08-07T16:07:13.367Z · score: 2 (1 votes) · LW · GW
If belief in determinism causes someone to make poorer choices

If determinism is actually true, this condition is false. Belief in determinism is correlated with worse outcomes, but one doesn't cause the other; both are determined by the state and process of the universe. Minds thinking about counterfactuals is just part of the (determined) universe, and their conclusions are also determined. Free will is an illusion that some kinds of brains create after the fact to explain what happened to themselves.

Note that I don't actually know this is true, nor how I'd prove or disprove it. My intuition (which may be just a side-effect of the universe) is that there is something about me that makes decisions which influence future experiences. Since I can't think of any evidence that would shift my beliefs, I'm going to call it a modeling choice rather than a truth. I prefer (or am destined to prefer) a semi-free choice model, where there is a thing that has beliefs which correlate with the branch of the universe it finds itself experiencing. It doesn't matter whether counterfactual me experience something else, or if they don't exist, or if the causality is illusory. The correlation is so strong and clear that I'm forced to (heh) act like it's causal.

Comment by dagon on benwr's unpolished thoughts · 2019-08-06T23:18:04.729Z · score: 7 (3 votes) · LW · GW

That's helpful, I hadn't looked at that page, I generally just look at /daily, and sometimes at the main page for recommendations and recently curated.

Comment by dagon on Power Buys You Distance From The Crime · 2019-08-06T20:59:09.177Z · score: 4 (2 votes) · LW · GW

I don't find either of your main examples (taxes or blame apportionment) particularly compelling, and gave some reasons for that. And this makes me less likely to accept your thesis that power allows an incorrect perception of moral distance, or that it (necessarily) obscures information flow.

There probably is a relationship in there - power as a measure of potential impact on almost any topic means that power can do these things. It's not clear that it automatically or always does, nor that power is the problem as opposed to bad intentions of the powerful.

Comment by dagon on benwr's unpolished thoughts · 2019-08-06T16:33:13.878Z · score: 4 (2 votes) · LW · GW

I'm loving the shortform feature, and I'd appreciate enhancements to help me find those which I'd read before but have active comment threads, and I'd certainly like a "watch" feature for posts, comment trees, and shortform topics. I don't want (I'm not sure if I object, or just would not use) person-based feeds or filters. There are some posters I default to reading and most that I read based on topic, but I prefer to bias toward the latter.

Comment by dagon on Permissions in Governance · 2019-08-06T13:33:57.902Z · score: 2 (1 votes) · LW · GW

Ah, I think we need a more detailed model of what it means to want something. What a person says they want, what they think they want, and what they actually want at any given moment may differ. As verbal manipulators, humans tend to focus on what is said, but it's hard to see how that's actually the correct one.

If a group decided that it wants X, and the individual member doesn't want X enough to actually do it, the definition of "decided" seems to be in question. Maybe some members want X more than others.

(yes, I'm being a bit intentionally obtuse. I do want to be explicit when we're talking about coercion of others in order to meet your goals, as opposed to examining our own goals and beliefs. )

Comment by dagon on Raemon's Scratchpad · 2019-08-05T23:10:30.705Z · score: 5 (3 votes) · LW · GW

I think lack of long-term contract enforcement is one part of it - the US congress routinely passes laws with immediate costs and delayed revenue, and then either continually postpones or changes it's mind on the delayed part (while keeping the immediate part). I'd classify it as much as deception as of lack of enforcement. It's compounded by the fact that the composition of the government changes a bit every 2 years, but the fundamental problem is that "enforcement" is necessary, because "alignment" doesn't exist.

Trying to go meta and enforce far-mode stated values rather than honoring near-mode actual behaviors is effectively forcing people into doing what they say they want, as opposed to inferring what they actually want. I'm actually sympathetic to that tactic, but I do recognize that it's coercion (enforcement of ill-considered contract) rather than actual agreement (where people do what they want, because that's what they want).

Comment by dagon on Dagon's Shortform · 2019-08-05T22:36:50.675Z · score: 3 (2 votes) · LW · GW

I opted in to seeing downvotes, and I think my curiosity (both about personal interactions on the site and about the general topic of mechanism design) will compel me to continue. I'm not worried about them - as I say, they were low-value comments, though not much different from others that get upvoted more (and IMO shouldn't be). Mostly it shows that low-number-of-voters is mostly noise, and that's fine.

My main point was wondering how I can monitor my own voting behavior? (ok, and a little complaining that variable-point voting is confusing.) I think I upvote pretty liberally and downvote very rarely, but I really wonder if that's true, and I wonder things like how many votes I give (on multiple comments) on the same post.

Comment by dagon on Raemon's Scratchpad · 2019-08-05T20:49:14.770Z · score: 2 (1 votes) · LW · GW
My guess is to frame things in terms of skills to learn or particular attributes to acquire.

IMO, even this is too status-ey and centered on attributes of the person rather than crux-ey and centered on the discussion you want to have.

Frame things in terms of models of thinking and level of abstraction/generalization to apply here and now. There may be skills to learn (or even attributes that can't be acquired, making the conversation at that level impossible) in order to get there, but start with what you want to understand/communicate, not with an assumption of capability (or lack thereof).

Doing this is also a reminder that sometimes washing the dishes is just the fastest way to empty the sink - generalizing to some idealized division of labor and social reward scheme doesn't have to happen every time. It often works better to generalize when there's not an object-level decision to be made (but beware failing to tie it back to reality at all, or you'll ignore important details).

Comment by dagon on Permissions in Governance · 2019-08-05T19:33:18.294Z · score: 4 (3 votes) · LW · GW
If there were absolutely no rule, and everyone were allowed to do what they wanted, nothing would get done.

Wait. If everyone does what they want, and nothing gets done, that implies that everyone wants nothing done, doesn't it? What if doing what they want actually is DOING what they want? In that case, what they want gets done.

Comment by dagon on Dagon's Shortform · 2019-08-05T17:03:54.662Z · score: 6 (3 votes) · LW · GW

My daily karma tracker showed that 3 comments got downvoted. Honestly, probably justified - they were pretty low-value. No worries, but it got me thinking:

Can I find out how I'm voting (how many positive and negative karma I've given) over time periods? Can I find it out for others? I'd love it if I could distinguish a downvote from someone who rarely downvotes from one from a serial downvoter.

I've been downvoting less, recently, having realized how little signal is there, and how discouraging even small amounts can be. Silence or comment are the better response, for anything better than pure drivel.

Comment by dagon on Can we really prevent all warming for less than 10B$ with the mostly side-effect free geoengineering technique of Marine Cloud Brightening? · 2019-08-05T16:31:30.557Z · score: 7 (7 votes) · LW · GW

Raw answer: no. Law and politics work on , and the legal liability will outweigh any actual costs by orders of magnitude. And the actual costs are probably WAY underestimated, ignoring inefficiencies and overhead of any real-world group that could undertake it.

There is also good reason to be cautious about the undertaking itself. We have no data on what actual weather shifts will occur with this scale of geoengineering, and the models have pretty large error bars. I don't know enough to say whether we CAN undertake small-scale projects to measure, or whether it's just impossible to know until we can model global weather in detail.

All that said, I suspect we're near the point where failing to take control via geoengineering will doom us anyway, so the risky attempt is likely justified.

Comment by dagon on AI Alignment Open Thread August 2019 · 2019-08-05T16:19:21.256Z · score: 3 (2 votes) · LW · GW

Opposition to and heavy regulation of nuclear reactors is mostly about accidents, not weapons (though at least some of the effort into tracking the material is about weapons). Everyone agrees we don't want accidents, not everyone agrees how much we should give up to prevent 100% of accidents. We have, in fact, had significant accidents.

Also, accidents with weapons are definitely a thing. Human regulation and cooperation is unsolved, so even knowing the difference between accident and intent is actually somewhat hard to define for many group activities.

Comment by dagon on Raemon's Scratchpad · 2019-08-05T16:10:40.549Z · score: 3 (2 votes) · LW · GW
How (or under what circumstances), can people talk openly about their respective development stages?

Talking about one's own is easy. Talking about someone else's is, as you note, fraught. I'd like to focus on the "how can such conversations be effective" and "what do we want from such conversations" part of the issue.

I think a lot of harm is done by framing it as a linear set of stages, rather than a mesh of abstractions, and recognizing that object-level results are ALWAYS relevant, and the stages are mostly ways to take more factors into account for the models and beliefs that lead to results.

When it's a stage-based system, it implies such an overt status signal that it's hard to actually discuss anything else. People of higher levels can't learn anything from those lower, and lower levels just have to accept whatever the higher-level says. This is not useful for anything.

Basically, the social move of "you seem like you're at a lower development level than me" is often legitimately read as a status attack, and worse, a plausibly-deniable status attack*.

Go further. Phrased this way, it _IS_ a status attack. There's no possible useful further discussion. This is not plausibly-deniable, it's just plain asserting "I'm thinking deeper, so I'm right".

If you phrase it not about the participants, but about the discussion, "consider this higher-level abstraction - does it not seem relevant to the point at hand?", then you've got a hook to talk about it. You don't need to bring up cognitive stages or categorize the participants, you only need to make clear what levels THIS discussion is about.

There _MAY_ be a place for talking directly about what levels someone can operate at, for elitists discussing or reinforcing a membership filter. "Don't hire a CEO who can't handle level-5 thinking" is good advice. And in such cases, it's STILL entangled with status games, as the strong implication is that if you're not on that level, you're not part of the group.

Comment by dagon on Sleeping Beauty: Is Disagreement While Sharing All Information Possible in Anthropic Problems? · 2019-08-04T15:57:02.106Z · score: 2 (1 votes) · LW · GW
But we do not need probability to wager. Probability is however needed to come up with a betting strategy

This may be near to a crux for me. Other than making decisions, what purpose does a probability distribution serve? Once we've agreed that probability is about an agent's uncertainty rather than an objective fact of the universe, it reduces to "what is the proper betting strategy", which combines probabilities with payoffs.

If you are a Boltzmann brain (or if you somehow update toward that conclusion), what will you do differently? Nothing, as such brains don't actually act, they just exist momentarily and experience things.

Comment by dagon on Power Buys You Distance From The Crime · 2019-08-03T05:29:36.720Z · score: 1 (1 votes) · LW · GW

Mostly that this is what happens, and it wasn't clear whether you were describing the same thing, or your preferred configuration.

Comment by dagon on Power Buys You Distance From The Crime · 2019-08-02T23:38:35.120Z · score: 6 (4 votes) · LW · GW
that people without power should be considered less morally culpable for their actions

I strongly disagree with this. People without power often have less impact from their actions, and actions that do less harm should be judged less harshly. But this is a judgement of the degree of wrongness of the action, not the blame-ability of the person.

Also, moral culpability is not zero-sum. There's plenty of blame for everyone making harmful decisions, and "just following orders" is not a valid defense. Giving bad orders is clearly more harmful than following, but in fact more followers adds to the total and to the individual blame, rather than distributing it.

Comment by dagon on Power Buys You Distance From The Crime · 2019-08-02T22:49:49.925Z · score: -3 (4 votes) · LW · GW

Thanks for writing this, but I don't believe there's broad agreement on either of your main examples.

I don't know anyone who claims taxes should be only proportional to wealth or income. There are those who say it should be super-linear with income or wealth (tax the rich even more than their proportion of income), and those who say not to tax based on wealth or income, but on consumption (my preference) or the hyper-pragmatic "tax everyone, we need the money". And of course many more nuanced variations.

I likewise don't think that most blame goes to people had power but failed to stop a harm - it goes to people who were active in the harm ( ideally; but often people who are active near the harm, even if not causal. ).

Agreed that in both cases, some dimensions of power can distort the straightforward application of principles, but I argue that that's true mostly BECAUSE the principles are not clearly agreed to by most people. Power exploits the disagreements by diverting attention toward the interpretations that favor the power. "Obscures information flow" is a misleading framing. It's closer to "diverts attention toward different models". And mostly on topics complicated enough that it's hard to say what's factually best, only what preferences are prioritized.

Comment by dagon on Permissions in Governance · 2019-08-02T20:21:13.101Z · score: 5 (4 votes) · LW · GW

An important consideration here is trust. Governance is _easy_ for small groups of people who are generally aligned on goals and expectations (and on scope of interaction; my boss can't tell me where to go on vacation, only some negotiation of when). I have to ask my boss for any major purchase or travel, but I have a very high degree of confidence that he will approve anything I request (or suggest a better alternative), and this is because he has a high degree of trust that I won't try to sneak a wasteful or harmful purchase past him.

The adversarial cases are _much_ harder. Most attempts to minimize cost of compliance _also_ expose opportunities for un-enforced non-compliance. And as the number and diversity of agents being governed (or claiming to self-govern) increases, the likelihood that there are adversarial in addition to cooperative agents increases very quickly. Other things that increase the probability of adversarial governance (or semi-adversarial; any time the rule is being followed only based on rule-enforcing consequences, not because of actual agreement that it's best) are weirdness of topic (rules about things that differ from that group's norms), intrusiveness (on both dimensions: how often the rule applies and how difficult it is to know whether the rule applies), and opposing incentives (which can include stated or unstated rules from a different authority).