display an aggregated "what's this user's rep" function

Agreement votes must never be aggregated, otherwise there is incentive for uncontroversial commenting.

I suspect it's a half-solution that will decay back to mostly-people-just-use-the-first-vote

Regardless of whether it's a bad solution in other respects, I predict that people will use the agree/disagree vote a ton, reliably, forever.

I don't think it lets me grok the quality of the reaction to a comment at a glance; I keep having to effortfully process "okay, what does—okay, this means that people like it but think it's slightly false, unless they—hmm, a lot more people voted up-down than true-false, unless they all strong voted up-down but weak-voted tru—you know what, I can't get any meaningful info out of this."

I mostly care about agree/disagree votes (especially when it comes to specifics). From my perspective, the upvotes/downvotes are less important info; they're mostly there to reward good behavior and make it easier to find the best content fast.

In that respect, the thing that annoys me about agree/disagree votes isn't any particular relationship to the upvotes/downvotes; it's that there isn't a consistent way to distinguish 'a few people agreeing strongly' from 'a larger number of people agreeing weakly', 'everyone agrees with this but weakly' from 'some agree strongly but they're being partly offset by others who disagree', or 'this is the author agreeing with their own comment' from 'this is a peer independently vouching for the comment's accuracy'.

I think all of those things would ideally be distinguishable, at least on hover. (Or the ambiguity would be eliminated by changing how the feature works -- e.g., get rid of the strong/weak distinction for agreevotes, get rid of the thing where users can agreevote their own comments, etc.)

The specific thing I'd suggest is to get rid of 'authors can agree/disagree vote on their own comments' (LW already has a 'disendorse' feature), and to replace the current UI with a tiny bar graph showing the rough relative number of strong agree, weak agree, strong disagree, and weak disagree votes (at least on hover).

It was going to be really important to have a single click, not multiple clicks (i.e. any two-separate-votes system was going to have people overwhelmingly just using one of the votes and largely ignoring the second one)

I feel like it's slightly less work for me to consider one axis and up/downvote it, and then consider the second axis and up/downvote it, than it'd be to vote on two axes with a single click. The former lets me consider the two separately, "make one decision and then forget about it", whereas the latter requires me to think about both at the same time. That means that I'm (slightly) more likely to cast two votes on a multiple-click system than on a single-click system. 

Though I do also consider it a feature if the system allows me to only cast one vote rather than forcing me to do both. E.g. in situations where I want to upvote a domain expert's comment giving an explanation about a domain that I'm not familiar with, so don't feel qualified to cast a vote on its truth even though I want to indicate that I appreciate having the explanation.

After using the new system for a couple of days, I now believe that a single-click^[1] system, like the one Duncan describes, would probably be preferable for interaction efficiency / satisfaction reasons. (Having to click on two different UI widgets in two different screen locations—i.e., mouse move, click, another mouse move, click—is an annoyance.)

One downside of Duncan’s proposed widget design would be that it cannot accommodate the full range of currently permissible input values. The current two-widget system has 25 possible states (karma and agreement can each independently take on any of five values: ++, +, 0, −, −−), while the proposed “blue and orange compass rose” single-widget system has only 9 possible states (the neutral state, plus two strengths of vote × four directions).

It is not immediately obvious to me what an ideal solution would look like. The obvious solution (in terms of interaction design) would be to construct a mapping from the 25 states to the 9, wherein some of the 25 currently available input states should be impermissible, and some sets of the remainder of the 25 should each be collapsed into one of the 9 input states of the proposed widget. (I haven’t thought about the problem enough to know if a satisfactory such mapping exists.)

I find all of the four-way graphical depictions in this subthread to be horribly confusing at an immediate glance; indeed I had to fight against reversing the axes on the one you showed. I already know what a karma system is, as imperfect as that is, and I already know what agreement and disagreement are—and being able to choose to only use one of the axes at any given moment is an engagement win for me, because (for instance) if I start out wanting to react “yes, I agree” but then have to think about “but also was the post good in a different sense” before I can record my answer, or vice versa, that means I have to perform the entire other thought-train with the first part in my working-memory stack. And my incentive and inclination to vote at all doesn't start out very high. It's like replacing two smaller stairs with one large stair of combined height.

A more specific failure mode there is lack of representation for null results on either axis, especially “I think this was a good contribution, but it makes no specific claims to agree or disagree with and instead advances the conversation some other way” and “I think this was a good contribution, but I will have to think for another week to figure out whether I agree or disagree with it on the object level, and vaguely holding onto the voting-intention in the back of my mind for a week gives a horrible feeling of cruftifying my already-strained attention systems”.

To try to expand the separate-axes system in this exact case, I have upvoted your comment here on the quality axis and also marked my disagreement (which I don't like the term “downvote” for, as described elsewhere), because I think it's a good thing that you went to the effort of thinking about this and posting about it, and I think the explanation is coherent and reasonable, but I also think the suggestion itself would be more complicated and difficult and overall worse than what's been implemented, largely because of differing impressions of the actual world. I think this is much better than having the choice of downvoting the comment and thus indicating that I wish you hadn't posted this, which is false, or upvoting it and risking a perception that I wanted the proposal to be implemented, which is also false.

I have meta-difficulty grasping why you find the at-a-glance compounding of info difficult under the separate-axes system. Do you feel inclined to try to explain it differently? In particular, I do not understand why “a lot more people” and “strong/weak votes” play so heavily into your reported thought process. I process the numbers as pre-aggregated information with a stock “congealed” (one-mental-step access) approximate correction-and-blurring model of the imperfections of the aggregation and selection most of the time. Trying to dig further is rare and mainly happens if I'm quite surprised by what I see initially.

I think many of the UI ideas here are potentially interesting, but one major issue is the amount of space we have to work with. The design here is particularly cool because it's a compass rose which matches the LW logo, but... I don't see how we could fit a version into every comment that actually worked. (Maybe if it was little but got big when you hovered over it?)

(to be clear these seem potentially fixable, just noting that that's where my attention goes next in a problem-solving-y way)

FYI In my mind there's still some radically different solutions that might be worth trying for agree/disagree, I'm still pretty uncertain about the whole thing.

↑ comment by Duncan Sabien (Deactivated) (Duncan_Sabien) · 2022-06-26T06:55:05.740Z · LW(p) · GW(p)

Yeah, the UI issues seem real and substantial.

In my mind, the thing is roughly as tall as the entire box holding the current vote buttons.

Replies from: Duncan_Sabien

↑ comment by Duncan Sabien (Deactivated) (Duncan_Sabien) · 2022-06-26T23:56:41.996Z · LW(p) · GW(p)

Replies from: Raemon

↑ comment by Raemon · 2022-06-27T01:55:51.142Z · LW(p) · GW(p)

hmm, I could see that working. Click-target seems small-ish but maybe fixable with some-UI magic (and maybe it's actually just fine?)

I found the current UI intuitive. I find the four-pointed star you suggested confusing (though mayyyybe I'd like it if I got used to it?). I tend to mix up my left and my right, and I don't associate left/right with false/true at all, nor do I associate blue with "truth". (if anything, I associate blue more with goodness, so I might have guessed dark-blue was 'good and true' and light-blue was 'good and false')

A version of this I'm confident would be easier for me to track is, e.g.:

It's less pretty, but:

• The shapes give me an indication of what each direction means. ✔ and ✖ I think are very useful and clear in that respect: to me, they're obviously about true/false rather than good/bad.
• Green vs. red still isn't super clear. But it's at least clearer than blue vs. red, to me; and if I forget what the colors mean, I have clear indicators via 'ah, there's a green X, but no green checkmark, because the heart is the special "good on all dimensions" symbol, and because green means "good" (so it would be redundant to have a green heart and a green checkmark)'.
• The left and right options are smaller and more faded. Some consequences:
  • (a) This makes the image as a whole feel less overwhelming, because there's a clear hierarchy that encourages me to first pay attention to one thing, then only consider the other thing as an afterthought. In this case, I first notice the heart and X, which give me an anchor for what green, red, and X mean. Then I notice the smaller symbols, which I can then use my anchors to help interpret. This is easier than trying to parse four symbols at the exact same moment, especially when those symbols have complicated interactions rather than being primitives.
    • I think this points at the core reason Duncan's proposal is harder for me to fit in my head than the status quo: my working memory can barely handle four things at once, and the four options here are really ordered pairs. At least, my brain thinks of them as ordered pairs rather than as primitives: I don't have four distinct qualia or images or concepts for (true, good), (true, bad), (false, good), and (false,bad), I just have the dichotomies "true v. false" and "good v. bad". Trying to compare all four options at once overloads my brain, whereas trying to compare two things (good v. bad) and then two other things (true v. false) is a lot easier for me.
    • Having a "heart" symbol is a step in the right direction in that respect, because it's closer to "a unitary concept" in my mind rather than "an ordered pair". If I had four very clearly distinct symbols for the four options, and they all made sense to me and were hard to confuse for each other, then that might more-or-less solve the problem for me.
  • (b) This makes it easier for me to chunk the two faded options as a separate category, and to think my way to 'what does these mean?' hierarchically: first I notice that these are the two 'mixed' options (because they're small and faded and off to the sides), then I notice which one is 'true mixed' versus 'false mixed' (because true mixed will have a check, while false mixed has an X).

↑ comment by Rob Bensinger (RobbBB) · 2022-06-24T20:12:45.420Z · LW(p) · GW(p)

Here's a version that's probably closer to what would actually work for me:

Now all four are closer to being conceptual primitives for me. 💚 is 'good on all the dimensions'; ❌ is 'bad on all the dimensions'.

The facepalm emoji is meant to evoke a specific emotional reaction: that exasperated feeling I get when I see someone saying a thing that's technically true but is totally irrelevant, or counter-productive. (Colored purple because purple is an 'ambiguous but bad-leaning' color, e.g., in Hollywood movies, and is associated with villainy and trolling.)

The shaking-head icon is meant to evoke another emotional reaction: the feeling of being a teacher who's happy with their student's performance, but is condescendingly shaking their head to say "No, you got the wrong answer". (Colored blue because blue is 'ambiguous but good-leaning' and is associated with innocence and youthful naïveté.)

Neither of these emotional reactions capture the range of situations where I'd want to vote (true,bad) or (false,good). But my goal is to give me a vivid, salient handle at all on what the symbols might mean, at a glance; I think the hard part for me is rapidly distinguishing the symbols at all when there are so many options, not so much 'figuring out the True Meaning of the symbol once I've distinguished it from the other three'.

Replies from: RobbBB, gjm

↑ comment by Rob Bensinger (RobbBB) · 2022-06-25T08:04:52.644Z · LW(p) · GW(p)

I don't like my own proposals, so do the disagree-votes mean that you agree with me that these are bad proposals, or do they mean you disagree with me and think they're good? :P

Replies from: RobbBB, habryka4

↑ comment by Rob Bensinger (RobbBB) · 2022-06-25T08:05:39.882Z · LW(p) · GW(p)

(I should have phrased this as a bald assertion rather than a question, so people could (dis)agree with it to efficiently reply. :P)

↑ comment by habryka (habryka4) · 2022-06-25T17:20:34.672Z · LW(p) · GW(p)

For me it meant "I think this is a bad proposal".

↑ comment by gjm · 2022-06-26T15:11:10.649Z · LW(p) · GW(p)

For what it's worth, the head icon doesn't read to me at all like a condescending head-shake. My brain parses it as "contented  face plus halo".

With two axes, each on a scale -strong/-weak/null/weak/strong, there are 24 non-trivial possibilities. Why have you chosen these four, excluding such things as "this is an important contribution that I completely agree with", or "this is balderdash on both axes"?

I'm also struggling to interpret cases where karma & agreement diverge, and would also prefer a system that lets me understand how individuals have voted. E.g. Duncan's comment above currently has positive karma but negative agreement, with different numbers of upvotes and agreement votes. There are many potential voting patterns that can have such a result, so it's unclear how to interpret it.

Whereas in Duncan's suggestion, a) all votes contain two bits of information and hence take a stand on something like agreement (so there's never a divergence between numbers of votes on different axes), and b) you can tell if e.g. your score is the result of lots of voters with "begrudging upvotes", or "conflicted downvotes" or something.

The alternative solutions you are gesturing at do communicate the problems of the current solution, but I think they are worse than the current solution, and I'm not sure there is a feasible UI change that's significantly better than the current solution (among methods for collecting the data with the same meaning, quality/agreement score). Being convenient to use and not using up too much space are harsh constraints.

all else equal, I do want to recommend comments more insofar as I agree with them more

It's a fair point. Sometimes the point of a thread is to discuss and explore a topic, and sometimes the point of a thread is to locally answer a question. In the former I want to reward the most surprising and new marginal information over the most obvious info. In the latter I just want to see the answer.

I'll definitely keep my eye out for whether this system breaks some threads, though it seems likely to me that "producing the right answer in a thread about answering a question" will be correctly upvoted in that context.

Partly because I find it cluttered and confusing, but also because I think audience agreement/disagreement should in fact be a key factor influencing comment rankings.

I have a different ontology here. I'd say that "truth-tracking" is pretty different from "true". A comment section with just the audience's main beliefs highly upvoted is different from one where the conversational moves that seem truth-tracking are highly upvoted. The former leans more easily into an echo-chamber than the latter, which better rewards side-ways moves and thoughtful arguments for positions most people disagree with.

I would be extremely surprised if karma does not track with agreement votes in the majority of cases. I only expect them to diverge in a narrow range of cases like excellently stated arguments people disagree with,  extremely banal comments that are true but don't really add anything, actual voting [LW · GW], and high social conflict posts. If we can operationalize this prediction I'm interested in a bet.

Even a completely wrong claim occasionally contributes relevant ideas to the discussion. A comment can contain many claims and ideas, and salient wrongness of some of the claims (or subjective opinions not shared by the voter) can easily coexist with correctness/relevance of other statements in the same comment. So upvote/disagree is a natural situation. Downvote/correct corresponds to something true that's trivial/irrelevant/inappropriate/unkind. Being forced to collapse such cases into a single scale is painful, and the resulting ranking is ambiguous to the point of uselessness.

I think I'd change the left/right for regular karma to up/down, to match common usage.  I agree with dissatisfaction for the agree/dis icons, but I'm not sure what's better.  Perhaps = and ≠, but that's not perfect either.  Perhaps a handshake for agree, but I don't know the opposite for disagree.

edit: I'd also swap the icons.  Good on the left, bad on the right.  Only works if the votes are no longer less-than/greater-than symbols, though.

Aesthetically speaking, this current implementation still looks rather ugly to me. Specific things I find ugly:

• Left-right arrows in the comments vs. down-up arrows on LW posts.
• The visible boundary box around normal votes & agree-disagree votes.
  • I might understand vertical lines between date & normal upvotes, and between normal upvotes & agree-disagree votes. But why do we need boundary lines at the top & bottom?
  • And rather than even vertical lines, maybe just extra whitespace between the various votes might already be enough?
• The boundary boxes even seem to push some of the other UI elements around by a few pixels:
  • See this screenshot from desktop Firefox: the boundary box creates a few pixels of extra whitespace above and below the comment headline. This creates undesirable wasted space.
  • Also, the comment menu button on the right (the three vertical dots) are not aligned with the text on the left, but rather with the upper line of the boundary box.
• None of the comment hover tooltips are aligned: That is, when hovering over comment username, date, normal downvote & upvote button, normal karma, agree & disagree vote button, and agreement karma, the tooltips just seem to pop up at semi-random but inconsistent positions.

What about something like text buttons?  

When I'm designing a UI, I try to use text if there is not a good iconographic way of representing a concept.

Something like:

AGREE (-12) DISAGREE

I'm not sure how that would look with the current karma widget. Would require some experimentation.

Huh, are all the disagreement votes here meaning "the current icons are not cluttered looking?" I'm hella surprised, I was not expecting this to be a controversial take since the current UI was whipped up really quickly.

I think this is worth some experiments at least. I do think any number that is visible on every comment really needs to pass a very high bar, though this one seems like it could plausibly pass it.

+1

I may get this in the current setup from looking at agreement compared to karma

I don't think this is reliable enough.

I very much do not believe that "agree or disagree" is a good proxy for that/tracks that. I think that it doesn't train[ LWers to distinguish their sense of truth or falsehood from how much their monkey brain wants to signal-boost a given contribution. I don't think it is going to nudge us toward better discourse and clearer separation of [truth] and [value].

See my other comment. I don't think agree/disagree is much different from true/false, and am confused about the strength of your reaction here. I personally don't have a strong preference, and only mildly prefer "agree/disagree" because it is more clearly in the same category as "approve/disapprove", i.e. an action, instead of a state.

I think the hover-over text needs tweaking anyways. If other people also have a preference for saying something like "Agree: Do you think the content of this comment is true?" and "Disagree: Do you think the content of this comment is false?", then that seems good to me. Having "approve/disapprove" and "true/false" as the top-level distinction does sure parse as a type error to me (why is one an action, and the other one an adjective?).

I also think we should definitely change the hover for the karma-vote dimension to say "approve" and "disapprove", instead of "like" and "dislike", which I think captures the dimensions here better.

I am self-aware about the fact that this might just mean "this isn't your scene, Duncan; you don't belong" more than "this group is doing something wrong for this group's goals and values."

Like, the complaint here is not necessarily "y'all're doing it Wrong" with a capital W so much as "y'all're doing it in a way that seems wrong to me, given what I think 'wrong' is," and there might just be genuine disagreement about wrongness.

But I think "agree/disagree" points people toward yet more of the same social junk that we're trying to bootstrap out of, in a way that "true/false" does not. It feels like that's where this went wrong/that's what makes this seem doomed-from-the-start and makes me really emotionally resistant to it.

I do not trust the aggregated agreement or disagreement of LW writ large to help me see more clearly or be a better reasoner, and I do not expect it to identify and signal-boost truth and good argument for e.g. young promising new users trying to become less wrong.

You would prefer, if I am understanding you right (I remark explicitly that of course I might not be), a world where the thing people do besides approving/disapproving is separating out specific factual claims and assessing whether they consider those true or false. I think that (1) labelling the buttons agree/disagree will not get you that, (2) there are important cases in which something else, closer to agree/disagree, is more valuable information, (3) reasonable users will typically use agree/disagree in the way you would like them to use true/false except in those cases, and (4) unreasonable users would likely use true/false in the exact same unhelpful ways as they would use agree/disagree.

Taking those somewhat out of order:

On #2: as has been mentioned elsewhere in the thread, for comments that say things like "I think X" or "I like Y" a strict true/false evaluation is answering the question "does the LW readership agree that Duncan thinks X?" whereas an agree/disagree evaluation is answering the question "does the LW readership also think X or like Y?", and it seems obvious to me that the latter is much more likely to be useful than the former.

On #4: some people don't think very clearly, or aren't concerned with fairness, or have a grudge against a particular other user, or are politically mindkilled, or whatever, and I completely agree with you that those people are liable to abuse an agree/disagree button as (in effect) another version of aspprove/disapprove with extra pretentions. But I would expect those people to do the same with true/false buttons. By definition, they are not trying hard to use the system in a maximally helpful way, attending to subtle distinctions of meaning.

Hence #1: labelling the buttons true/false will not in fact make those people use them the way you would like them to be used.

On #3: Users who are thinking clearly, trying to be fair, etc., will I think typically interpret agree/disagree buttons as asking whether they agree with the factual content of the text in question. There will of course be exceptions, but I think they will mostly be situations like the ones in #2 where pure factual-content-evaluation is (at least in my view) the Wrong Thing.

(Another class of situations where true/false and agree/disagree might diverge: a comment that both asserts facts and makes an argument. Maybe true/false is specifically about the facts and agree/disagree is about the argument too. My expectation would be that when the argument rather than the factual claims is the main point -- e.g., because the factual claims are uncontroversial -- agree/disagree will be applied to the argument, and otherwise they will be applied to the factual claims. That seems OK to me. You might disagree.)

In a two-factor voting system, what happens if I'm not sure if I agree or disagree, e.g. because I am still thinking about it?

If agree means "I endorse the claims or reasoning and think that more people should believe them to be true", I would probably default to no (I would endorse only if I'm pretty sure about something, and not endorsing doesn't mean I think it's wrong), so it's more like +1/0 voting. But if agree means "I think this is true", disagree would then mean saying "I think this is false", i.e. more like +1/-1 voting, so I would probably abstain?

Also, also - it's a bit confusing that karma defaults to a normal upvote by the poster, but the agreement defaults to none (but it can be added by the poster if they actually agree with themselves)?

On this point, I suggest making it so that people cannot vote agree/disagree on their own comments. It's one thing to say "I find my own comment here so valuable that I use a strong upvote on it so more people see it" - that's weird and somewhat discouraged by the community, but at least carries some information.

But what's the equivalent supposed to be for agreement? "I find my own comment so correct that I strongly agree with it"? Just disallow that in the software.

Currently, it feels like complexity and distraction for pretty low information value.  

Strong disagree

Also, why the indirect metaphor text of "agreement up/down vote", rather than much more straightforward "agree/disagree" labels?

Agree

Also, why the indirect metaphor text of "agreement up/down vote", rather than much more straightforward "agree/disagree" labels?

Yeah, this does just seem weird. Agree that we should just label them "agree/disagree".

Something I'm hoping to see (and would constitute positive evidence for me) would be seeing a comment with a high/low agree score, and someone responding along the lines of "Huh, seems like lots of people agree/disagree with this comment, which seems wrong to me, let me flesh out a counterargument here" and that post leading to many users changing their minds, and future comments about that point getting a v different agree/disagree score.

This is more confusing if the comment makes multiple claims, though, but that's a failure mode of the agree and disagree also.

It's also more confusing if the original comment made a claim like "The sky is blue, with 70% probability." Then if a user assigns 40% probability to that comment, it's not clear whether they mean:

• I think it's 40% likely that the sky is blue.
• I think it's 40% likely that you assign 70% probability to the sky being blue. (E.g., maybe you're going back and forth about what your true belief is, and I want to weigh in on what I think your view is.)
• I think it's 40% likely that you're correct in assigning 70% probability to the sky being blue. (E.g., maybe I think you're underconfident and the true probability is 90%; or maybe I think you're overconfident and the true probability is 50%; etc.)

I just made a bunch of claims all at once... that is indeed a failure mode of this system which is going to regularly occur.

I think the current system isn't ideal, but I don't particularly mind this specific issue. It's already a problem for upvotes/downvotes, and I think upvotes/downvotes are a good feature on net in spite of this. (And it's at least plausible to me that adding more UI complexity in order to let someone upvote/downvote parts of posts/comments would be net-negative.)

Part of why I'm fine with this issue is that I think it's just good for people to be separately tracking agree/disagree and good/bad. Even if they don't end up voting 'agree/disagree' that often, I expect positive affects from the mental activity alone. (E.g., prompting people to think in this mode might cause them to notice that they agree with the first half of a comment but not the second half; in which case we're already making good things happen, whether or not they write a follow-up comment explicitly saying 'I agree with the first half but not the second'.)

After 3 months, I still stand by this assessment.

Claim 2: Agree/disagree buttons are confusing or even harmful for comments that are making multiple claims. This is significant enough that there should not be an agree/disagree button for comments where agree/disagree buttons are not suitable.

• Agree: The negative consequences are significant enough that there should not be agree/disagree buttons for certain types of comments. For example, authors may be able to decide if they will allow agree/disagree votes on their comment.
• Disagree: It is acceptable to have agree/disagree votes even for posts/comments where this does not make sense, e.g. because people will adjust accordingly. We can add in a feature to disable agree/disagree votes for certain comments, but it is also okay if we don't.

Claim 1A:  Agree/disagree buttons disincentivizes productive conversations because clicking the disagree button satisfies the need for expressing disagreement (or agreement) with lower cost (less effort & no reputational cost since votes are anonymous) than writing out a reply. This is a significant enough concern that we should consider its effects when deciding whether or not to go with the new voting system.

• Agree: This matches my experience: I am less likely to write replies expressing agreement/disagreement because I am now able to vote agree/disagree.
• Disagree: This does not match my experience: If I was already going to write a reply, I would still write one even if I can just vote agree/disagree.

It's true! Not fully clear how to fix this, since the whole architecture we've chosen kind of assumes the voting-system is set at the post-level.

Giving a post's creator the option to enable/disable this secondary axis voting seems valuable. A post creator will probably know when his post will generally need nuanced comments with differing opinions, or is more lightweight (ie. what's your favourite icecream) and would appreciate the lighter UI. 

Posts tend to make a lot of claims at the same time, such that "agree/disagree" became less meaningful, and it also came with more substantial UI challenges (having a whole second number visible from the frontpage per post would add a lot of clutter).

(I have changed the post title from saying it's on all new posts to saying it's on all new comment threads.)

too radical

Hmm didn't really find anything similar, but here are some examples of rating systems I found that looked interesting (though not necessarily relevant):

2-factor rating systems

SaidIt: (1) Insightful & (2) Fun

SaidIt is a Reddit alternative which seeks to "create an environment that encourages thought-provoking discussion". SaidIt has two types of upvotes to choose from: 1) insightful, and 2) fun.^[1]

Goodfilms: (1) quality & (2) rewatchability

Goodfilms is a movie site for users to rate, review, share films and find movies to watch. Users rate movies on two dimensions: quality and rewatchability. The ratings are displayed as a scatterplot,  giving users a better sense of the type of movie (e.g. most people agree it is highly rewatchable, but there is disagreement on its quality => may not be very good, but is fun to watch).^[2]

Suggestion by Majestic121: (1) Agree/Disagree & (2) Productive/Unproductive

A Hacker News comment by Majestic121 suggests a 2-factor voting system:

Up/Down : Agree/Disagree Left/Right : Makes the discussion go backward/forward

This way you could express disagreement while acknowledging that the point is interesting, or like a joke without having it drown a conversation

Suggestion by captainmuon: (1) Promote/Bury & (2) Reward

Hacker News comment by captainmuon: Promote/Bury and Reward buttons 

Up/downvotes always have multiple conflicting dimensions.

• The post is factually right / wrong 
• Confirms to the site rules / breaks the rules
• I agree / disagree
• I want to promote / bury this post
• Reward poster with XP / punish

I am usually very pragmatic and upvote a post when I want other people to read it (because I want to see the discussion, or because I want to spread the idea). I also upvote to reward the poster.

I don't tend to downvote factually wrong posts when they are still interesting, because that limits the chance they get good discussion, and because I don't want to punish somebody for Being Wrong On The Internet. I do downvote positions that I find bad in order to reduce their reach.

It would be probably possible to have a site that implements two dimensions as a cross (maybe only for users with a certain XP) although the UX might not be to great. Maybe it is a good idea to have "promote/bury" and "reward" buttons?

Others

Pol.is: displays a cluster graph of participants based on their voting patterns on statements

Pol.is is a platform where participants submit statements for others to vote on (agree/disagree/pass), and participants are then clustered based on their votes. People can see this map of voters and are incentivized to craft statements that also appeal to members of other groups to gain more support, thus converging on a consensus.^[3]

Description of graph extracted from Pol.is report:

In this graph, statements are positioned more closely to statements which were voted on similarly. Participants, in turn, are positioned more closely to statements on which they agreed, and further from statements on which they disagreed. This means participants who voted similarly are closer together.

Example case study: vTaiwan Uber

People in Taiwan were invited to discuss the regulation of Uber. At the beginning, there were two groups: pro-Uber and anti-Uber. As people tried to submit statements that would gain more supporters, they converged on a set of seven comments that majority agreed on, such as "It should be permissible for a for-hire driver to join multiple fleets and platforms." These suggestions shaped the regulations that were eventually adopted by the government. ^[3]

Other Pol.is case studies and reports: https://compdemocracy.org/Case-studies

Tweakers: users can assign scores from -1 to +3, with detailed guidelines on how to vote

Tweakers is a Dutch technology website. Users can assign a score of +3 (insightful), +2 (informative), +1 (on-topic), 0 (off-topic/irrelevant), -1 (unwanted) to comments. The median score is displayed, but users can click to view the breakdown of votes.^[4] 0 scores with 0 votes are displayed in a different color from 0 votes with >= 1 votes.

There are detailed guidelines on what each score means, as well as how to handle common scenarios, such as:

• Comments about spelling, typos etc.: -1, because they should be reported in the Dear Editorial Forum instead 
• Inaccurate comments: no less than 0 if it is well-intentioned, -1 is for socially inappropriate behavior
• Opinions you disagree with: comments should not receive lower ratings simply because they conflict with your opinion. To express dissatisfaction, reply the comment with your refutation.
• Strategic voting e.g. upvoting a comment because you think its score is too low, rather than because you think it deserves the high score: do not do this

The guidelines also vary based on the context, with the guidelines explaining how moderation practices should differ for downloads, product reviews, and other pages.

(Disclaimer: site was in Dutch so I used Google Translate)

Slashdot: voting by assigning different types of labels (e.g. insightful, redundant)

Moderators can assign different labels to comments, which will add or deduct a point from the comment's score. There are descriptions of what each label means in the FAQ. The labels are as follows:

• Normal (default setting)
• Offtopic
• Flamebait
• Troll
• Redundant
• Insightful
• Interesting
• Informative
• Funny
• Overrated
• Underrated

Placebo button :p

Hacker News comment by kevin_thibedeau: 

They should have dummy agree/disagree buttons that disappear once selected and has no function other than to satisfy the malcontents that the machine has recorded their opinion.

1. ^{^}

   https://saidit.net/s/SaidIt/comments/37r/welcome_to_saiditnet/ 

2. ^{^}

   https://web.archive.org/web/20200812094557/http://goodfil.ms/blog/posts/2012/08/22/why-ratings-systems-dont-work/ 

3. ^{^}

   https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/

4. ^{^}

   https://tweakers.net/info/faq/karma/#tab:1-2 

5. ^{^}

   https://pol.is/report/r32beaksmhwesyum6kaur

I think I'm confused why the chosen distinction is something like "good/bad vs. agree/disagree" rather than "approve/disapprove vs. true/false."

I do not have faith that people will use the agree/disagree voting for assessments of truth, which was the thing I personally wanted added to our voting system. Right now it feels like there are just two S1 monkey buttons and no nudge toward teasing out different strands of value.

It's possible there's a bug in the comment ordering here that we should look into, but it's very unlikely to be because the agreement voting is being taken into account.

FWIW, if you ever reintroduce multi-axis voting of some sort, the primary axis I'd like to see is "novelty" (can't be downvoted), for the same reasons as above.

The same as always. Karma score, with a hint of magic (i.e. putting new comments higher for a period on the order of a few hours). As it says in the OP section titled "How the system works", agree/disagree voting has no effect on sorting.

Upvoting/downvoting self

• Sorting importance

'Agreeing'/'Disagreeing'

• 'I have discovered that this (post (of mine)) is wrong in important ways'
• or
• Looking back, this has still stood the test of time.

These methods aren't necessarily very effective (here).

Arguably, this can be done better by:

Having them be public (likely in text). What you think of your work is also important. ('This is wrong. I'm leaving it up, but also see this post explaining where I went wrong, etc.')

See the top of this article for an example: https://www.gwern.net/Fake-Journal-Club 

⁠certainty: log ⁠importance: 4

Nothing much. It's probably the right call to just remove self-agreeing.

If you're really into manipulating public opinion, you should also consider strong upvoting posts you disagree with but that are weakly written, so as to present an easily defeated strawman. 

I'd say you're correct this new addition does not change much to the previous incentives that exist in manipulating comment visibility, but that's not the point of this new addition, so not a negative of this update. 

[Edited for clarity thanks to Pattern's comment]

My first thought against would be that it would end up pretty misleading. Like, suppose the recent AGI lethalities post had this, and Eliezer picked "there is at least a 50% chance of extinction risk from AGI" as the claim. Then I think many people would agree with it, but that would look (on first glance) like many people agreed with the post, which actually makes a way more detailed and in-depth series of claims (and stronger claims of extinction), and create a false-consensus.

(I personally think this [LW(p) · GW(p)] is the neatest idea so far, that allows the post author to make multiple truth-claims in the post and have them independently voted on, and doesn't aggregate them for the post overall in any way.)

A few responses:

• Opt-in sounds like a lot of cognitive overhead for every single comment, and also (in-principle) allows for people to avoid having the truth value of their comments be judged when they make especially key claims in their argument.
• Re "what if a single comment states multiple positions? You might agree with some and not with others" <- I expect the result is that (a) the agree/disagree button won't be used that much for such comments, or (b) it just will be less meaningful for such comments. Neither of these seem very costly to me.
• Re "what if you're uncertain if you've understood the commenter's position... The vote is biased by people who think they correctly understood the position." <- If lots of people agree with a given comment because of a misunderstanding, making this fact known improves others' ability to respond to the false belief. In general my current model is that while consensus views can surely be locally false, understanding what is the consensus helps to respond to it faster and with more focus and discover the error.
• Re "what if the comment isn't an opinion, it's a quote or a collation of other people's perspectives?" <- Seems like either the button won't get much use or will be less meaningful than other occasions. Note that there are many comments on the site who also don't get many upvote/downvotes, and I don't consider this a serious reason to make up/downvoting optional on comments just because it's often not used.

One more thing is that my guess is the agree/disagree voting axis will encourage people to split up their comments more, and state things that are more cleanly true or false. (For example, I felt this impulse to split up these [LW(p) · GW(p)] two [LW(p) · GW(p)] comments upthread.)