Something that seems fairly important is the ability to mark your own answer before seeing the others, to avoid anchoring. (I don't know that everyone should be forced to do this but it seems useful to at least have the option. I noticed myself getting heavily anchored by some of the current question's existing answers)

↑ comment by Davidmanheim · 2020-11-22T20:40:43.216Z · LW(p) · GW(p)

Strong +1 to this suggestion, at least as an option that people can set.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2020-11-22T20:54:07.151Z · LW(p) · GW(p)

You actually can since last evening! In your user settings you should have: 

Replies from: adamShimi

↑ comment by adamShimi · 2020-11-22T23:05:22.395Z · LW(p) · GW(p)

Weird, I don't see this option in my settings.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2020-11-23T00:00:26.752Z · LW(p) · GW(p)

Sorry, my bad. Accidentally made the option admin-only. Will be fixed within the next two hours.

Another feature that we also launched at the same time as this: Metaculus embeds!

Just copy paste any link to a Metaculus question into the editor, and it will automatically expand into the preview above (you can always undo the transformation with CTRL+Z).

I wish I could have my user settings so that I didn't see everybody else's predictions before making my own.

An update: We've set up a way to link your LessWrong account to your Elicit account. By default, all your LessWrong predictions will show up in Elicit's binary database but you can't add notes or filter for your predictions. 

If you link your accounts, you can:
* Filter for and browse your LessWrong predictions on Elicit (you'll be able to see them by filtering for 'My predictions')
* See your calibration for LessWrong predictions you've made that have resolved
* Add notes to your LessWrong predictions on Elicit
* Predict on LessWrong questions in the Elicit app

If you want us to link your accounts, send me an email (amanda@ought.org) with your LessWrong username and your Elicit account email!

This is awesome, congrats!

I predict that having to go to elicit.com/binary and paste the URL here will be a major barrier to usage and that if there was a button/prompt in the LW text editor it'd significantly reduce that barrier. People have to a) remember that this is an option available to them, b) remember or be able to look up the URL they need to go to (probably by visiting this LW post), and c) be motivated enough to continue. That said, I also think it makes sense to have it work the way it currently does as a version one.

Also, I'm curious to know how difficult it is to get this integrated on other websites?

This looks so good! Great work by both Ought and the LW mods.

Perhaps this can also gauge approval (from 0% (I hate it) to 100% (I love it)), or even function as a poll (first decile: research agenda 1 seems most promising; second decile: research area 2...)... or Ought we not do things like that?

↑ comment by jungofthewon · 2020-11-20T21:51:06.686Z · LW(p) · GW(p)

try it and let's see what happens! 

Elicit Prediction (elicit.org/binary/questions/7i0Xxypzs)

Replies from: TurnTrout, TurnTrout, jacobjacob, Amandango

↑ comment by TurnTrout · 2020-11-21T19:37:18.403Z · LW(p) · GW(p)

This question resolves yes. [LW · GW]

Replies from: Benito

↑ comment by Ben Pace (Benito) · 2020-11-21T19:43:49.016Z · LW(p) · GW(p)

woop woop making predictions in posts is the way to go

Replies from: TurnTrout

↑ comment by TurnTrout · 2020-11-21T20:37:25.007Z · LW(p) · GW(p)

I actually waited for this feature to post!

↑ comment by TurnTrout · 2020-11-20T22:01:50.634Z · LW(p) · GW(p)

You've just handed me so much power to prove people right or wrong on the internet... 

↑ comment by jacobjacob · 2020-11-21T02:28:44.910Z · LW(p) · GW(p)

Your prior should be that "people don't do stuff", community looking way too optimistic at this time

Replies from: TurnTrout, Benito

↑ comment by TurnTrout · 2020-11-21T03:08:37.974Z · LW(p) · GW(p)

Smiles knowingly from behind steepled hands

↑ comment by Ben Pace (Benito) · 2020-11-21T03:11:08.395Z · LW(p) · GW(p)

I think TurnTrout is a pretty good example [? · GW] of someone who did stuff when anyone could but nobody did :)

↑ comment by Amandango · 2020-11-20T21:55:03.781Z · LW(p) · GW(p)

I'm counting using this to express credence on claims as a non-prediction use!

this is too much fun to click on

I liked this post a lot. In general, I think that the rationalist project should focus a lot more on "doing things" than on writing things. Producing tools like this is a great example of "doing things". Other examples include starting meetups and group houses.

So, I liked this post a) for being an example of "doing things", but also b) for being what I consider to be a good example of "doing things". Consider that quote from Paul Graham about "live in the future and build what's missing". To me, this has gotta be a tool that exists in the future, and I appreciate the effort to make it happen.

Unfortunately, as I write this on 12/15/21, https://elicit.org/binary. is down. That makes me sad. It doesn't mean the people who worked on it did a bad job though. The analogy of a phase change in chemistry comes to mind.

If you are trying to melt an ice cube and you move the temperature from 10℉ to 31℉, you were really close, but you ultimately came up empty handed. But you can't just look at the fact that the ice cube is still solid and judge progress that way. I say that you need to look more closely at the change in temperature. I'm not sure how much movement in temperature happened here, but I don't think it was trivial.

As for how it could have been better, I think it would have really helped to have lots and lots of examples. I'm a big fan of examples, sorta along the lines of what the specificity sequence [LW · GW] talks about. I'm talking dozens and dozens of examples. I think that helps people grok how useful this can be and when they might want to use it. As I've mentioned elsewhere though, coming up with examples is weirdly difficult.

As for followup work, I don't know what the Elicit team did and I don't want to be presumptuous, but I don't recall any followup posts on LessWrong or iteration. Perhaps something like that would have lead to changes that caused more adoption. I still stand by my old comments about there needing to be 1) [LW(p) · GW(p)] a way to embed the prediction directly from the LessWrong text editor, and 2) [LW(p) · GW(p)] things like a feed of recent predictions.

This was a neat feature on Arbital, nice to see it here as well.

How does this differ from PredictionBook besides being a much more pleasing interface and actually used for reasonable things (and also the nice embedding)?

Oh, I guess I just explained how.

Really nice site I like it.

I notice myself checking back on this post because I want to submit more predictions. I think it'd be cool if there was a feed of predictions similar to Recent Discussion section on the LW home page.

I'm curious what this means for Ought. Is Ought planning on doing anything with the data, and if so, what are the current speculations about that? How will Elicit evolve if the primary user base is LessWrong?

I looked at some of the stuff about Elicit on the Ought website, but I don't see how LessWrong embeds obviously helps develop this vision.

(To be clear, I'm not complaining -- if Elicit integration is good for LW with no direct benefit for Ought that's not a problem! But I'm guessing Ought was motivated to work with LW on this due to some perceived benefit.)

Do predictions resolve? (I guess just by the question author)

Cool feature!

Is there any info on implementing Elicit embeds on other sites? (Like, say, GreaterWrong? :) I looked on elicit.org and didn’t find anything.

EDIT: Same question re: Metaculus embeds…

ElicitBlockData(questionId: String): ElicitBlockData // For querying
MakeElicitPrediction(questionId: String, prediction: Int): ElicitBlockData // For predicting

type ElicitBlockData {
  _id: String
  title: String
  notes: String
  resolvesBy: Date
  resolution: Boolean
  predictions: [ElicitPrediction]
}

type ElicitPrediction {
  _id: String
  predictionId: String
  prediction: Float
  createdAt: Date
  notes: String
  creator: ElicitUser
  sourceUrl: String
  sourceId: String
  binaryQuestionId: String
}

type ElicitUser {
  isQuestionCreator: Boolean
  displayName: String
  _id: String
  sourceUserId: String
  lwUser: User
}

mutation ElicitPrediction($questionId: String, $prediction: Int) {
  MakeElicitPrediction(questionId: $questionId, prediction: $prediction) {
    _id
    title
    notes
    resolvesBy
    resolution
    predictions {
      _id
      predictionId
      prediction
      createdAt
      notes
      sourceUrl
      sourceId
      binaryQuestionId
      creator {
        _id
        displayName
        sourceUserId
        lwUser {
          _id
          displayName
          __typename
        }
        __typename
      }
      __typename
    }
    __typename
  }
}

query Variables: {
  "questionId": "EhJt1xwVh",
  "prediction": 91
}

So cool, congrats on shipping! 

This feature seems to be making the page wider and allowing horizontal scrolling on my mobile (iPhone) which degrades the post reading experience. I would prefer if the interface got shrunk down to fit the phone’s width.

Very cool, looking forward to using this!

How does this work with the alignmemt forum? It would be amazing if AFers predictions were tracked on AF, and all LWers predictions were tracked in the LW mirror.

Calling a feature that's about getting a numeric value between 1% and 99% prediction suggests it shouldn't be used for asking questions that aren't predictions. Is that a conscious choice?

Is there a way to see all the users who predicted within a single "bucket" using the LW UI? Right now when I hover over a bucket, it will show all users if the number of users is small enough, but it will show a small number of users followed by "..." if the number of users is too large. I'd like to be able to see all the users. (I know I can find the corresponding prediction on the Elicit website, but this is cumbersome.)

Great feature! By default, I'm not (yet) into forecasting. But I can see the point of this even just for feedback. I'll definitely try to integrate it in my posts!

It looks like people can change their predictions after they initially submit them. Is this history recorded somewhere, or just the current distribution?

Is there an option to have people "lock in" their answer? (Maybe they can still edit/delete for a short time after they submit or before a cutoff date/time)

Is there a way to see in one place all the predictions I've submitted an answer to?

↑ comment by habryka (habryka4) · 2020-11-20T23:22:09.459Z · LW(p) · GW(p)

I am pretty sure Ought is keeping track of all of your predictions. But I don't think there (yet) exists UI for seeing them, and also, we don't yet have a way of connecting your LW account with your elicit account, which would be necessary to allow you to see the history of predictions you made on LW. The Elicit API currently only exposes uncancelled predictions you've made.

Replies from: Amandango

↑ comment by Amandango · 2020-11-20T23:25:54.763Z · LW(p) · GW(p)

You can search for the question on elicit.org/binary and see the history of all predictions made! E.G. If you copy the question title in this post, and search by clicking Filter then pasting the title into "Question title contains," you can find the question here.

Replies from: jmh

↑ comment by jmh · 2020-11-21T17:10:19.669Z · LW(p) · GW(p)

Is it just me or do all those prediction assessments from jungofthewon point to a rather undesirable feature of the tool?

Replies from: jungofthewon

↑ comment by jungofthewon · 2020-11-21T18:57:18.239Z · LW(p) · GW(p)

you mean because my predictions are noisy and you don't want to see them in that list? 

Replies from: jmh

↑ comment by jmh · 2020-11-22T01:28:23.560Z · LW(p) · GW(p)

Unless I'm off base it looks like you form four different predictions on the same question. That seems odd to me. I would expect a one prediction per person making a prediction -- so later predictions would update rather than provide a new value along with the prior ones. It looks like you hold all four positions simultaneously. Also, if they are all considered current predictions then that might skew the average.

But maybe I am just not getting something about what's going on. Have not really looked beyond the LW post and comments.

Replies from: jungofthewon

↑ comment by jungofthewon · 2020-11-22T14:02:40.172Z · LW(p) · GW(p)

I see what you're saying. This feature is designed to support tracking changes in predictions primarily over longer periods of time e.g. for forecasts with years between creation and resolution. (You can even download a csv of the forecast data to run analyses on it.)

It can get a bit noisy, like in this case, so we can think about how to address that. 

Do predictions have resolutions?

Is it possible to hide the values of other predictors? I'm worried that seeing the values that others predict might influence future predictions in a way that's not helpful for feedback or accurate group predictions. (Especially since names are shown and humans tend to respect others' opinions.)

Is it possible to have answers given in dates on https://forecast.elicit.org/binary, like it it is for https://forecast.elicit.org/questions/LX1mQAQOO?

Note that for embedding Elicit predictions to work you must have the "Activate Markdown Editor" setting **turned off** in your [account settings](https://www.lesswrong.com/account).

I'm having trouble embedding forecasts. Does anyone know what I am doing wrong?

Elicit Prediction (https://forecast.elicit.org/binary/questions/el3utYd8Z)