Over the summer, Eliezer suggested (approximately, I am repeating this from memory) the following method for making an important decision:

1. write down a list of all of the relevant facts on either side of the argument.
2. assign numerical weights to each of the facts, according to how much they point you in one direction or another.
3. burn the piece of paper on which you wrote down the facts, and go with your gut.

This was essentially the method I used in coming to my (probably slightly low) estimate of the probability that Knox and Sollecito were innocent. It just felt like they were innocent, and I saw essentially no reason to suspect they were guilty. I will note that the 'pro-guilt' site that komponisto linked to was just horribly devoid of anything that I might consider evidence (if anything, that site did more to convince me of Knox's innocence than the pro-innocence site), and I did spend probably about 10 minute trying to find some evidence that they had missed, but completely failed.

On a different not, as I said at the time, 0.95 and 0.05 were just proxies for "pretty damn sure" and "pretty damned unlikely" - I have very little idea what 5% probability feels like, and I'm sure that if arbitrary scientific convention had settled on some different number for significance, I'd have picked that one instead. I have made some progress since a year ago on calibrating my estimates of small probabilities, but I absolutely do not think that I would be wrong approximately 1 time in 20 when making predictions to which I assign a probability of 0.95.

Despite the fact that my opinion on the case has hardly changed at all, these posts -- and thinking about the case in general -- were a tremendous learning experience for me. Some of the lessons include:

• Less Wrong is good at getting the right answer. Believe it or not, the strong survey consensus in favor of innocence -- prior to my second post -- came as something of a pleasant surprise to me. You don't find this in many other places, despite the fact that the case is a no-brainer. I had assumed there would be more wishy-washiness and probabilities close to 50% than there turned out to be. (There was some of this, but less than I expected.)

• People in general are bad at getting the right answer. As shown by the original verdict, not to mention all the numerous pro-guilt commentators on internet forums and elsewhere. What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode. Even on a mundane question such as this, people are susceptible to strange cognitive biases that can severely distort their assessment of evidence.

• Confidence should depend on how much you know. I erred twice during the original sequence of posts on the case: once when I implied that participants in my survey ought to have come up with numbers like mine (0.001), and then again when I reacted to the indignation that followed by downgrading my own confidence. In truth, because (even before the verdict) I had followed the case more than most LW readers, my level of confidence should have been higher than most readers'. For someone who spends ten minutes reading the two sites I linked to, 0.9 probability of innocence (as opposed to 0.99) is perfectly reasonable; whereas someone like me should expect to be more confident, even if only on account of the meta-information that any additional information not on those sites isn't particularly relevant.

(More comments possibly to be added later...)

See my added comment. I did not assign a probability of 15%. I said that if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots. My probability that Knox and Sollecito were guilty was "that's privileging the hypothesis", i.e., "I see no real evidence in its favor so same as prior probability", i.e., "really damned' low".

The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.

All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)

On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect to be right all the time - sometimes, with minimal knowledge of the relevant facts, a 50% prediction is in fact not so bad.

Then again, some of us were also apparently very confident in the answer that "in fact" turned out to be the correct one. Then again, we all get lucky from time to time - that too is the nature of the beast...

My intention is to continue to learn, to continue to get better at predicting, to become better calibrated and more discriminating over time.

I want to see the regression of LW karma (or the log of LW karma) on probability-Amanda-is-guilty!

You ought to leave Eliezer out of the equation, or assign him a Karma value equal to Yvain's, or else he'll dominate the regression.

Amazing job putting this together.

I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles.

My .13 probability in the first thread definitely went down following further discussion and, in particular, komponisto's second post. In the last year I've been comfortable using "She's definitely innocent" talking about the case with non-LWers.

I updated my 60% guilt for Knox/Sollecito almost immediately after reading the follow-up article. As I noted on that page, my 60% judgement was a clear case of anchoring. I started with the pro-guilt evidence and only managed an 80% guilt after reading only their evidence. That I was then only able to re-adjust down to 60% was absurd. Since I was (and still am) fairly weak as a rationalist, I probably should have withheld any kind of assessment until after I had read the pro-innocence website, rather than try to make an early assessment and update it with the evidence I knew would be coming.

I didn't put a number on my update at the time, I just went from "There isn't enough evidence to convict" to "Why were they still suspects?". If I had to put a number on it, I'd say this was a <5% chance of guilt.

Since the acquittal, given the judge's statements about the DNA evidence and the handling thereof (I didn't read the full assessment by the independent expert), I'd have to adjust it now to <1%.

ETA: I just went through the independent analysis, and the conclusions were even stronger than I thought they were.

I thought that the conclusions were of the nature: "this DNA cannot be conclusively matched to anyone".

In fact, they were of the nature: "There is no organic material present."

They did find a few grains of starch on the bra clasp, but that was it. How did the Scientific Police screw it up that badly? They got a shakey DNA match where no DNA existed!

I have to update my assessment yet again, to the lowest I'm willing to put a number on, and that's <0.1% chance that either of them are guilty. There is literally zero evidence that either of them were involved.

Strange thing about this is, if I've calculated it right, the average probability estimate of Guede's guilt is only ~87%. It seems to me that if this were your real probability estimate of his guilt, and you were on the jury at the guy's trial, you would be obligated to vote innocent. If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent. That is (let me check) more than one in ten, which strikes me as rather a lot. I think my own estimate of Guede's guilt is above 99%, so I would vote guilty, but I'm surprised the average here is so low.

One person whose reflections would be particularly welcome, of course, is Rolf Nelson.

Our debate (which is currently stalled due to my fault -- it's my turn to reply) has dealt with one of the few important pieces of evidence to emerge after the original verdict: the incompatibility of the digestive evidence with the prosecution's hypothesized time of death. (This was covered during the original trial, but was never a focus of discussion among outside commentators until the folks at the JREF forum brought it to attention last year.)

Since I was randomly chosen to comment on this, I'll throw in my two cents. I haven't thought about too much and my first instinct was to trust whatever value judgements I had made at the time, which I thought were something like 5-5-95, but were actually 1-1-99. Since me-at-the-time was much more familiar than me-right-now, I'd still probably defer to his judgement; if anything, her exoneration and other evidence should move those numbers slightly closer to the extremes.

I assigned a relatively high probability of guilt for Amanda Knox because of a combination of ignorance and over-correction. I read material on the website of the innocent side first, and felt fairly convinced largely because of their assessment of the DNA evidence. The other website had a conflicting assessment of the DNA evidence, which I didn't know how to adjudicate without trying to learn more about how DNA evidence works. I didn't do this, and so remained uncertain but still thought that Knox was innocent with fairly high probability.

Additionally, I recall at the time that I was afraid of being "wrong." The original post had the feel of setting up a "gotcha" and my instinct was to avoid being gotcha'd, so I started to hedge. I realized this wasn't a good reason to change my assessment of the situation, but I discovered something else to blame it on: my own personal biases. I used to be a pretty hardcore libertarian and am wary of politics hijacking my mind (again), so I blamed the hedging on a correction of my own biases - I think I even mentioned this bias in my original comment. I also probably overweighted the fact that Knox was accused in the first place in my assessment.

In any case, I also recall changing my mind after reading the followup article and comments - partially because more research was put into that article than I put into my own assessment, but also partially because they considered the relevant data more honestly than I had. I don't really recall the details of this update or, for that matter, the real reason why I now think Amanda Knox is innocent - it's all cached thoughts at this point.

My high estimate came from spending insufficient mental energy to come to a stable estimate as well as, as Eliezer said "if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots."

So I'm a few months late to the game here, but I was one of the people selected to give my retrospective on this, so here goes:

My original estimates were .01, .01 and .99. I realize now that my calibration was off in the same way Komponisto has conceded his was: those numbers are way too strong to be rational estimates for something you read about on the internet for an hour or two. If I had it to do over again with benefit of hindsight, I'd probably say something more like .1 .1 .9.

The thing is though, in the time since then I've done quite a bit more reading on the topic, to the point where I now feel much more justifiably confident in my conclusions... so if I had to choose probabilities TODAY, I'd probably pick something pretty similar to my original .01 .01 .99.... ORIGINALLY those numbers were too strong to be justified based on the state of my knowledge at the time, but that's no longer the case.

So basically in the last two years I've come full circle back to my original estimates.

Of the 3 new pieces of evidence you mentioned, #1 is the only one that you might say I updated on. I didn't see any reason to update on the expert report, since I assume it just contains all the same evidence that I'd already updated on.

Hmm... I'm a bit confused about what was supposed to be predicted here. Were we supposed to predict whether Knox would be convicted, or predict whether Knox actually committed a murder? If I had been involved in the original conversations, I would have assigned a very low probability to Knox's actual guilt, but a higher probability to her being found guilty. One is a question that specifically pertains to Knox herself, and the other is a commentary on the state of the Italian justice system.

I can't remember exactly what I said in the last thread, but I think my opinion is basically the same now. I am reasonably confident that Knox and Sollecito were involved in the murder and very confident that Guede was.