A probability question

It can be done, but it's a lot easier if you use odds ratios, as shown in Share likelihood ratios, not posterior beliefs. That being said, experts will tend to know a lot of the same information, so using this will involve major double counting. Also, they tend to know each other's opinions. This means that you just have to accept the opinion they're guaranteed to share via Aumann's agreement theorem, or more likely, you have to accept that they're not acting rationally, and take their beliefs with a grain of salt.

In your example:

A = 7:3, B = 8:2, P(Q) = 4:6

First, calculate the odds ratio expert B has:

(8:2)/(4:6) = 48:8

= 6:1

Then just multiply that by what expert A had to update on his opinion:

(7:3)(6:1) = 42:3

= 14:1

Thus, there's a 14/15 = 93.3% chance of Q

I think the first order of business is to straighten out the notation, and what is known.

• A - measurement from algorithm A on object O
• B - measurement from algorithm B on object O
• P(Q|I) - The probability you assign to Q based on some unspecified information I.

Use these to assign P(Q | A,B,O,I).

You have 2 independent measurements of object O,

I think that's a very bad word to use here. A,B are not independent, they're different. The trick is coming up with their joint distribution, so that you can evaluate P(Q | A,B,O,I).

The correlation between the opinions of the experts is unknown, but probably small.

If the correlation is small, your detectors suck. I doubt that's really what's happening. The usual situation is that both detectors actually have some correlation to Q, and thereby have some correlation to each other.

We need to identify some assumptions about the accuracy of A and B, and their joint distribution. A and B aren't just numbers, they're probability estimates. They were constructed so that they would be correlated with Q. How do we express P(QAB|O)? What information do we start with in this regard?

For a normal problem, you have some data {O_i} where you can evaluate P(A), your detector, versus Q and get the expectation of Q given A. Same for B.

The maximum entropy solution would proceed assuming that these statistics were the only information you had - or that you no longer had the data, but only had some subset of expectations evaluated in this fashion. I think Jaynes found the maximum entropy solution for two measurements which correlate to the same signal. I don't think he did it in a mixture of experts context, although the solution should be about the same.

If instead you have all the data, the problem is equally straightforward. Evaluate the expectation of Q given A,B across your data set, and apply on new data. Done. Yes, there's a regularization issue, but it's a 2-d -> 1-d supervised classification problem. If you're training A and B as well, do that in combination with this 2-d->1d problem as a stacked generalization problem, to avoid over fitting.

The issue is exactly what data are you working from. Can you evaluate A and B across all data, or do you just have statistics (or assumptions expressed as statistics) on A and B across the data?

If you have a lot of experts and a lot of objects, I might try a generative model where each object had unseen values from an n-dimensional feature space, and where experts decided what features to notice using weightings from a dual n-dimensional space, with the weight covectors generated as clustered in some way to represent the experts' structured non-independence. The experts' probability estimates would be something like a logistic function of the product of each object's features with the expert's weights (plus noise), and your output summary probability would be the posterior mean of an estimate based on a special "best" expert weighting, derived using the assumption that the experts' estimates are well-calibrated.

I'm not sure what an appropriate generative model of clustered expert feature weightings would be.

Actually, I guess the output of this procedure would just end up being a log-linear model of the truth given the experts' confidences. (Some of the coefficients might be negative, to cancel confounding factors.) So maybe a lot easier way to fix this is to sample from the space of such log-linear models directly, using sampled hypothetical imputed truths, while enforcing some constraint that the experts' opinions be reasonably well-calibrated.

You have 2 independent measurements

I ignored this because I wasn't sure what you could have meant by "independent". If you meant that the experts' outputs are fully independent, conditional on the truth, then the problem is straightforward. But this seems unlikely in practice. You probably just meant the informal English connotation "not completely dependent".

If you don't get any information after the fact on whether O was Q or not, there's not one right way to do it. JRMayne's recommendation of averaging the expert judgments works, as does DanielLC's recommendation of assuming that the experts are entirely uncorrelated. The trouble with assuming they're uncorrelated is that it can give you pretty extreme probability estimates- but if you're just making decisions based on some middling threshold ("call it a Q if P(Q)>.5") then you don't have to worry about extreme probability estimates! If you make decisions based on an extreme threshold ("call it a Q if P(Q)>.99"), then you have to worry. One of the things that might be helpful is plotting what these formula will result in A,B space, and seeing if that graph looks like what you / experts in this domain would expect.

If you do get information after the fact, you'll want to use what's called a Bayesian Judge. Basically, it learns P(Q(O)|A,B,P(Q)) through Bayesian updates; you're building an expert that says "if I consider all of the n times A said a and B said b, nP times it turned out to be Q, so P(Q)=P."

The other neat thing about Bayesian judges is that they fix calibration problems with experts- it will quickly learn that when they say .9, they actually mean .7.

The trouble with the Bayesian judge is that it will starve if you can't feed it data on whether or not O was Q. I won't type up the necessary math unless this fits your situation, but if it does I'd be happy to.

It seems like there are a few missing details. How do the experts arrive at their opinions? Presumably they have data about O (maybe different from each other) and are updating on that data and reporting the result. So what they're really telling you is their odds ratio given the data. Another concern that you need to take into account is how much of a track record the experts have. Maybe A is more experienced than B (or runs a more sophisticated algorithm). Maybe A's data about O is more significant than B's data about O.

There simply is no right answer to your question in general. The problem is that most of the time you simply have no way of knowing whether the experts' opinions are independent or not. The thing to realise is that even if the experts don't talk to each other and have entirely separate evidence, the underlying reality can still create a dependence. Vaniver's comment actually says it pretty well already, but just to hammer it home let me give you a specific example.

Imagine the underlying process is this: A coin is flipped 6 times, and each time either a 1 or a 0 is written a the side of a 6 sided die. Then the die is rolled, and you're interested in whether it rolled a 1. Obviously your prior is 0.5. Now imagine there are 3 experts who all give a 2/3 chance that a 1 was rolled.

Situation 1: Each expert has made a noisy observation of the actual die roll. Maybe they took a photo, but the photos are blurry and noisy. In this case, the evidence from each of the three separate photos is independent, and the odds combine like DanielLC describes to give an 8/9 chance that a 1 was rolled. With more experts saying the same thing, the probability converges to 1 here. Of course if they all had seen copies of the same photo it would be a different story...

Situation 2: No-one has seen the roll of interest itself, but each of the experts has seen the result of many other rolls of the same die (different other rolls for each expert). In this case, it's clear that all you have is strong evidence that there are four 1s and two 0s on the die, and the probability stays at 2/3. Note that the experts haven't spoken to each other, nor have they seen the same evidence, they're correlated by an underlying property of the system.

Situation 3: This one is a little strained I'll admit, but it's important to illustrate that the odds can actually be less than your prior even though the experts are individually giving chances that are higher than it. Imagine it's common knowledge among experts that there are five 1s on the die (maybe they've all seen hundreds of other rolls, though still different rolls from each other). However, each of them also has a photo of the actual roll, and again the photos are not completely clear but in each case it sure does look a little more like a 0 than a 1. In this case, the probability from their combined knowledge is actually 8/33! Ironically I used DanielLC's method again for this calculation.

The point of this then is that the answer to what their combined information would tell you could actually be literally anything at all. You just don't know, and the fact that the experts don't talk to each other and see separate evidence is not enough to assume they're uncorrelated. Of course there has to be a correct answer to what probability to assign in the case of ignorance on the level of correlation between the experts, and I'm actually not sure exactly what it is. Whatever it is though there's a good chance of it still turning out to be consistently under- or over-confidant across multiple similar trials (assuming you're doing such a thing). If this is a machine learning situation for instance (which it sounds like) I would strongly advise you simply make some observations of exactly how the probabilities of the experts correlate. I can give you some more detailed advice on how to go about doing that correctly if you wish.

Personally I would by default go with averaging the experts as my best guess. Averaged in log-odds space though of course (=log(p/(1-p)), not averaging the 0 to 1 probabilities. DanielLC's advice is theoretically well founded but the assumption of statistically independent evidence is, as I say, usually unwarrented. I would expect his method to generally give overconfident probabilities in practice.

Clemen and Winkler (1990) discuss parts of this problem.

Abstract:

When two forecasters agree regarding the probability of an uncertain event, should a decision maker adopt that probability as his or her own? A decision maker who does so is said to act in accord with the unanimity principle. We examine a variety of Bayesian consensus models with respect to their conformance (or lack thereof) to the unanimity principle and a more general compromise principle. In an analysis of a large set of probability forecast data from meteorology, we show how well the various models, when fit to the data, reflect the empirical pattern of conformance to these principles.

See also Clemen and Winkler (1999) for the more general problem of combining probability distributions rather than probabilities.

I think I misunderstand the question, or I don't get the assumptions, or I've gone terribly wrong.

Let me see if I've got the problem right to begin with. (I might not.)

40% of baseball players hit over 10 home runs a season. (I am making this up.)

Joe is a baseball player.

Baseball projector Mayne says Joe has a 70% chance of hitting more than 10 home runs next season. Baseball projector Szymborski says Joe has an 80% chance of hitting more than10 home runs next season. Both Mayne and Szymborski are aware of the usual rate of baseball players hitting more than 10 home runs.

Is this the problem?

Because if it is, the use of the prior is wrong. If the experts know the prior, and we believe the experts, the prior's irrelevant - our odds are 75%.

There are a lot of these situations in which regression to the mean, use of averages in determinations, and other factors are needed. But in this situation, if we assume reasonable experts who are aware of the general rules, and we value those experts' opinions highly enough, we should just ignore the prior - the experts have already factored that in. When Nate Silver gives you the odds that Barack Obama wins the election, you shouldn't be factoring in P(Incumbent wins) or anything else - the cake is prebaked with that information.

Since this rejects a strong claim in the post, it's possible I'm very seriously misreading the problem. Caveat emptor.