Question about application of Bayes

Well, the problem is that you have uncertainty over the probability of the code crashing with or without fixing the particular bug you're looking for. What you need, in order to apply Bayes, is:

• A prior model 'pr(f)' of the frequency of the code crashing with the bug present. Which is a distribution over crash frequencies. 1/(f(1-f)) is an ignorance prior that might do the job.
• I'd assume the probability of the code crashing with the bug fixed is 0, even if that's not strictly true (since there could be other bugs).
• A prior likelihood 'b' of the bug being on that line of code. This could depend (for instance) on how you identified that particular line to comment out in the first place. I'll call your evidence from running the program E1 (2 crashes out of 7) and E2 (10 runs no crashes)

P(bug on that line| E1, E2) = P(E1, E2 | bug on line) P(bug on that line) / (P(E1, E2)

P(bug on that line) = b

P(E1, E2) = P(E1, E2 | bug on line) + P(E1, E2 | bug elsewhere)

P(E1, E2 | bug on line) = P(E1)P(E2 | bug on line) (since the 2 crashes out of 7 are independent of the bug location)

P(E1, E2 | bug elsewhere) = P(E1)P(E2 | bug elsewhere) (as above)

P(E2 | bug on line) = 1

Given a frequency of crashing 'f', P(E2 | f, bug elsewhere) = (1-f)^10

P(E1|f) = f^2 * (1-f)^5

So, then you need to integrate over all possible values of 'f': P(E1) = Integral over [0,1] of: P(E1|f)pr(f)df

P(E2 | bug elsewhere) = Integral over [0,1] of: P(E2 | f, bug elsewhere)pr(f)df

That's everything you need, the rest is just picking those priors and integrating back up the line. Of course the results are only as good as the priors. A much easier solution is: "The chance of a crash appears to be about 2/7. The chance of getting 10 non-crashes is (5/7)^10 ~= 3.5% " Note that this is not the same as the above, it's an approximation, but it's probably going to be just as good as doing it the hard way.

Incidentally, you need to be aware that, particularly with intermittant bugs, just because commenting out a line stops the crash (even when you're 100% sure of the correlation) that doesn't mean the line itself is the problem. Bugs can be absolutely pathological. For example, if the problem is, say, freeing the same memory twice, then any line that calls a lot of memory allocations will increase the frequency of crashes even if the problem is actually earlier in the code. Also, if the bug is overrunning the end of an array, taking out any line can have a chaotic effect on the optimiser, moving the relative locations of things in memory around and causing the bug to disappear without fixing it (only for it to reappear later). On a simpler level, taking a line out might change the execution path avoiding the bug without fixing it. There seems to be no end to ways in which impossible seeming things can happen in computer code.

Is this program written in C or C++, by any chance? In that case, you can't conclude anything from probabilistic data - because you're probably hunting memory corruption, commenting out lines affects memory layout, and memory layout affects the symptoms of memory corruption. Actually, this problem is pretty general; anything that looks like a probabilistic software crash needs a source of entropy to randomize it, and usually this "entropy" will be in the form of a fragile dependency on timing or memory arrangement which interacts with attempts to measure it in impossibly confusing ways. (This is why using C and C++ is usually a terrible idea, and why threads are so feared.)

If you are talking about what "causes" what, maybe you should think about the problem causally first, not as a standard hypothesis testing problem (although hypothesis testing may reappear later). What does it mean for a line to be a cause of bad behavior? What is "a causal effect"? Often people formalize causal effect as a difference of means between the "control group" and a "test group." In your case the control group is the original program, and the test group is the program where you intervened to comment out the offending line, say line 500.

You have program output O, which is either good (o1) or bad (o2). You have the original program, where nothing was done to it, where you get crashes sometimes P(O = o2) > 0. You also have an altered program, where you intervened to comment out line 500, say, where P(O = o2 | do(line500 = noop)) = 0.

The statistic you want is, for example, E(O) - E(O | do(line500 = noop)). If this statistic is not zero, we say line 500 has a causal effect on the crash.

Since you can just intervene directly in your system, you can just gather enough samples of this statistic to figure out whether there is a causal effect or not. In systems that are not computer programs, people often cannot intervene directly, and so resort to "trickery" to get statistics like the above mean difference.

If this seems simple, that's because it is. This setup mirrors how people actually debug -- they intervene in systems and compare with "the test group," sometimes doing multiple runs if the bug is a "Heisenbug."

There is also the issue of whether you can really treat the outputs of repeated program runs as iid samples. Sometimes you can, often you cannot, as other posters pointed out.

If the example is intended to be concrete rather than abstract... First understand what that line does. Shotgun debugging is seldom a good idea.

I'd recommend using the beta distribution. I'd recommend Jeffrey's prior (beta(1/2,1/2)), though I don't fully understand it.

I've only ever used it to figure out the probability of getting heads on the next step, but you could just multiply these together to get the probability of the sequence, so it's 0.5/1x1.5/2x2.5/3x3.5/4x4.5/5x4.5/6x4.5/7 if that is where the bug is (since it failed five times then passed twice, or might as well have since order doesn't really matter, and clearly it will pass the next ten) and 0.5/1x1.5/2x2.5/3x3.5/4x4.5/5x4.5/6x4.5/7x5.5/8x6.5/9x...x14.5/17 if you didn't find it. The first seven terms will cancel, so you just get that it's 5.5/8x...x14.5/17 = (14.5!/4.5!)/(17!/7!) = 0.0906 times as unlikely if there's no bug.

You need a probability distribution over something like pairs (crash probability without this line, crash probability with this line). So the likelihood for (p,q) is Pr(2 crashes of 7 with the line | p) Pr(0 crashes of 10 without the line | q) = 21 p^2 (1-p)^5 (1-q)^10, and presumably you've got a prior that prefers small p, small q, and p ~= q.

Then you could look deeper into the kind of thing that might be going on and consider hypotheses like "there's a race condition that happens early in the program when debug logging is active, because of the extra time taken writing to the log" but it's going to be really tough assigning the relevant conditional probabilities. In practice, if a lot of your probability in the initial (p,q) thing ends up in regions where p<q by a substantial margin, it's probably reasonable to say that adding the line causes the program to misbehave.

The first step is to be clear about what you're asking and what you're trying to accomplish.

A whole bunch of things could be said to "cause" the crash, such as the power supplied to the system. You want a program that does what you want, and doesn't crash. You could immediately exit the program and likely always avoid a crash. The lack of such an exit could be said to be "the cause" of the crash. But what good does identifying such a cause do you?

A cause isn't necessarily "the thing that needs to be changed", though people often treat it that way.

Do I need a probability distribution over crash frequencies?

I think the data you have, and some ignorance prior, and an independence assumption of crash trials (which I think is likely a false assumption), you have Bernoulli trials and can assign a probability distribution to each of the two states - with and without the line.

But I don't know what good those distributions do you.

Why are you assigning probability distributions instead of doing more debugging? What do you expect to do with those distributions? What does the line do? Why not just remove it?

Basically, what's the problem you're trying to solve?

As a relatively new member of this community I was wondering what implications you could foresee because of this:

http://understandinguncertainty.org/court-appeal-bans-bayesian-probability-and-sherlock-holmes

I know this might not be the right place to post, but Karma disallowed me from posting anywhere else, and I was hoping to get some thoughts on this.

. Do I need a probability distribution over crash frequencies? That was the solution the last time I was confused over Bayes, but I don't understand what it means to say "The probability of having the line, given crash frequency f", which it seems I need to know to calculate a new probability distribution.

What you want to find is P(line causes crash|observed crashes)

let's call

L: "line is cause of crash"
R: "crash is random" (so basically R = ~L)
O: "observed crash distribution"

p(L|O) = p(O|L)p(L)/p(O) = p(O|L)p(L)/(p(O|L) + p(O|R))

If your prior is that L and R are equally likely, this becomes

p(L|O) = 0.5/(1+p(O|R)/p(O|L))

... so you just need the ratio between p(O|R) and p(O|L).

To get that, you may want to compare two models, one where the crash occurs randomly with probability pr, one when the crash occurs only when the line is present with probability pl (pr and pl with some simple prior distribution like Jeffrey's prior, as DanielLC recommends).