Coding Rationally - Test Driven Development

CSS is hard to unit-test because nearly all the places where it can be messed up are detected by a human who says "Hey, this looks really ugly/hard to read/misorganized", a category of problems that is generally hard to write automated tests for. I don't think it's a fault in the language, but the application domain.

C++ is also hard to unit-test, but in that case I agree that it really is part of the language. I enjoy working with C++ and use it for some of my own projects, but if I'm being honest I have to admit that its near-total lack of reflectivity and numerous odd potholes and tripwires makes it much less convenient to do certain sorts of things with it, in-language automated testing being a prominent one of those.

I'm optimistic about Vala, an in-development C#/Javaish language that compiles to Glib-using C and supports native (but language-mediated) access to C libraries, so you get all the performance and platform-specific benefits of working in C/C++, but with modern language features and a noticeable lack of C++'s slowly expanding layers of cruft.

What you're saying is too abstract, I can't understand any of it. What would be the "preconditions and postconditions" for Google Maps? "The tiles must join seamlessly at the edges"? "When the user clicks and drags, all tiles move along with the cursor"? How do you write automated tests for such things?

In a child comment wnoise says that "every bug that is found should have a unit test written". For the record, I don't agree with that either. Take this bug: "In Opera 10.2 the mousewheel delta comes in with the wrong sign, which breaks zooming." (I can't vouch for the exact version, but I do remember that Opera pulled this trick once upon a minor version update.) It's a very typical bug, I get hundreds of those; but how do you write a test for that?

You could say web development is "special" this way. Well, it isn't. Ask game developers what their typical bugs look like. (Ever try writing a 3D terrain engine test-first?) Ask a Windows developer fighting with version hell. Honestly I'm at a loss for words. What kind of apps have you seen developed with TDD start to finish? Anything interesting?

Maybe related: Ron Jeffries (well-known Extreme Programming guru) tries to write a Sudoku solver using TDD which results in a slow motion trainwreck: 1, 2, 3, 4, 5. Compare with Peter Norvig's attempt, become enlightened.

Thank you for the quite clear specification of what you mean by TDD.

Personally, I love unit tests, and think having lots of them is wonderful. But this methodology is an excessive use of them. It's quite common to both write overly complex code and to write overly general code when that generalization will never be needed. I understand why this method pushes against that, and approve. Never the less, dogmatically writing the "wrongest possible" code is generally a huge waste of time. Going through this sort of process can help one learn to see what's vital and not, but once the lesson has been internalized, following the practice is sub-optimal.

Every bug that is found should have a unit test written. There are common blind-spots that programmers make[*], and this catches any repeats. Often these are ones that even following your version of TDD (or any) won't catch. You still need to think of the bad cases to catch them, either via tests, or just writing correct code in the first place.

[*]: Boundary conditions are a common case, where code works just fine for everywhere in the middle of an expected range, but fails at the ends.

You're both losing me, I'm afraid. I wasn't parsing cousin_it's argument as saying "it's the style of development that's the more important output of TDD". How do you get that?

I'll agree that a style of development can be useful in and of itself. The style of development that tends to be useful is one where complex behaviour is composed out of smaller and simpler elements of behaviour, in such a way that it is easy to ascertain the correctness not only of the components but also of the whole.

So we have one question here amenable to empirical inquiry: does the approach known as TDD in fact lead to the style of development outlined above?

But it also seems to me that a suite of tests is a useful thing to have in many circumstances. The empirical question here is, does having a comprehensive test suite in fact yield a practical benefit, if so on which dimensions, and at what cost?

If the tests are of the kind that can be automated, then I see little benefit in having human testers manually follow the same instructions. The outcome is the same - possibly detecting a new defect - but the cost is a million times the cost of computer execution. The main cost of automation is the cost of thinking about the tests, not the cost of typing them in so a computer can run them. So ex hypothesi that cost is incurred anyway.

To more information about the TDD-based mind hacking that you mentioned.

Well, at the moment that's mostly in not-publicly-available stuff, though there are bits and bobs in my blog, and a bit in chapter 4 of the first draft of Thinking Things Done. Most of the meat is in audios and videos of recorded workshops for the Mind Hackers' Guild, though.

Short version: notice what happens in your body in response to a thought. Notice what flashes through your mind right before the response in your body. Observe whether these reactions are both spontaneous (i.e., "you" aren't doing them) and repeatable (i.e., you get roughly the same response to the thought each time).

This is now your "test". To borrow TDD language, you now have a test that's "on red". You can now use any method you like to try to change the output of the test, except for consciously overriding the response. (Since you would then have to do it every time.)

You can make a test for any behavior or circumstance, by thinking about a specific instance of that behavior or circumstance. The key is it has to be near-mode (experiential) thinking, not far-mode (abstraction, categories, not sensory-specific).

Our behaviors are generally driven by our emotional (and physical) anticipations, so if you can change (via memory reconsolidation) the anticipations in question, then the behaviors will generally also change.

Keys to change:

• You must notice what memory/anticipation is flickering past, in order to alter it (see e.g. Eliezer's getting rid of his anticipation of a serial killer in the bathroom) -- this is a general principle known as memory reconsolidation

• To observe the fleeting memory or anticipation, it is generally helpful to intensify it, or connect it to its ultimate consequence, e.g. by asking, "What's bad about that?" (Sometimes you will get a flicker of something that seems like a neutral or not-too-bad anticipation, yet it is accompanied by a negative physical response... Often you can pull up the actual anticipation by focusing on the feeling itself.)

• Sometimes the anticipation is about what happens if you fail. Sometimes it's about if you succeed. Sometimes, it's about something good you'll get by failing, and lose if you succeed. Sometimes, it's the kind of person you think you'll be, or that others will think you are. And not a one of these anticipations will make a lick of sense to your conscious mind, which will insist that these ideas are not yours at all. And that, is the best possible evidence that what you're doing is actually working. If you are not surprised by what you discover, you are probably doing something wrong.

• Corollary: real change is also surprising. When you successfully mindhack, two things usually happen: first, you will surprise yourself by acting differently in some area where you didn't expect it, and then realize that there's a connection you didn't see to the thing you changed. Second, you will begin to forget that you ever did things differently, and what it was like to think the way you did before.

• There are only a handful of techniques that work, but they are dressed in many superficially different outward forms. ALL of them appear to work via some kind of memory consolidation; they mostly differ in the details of how they get to the memory, and how they get you to either disrupt the trace or lay down a new one. The epistemology of all these techniques is 100% horseshit: nobody really knows how or why they work (even my reconsolidation hypothesis is just that: a hypothesis), so don't let a stupid theory stop you from actually doing something.

• To get good at this, you need one physical technique and several mental ones. Physical techniques like EFT or the Matherne Speed Trace are good for dealing with strong reactions, and will fix a lot of things. Pick one, and learn it cold by testing. If you can take, say, a physical reaction you get to seeing, smelling, or just imagining a food that disgusts you as your test, and you can make it go away using a physical technique, then you know you learned the technique. If you use it on something else, and it doesn't work, this is now evidence that the technique won't work for what you're trying to fix, rather than evidence that the technique is bad or you don't know how to do it.

• When you have tried the physical technique on everything you want to change in your life that you can successfully get a strong spontaneous reaction to, you're ready to move up to mental techniques. The physical ones are not so good at changing things like misplaced moral judgments, warped values, crazy definitions, and other kinds of broken brain patterns that you currently don't know you have. (Side note: if you're not a mind hacker, chances are good you have no idea just how crazy your own brain is by default. Just because you have the logical side of things wrapped up tight, doesn't mean your emotional brain isn't a fucking basket case!)

• For a first mental technique -- or at least, of the ones you can learn inexpensively -- I suggest either the Work of Byron Katie, or the Lefkoe belief-change process. To successfully learn either, though, you need to be rigorous about testing. Identify a response that you have (that your physical technique failed on), and attempt to change it using the technique you're learning. If you can't get that one to change, try another, until you learn the technique itself. Katie and Lefkoe are good examples of intermediate-level mental techniques; I have a few easier-to-learn ones, but the learning materials are more expensive compared to K&L. ;-)

That's pretty much the gamut of what you can do with free or nearly-free materials out there. (Katie and Lefkoe may both be in your local library, while EFT and Matherne are just a Google away.) This is (sort of) the path I followed myself, except that I'm cutting out lots of dead ends and leaving out all the good bits I personally discovered, tweaked, or adapted, but which require too many details to describe here.

(There's also a good reason that the mental-level techniques are generally taught in workshop form, or as workshop transcripts; many things are difficult to grok except by experience, or at least the vicarious "experience" of seeing how someone else's bug or blind spot resembles your own. Even Katie and Lefkoe put a lot of session transcripts in their books.)

A final note: the information in this comment, even if combined with the sources I've mentioned, is not sufficient to prevent you from going down a thousand blind alleys. The only thing that will get you out of them is an absolute desire to succeed, combined with an implacable determination to rigorously test, and to assume that anything that doesn't result in a clear "red" or clear "green" is just confusion and time wasting.

Always define the test first, and get a repeatable red before you try to change. Always recheck the test after you "do" a technique. Assume that if you can't get any test to change using a given technique, it probably means you haven't learned to do the technique correctly. Once you have successfully made some tests change using a given technique, you can then assume that it not working on something probably means you're either applying it to the wrong part of the problem, or it's not applicable to that problem... but only experience will let you discover which is the case.

The changes you make will sometimes "stick" with one application of a technique, just often enough to make it really frustrating when you have a "bug" that keeps coming back. Generally, though, you'll find that your original "test" is still passing, but there is now a problem at some other part of your thought process. Either you missed it the first time, or your brain is generating new bugs in order to address a higher-level need.

If something keeps coming back, you generally have some emotional need for your problem, that has to be addressed with a different level of analytic technique; plain old K&L will probably not cut it. I spent almost two years in hell after deciding to write Thinking Things Done because I had mastered everything I just told you above, except for this final level of technique.

Let's use a more concrete example. Since I've recently worked in that domain, say we're implementing a template language like mustache.

We're starting from a blank slate, so our first test might be a test for the basic capability of variable substitution:

input: blah{{$foo}}blah
context: foo=blah
expectation: blahblahblah

Is this format clear? It's language-agnostic, so you could implement that test in Ruby or whatever.

We want to see this test fail. So we have to supply an implementation that is deliberately broken - a simple way to do that is to return an empty string, or perhaps return the exact same string that was passed as input - there are many ways to be broken.

At this point an experienced TDDer will notice this arbitrariness, think "we could have started with an even simpler test case", and write the following:

input: blahblah
context: foo=whatever
expectation: blahblah

We've narrowed it down to only one non-arbitrary way to make the test fail: return the empty string. And to make the test pass we'll return the original input.

See how this works? Because I'm not yet thinking about my implementation, but thinking about how my tests pin down the correct implementation I'm free to come up with non-confirmatory examples. My tests are drawing a box around something, but I'm not yet concerned with the contents of the box.

Now we can go back to the simple variable substitution. An important part of TDD is that one failing test does not allow you, yet, to write the fully general code of your algorithm. You're supposed to write the simplest possible code that causes all tests to pass, erring on the side of "simplistic" code.

So for instance you'd write a small method that did more or less this:

return "blahblahblah" if template.contains("{")
else return "blahblah"

Many of my colleagues make a long face when they realize this is what TDD entails. "Why would you write such a deliberately stupid implementation?" Precisely to keep thinking about better tests, and to hold off on thinking about the fully general implementation.

So now I may want to add the following:

input: blah{{$bar}}blah
context: foo=blah
expectation: blahblah

And maybe this:

input: blah{$foo}blah
context: foo=blah
expectation: blah{$foo}blah

Which are important non-confirmatory test cases. And I want to see these tests fail, because they reveal important differences between a sophisticated enough implementation and my "naive" first attempt.

I will also probably be thinking that even this "basic" capability is starting to look like a fairly complex bit, a complexity which wasn't obvious until I started thinking about all these non-confirmatory test cases. At this point if I were coding this for real I would start breaking down the problem, and maybe narrow the problem down to tokenizing the template:

input: blah
expectation: (type=text, value="blah")

and

input: blah{{$foo}}blah
expectation: (type=text, value="blah"), (type=var, value="foo"),(type=text, value="blah")

(Now I'm not testing the actual output of the program, of course, but an intermediate representation. That's OK, since those are unit tests: they're allowed to examine internals of the code.)

At this point the only line of code I have written is a deliberately simplistic implementation of my expansion algorithm, and I have already gotten a half-dozen important tests out of my thinking.

 start with spaces for blocks.

See this other reply for more detail.