Just Try It: Quantity Trumps Quality

Long ago, I forget where, I saw a blog post that applied this to writing. It pointed out that if we model the quality of your writing as having a mean X and variance Y, then the only way to hit those unlikely exceptionally good texts is to write a lot. Yes, while doing so you might also come up with the same number of exceptionally bad texts, but nobody forces you to show those to anyone. Plus writing a lot will give you practice, gradually pushing up the mean.

From personal experience, I'd also err on the side of publishing even texts you're not personally all that impressed by. I've noticed that I'm relatively bad at estimating what's going to be popular. Some of my biggest hits have been blog posts I'd never have thought would be popular.

i agree with the spirit of this post but I think you're leaving out an important part. It seems from the other comments that the experiment described never really happened. I think if it were tried it wouldn't really work out as described. I know if i had been in that class and been in the quantity group, I would have made some really crappy, really thick pots and been done as quickly as possible in order to goof off for the maximum amount of time. If I had been in the quality group I wouldn't have theorized about it, I just would have iterated on a lump of clay. Making a pot and then not firing it or anything, just mashing it back into a lump and starting over until I got to a really good one. I think I would have learned a lot more about pot making in the quality group.

What I have read and also experienced is that producing quantity is necessary but not sufficient for producing quality. If you want to get really good at something, rather than just getting somewhat good and then plateauing, you have to not only do it a lot, but you have to care deeply about how good you are doing, identify your weaknesses and work specifically to improve those. The problem with your story is that the quantity kids have no incentive to produce quality, so they probably just won't.

For example, I'm a self-taught programmer. So, in the beginning I wrote some truly atrocious code. I got good at coding when my livelihood depended upon producing and maintaining a large complicated system. The fact that I have to maintain and improve upon this codebase in the future made me a lot better because it made me suffer for my sins, and really care about not repeating them. And obviously having my income depend on it made me care about bugs and performance and stuff a lot more. If I had just been tasked with writing a lot of code, and was paid based on line count or something, I bet I would have written a lot of code and it would have all sucked.

This story is about rapid iteration rather than quantity. The "quantity" is the detritus of evolution created while learning to produce a perfect pot. If a machine was producing pots it would generate great quantity but the quality would not vary from one iteration to the next.

There are many stories and heuristics in engineering lore suggesting rapid iteration converges on quality faster than careful design. See also: OODA loops, the equivalent military heuristic.

I'm brand new to Less Wrong, and very pleased that I found a topic right away that I have given a great deal of thought to already, since it's affected me throughout my life. I grew up with a mother who was constantly critical, and stingy or withholding of praise, with the result that my sister and I, who are both in our late 40s, still converse about the negative affect that my mother had on us when it comes to making mistakes, and attempting to do new things.

I used to feel that I was being scolded because I didn't know something I "ought to have known" in advance. I'm not referring to breaking some established rule in the family. I'm talking about being blindsided by sudden harsh words in a loud volume about something I had never heard about or considered before, something that I had had the nerve to "get wrong." This happened often enough that I began thinking that I had better not try to do things unless I knew EVERYTHING there was to know BEFORE I took any action. This, not surprisingly, had the affect of paralyzing me into inaction, fearing the reprisals for "mistakes", and of course, the judgment about what was right and what was wrong was based on what my mother thought about the issue, which was largely subjective.

As my sister and I got older, we began challenging my mother about her views and how she spoke to us. She was quite unhappy about being challenged by her daughters, who had once been so docile and albeit unhappily, accepting of her criticism and punishment. I did many years of therapy, starting in my early teens, I also read many books and articles about and took several courses in psychology, and did the est training (a personal growth seminar) in 1983. My sister and I both eventually came to the understanding that not only do you not have to know everything about an endeavor before embarking upon it, you CAN'T know what you need to know EXCEPT in the process of doing it. There is a reason it's called "trial and error". You don't learn anything when you know how to do something and get it right the first time. You learn when you make mistakes, and you find that you need to keep working at getting it, yes, "less wrong".

My son who is 15 now, is in his first year of High School at both his neighborhood HS and at an engineering program at a local magnet HS. His class had a project that incorporated their biology and engineering principles coursework that was due in January. They were assigned teams, and had to come up with a hemodialysis machine. There is a company that supplies the school with a synthetic blood, which is filled with a substrate, and the teacher provides a selection of components the kids have to use in their design. The team has to prove that their device filters the distillate material out of the synthetic blood.

Matthew told me that all the other teams designed a machine, and stuck to their original design, whereas he kept experimenting and coming up with different designs. His team didn't do any of the designing, they saw that Matthew knew how to take charge, and he just delegated to them the tasks that would assist him in completing the machine. They got very concerned that he kept changing his design, but when they asked him why he was doing that, he just said "I have to get it right, and until it's right, I won't use the design." He had the team present the paper explaining the machine they finally built, and he demonstrated how it worked. They got a 95%. I am very proud of him. I have worked hard on raising him without the same mistakes that my mother made (so I've given him a bunch of different mistakes ;) ) He has always been told that it's not only OK to make mistakes, they are necessary stepping stones on the pathway towards accomplishment and knowledge.

Fear of failure is a big problem in my life right now. Its why I don't have a job, since I'm silly and am afraid of being rejected. This reframed something I think I already knew, but I'm sure it will help anyway. Time to really get on to things now.

Paul Graham said something very similar about figuring out a program:

"I was taught in college that one ought to figure out a program completely on paper before even going near a computer. I found that I did not program this way. I found that I liked to program sitting in front of a computer, not a piece of paper. Worse still, instead of patiently writing out a complete program and assuring myself it was correct, I tended to just spew out code that was hopelessly broken, and gradually beat it into shape. Debugging, I was taught, was a kind of final pass where you caught typos and oversights. The way I worked, it seemed like programming consisted of debugging.

For a long time I felt bad about this, just as I once felt bad that I didn't hold my pencil the way they taught me to in elementary school. If I had only looked over at the other makers, the painters or the architects, I would have realized that there was a name for what I was doing: sketching. As far as I can tell, the way they taught me to program in college was all wrong. You should figure out programs as you're writing them, just as writers and painters and architects do."

As a CS student currently at university, my experience has been identical. I also can't help but notice a similarity between these ideas and the methods of agile software development, where at each iteration you produce a working approximation of the final software, taking into account new information and new specification changes as you go.

I wish to expand on your conclusions and look for their limits. It might be more relevant to the "Go Try Things" post, but it being a kind of series of posts, I suppose it makes sense most to comment here.

So, data collection is good. But aside of getting one better at some area in which one tries to reach expertise or improvement, data collection is also good for discovering almost totally new facets of reality, territory that is outside the map's margins.

Data collection bring to light not only known unknowns, but unknown unknowns too. There's a risk involved, however. It seems that for the most part, the opportunity cost of researching unknown unknowns is greater than researching known unknowns: When practicing anything, the costs and possible benefits are pretty known. You know what you have to do to get better at playing an instrument, build better robots, programming or dancing tango. You also pretty much know what are the fruits of that labor (though perhaps not entirely, especially when it is many "quantum steps" away in terms of skill expertise).

On the other hand, when you consider whether to delve into some new unknown territory, you’re less familiar with the costs (you don’t know whether you’ll enjoy “uncovering data” or not, for example) as well as with the possible utility. Let's say some person A is invited to a salsa dancing party or class. He considers the idea but decides not to go. He thinks how It will obviously take a few hours which he could invest in more familiar activities that yield more utility than dancing; it will probably have some social costs involved, as in any new endeavor which is unfamiliar and especially one involves the moving of one’s body; even if he will enjoy it he doesn’t think he’ll have the time to invest on more such occasions, and he doesn’t think doing it once will be very useful, etc. etc. etc.

However, what if this person is unaware that salsa, if he were to try it out, will greatly benefit him? Elevate his spirit, exercise his body and provide some new kind of social interactions which will benefit him on non-dancing social occasions, and that if he decided to fully incorporate it in his life, it would provide excellent rest from his usual activity (say, his profession) and even benefit it in other ways?

So it must be benefitial to also collect data outside of the map, to explore new frontiers and horizons. But there must be a limit to this. The great many activities the world provides can probably fill a few life-times of human beings (or maybe not?). But either way, there must be some point where more exploration is actually adverse in its effects, if no activity is being engaged more than superficially. So how can one decide whether to embark on exploration or not?

Of course, there is meta-data available on activities. There is some text on the internet for probably most tried-out activities out there, friends share their experience with things they’ve done, movies and books tell us about activities unknown to us, and so on. But would such data actually help a person decide whether to engage in an activity or not, is it overwhelming enough to “change his mind” from not-doing the activity to doing it? My guess is not. Most people (as noted on “Hold off proposing solutions”) probably decide if they want to engage in the activity upon first hearing about the opportunity to engage in it, and, more than that, I suspect that their decision is based less upon the nature of the activity and more upon the nature of the “activists”, the people who are commonly engaged in that activity. Many activities produce some kind of culture around them, which hardy can be ignored. Since for an activity to exists it needs to be done, and if it is being done then someone must be doing it, so to imagine that activity one must imagine someone doing that activity, or imagine oneself as the kind of person who does that (of course, if it is taken more seriously, one can imagine the activity more “naturally”, ignoring the nature of other people who engage in that).

To actually decide whether to engage in some new activity, one needs to take the decision seriously. But then, to avoid such “paralysis analysis”, it would probably be easier just to start “doing it” instead of thinking about it (with the exception of activities with really high costs such as exploring the south pole or conceiving a child). But then again, there must be a limit to the amount of “new things” a person can do. Some people are likely (have high probability) to greatly beefit from exploration, while others are unlikely to benefit from it. How can one recognize which one she/he is?

What do you think?

So far as I can tell, the real issue in telling someone that the only important thing is quality, is that it leads to a phenomenon known in some circles as "paralysis by analysis." For instance, a writer could spend a day debating whether or not a particular place needed a comma or not, and miss that the whole page is rubbish. In sports, it is often what is meant when someone is accused of "thinking too much." In football, a receiver might spend his time thinking about how to get around a defender once he has the ball, and forget to catch the ball.

Like Jeff Atwood, I am a programmer. Unlike Jeff Atwood, I do not have a Wikipedia entry -rightfully so. Also, unlike Jeff, I'm pretty new: unseasoned. So, unlike Jeff Atwood, I still remember the process of learning how to be a programmer.

As far as I can tell, this entry fits with my experiences so far in improving myself as a programmer. I didn't get better at it by theorizing about how to make a beautiful program; in fact, when I tried, I found out the basic truth every good programmer knows; "If you're just barely smart enough to write it, you are, by definition, not smart enough to debug it." I spent weeks thinking about it, getting nowhere, before I used a brute force technique to fix the trouble spot within hours, and still ended up with a pretty nice program.

I must take issue with what Jeff Atwood wrote though. The vast majority of time in a nontrivial program is spent thinking, whether beforehand, or while you're trying to parse the kludge that steadfastly refuses to work. The kludge, insoluble mass that it is, can be immensely harder to fix than replace, but the natural mechanism is always to fix. It isn't immediately obvious, but the solution you have written has an immense hold on your mind, especially since the actual act of entering it took considerable time and effort, so some due diligence in the initial decision is highly warranted.

Many programmers love iteration, which would be analogically described as follows. Take a lump of clay that is to be a statute of a man. Make it the size of the man. First iteration complete. Form arms and legs and a head roughly where they go. Iteration two complete. Carefully delineate the broad details, such as knees, elbows wrists necks, ankles, shape of torso. Iteration three complete. Make clear the placement of medium details, such as fingers, toes, ears eyes, mouth, nose. Iteration four complete. Add the fine details, roughly. Delineate joints, add nails, add scars, add hair. Iteration five complete. Make the joints , and nails, and scars, and hair, and all the other little details just about right. Iteration six complete. Decide what to improve and do so. Iteration seven complete. Check for errors. Fix them. Repeat until done. Iteration eight complete.

The analogy is actually pretty close. Only problem? Each iteration listed above could, and usually would, actually involve several iterations, and testing steps.

The other major solution is far more straightforward. Make the clay look like a man. Step one complete. Is it correct? If no, repeat. Done.

The second way has a much greater quantity of results, because it is a simpler, and quicker, way to make a figure of a man. The superiority of the iterative approach comes in when it is clear you will not get it right in any one try. If I make no mistakes, I may take 24 steps in an iterative approach where I would take 2 in the one go approach. The steps are, however, not equal in length (the iterative steps are much shorter). Let us make it 8 to 2.

This still looks like a slam dunk for the all at once approach. With errors, it stays that way with a low quality standard. If any of the first three are right for it, all at once is still better. Once an extremely high quality standard for the end result is used, however, it is quite likely that not one of the first 25 all at once clay men will be good enough. Even with a high standard, iteration is likely to not need an amplification of more than 2 or 3. 50 versus 24 now makes it a slam dunk in favor of iteration.

In the end, both methods actually would give about the same amount of experience, (though I don't have space to justify that here, and good arguments against it could be made) almost regardless of the number of steps necessary. Somewhere in the middle must be a crossover point, between iteration and doing it all at once. It behooves us to figure out where we are in relation to it.

The long-winded point here is that quantity (iteration) can produce high quality quicker than quality (get it right straight up), but only some of the time. A low quality standard is analogous to your low cost of failure, whereas the high cost of failure is likewise to the high quality standard. For beginners, the standard is low, and just doing it is probably the best way (though they still need to understand in order to make a real attempt).

For pure personal learning, among the experienced, it is far trickier to be sure. True failure is highly costly, as you learn the wrong thing, but it also is less likely, and minor failures can be learned from.

I'm relatively sure Jeff Atwood understands all of this, of course, but it isn't immediately obvious from his writing here. I'm not some guru, and this isn't diving wisdom wither, but it is always a good idea to keep in mind what is so basic, that the expert has forgotten to mention it exists. After all, he is here to push for the deviation, not simply the status quo.

See also: NaNoWriMo.

Did you write the title backwards?

You should hold off on proposing solutions, but you also need to get around to actually trying the proposed solution.

I can almost swear that I was reading a post in the sequences where a statement was made to the effect that "If you already know where you think you'll end up on a decision/analysis/estimation, you should just head in that direction."

Does this ring any bells? I'd like to re-find that and have googled for it many, many ways and never found it again. It seems similar to the idea here, and thus I thought I'd ask for help here in finding it -- I really would like to read it again!

The scoring rule for group B is critically underspecified -- are they being graded on the best pot they produce? The worst? The mean? The median?

You should hold off on proposing solutions, but you also need to get around to actually trying the proposed solution.

I was under the impression that the whole point of holding off on proposing solutions was to make sure you came up with more than one idea.

As a general principle the right answer depends on the consequence of low quality and the balance of creativity versus practice. In medicine where the consequence of low quality (of a surgical operation for instance) may be death of the patient, there still is value to quantity and perfecting technique by frequent practice. Therefore simulations play a role. Ideally the surgeon makes use of much tacit knowledge in a successful operation, many instinctual elements that cannot be put into words. In many tasks like surgery, you want only intermittent creativity. Most efforts go into getting better and better at a task already mastered conceptually, not into improving the concept.

Wow, what a cool experiment.

I'm a bit incredulous about the experiment. The students' quantity of production can always be maximized by decreasing the quality. Why wouldn't they? (Even if the students maintain quality, how could the teacher justify assuming this in advance.)

Evaluating the study formally as an experiment, attention to quality is confounded with spending time on the work. (Reiteration is a different story, but in this case we seem to be talking about moving on to a different project, rather than perfecting a single design.) Did the students evaluated on quality produce lower quality because they spent time thinking rather than working; or because they spent too much time on a single item, trying to perfect it?

In writing, it's important to write rather than plan to write, but it doesn't follow that it's important to produce a great number of products rather than ones of high quality. From personal experience, one's writing improves by producing polished work, not by producing an abundance of it.