Beware Experiments Without Evaluation

Very useful - note that in many cases, this is intentional (or semi-intentional).  The proposer of the change is labeling it an experiment, but doesn't actually have any intention to learn anything or to roll back the change under foreseeable circumstances.

Your point is valid, but it's a mistake to think it can be fixed by trying to formalize the "experiment".  In many cases, when "experiment" is just a euphemism for "implement without discussion", it'll be more effective to just disagree with the policy (if you do) than to object to the "experimental setup".

This seems true in some cases, but very false in others. In particular:

• Humans tend to experiment too little, and so we should often encourage more experimentation.
• Qualitative observations are often much richer than quantitative observations. So even if you don't know what you're going to measure, making changes can help you understand what you were previously missing.

Your observations seems most true when there's institutional inertia, so that calling something an experiment can be a useful pretext.

I think you are making an important point.

A relevant follow up question could be: What makes it more (or less) likely that a group or an organisation does make plans to evaluate the results of an experiment?

Some ideas:

Culture. It would be helpful to have (or create) a culture that doesn't see a "failed" experiment as a failure but as an important learning opportunity.

Intellectual humility. Running a true experiment (and not just calling one's plan "experiment") requires accepting that one has less certainty about how the world works.

A few thoughts on this:

1. The person advocating such experiments is usually an advocate of the change rather than a curious seeker of truth. Consequently, the lack of evaluation is a feature not a bug.
2. It is very hard to get good data out of any experiment that has a sample size of one.
3. It is often very hard to measure the actual thing you desire, and so as a consequence, insofar as there is measurement it is measurement of something that can be measured rather than something that is a useful measurement.
4. Even insofar as it is possible to measure something useful, it is often impossible to get all experimenters to agree on which of the useful metrics to use.
5. To some degree these things can be dealt with honestly by taking an A/B approach. So, for example
   1. You have an open office plan on the first floor, and keep it as is on the second floor. 
   2. You introduce new forum rules for the month of December, and revert on January (and see if the forum members demand the return on the new rules.) 
   3. You spin off a new forum: my-fourm-expanded, that allows in the new users and see if the old users naturally migrate to the broader discussion. Failure and the new forum will automatically die.

However, as you say in your comment oftentimes such things are introduced more out of an implementation of a belief system about what is "right", rather than a curious investigation into which is more effective. It often requires utterly disastrously bad results before reverting the change is likely. 

This seems obvious to me, at least in the specific context that you mention (a community-level change). But the post is clear, short, and to the point, so it will surely help some readers.

Military strategy is probably more careful than science in how many contingencies it expects, yet how few are truly accurate. Legitimately, science is more careful to actually try and do things it thinks has merit, while wars are fought off of conjecture and often just error. I think you can find a great number of amazing examples of both careful planning and completely spurious decision making, often on purpose to discredit another leader, within war. von Falkenhayn's memoir of the Great War is a wonderful example. 

He also brings up an absolutely excellent point, that is often under-considered. 

"Whether the solution proved itself in actual life depended, to be sure, as in all things in this imperfect world, primarily upon the men who had to put these principles in practice."

offtopic: why is the editor so hard to use right now, I don't remember quote blocks being this fiddly.