Giving calibrated time estimates can have social costs

Thanks Alex :). Comment just on this section:
 

"The annoying thing here is that I believe the only difference between me and another task doer in this situation is that I have more accurate beliefs, or I have a higher belief threshold for making claims (or something similar, like that I only use statement for communicating beliefs and not for socially enforcing a commitment to myself)."

As someone who was in this situation with Alex recently (wanting a commitment from him, in order to make plans with other people that relied on this initial commitment), I think there's maybe an additional thing in my psychology and not in Alex's which is about self-forcing. 

I'm careful about situations where I'm making a very strong commitment to something, because it means that if I've planned the timing wrong, I'll get the thing done but with high self-sacrifice. I'm committing to skipping sleeping, or fun hangouts I otherwise had planned, or relaxing activities, to get the thing done by the date I said it'd be done. I'm capable and willing to force myself to do this, if the other person wants a commitment from me enough. It's not 100% certain I'll succeed -- e.g. I might be hit by a car -- but I'm certain enough of success that people would expect me to succeed barring an emergency, which is mostly what I expect from other people when they're for-real-for-real committing to something. 

So when I'm asking someone to for-real-for-real commit to me, I'm asking "are you ready to do self-sacrifice if you don't get it done by this date, barring an emergency. It's fine if it's a later date, I just want the certainty of being able to build on this plan". And I do think there's a bunch of different kinds of commitments in day-to-day life, where I make looser commitments all the time, but I do have a category for "for-real-for-real commitment", and will track other people's failures to meet my expectations when I believe they've made a "for-real-for-real" commitment to me. I might track this more carefully than other people do though-- feels like it kinda rhymes with autism and high conscientiousness, maybe also high-performance environments but idk?

Anyway, this all might be the same thing as "I only use statement for communicating beliefs and not for socially enforcing a commitment to myself". I'm not sure I'd use exactly the the "social enforcing a commitment to myself" phrase; in my mind, it feels like a social commitment and also feels like "I'm now putting my personal integrity on the line, since I'm making a for-real-for-real commitment, so I'd better do what I said I would, even if no one's looking".

Amusingly, I think Alex and I are both using self-integrity here, but one hypothesis is that maybe I'm very willing and able to force myself to do things, and this makes up the difference with respect to what concepts we're referring to with respect to (strong) commitment?

Always fun getting unduly detailed with very specific pieces of models :P.

My impulse here is to itemize - X hours for this step, Y for that step, Z as safety margin in case of P, Q, or R.

If your colleagues are regularly giving unrealistically optimistic estimates, and you are judged worse for giving realistic estimates, clearly your superiors don't care for the accuracy of the estimates all that much. You're trying to play a fair game in a situation where you will be rewarded for doing the opposite of that.

Personally I've had good mileage out of offering to lie to the people asking for estimates. When asked for estimates during a sprint, or the likes, and if I sufficiently trust the people involved I would say something like "You are asking us to do X, which I think will take 2 months. My colleagues are giving estimates of 2-3 weeks, but the previous times they gave estimates like that the project took 6-10 weeks. I'm committed to the project, and if you want to hear that we can do it in 3 weeks I'm happy to tell you that, but I don't think we will finish it within 2 months."

If after that you still find you are being punished for giving realistic estimates, consider not telling the truth?

This probably doesn't help your specific situation, but best practice for estimation in software projects is to use "ideal/naive" estimates and then use a velocity multiplier. What that means is, take everybody's estimates and then multiply them by a team-wide constant factor to allow for the fact that there are interruptions, meetings, unexpected problems, etc. This is usually a 2-4x multiplier, and can be treated as a thing to optimize at the team level (i.e., figuring out what things increase or decrease the multiplier for the team as a whole).

Of course, if that's what the team is explicitly doing, then you know to go ahead and give your naive estimate anyway and let the manager worry about the multiplier. But a lot of organizations don't do this explicitly, because the manager is just doing it in the back of their head and maybe not even acknowledging it to themselves because they think that if they do, it means everybody will act like they have more time. In essence, you may be being judged for defecting from the implicit norm of giving naive estimates for this process.

I often use the following way of explaining time estimates. "If I was to do thing X and not stumble across any issues I haven't accounted for / unknown unknowns / circumstances don't change, it will take me about 2 weeks" followed by "in my experience, most of the time issues do come up, therefore a more conservative estimate is on the order of magnitude of months, not weeks." This way I can calibrate with others who are estimating "how long does this take if everything goes completely smoothly" while also bringing up how unrealistic this is.

I ran into a similar problem. I was doing estimates of time and costs for projects which then went into the business case. As with OP my estimates were calibrated and usually fairly accurate. 

Others' estimates were massively biased to low $ and time and often wildly wrong - in one case too low by a factor of 12.5. This is not rare of course - Microsoft Word for Windows V1.0 took over 5 years but never had an "end date" more than 1 year out.

The problem is that the business units wanted lowball estimates so they could get their projects started. It was then not too hard to exploit the sunk cost fallacy to keep the project alive. They felt I was not a "team player" and so forth. 

See the extracts from Moral Mazes https://www.lesswrong.com/posts/45mNHCMaZgsvfDXbw/quotes-from-moral-mazes [LW · GW] for more on this kind of world.
 

I remember literally thinking to myself, "it would be nice if this got finished in 2021". I currently have my fingers crossed for it being finished in 2022.

Update: it has not finished in 2022.

Here's a related model for why it might be beneficial to give optimistic ETA, as well as an argument for this (perhaps) explaining why we're over-optimistic in the first place: https://sideways-view.com/2016/11/26/if-you-cant-lie-to-others-you-must-lie-to-yourself/

I aim to give well-calibrated estimates. Sometimes projects finish faster than my median estimate, and sometimes they finish slower. The ones that finish faster then become convenient examples for deflecting social problems. Hypothetical simplified conversation:

• Alice: "I think this will take a couple of weeks"
• Martin: "Maybe two weeks if everything goes right, but I expect something more like six to eight weeks. 80% confidence."
• Bob: "There's no way it will take six weeks! We just have to add the thing to the other thing."
• Martin: "Remember I thought Project X would take two weeks and then Alice knocked it out over the weekend? I'm not saying Alice is wrong, just giving my best estimate."

Naturally any final plan presented externally has to fit the external organizational culture, whatever that is, but within a team there are benefits to a diversity of optimism/pessimism and quantitative/qualitative.

Stress is a major motivator for everyone. Giving a fake overly optimistic deadline means that for every single project, you feel stressed (because you are missing a deadline) and work faster. You don't finish on time, but you finish faster than if you would have given an accurate estimate. I don't know how many people internalize it, but I think it makes sense that a manager would want you to "promise" to do something faster than possible - it'll just make you work harder. 

Taking this into account, whenever I am asked to give an estimate, I try to give a best-case estimate (which is what most people give naturally). If I'd take the time I spent aimlessly on my phone into account when planning, I'd just spend even more time on my phone, because I wouldn't feel guilt.

...sometimes people will ask me "can you get this done by [time]?" and I'll say something like "seems reasonable" but as the discussion carries on, it becomes clear that they want me to basically *promise* that I'll get the task done by then.

I think you're right in this diagnosis, but I don't think that this is strange or unreasonable. In complex organizations, people do often need to know with reasonable certainty that someone else's task will be done by a certain time. So in many work cultures the counterparty is actually asking "can you promise to work as hard as it takes to get it done by X time", not asking for a calibrated estimate for how long it would take when you're working an average 40 hour week.