On retrospective coherence – Part 1

by admin

I met up with my friend Michael last night. Michael’s a great guy, an expert in many fields, but one thing he really does well is throw a party. I have many happy (and sometimes drunken) memories of bashes held at his place.

One of the reasons I met up with him was because I’m planning a small surprise party for another close friend, and I wanted to benefit from his experience and advice. What made Michael’s parties so good? During our chat, we realized that it was it the people attending, and their conversations. Not only that, it was also the music. Of course, we couldn’t forget the food and drink.

In the end, I decided to ask Michael to help me organize the party. He knew exactly what needed to be done for a party to be a success, and I was convinced that, if we just did the same things he always did, my party would turn out great. So, what was the first thing he said?

“Joseph, if you want to have a great party, you have to invite all of my friends.”

I thought I didn’t hear right. I wanted to invite my own friends, but on the other hand, I knew that with his friends, the party would be a smash, like his parties. Sensing my disappointment, he tried to cheer me up by saying that he could save me a lot of money. I wouldn’t need a DJ, he said, because he noted all the songs his DJ played, and he could just give me a playlist for my iPod. I’d have to lock the iPod away, so that no one could manipulate the song order, but he was sure that with these specific songs, played exactly in this order, the party would be a success. Also, it seems he videotaped his last party, and noted what everyone ate and drank. He’d provide me with scripts for all the guests, with precise timings, so that everybody ate and drank the proper things at the proper time. Of course, all conversation and interaction would also be scripted.

So, just imagine you plan to have a party, and it must be as good as the last one you held. You invite the same people, and just to make sure you’ve covered everything, you tell them to wear the same things, and speak only to the same people as last time. You’ve got the same music playlist, the same food, drinks and layout. Will this be a good party? Does something seem wrong with this story?

The answers to these last questions seem logical, although you can’t easily explain why. There is a reason, though, and this reason is key to understanding the basic issues surrounding complex problems and agile methods.

It’s called retrospective coherence.

One thing that makes complex systems complex is their causality. In an ordered system, if you do something, you expect a specific result. Do it again, expect the same result. It’s that simple. In a complex system, though, the causality emerges as the system emerges. As the party goes on, the reasons for its success become established, not before. After it’s over, you can say that a party was a success, and that these people were there, this music was played, and this much beer was downed  – but you cannot say that the party was a success because these people were there, this music was played, and this much beer was downed!

In a complex system, causality emerges as the system itself emerges, so that at the end, you can say how you got to where you are, but you can’t guarantee that by doing exactly the same things, you’ll get to the same place again – and you probably won’t. In complex systems, we say the causality is retrospectively coherent.

So let’s take a closer look, and consider what it actually was that made Michael’s party so good. It wasn’t any one factor, but how they interacted. It wasn’t the components, but the emergent complex system, that was a success. An interesting thing to note here is that the things that people often interpret as being the causes of a success aren’t that, they’re the symptoms. For instance, was it the people that made the party a good one? The music that was played? His choice of fine food and drink? Well, yes and no. Michael played a significant part because he’s a great host. He’s constantly walking around, looking, questioning. Is anyone standing all alone? Introduce them to a group! Does everyone have a drink? If not, get some more beer! Are people dancing? Trying to talk? Put on upbeat or down-tempo music as appropriate! All the time, the good host is surveying the scene and making small adjustments.

As a good host, Michael drew some boundaries. His bedroom was off-limits. His single-malt collection locked away. And, he set up a few things that people would be attracted to: a wide-screen television, a keg of beer, some games maybe. Every party needs attractors. Did you ever wonder why so many people congregate in the kitchen at a party, even if it’s the smallest room?
One thing he didn’t do is to define success, prescribe or proscribe behaviour (could you imagine saying that a party would only be a success if someone danced on the table with a lampshade on their head?). He let the party evolve, observed which patterns emerged, gently supported the good patterns, disrupted the bad patterns as soon as he noticed them, but generally let the party run itself.

Now shift the focus to a project. We tend to make the mistakes above when it comes to project planning. What worked last time? Why? Well, it must have been the people, the methodology, the meeting schedule – let’s do it the exact same way again. Like the repeat party, a project planned on this basis is likely to be a flop.

Contrary to Einstein’s definition, in a socially complex system, insanity is doing the same thing over and over again and expecting the same result!

(n.b. Dave Snowden has a classic story on organizing a child’s birthday party, and his story served as an inspiration for my version. You can find the story here. Thanks, Dave!)