Discover more from Decisions, Decisions by Martin Robbins
Data Theatre: Why the Digital Dashboards of Dominic Cummings may not help with COVID
More data does not necessarily lead to better decisions.
In every job I’ve ever had as a product manager, somebody has suggested putting a big screen on the wall and displaying a dashboard on it. Occasionally that person has been me. Last week, in a very different organisation, it was Dominic Cummings, and newspapers gladly filled space with mock-ups of what his ‘Mission Control’ might look like.
The data dashboard is a seductive idea. On some deep psychic level it represents control over destiny, mastery of fate. If we could only record the trajectory of every atom, every particle of dust in the universe, then we would fully understand the past, present and future. We could resolve all mysteries, answer all riddles, or achieve a 10 point increase in our Net Promoter Score.
The tech industry is an increasingly metrics- and data-obsessed culture. This isn’t necessarily a bad thing: product managers who expose themselves to user research studies and engagement analytics will tend to make smarter decisions, on average, then those who ignore them. The problem, as with any technique or approach, is when data becomes the end rather than the means; when teams and managers start to develop cargo-cult attitudes toward it.
Putting data on a dashboard isn’t automatically helpful. For it to be worthwhile, you need at least four things. The first three are straightforward: frequent updates, the ability to react to that data on a similar cadence, and regular team rituals centred on the dashboard. Without those you may as well hang a painting, or some nice wallpaper. Nobody will pay regular attention to something if it doesn’t change, or if they can’t do anything useful when it does change; and unless the team are going to come together and interact with the screen in some way, you’d be better off firing a Slack message.
The fourth is much harder, but much more important: you need perspective.
The heart of strategy, of effective decision-making, is the ability to separate signal from noise. Each new day brings thousands of pieces of information from users, engineers, stakeholders, analytics, the news, competitors, you name it. Successful managers aren’t the people who absorb all that information, they’re the people who are most effective at forgetting it, filtering out all the distracting noise.
Take your weight, for example. If you step on the scales every morning, you’ll see that your body mass changes by a pound or two each day. People get excited about this, they talk of how they “lost three pounds since last week”, or “put on a pound after that heavy dinner yesterday”, but it’s all complete and utter tosh. Your weight on any given day is just noise, variation in water retention or bowel movements. It’s only by averaging over weeks or months that you can start to pick out the impact of a holiday, your period, or a new exercise regime.
That’s not to say that you shouldn’t weigh yourself every day - you absolutely should, in order to get the raw data to feed that rolling average - but looking at that datapoint each day, putting it up on the wall on a dashboard, is not just unhelpful but downright unhealthy. It erodes your mental and emotional bandwidth reacting to rises and falls that don’t actually exist.
That’s why being obsessively ‘data-driven’, as opposed to ‘data-supported’, can be so damaging. If you’re sat watching engagement metrics tick along a screen on a minute-by-minute basis, or if you’re reacting to every user research report as if it’s the Pharos of Alexandria illuminating the path to glory, then the chances are that you’re overfitting your brain to the data.
One big sign of this, something I’ve seen in several start-ups, is that you’re making too many decisions. It’s true that flexibility and agility are important. Smart founders and product managers are happy to pivot, to have strong opinions weakly held. But if you’re constantly changing your mind, going back and forth over the same issues on the basis of a meeting you just had or a research report or a new trend on a graph, then the odds are that you’re following noise. You’re reacting to data, instead of thinking about it, questioning it, and incorporating it into a robust understanding of your world.
Which is not to say that a mission control for government is a bad idea. Dominic Cummings is a controversial figure, and I disagree with him about many things; but he also attracts a lot of knee-jerk criticism, which tends to obscure the times when he actually has a point, whether it’s questioning Britain’s failure to produce tech companies on the scale of Silicon Valley, or articulating the need for better use of data at the heart of government, building on the huge strides made by the likes of GDS in recent years.
At the same time, it’s hard to look at the government’s response to COVID over the last six months and conclude that problem was a lack of graphs. Johnson’s leadership of the crisis has been characterised by a phenomenal rate of U-turns, and to repeat: while flexibility is important, making too many decisions is a good sign that you’re over-fitting to data, chopping and changing on the basis of the last piece of information received.
The government’s problem isn’t that it lacks information (although more reliable figures on a range of metrics would obviously help), it’s that it struggles to build a robust understanding of the situation from the information it has, that can form the basis for coherent, consistent policy. Each day brings some new finding - on COVID science, on public opinion, the level of anger among its back-benchers - that completely upends the previous model and sends them scrambling to change course, as if physicists had to frantically rewrite the entire laws of the universe upon the discovery of each new particle.
Dashboards give us the illusion of control, but graphs and trendlines can also become a substitute for thought. The government can plaster screens on every wall of every building with metrics on teacher numbers and employment rates and COVID infections, but none of it really matters if they’re not asking the right questions, talking to the right people, spotting what’s already sitting in their inboxes, or doing the calm work of sitting down, bringing all that together, and assessing: what exactly is happening?