Month: April 2016

Software makes mistakes. So do users. So let’s deal with it.

I have a fantasy. OK, so I live a lot of my life in software-development and software-use land, so it’s a kind of prosaic fantasy. But bear with me: here goes anyway.

One day, my fantasy goes, an email will arrive in my Inbox from the vendor of some piece of software I’m using (Intuit, for the sake of example) which will go something like this:

Dear davidkarlin,

Our monitoring systems have detected that on 20th January 2015, you received an error message “Error 407: Unable to update bank transactions. Please try later or contact support.” We have now analysed the cause of this error and are glad to tell you that a fix was deployed in last night’s release.

We trust that this fix has been effective, but if the error should recur, please contact our developers at quoting incident no. 123456789.


The Intuit Development team

Sadly, when I’ve woken up, reality is very different. What actually happens is this:

  1. Intuit certainly don’t proactively look at error messages they generate for me and deal with them on my behalf. What actually happens is that I phone the support line; when I’ve negotiated their IVR system, I get put through to an agent whose first reaction to all problems is to ask me to clear cookies and try again.
  2. Once it’s been verified that my error is unaffected by cookies (no surprises there), I get asked to uninstall and re-install as much of the system as possible.
  3. Once that’s failed, we’re into “it’s all terribly difficult, isn’t it: maybe you can try again tomorrow” territory.
  4. I then receive a survey asking me the now-ubiquitous “Net Promoter” question (the one that begins “on a scale of 0 to 10, would you recommend…”), followed by an email about the latest upgrade, which contains some delightful new feature set I didn’t ask for.

By the way, I’m not singling out Intuit here: their support line is actually one of the better ones I deal with. But the general tenor of the experience is common to most technology vendors that I’ve either worked in or whose products I’ve used: software houses prioritise cool new features over the simple business of eliminating errors.

What’s particularly striking is how bad software developers are at dealing with intermittent faults: if you can’t replicate the problem to order, that’s pretty much end of story in terms of getting anyone to take it seriously.

In my view, *any* error message is a bad thing. If it’s as a result of a software bug, there should be zero tolerance. If it’s as a result of user error, I should be thinking “how could I have designed the interface better so that the user would have been less likely to make that mistake”. Eventually, of course, there’s a law of diminishing returns here. But the vast majority of software, I would argue, is a country mile from reaching the point where a significant improvement in user experience would no longer be generated by a straightforward analysis of the rate at which error messages are generated and their most frequent causes.

And here’s an important thing: technically, it’s not all that difficult to keep logs of enough diagnostic information to enable a developer to find out what went wrong, even for the intermittent stuff. It comes down to a matter of choice: do you or do you not make the effort to log the data and then make it someone’s job to look through the logs and find the root causes. The software companies who make engine management or process control  systems keep this kind of log data as a matter of course: it’s completely understood that some particular vibration pattern might only happen once in a long test run, that testers can’t predict when it will happen and that analysis needs to be done after the event.

As well as the technology being there to keep and analyse logs, storage is now becoming so cheap that it’s possible to take logs in a lot more detail. The toughest issue, these days, is ensuring the privacy of all this log data – which is tricky, but not insurmountable.

So here’s my plea to all you providers of software and software-based systems:

  1. Analyse your incidence of error messages, and gather a metric along the lines of “number of errors per user per hour of usage”. Allocate more resources to reducing this metric than you do to providing the latest cool features.
  2. Adopt a zero-tolerance approach to bugs, including intermittent ones. Get rid of the “if you can’t replicate a bug, it doesn’t really exist” mentality, and replace it by “if a bug happens even once, we want to find out why and kill it”.
  3. Invest in instrumentation so that your developers can review logs of one-off events in enough detail to fix them.
  4. And if you really want to delight me, make my own crash data personally identifiable (with my permission, of course) so that you can proactively tell me about the good things you’ve done for me.

After writing this, I made a resolution to put my money (well, time) where my mouth is, so on Friday, I looked through the error logs on Bachtrack’s web server. Surely enough, there was a consistent “page not found” log that occurred over a hundred times in March. That’s not a lot, in the grand scale of things (we get 200,000 page views a month), but it only took an hour or so to find and fix it. If I can keep doing that for a few hours each week, that adds up to a lot of people whose user experience is going to be improved. None of them, by the way, called in to complain.

As software suppliers, let’s all take this stuff a lot more seriously. It really will help the world out there.

Can someone now start the real EU debate, please?

We arrived home from holiday last week to find campaign literature from both sides of the EU referendum campaign in the letterbox. I was dismayed by both: the arguments presented were dubious, to say the least, and it seemed to me that neither side dares to say what it really believes.

Humans are tribal creatures and the EU debate is ultimately about the size of the tribe. Do we want a tribe which is large and strong (if possibly fragmented and slightly fractious), or do we want our tribe to be small and cohesive (if possibly short of resources and clout)?

The Eurosceptic viewpoint seems to me to be driven by two fundamentals. The first is the image of Gulliver strapped down by the Lilliputians: Eurosceptics hark back to the days when the United Kingdom was a great power in its own right and feel that it could be so again if it were not enmeshed in a web of European bureaucracy and compromise. The second is a deep discomfort with immigrants, the idea that we are losing our country to invaders and not even putting up a fight over it.

European history My own beliefs are the opposite. I was born in 1958, which makes me pretty much the first generation for as long as anyone can remember to live sixty years without a major European war, and I attribute this not merely to nuclear weapons but largely to the EU – not to its specific institutions, but to the change in mindset that makes European governments start with an assumption of co-operation.

I don’t buy the “UK can be great again” argument. The UK is great: I love my country for the creativity, humour and fundamental sense of decency of its people, not its empire – which was lost not because of the EU but because the UK bankrupted itself over two world wars and because the prevailing ethical climate made it impossible to continue with the colonialist principles on which that empire was built. I’m only too pleased that historical episodes such as slavery, the opium wars and the salt tax  are well behind us, and if a Brit ever feels the need to preach to the world about genocide, they would do well first to consider the question “why are there no aborigines in Tasmania.” (Answer: because we killed them all). And I fundamentally believe in government by compromise – at the European or the UK level. To get technical for a moment, I actually believe in stuff like pooled sovereignty and subsidiarity.

Turning from those who hark to the olden days to those who merely say that Britain is the world’s fifth largest economy and can stand perfectly well on its own, I would reply “yes, but for how long?” As education globalises, can a heavily populated small country with depleted natural resources really maintain the productivity lead that we have had in the past?

These are huge issues, and I’m more than happy to debate them with an open mind – and I believe we should be debating them. In my view, it is neither evil nor racist to feel uncomfortable at living in a city where you often can’t understand the language of the majority of the people in your street, or to be concerned at democratic deficit. But that’s not the debate we’re having.

Rather, on both sides, the debate so far has been about nickels and dimes, with statements being made that are deeply misleading. Consider this one, from the “Leave EU” side:

“We can remove our politicians who are answerable to us. Unlike unelected European commissioners”.

European commissioners are the equivalent of our senior civil servants, who are every bit as unelected. The laws in the UK are written by civil servants and voted on by politicians, just as EU laws are written by commissioners (and their staff) and voted on by the Council of Ministers and/or the European Parliament, who are people we elect. At heart, it’s not the existence of appointed officials that Eurosceptics dislike – otherwise, they should be trying to get rid of the whole of Whitehall – it’s the idea of appointments being made by what they consider to be the wrong people.

The Stay camp hardly fares better. “AA warns of pain at the pump with possible 19p rise if Britain leaves EU,” trumpets the leaflet, quoting the authoritative source of The Sun. Really? Even if it’s true, which sounds highly dubious, is the future of our country’s international relations really to be decided on the basis of motorists’ worries about their next petrol bill?

Or “Good for women, with the EU protecting women’s rights in the workplace, including vital anti-discrimination and equal pay laws”. I’m sorry, I may be an ardent supporter of the staying in campaign, but I can’t accept the idea that the UK Parliament is in some way incapable of enacting wise gender equality laws without having to rely on the EU to police it.

So please, let’s debate the real issues. Will a Brexit do irreparable damage to our relationship with European countries? Can an independent UK continue indefinitely to punch above its weight economically? Is it real or illusory that ordinary people will gain more control over the decisions that affect them? Should we wean ourselves off immigration, which we have relied on to cope with the economics of an ageing population? And if the answer is yes, is it feasible for an independent UK go about doing so?

For me, the answer is simple and emotional. It lies in a clip from the unashamedly anti-Brexit Great European Disaster Movie, in which a German lady lays out on her table a series of Iron Crosses, each representing a parent / grandparent / great-grandparent who died in a European war. She is of the first generation not to add to that list, and is unspeakably proud of it. Long may that trend continue. Being part of a whole, peaceful Europe enhances my life.