Wednesday, November 03, 2010

Deconstructing a process failure

I blogged a while back about a pernicious intermittent problem with our internet connection that has been going on for months. To my surprise, someone from Comcast posted a comment on that post that offered to help, and yesterday, finally, after three months and I don't know how many service calls the problem was finally fixed, at least as far as I can tell. (With intermittent problems you never know for sure.) The cause turned out to be fairly prosaic: a bad modem.

In the process of dealing with this issue I learned a lot about how Comcast operates and got a small glimpse into their corporate culture. Somewhat to my surprise I discovered that the company has a lot of people who actually care about making their customers happy, but the company seems to go to extraordinary efforts to hide these people from their customers. I probably would never have met any of them had I not blogged about my frustrations. In retrospect, this is not too surprising. Comcast is a big company. It would probably be an understatement to say that not all of their customers are technically competent or even reasonable. They are probably as frustrated by the number of calls they get for problems where rebooting the computer is the right answer as we geeks are at being told for the bazillionth time to reboot our computers.

Where their system broke down, though, in my case was in not recognizing that it was stuck in a loop. I would call, they'd send a technician out, he (they were all men) would be unable to reproduce the problem (because it was intermittent), he'd fiddle with something, report back to the central office that the problem was fixed (because as far as he could tell it was), and then I'd have to start all over again from square one. The only reason the problem got fixed at all was that I blogged about it, and I was able to figure out a way to reproduce it, which I was only able to do because I know a lot about how the internet works. If I were an ordinary consumer I'm not sure I could ever have gotten it fixed.

There is, it seems to me, a pretty simple solution for this problem: provide some kind of mechanism for the customer to weigh in on whether or not a problem has been fixed rather than letting the technician make this call. Then after two or three service calls where the customer says the problem still exists, escalate up to the people who actually know what they are doing instead of just following the usual scripts. That simple measure would have been the difference, for me, between perceiving Comcast as a company that deserves their legendary reputation for bad customer service, and my current impression of it as a company that mostly does a pretty good job but has one or two broken processes.

If anyone from Comcast is reading this, I'm available for consulting :-)


labwork07 said...

Tell us how you were able to reproduce it.

Ron said...

By writing a little program that put a very high load on the network. Essentially emulating a few hundred people browsing the web at the same time. That would cause the modem to reliably crash within a minute or two.

The reason I didn't suspect the modem was:

1. Many of our neighbors were having problems with their networks and/or Comcast phones as well.

2. The background failure rate got much worse when it rained.

Those two things convinced me that the problem was upstream of our house. (It may well turn out that there is still an upstream problem as well. The next time it rains will be interesting.)