Debugging In Customer Situations

When you are learning the art of software development in school, you learn to debug your programs in your own little cocoon. You are the customer, and your only goal is to make the program work. You can use a debugger to look at the program as is runs. You can run your code multiple times, each time with additional tweaks to try to solve the problem. You can use all your available tools at your discretion.

However, when you are providing software service in a professional environment, you are not the customer. You have to deal with someone else’s reality, which is much more constrained than what you experienced as you were learning. Customers have the concept of “production” systems, on which their day-to-day business runs. Since the customer is betting their business on these systems, changes to them are highly controlled. Any changes have to be demonstrated to work before they are deployed. There is no way any customer will agree to just “trying out” a fix on a production system.

Instead, customers have "working," "test," or "development" systems. These are the places where fixes and new applications are tried out before they get to production. If you have a fix for a customer to try, these types of systems will be the place. However, there are going to be many restrictions:

• Without production workloads, it may be difficult to reproduce the problem you are trying to solve.
  
• You are not going to have direct access to the system. In particular, you are not going to get the ability to use a debugger.
  
• Reproducing the problem will require involving the customer. That means working on their schedule, among other things.
  
• The fix must be packaged in such a way that the same fix can be applied to a production system reliably.
  
• You will have to deal with the politics of the customer relationship. In particular, you need to be careful to avoid the appearance of wasting the customer’s time, and you need to make sure they see constant progress towards a solution.
  

To net it out, remember that the customer is not your partner in the debugging process. They are looking to you to fix their problem, and they want a minimum of interaction other than status updates. So, this requires a different style of debugging than you learned in school.

Next, I'll give a real world example to show how this really happens. I'll also talk about techniques you can use in these situations.

Introduction

I'm Jim Babka, and have been a software engineer for 24 years. Over the course of my career, I have often been called upon to debug some very difficult problems, both in the office and at customer sites. This has given me a good perspective on designing code in the first place to make debugging easier. However, as I reflect upon my career, and as I have talked to many other highly skilled software engineers, I have realized that there is a wide disparity in practical debugging skills, even when people are otherwise equally skilled at writing code.

The reason for this is that many software developers have been left to their own devices to learn debugging. So, they have learned to use a “gunslinger” approach to debugging. This approach works well for class projects. However, it is unsuitable for professional development. The professional developer needs to put aside their early training (or lack thereof), instead incorporating a more disciplined and thorough approach to debugging that starts with good serviceability development techniques and ends up with deliberate methodology for debugging hard problems. While this may cause debugging to take longer in some cases, it will pay huge dividends when difficult problems get much scrutiny.

So, I decided to start this blog to discuss the serviceability and debugging techniques that are required for any enterprise software developer to be truly successful. It is my hope that some of the ideas here may eventually lead to improvements in the education and training of software engineers.