Debugging In Customer Situations

When you are learning the art of software development in school, you learn to debug your programs in your own little cocoon. You are the customer, and your only goal is to make the program work. You can use a debugger to look at the program as is runs. You can run your code multiple times, each time with additional tweaks to try to solve the problem. You can use all your available tools at your discretion.

However, when you are providing software service in a professional environment, you are not the customer. You have to deal with someone else’s reality, which is much more constrained than what you experienced as you were learning. Customers have the concept of “production” systems, on which their day-to-day business runs. Since the customer is betting their business on these systems, changes to them are highly controlled. Any changes have to be demonstrated to work before they are deployed. There is no way any customer will agree to just “trying out” a fix on a production system.

Instead, customers have "working," "test," or "development" systems. These are the places where fixes and new applications are tried out before they get to production. If you have a fix for a customer to try, these types of systems will be the place. However, there are going to be many restrictions:

• Without production workloads, it may be difficult to reproduce the problem you are trying to solve.
  
• You are not going to have direct access to the system. In particular, you are not going to get the ability to use a debugger.
  
• Reproducing the problem will require involving the customer. That means working on their schedule, among other things.
  
• The fix must be packaged in such a way that the same fix can be applied to a production system reliably.
  
• You will have to deal with the politics of the customer relationship. In particular, you need to be careful to avoid the appearance of wasting the customer’s time, and you need to make sure they see constant progress towards a solution.
  

To net it out, remember that the customer is not your partner in the debugging process. They are looking to you to fix their problem, and they want a minimum of interaction other than status updates. So, this requires a different style of debugging than you learned in school.

Next, I'll give a real world example to show how this really happens. I'll also talk about techniques you can use in these situations.