How do I fix a really difficult bug in programming?

Here was the question:

"How do I fix a really difficult bug in programming?"

Here was my first answer:

There is no such thing as a “difficult bug.”

I suspect my answer requires further explanation. First of all, I doubt that you have experienced actual bugs in your computer, the kind with 8 legs that bite and swarm. I have, a couple of times, but they are rare, and usually not difficult to eradicate.

Perhaps you are talking about errors, but using inaccurate language. In that case, I will assert “there is no such thing as a difficult error.” The same error might be handled easily by a different person. I have seen that circumstance often. For instance, I once spent a month trying to pinpoint a coding error. When I finally asked the help of a colleague, she found it in less than two minutes.

No, there are no difficult errors, but there are people who have difficulty with an error. We have all been there, and we tend to want to blame the error rather than ourselves.

So, the first thing you need to do to handle a “difficult bug” is to ask yourself,

“What is it about me that is making this error so difficult to handle?”

Perhaps you are having difficulty because you are impatient, or think failure to handle the error will make you look bad to your boss or colleagues.

Perhaps pressure to handle the error is throwing you off your center, distorting your thinking.

Perhaps you do not know enough about the system with the error, or the language in which the program is written.

Perhaps your mind is on other things in your life, things distracting you because they are more important to you than this darn “bug.”

Maybe you should discuss this error with a colleague or two, What is it about you that is keeping you from doing that?

Anyway, good luck in your quest for resolution.

Take a look at Errors: Bugs, Boo-Boos, and Blunders

My most challenging experience as a software developer

Here is my detailed answer to the question, "What is the most challenging experience you encountered as a software developer?:

We were developing the tracking system for Project Mercury, to put a person in space and bring them back alive. The “back alive” was the challenging part, but not the only one. Some other challenges were as follows:

- The system was based on a world-wide network of fairly unreliable teletype connections. 

- We had to determine the touchdown in the Pacific to within a small radius, which meant we needed accurate and perfectly synchronized clocks on the computer and space capsule.

- We also needed to knew exactly where our tracking stations were, but it turned out nobody knew where Australia's two stations were with sufficient precision. We had to create an entire sub-project to locate Australia.

- We needed information on the launch rocket, but because it was also a military rocket, that information was classified. We eventually found a way to work around that.

- Our computers were a pair of IBM 7090s, plus a 709 at a critical station in Bermuda. In those days, the computers were not built for on-line real-time work. For instance, there was no standard interrupt clock. We actually built our own for the Bermuda machine.

- Also, there were no disk drives yet, so everything had to be based on a tape drive system, but the tape drives were not sufficiently reliable for our specs. We beat this problem by building software error-correcting codes into the tape drive system.

We worked our way through all these problems and many more smaller ones, but the most challenging problem was the “back alive” requirement. Once we had the hardware and network reliability up to snuff, we still had the problem of software errors. To counter this problem, we created a special test group, something that had never been done before. Then we set a standard that any error detected by the test group and not explicitly corrected would stop any launch.

Our tests revealed that the system could crash for unknown reasons at random times, so it would be unable to bring down the astronaut safely at a known location. When the crash occurred in testing, the two on-line printers simultaneously printed a 120-character of random garbage. The line was identical on the two printers, indicating that this was not some kind of machine error on one of the 7090s. It could have been a hardware design error or a coding error. We had to investigate both possibilities, but the second possibility was far more likely.

We struggled to track down the source of the crash, but after a fruitless month, the project manager wanted to drop it as a “random event.” We all knew it wasn’t random, but he didn’t want to be accused of delaying the first launch.

To us, however, it was endangering the life of the astronaut, so we pleaded for time to continue trying to pinpoint the fault. “We should think more about this,” we said, to which he replied (standing under an IBM THINK sign), “Thinking is a luxury we can no longer afford.”

We believed (and still believe) that thinking is not a luxury for software developers, so we went underground. After much hard work, Marilyn pinpointed the fault and we corrected it just before the first launch. We may have saved an astronaut’s life, but we’ll never get any credit for it.

Moral: We may think that hardware and software errors are challenging, but nothing matches the difficulty of confronting human errors—especially when those humans are managers willing to hide errors in order to make schedules.

https://leanpub.com/errors

Can they charge me for bugs?

How likely is it that you can create 0 software bugs?

A contract programmer told us, "For years, my client has aimed for 0 bugs on every software release. However we can't control the bugs that closely. Now the client has come out with an idea of charging me a penalty—a cost refund as much as 3% per bugs from what I charge them. What can I do?"

First of all, stop calling them “bugs.”  They are not independently reproducing life forms. They are made by us humans, and there are no perfect humans.

Next, listen to what experienced S/W developers will tell you. Perfect software is a myth, an illusion.

But suppose you did produce a piece of zero-error software. How would you know that’s what you had? I’ve known software that was thought to be error-free for 30+ years, then an error turned up. Are they still going to be charging you penalties thirty years from now?

Quite simply, perfect software violates the Second Law of Thermodynamics. Then, too, software that might be perfect yesterday can become imperfect because of changes in the world today.

But, if they want to charge you for errors detected in software you built, that’s okay. What you need to do is charge them more for the software to begin with, to account for what you will eventually have to pay back. Just set a time limit—maybe a year or so, or until someone else modifies the code. And be sure you have an agreed definition of what constitutes an “error.”

This is not a simple question. I’ve written at least two books on the subject, and ultimately they don't cover every possible variation. But at least give your client a copy of the books so you can begin your negotiation with some intelligent information, not just myths and illusions:

Errors: Bugs, Boo-Boos, and Blunders

Perfect Software And Other Illusions About Testing

Must There Always Be Inferior Code?

Some people claim that when you learn high software standards you will never again develop in inferior ways. Is this true?

I think you can arrive at a meaningful answer by using an analogy:

Some people claim that when you learn high medical standards, a doctor or nurse will never again treat a patient in inferior ways. Is that true?

Seen in this light, the answer is obvious. Most doctors and nurses will not treat patients in inferior ways—unless it's an emergency, like an explosion or a fire in which many people need saving in a hurry. If that happens, the doctor or nurse will return to those patients when the emergency has calmed down. Same in software.

But there do exist a few medical professionals who don’t live up to such high standards. They are, after all, human beings. Yet in spite of their inferior practices, some of their patients do get better. Why? Because humans have built-in healing mechanisms—but software does not.

Software with sick code doesn’t heal itself. Those programmers who develop in inferior ways will eventually produce troublesome code. But the key word is "eventually."

The inferior programmer may not be around any longer when the code's trouble makes itself known, so some inferior programmers can get away with hacky ways for an entire career.

It’s a good manager's job to recognize these inferior programmers and replace them and their code before the true costs of their inferior work become evident.

Some managers overuse the tactic of forcing programmers to code in a hurry, as if there's always an emergency. Just as in medicine, emergency treatment of code tends to produce inferior results. Managers who care only about the short-term will not do anything about their inferior programmers, but they, too, may move out before the consequences of their inferior management become apparent.

That’s why inferior programming practices persist. And, as long as programmers and managers are human, inferior practices will always persist. But they don't have to persist in your world. It's up to you. \

Code in haste, debug forever.

Must a Developer Know the Language?

We were asked, "Have you ever applied for a software developer job where you didn't know the language?"

My story is not exactly the same as others might have, for several reasons, but I think it does answer the question.

There are two phases to my story. My first job developing software was at IBM, in 1956. At that time, I didn’t know any programming language, largely because there really weren’t any languages other than machine code. So, I spent two weeks in a closet learning my first computer language.

Actually, it was three languages at once: machine codes for the IBM 704 and 650, plus the wired “language” for the IBM 607.

The second phase of my story takes place some years later, when I became a consultant. In that role, I have helped many, many clients who were using languages I didn’t know—even though I knew quite a few by that time, including LISP, Smalltalk, APL, PL/I, COBOL, FORTRAN, C, Pascal, Simula, several home-grown special application languages, and the machine code for the IBM 7090, 1410, 705, STRETCH, Dec’s PDP-1 and a few other machines. I had also studied in a bookish way quite a few other machines while doing competitive analyses for IBM.

I was able to help those clients largely because their problems seldom had much to do with the details of their chosen language(s). Instead, they were people problems of all sorts. The problems that did wind up with a language embodiment were usually easy to spot using my general knowledge of computer languages and typical errors people made in using them. That’s why I’ve always insisted that professional developers should know at least a handful of different language.

I think there's an analogy here with the term "mathematical maturity," something we might call "programming maturity." Here's how Wikipedia defines mathematical maturity:

Mathematical maturity is an informal term used by mathematicians to refer to a mixture of mathematical experience and insight that cannot be directly taught. Instead, it comes from repeated exposure to mathematical concepts. It is a gauge of mathematics student's erudition in mathematical structures and methods.

For instance, a mature mathematician is able to transcend notational differences, unlike my tutorial student who flunked algebra because he had learned to "solve for x," but said, "You didn't teach me to solve for y."

We could easily use most of those words to define "programming maturity," the ability that allows you to succeed in a developer job using a language in which you have no previous experience.

The Psychology of Computer Programming: Silver Anniversary Edition

Survey: The Worst Error Message

I participated in a recent survey. The question was:

What's the Worse Error Message You've Ever Seen?

This was my contribution, which received thousands of up-votes:

A graduate student brought me an error message that said:

THIS IS AN IMPOSSIBLE ERROR. THIS ERROR CANNOT OCCUR.

IF THIS ERROR OCCURS, SEE YOUR IBM REPRESENTATIVE.

I told the student I wasn’t the right person to see, but he should see his IBM representative.

He said, “But I’m my IBM representative.”

—————————-

I'd like to collect some of these lousy error messages for my book, ERRORS.

If you've got a good (i.e. bad) one, please share it with us in a comment to this post.

Thanks for contributing.

Classifying Errors

I received an email the other day from Giorgio Valoti in Italy, but when I wrote a response, it bounced back with "recipient unknown." It may have been a transient error, but it made me think that others besides Giorgio might be interested in discussing the issue of classifying errors, so I'll put my answer here and hope Giorgio will see it.

Here's the letter: 

Dear Mr. Weinberg,
My problem is that I’m looking for good way — maybe a standard, more likely a set of guidelines — to classify and put a some kind of label on software defects.

Is there such a thing? Does it even make sense trying to classify software defects?

And here's my reply:

https://leanpub.com/errors

Hello, Giorgio

It can certainly make sense to classify errors/defects, but there are many ways to classify, depending on what you're trying to accomplish. So, that's where you start, by answering "What's my purpose in classifying?"

For instance, here are a few ways my clients have classified errors, and for what purposes:

- cost to fix: to estimate future costs

- costs to customers: to estimate impact on product sales, or product market penetration

- place of origin in the development cycle: to decide where to concentrate quality efforts

- type of activity that led to the error: to improve the training of developers

- type of activity that led to detecting the error: to improve the training of testers

- number of places that had to be fixed to correct the error: to estimate the quality of the design

- and so on and on

I hope this helps ... and thanks for asking.

--------------end of letter-----------

As the letter says, there are numerous ways to classify errors, so I think my readers would love to read about some other ways from other readers. Care to comment?

Foreword and Introduction to ERRORS book

Foreword

Ever since this book came out, people have been asking me how I came to write on such an unusual topic. I've pondered their question and decided to add this foreword as an answer.

As far as I can remember, I've always been interested in errors. I was a smart kid, but didn't understand why I  made mistakes. And why other people made more.

I yearned to understand how the brain, my brain, worked, so I studied everything I could find about brains. And then I heard about computers.

Way back then, computers were called "Giant Brains." Edmund Berkeley wrote a book by that title, which I read voraciously.

Those giant brains were "machines that think" and "didn't make errors." Neither turned out to be true, but back then, I believed them. I knew right away, deep down—at age eleven—that I would spend my life with computers.

Much later, I learned that computers didn't make many errors, but their programs sure did.

I realized when I worked on this book that it more or less summarizes my life's work, trying to understand all about errors. That's where it all started.

I think I was upset when I finally figured out that I wasn't going to find a way to perfectly eliminate all errors, but I got over it. How? I think it was my training in physics, where I learned that perfection simply violates the laws of thermodynamics.

Then I was upset when I realized that when a computer program had a fault, the machine could turn out errors millions of times faster than any human or group of humans.

I could actually program a machine to make more errors in a day than all human beings had made in the last 10,000 years. Not many people seemed to understand the consequences of this fact, so I decided to write this book as my contribution to a more perfect world.

Not perfect, of course, but more perfect. I hope it helps.

Introduction

For more than a half-century, I’ve written about errors: what they are, their importance, how we think about them, our attempts to prevent them, and how we deal with them when those attempts fail. People tell me how helpful some of these writings have been, so I felt it would be useful to make them more widely known. Unfortunately, the half-century has left them scattered among several dozen books, so I decided to consolidate some of the more helpful ones in this book.

I’m going to start, though, where it all started, with my first book where Herb Leeds and I made our first public mention of error. Back in those days, Herb and I both worked for IBM. As employees we were not allowed to write about computers making mistakes, but we knew how important the subject was. So, we wrote our book and didn’t ask IBM’s permission.

Computer errors are far more important today than they were back in 1960, but many of the issues haven’t changed. That’s why I’m introducing this book with some historical perspective: reprinting some of that old text about errors along with some notes with the perspective of more than half a century.

1960’s Forbidden Mention of Errors
From: CHAPTER 10
Leeds and Weinberg, Computer Programming Fundamentals PROGRAM TESTING
When we approach the subject of program testing, we might almost conclude the whole subject immediately with the anecdote about the mathematics professor who, when asked to look at a student’s problem, replied, “If you haven’t made any mistakes, you have the right answer.” He was, of course, being only slightly facetious. We have already stressed this philosophy in programming, where the major problem is knowing when a program is “right.”

In order to be sure that a program is right, a simple and systematic approach is undoubtedly best. However, no approach can assure correctness without adequate testing for verification. We smile when we read the professor’s reply because we know that human beings seldom know immediately when they have made errors—although we know they will at some time make them. The programmer must not have the view that, because he cannot think of any error, there must not be one. On the contrary, extreme skepticism is the only proper attitude. Obviously, if we can recognize an error, it ceases to be an error.

If we had to rely on our own judgment as to the correctness of our programs, we would be in a difficult position. Fortunately the computer usually provides the proof of the pudding. It is such a proper combination of programmer and computer that will ultimately determine the means of judging the program. We hope to provide some insight into the proper mixture of these ingredients. An immediate problem that we must cope with is the somewhat disheartening fact that, even after carefully eliminating clerical errors, experienced programmers will still make an average of approximately one error for every thirty instructions written.

We make errors quite regularly
This statement is still true after half a century—unless it’s actually worse nowadays. (I have some data from Capers Jones suggesting one error in fewer than ten instructions may be typical for very large, complex projects.) It will probably be true after ten centuries, unless by then we’ve made substantial modifications to the human brain. It’s a characteristic of humans would have been true a hundred centuries ago—if we’d had computers then.

1960’s Cost of errors
These errors range from minor misunderstandings of instructions to major errors of logic or problem interpretation. Strangely enough, the trivial errors often lead to spectacular results, while the major errors initially are usually the most difficult to detect.

“Trivial” errors can have great consequences
We knew about large errors way back then, but I suspect we didn’t imagine just how much errors could cost. For examples of some billion dollar errors along with explanations, read the chapter “Some Very Expensive Software Errors.”

Back to 1960 again
Of course, it is possible to write a program without errors, but this fact does not obviate the need for testing. Whether or not a program is working is a matter not to be decided by intuition. Quite often it is obvious when a program is not working. However, situations have occurred where a program which has been apparently successful for years has been exposed as erroneous in some part of its operation.

Errors can escape detection for years
With the wisdom of time, we now have quite specific examples of errors lurking in the background for thirty years or more. For example, read the chapter on “predicting the number of errors.”

How was it tested in 1960
Consequently, when we use a program, we want to know how it was tested in order to give us confidence in—or warning about—its applicability. Woe unto the programmer with “beginner’s luck” whose first program happens to have no errors. If he takes success in the wrong way, many rude shocks may be needed to jar his unfounded confidence into the shape of proper skepticism.

Many people are discouraged by what to them seems the inordinate amount of effort spent on program testing. They rightly indicate that a human being can often be trained to do a job much more easily than a computer can be programmed to do it. The rebuttal to this observation may be one or more of the following statements:

1. All problems are not suitable for computers. (We must never forget this one.)
   
2. The computer, once properly programmed, will give a higher level of performance, if, indeed,
   the problem is suited to a computer approach.
   
3. All the human errors are removed from the system in advance, instead of distributing them
   throughout the work like bits of shell in a nutcake, In such instances, unfortunately, the human errors will not necessarily repeat in identical manner. Thus, anticipating and catching such errors may be exceedingly difficult. Often in these eases the tendency is to overcompensate for such errors, resulting in expense and time loss.
   
4. The computer is often doing a different job than the man is doing, for there is a tendency– usually a good one—to enlarge the scope of a problem at the same time it is first programmed for a computer. People are often tempted to “compare apples with houses” in this case.
   
5. The computer is probably a more steadfast employee, whereas human beings tend to move on to other responsibilities and must be replaced by other human beings who must, in turn, be trained.
   

In other words, if a job is worth doing, it is worth doing right.

Sometimes the error is creating a program at all.
Unfortunately, the cost of developing, supporting, and maintaining a program frequently exceeds the value it produces. In any case, no amount of fixing small program errors can eliminate the big error of writing the program in the first place. For examples and explanations, read the chapter on “it shouldn’t even be done.”

The full process, 1960
If a job is a computer job, it should be handled as such without hesitation. Of course, we are obligated to include the cost of programming and testing in any justification of a new computer application. Furthermore we must not be tempted to cut costs at the end by skimping on the testing effort. An incorrect program is indeed worth less than no program at all because the false conclusions it may inspire can lead to many expensive errors.

We must not confuse cost and value.
Even after all this time, some managers still believe they can get away with skimping on the testing effort. For examples and explanations, read the section on “What Do Errors Cost?”

Coding is not the end, even in 1960
A greater danger than false economy is ennui. Sometimes a programmer, upon finishing the coding phase of a problem, feels that all the interesting work is done. He yearns to move on to the next problem.

Programs can become erroneous without changing a bit.
You may have noticed the consistent use of “he” and “his” in this quoted passage from an ancient book. These days, this would be identified as “sexist writing,” but it wasn’t called “sexist” way back then. This is an example of how something that wasn’t an error in the past becomes an error with changing culture, changing language, changing hardware, or perhaps new laws. We don’t have to do anything to make an error, but we have to do a whole lot not to make an error.

We keep learning, but is it enough?
Thus as soon as the program looks correct—or, rather, does not look incorrect—he convinces himself it is finished and abandons it. Programmers at this time are much more fickle than young lovers.
Such actions are, of course, foolish. In the first place, we cannot so easily abandon our programs and relieve ourselves of further obligation to them. It is very possible under such circumstances that in the middle of a new problem we shall be called upon to finish our previous shoddy work—which will then seem even more dry and dull, as well as being much less familiar. Such unfamiliarity is no small problem. Much grief can occur before the programmer regains the level of thought activity he achieved in originally writing the program. We have emphasized flow diagramming and its most important assistance to understanding a program but no flow diagram guarantees easy reading of a program. The proper flow diagram does guarantee the correct logical guide through the program and a shorter path to correct understanding.

It is amazing how one goes about developing a coding structure. Often the programmer will review his coding with astonishment. He will ask incredulously, “How was it possible for me to construct this coding logic? I never could have developed this logic initially.” This statement is well-founded. It is a rare case where the programmer can immediately develop the final logical construction. Normally programming is a series of attempts, of two steps forward and one step backward. As experience is gained in understanding the problem and applying techniques—as the programmer becomes more immersed in the program’s intricacies—his logic improves. We could almost relate this logical building to a pyramid. In testing out the problem we must climb the same pyramid as in coding. In this case, however, we must take care to root out all misconstructed blocks, being careful not to lose our footing on the slippery sides. Thus, if we are really bored with a problem, the smartest approach is to finish it as correctly as possible so we shall never see it again.

In the second place, the testing of a program, properly approached, is by far the most intriguing part of programming. Truly the mettle of the programmer is tested along with the program. No puzzle addict could experience the miraculous intricacies and subtleties of the trail left by a program gone wrong. In the past, these interesting aspects of program testing have been dampened by the difficulty in rigorously extracting just the information wanted about the performance of a program. Now, however, sophisticated systems are available to relieve the programmer of much of this burden.

Testing for errors grows more difficult every year.
The previous sentence was an optimistic statement a half-century ago, but not because it was wrong. Over all these years, hundreds of tools have been built attempting to simplify the testing burden. Some of them have actually succeeded. At the same time, however, we’ve never satisfied our hunger for more sophisticated applications. So, though our testing tools have improved, our testing tasks have outpaced them. For examples and explanations, read about “preventing testing from growing more difficult.” 

If you're as interested in errors as I am, you can obtain a copy of Errors here:

ERRORS, bugs, boo-boos, blunders

SaveSave

How does a tester help with requirement gathering?

The question was posed: "How does a tester help with requirement gathering?" A number of good, but conventional, answers were given, so I decided to take my answer to a meta-level, like "what is the tester's job, generally, and how does that relate to this question?"

The tester’s job is to test, that is, to provide information about the state of a system being built or repaired. Therefore, the tester should help with requirement gathering or any other phase of development where the job of testing might be affected.

A professional tester will involve her/himself in all phases, not such much to “help” others do their job, but to assure that s/he will be able to do a professional job of testing. So, for example, in the requirements work, the tester should obviously monitor any requirements to ensure that s/he can test to see if they’re fulfilled. The tester should block any vague statements such as “user-friendly” or “efficient” or “fast”—to take just one example.

Moreover, requirements are “gathered” in many ways besides an official requirements gathering phase, so the tester must always be on the alert for any way requirments can creep into the project. For instance, developers frequently make assumptions based on their own convenience or preferences, and such assumptions are not usually documented. Or, salespeople make promises to important customers, or put promises in advertisements, so a tester must spend some time speaking (at least informally) to customers—and also reading ads.

In short, a tester must be involved right from the very first moment a project is imagined, with eyes, ears, and, yes, nose always on the alert for what must be tested, how it must be tested, and whether or not in can actualliy be tested. That's the way a tester "helps" the requirements gathering and all other phases of a project's life, on alert for anything that could affect accurately documenting the state of the system.

As a consequence of this broad view of the tester's job, a tester has to resist any assertion that "you don't have to be involved in this phase," regardless of what phase it is. Obstacles to testing can arise anywhere.

Refer to Exploring Requirements Series

Quality Software Series