There's a popular question on the Programmers Stack Exchange site asking, "Are unit tests really that useful?" and the second answer seems to be responding to a straw-man version of automated testing. The comments show that a lot of people seem to think it's a reasonable description, so I thought it was worth addressing in order to explain how automated testing can be useful, but only if you view it as a tool and not as magic (which is unfortunately how some people teach it).

For simplicity's sake, I'll assume that any readers already agree that some sort of testing is an integral part of software development, but may not believe that automated testing is useful. Note that testing can mean a lot of things, like:

  • Compiling -- Testing that your code meets the requirements of the language (syntax, typing, etc.).
  • Static code analysis -- Testing that your code doesn't have any obvious mistakes (and sometimes testing adherence to style guidelines). Compilers tend to do this to a limited extent.
  • Manual testing -- Testing that it runs at all, and that it does what you're expecting. This frequently involves outside testers, since the person writing the code may not use it in the same way that a "normal" user would.
  • Running automated tests -- Similar to manual testing, except you make the computer do it for you.

Arguments Against Automated Testing

The arguments I've seen against automated testing tend to fall into similar groups, which I would categorize as:

  • Automated testing can't catch all errors, so it's not useful.
  • Automated testing takes more time than manual testing, so it's not cost-effective.

Automated Testing Isn't Useful

The argument that automated testing isn't useful goes something like this:

The concept behind unit tests is based on a premise that has been known to be false since before unit testing was ever invented: the idea that tests can prove that your code is correct.

Having lots of tests that all pass proves one thing and one thing only: that you have lots of tests which all pass. It does not prove that what the tests are testing matches the spec. It does not prove that your code is free from errors that you never considered when you wrote the tests. And last but not least, it does not prove that the tests, which are code themselves, are free from bugs.

This argument could be summarized as:

  1. Unit tests can't prove that your code is 100% correct.
  2. ???
  3. Unit tests aren't useful.

You may wonder why I left "???" as a second premise, and the answer is that without another premise, this argument is invalid. The first premise does not lead to the conclusion. The only premise I can think of that would fill in that blank is:

  • Unit tests are only useful if they catch 100% of errors.

I would argue that this premise would be very difficult to support, especially if you claim the opposite for manual testing.

The truth about automated testing (and testing in general) is that testing can't prove that your program is perfect, but it can prove that it performs a task correctly in a given situation.

Consider a program for generating the nth Fibonacci number:

def fibonacci(n):
    ...

There are infinitely many Fibonacci numbers, so we clearly can't test every input for this function. However, we could test some inputs:

def test_first_10_fibonacci_numbers():
    # From Wikipedia: https://en.wikipedia.org/wiki/Fibonacci_number
    expected = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
    actual = [fibonacci(n) for n in range(0, 10)]
    assert(expected == actual)

Is this test sufficient to prove that my implementation is good? No, it's not:

NUMS = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
def fibonacci(n):
    return NUMS[n]

However, if one of your programmers writes a function like this, you have bigger problems than testing. Unless your developers are activitly malicious, you'll get a version that they think works, and your tests can add one more level of verification which almost no effort. This test doesn't prove that the function gives correct inputs for all output, but no test can prove that. It does prove that in at least one case, for valid inputs, we got valid outputs, and that's exactly as much information as a manual tester could give you. It's also far more valuable than some people understand. Most of the time, when a function doesn't work right, it doesn't work.

Automated Testing Takes Too Much Time

How this is useful becomes more obvious with more complicated functions:

def test_get_first_user_from_database():
    # User ID 1 is an admin, so we assume it will always exist
    id = 1
    user = get_user_from_database(id)
    assert(user is not None)
    assert(user.id == id)

This test trivial to write, but it tests a number of vital things without wasting a tester's time:

  • Can we access the database?
  • Can we get something we recognize as a user out of it?
  • Does the user we got back have the ID we requested?

Or try another basic test:

def test_search_for_movie_by_actor_kevin_bacon():
    movies = search_for_movies_by_actor("Kevin Bacon")
    assert(movies is not None)
    assert(len(movies) > 0)
    for movie in movies:
        assert("Kevin Bacon" in movie.actors)

This time the test is checking:

  • Can we access our list of movies?
  • Does the search engine return false positives when doing actor searches?

Now extend this to everything that's:

  • Easy to write automated tests for
  • Likely to break

And remember that these are run automatically by a computer, so we can afford to run them after every build. How many companies are willing to pay a tester to run through their whole testing list after every build? If they're checking things like this, they're wasting their time.

Again, these tests could miss things, but everything they do catch is something that can be fixed more easily, because the developer knows immediately that something is wrong, and don't waste the time of testers.

Manual Testing

I've focused a lot on wasting tester time, and I think that's important. Remember that no one has unlimited time or money, so making things easier for testers means the company can either save money (and possibly hire more developers), or use the testers time on more important things (the things that are hard to write automated tests for). Why have a tester spend time checking if things work when they can check if things look right and work how they expect.

They'll inevitably find bugs that could have been caught by automated tests while they do their more important work, but at least they won't be spending all of their time on it, and when it does happen, you can create new automated tests so at least that same problems will never come back.

My opinion is basically this: Manual testers are much too valuable to waste on things that can be checked by the computer. The computer can't check everything, but as a first line of defense, it's much faster and much cheaper than human testers.

Conclusion

The point of this article isn't to convince you to write 100% coverage automated tests all of the time. In fact, I don't think that's a good idea because you'll waste time writing tests for obvious things, or making tests that are so complex you need to fix them after every change.

The point is that automated testing has a place in software development, and using it can be a huge productivity boost. The argument that it's not always useful is meaningless. Like most things, automated testing is useful if you use it for what it's good at, which is catching bugs quickly when it's easy to do so.