Test naming tips

No naming scheme is going to guarantee good names, so here are a few intentionally vague test naming guidelines:

Should X or Y is probably testing two branches, which should be two tests. A sure sign of this is the use of if/else in the test, which should just be outlawed. I can’t think of a single case where it would be better to have a branch in a test than two tests, but I’d be interested to hear of any.

Should X and Y can be caused by many things. Sometimes the test is fine, there just isn’t a single collective term for what the code is doing. Other times the test is verifying more than one thing at a time, such as the return value and a side effect. Unless the test is extremely slow this should be split into two tests.

The name should say something about either a side effect or a return value. As a counterexample, should parse input does neither. If the parsed result is stored or returned, it should say so. If the test is actually checking that nothing is thrown when parsing valid input then that should be part of the name.

When testing errors the name should be more descriptive than should fail when …. There are many ways code can fail, and many ways the surrounding code can react to that failure, so a better name would be should return error code 5 when … or should throw not found exception when ….

Context is everything. If the test filename contains “foo” it is probably not useful to also add “foo” to any of the test names. Similarly, if you have several test names which share a context it might be useful to split those out into something with a name corresponding to that context and removing that now redundant part from the test names. As a simple example, you might want to split the “app” tests into “view” and “model” tests. After doing so you can remove “view” from the view test names and “model” from the model test names.

If it’s really difficult to come up with a good name for a test I’ve found that it’s often because I don’t have a sufficiently clear idea of what the test is meant to assert. At this point it might be useful to step back and see whether it’s possible to split up the task some more to reach some easily testable next step.

Start test names with “should”

The purpose of a test is not just to enforce some behaviour on the code under test. When the test fails it also should provide enough information to understand which behaviour failed, where it failed, and (at least superficially) why it failed. If the only output of a failing test is just a binary value like “FAIL”, that test is only giving one bit of information to the developer. A good test framework will also print the test name and a call stack. We can’t change the call stack much, at least not without changing the semantics of the test, and how to actually write the test itself is a whole other story, but what does a useful test name look like?

A trick I learned from Dave Hounslow, a former colleague, is to start test names with “should”¹. This has a few advantages over test [function name]:

  • It removes redundancy, because the function name should already be in the call stack.
  • It is falsifiable, that is, a person reviewing the test can decide to which degree the name agrees with the actual test. For example, they could point out that should replace children when updating instance verifies that new children are added, but not that old children are removed.
  • It encourages testing one property of the function per test, like should apply discount when total cost exceeds 100 dollars, should create record for valid input, and should return error code 1 for unknown error. test [function name] encourages testing everything the function does (branches, side effects, error conditions, etc.) in one test.
  • It invites the developer to write something human readable. I usually find “test …” names to be clunky to read. This may just be bias after years of using this technique.
  • It is better than a comment explaining what the test does, because the comment will not be shown when the test fails.

¹ Some frameworks require that test names are formatted in some special way, like starting with “test”, using snake case, camel case or other. I’m ignoring that part of the naming for brevity and to avoid focusing on a specific language.

Unit testing Python + MySQLdb warnings

There seems to be several methods out there, based on elevating warnings to errors using warnings.simplefilter. Here’s another method, based on recording warnings in a variable, and checking that the last one is a MySQLdb.Warning. Hopefully to be integrated in INSPIRE.

import MySQLdb
import unittest
import warnings
class TestTagInsert(unittest.TestCase):
    def test_too_long_tags(self):
        with warnings.catch_warnings(record=True) as warn:
            [Run SQL statement]
            self.assert_(len(warn) == 1) # Ensures that the next statement won't break the testing
            #If you also want to check the text of the warning:
                'truncated' in str(warn[-1].message))