Python unittest: Regression Testing Your Code

Udemy Editor

Regression Testing for Robust Software Development

During the course of the software development, in Python, there are deliverables for customers, and deadlines, but developers will find bugs, and issues in their design, or execution, or both. How do software developers build programs, and maintain it across the years, with changing feature requests?

On a simpler level how do change software, code – programs, and algorithms – without losing the original functionality? Software development learns from a maxim, ‘History repeats itself, if you don’t learn from your mistakes‘. In this blog post we will learn about writing a simple Python program and updating it as requirements change, all the time using unit tests to capture this behavior.

Simple Example – Testing Factorial Function

Factorial of a number, N, is defined to be the product of numbers from 1 to N; i.e. N! = 1*2*3* … N. Clearly a straightforward way to calculate factorial is using a for-loop where temporary initialized to 1, will have to start incrementing a counter upto N, and keep track of the product. At the end of the iterations the temporary carries the result of the factorial of N.

However yet another equivalent, and accurate, mathematical definition of factorial function is using the recursive notation; i.e. N! = (N-1)!*N

In other words it is a mathematical statement saying, if you want factorial of N, N!, and you know the factorial of (N-1), (N-1)!, then here is how you can calculate the factorial of N.

Clearly a way of writing this as a program in iterative and recursive ways would be,

# recursive way of witing factorial

def fact( n ):

  if ( n == 0 ):

    return 1.0

    return fact( n - 1)*n;

if __name__ == "__main__":

   for i in range(0,10+1):

      print("%d! = %g"%(i,fact(i)))0! = 1

Running the Program

You can download Python package for your platform from the source website, Python.org, and run the tests and programs as, $ python factorial_v1.py, is the command to interpret the code and then run the program

0! = 1

1! = 1

2! = 2

3! = 6

4! = 24

5! = 120

6! = 720

7! = 5040

8! = 40320

9! = 362880

10! = 3.6288e+06

Finding Bugs:

Not all is well with our small recursion program. Do you know what happens when you call fact function with a negative number? Lets try, by changing the code to be,

 if __name__ == "__main__":

     for i in range(0,10+1):

         print("%d! = %g"%(i,fact(i)))0! = 1

         fact( -5 )

and running the code as before you see the error message,

 File "factorial_v1.py", line 11, in fact

 return fact( n - 1)*n;

 File "factorial_v1.py", line 11, in fact

 return fact( n - 1)*n;

 File "factorial_v1.py", line 11, in fact

 return fact( n - 1)*n;

 File "factorial_v1.py", line 11, in fact

 return fact( n - 1)*n;

 RuntimeError: maximum recursion depth exceeded

This is a signature of infinite recursion, which is a serious bug to program performance. Somewhere in your code, the program is getting past a fence-post.

Clearly the if-condition in the factorial is this fence-post. If the definition can be somehow modified to stop the recursion, when the numbers are negative, we will not have issues with the negative input to this function. Our solution will involve raising an exception if we have factorial function is called with a negative input. The correct program listing will be,

 # recursive way of writing factorial

 def fact( n ):

    if ( n == 0 ):

       return 1.0

    elif ( n < 0 ):

       raise Exception("Cannot calculate factorial for negative numbers")

    return fact( n - 1)*n;

if __name__ == "__main__":

    for i in range(0,10+1):

       print("%d! = %g"%(i,fact(i)))

       print("%d! = %g"%(-5,fact( -5 )))

Complex numbers

What happens when you try using complex number input? In Python the complex numbers are written like, 5 – 4j, with the real and imaginary part suffixed with ‘j’. Changing the code to be,

	if __name__ == "__main__":

		i = 5 - 4j

		print("%d! = %g"%(i,fact(i)))

and rerunning your program, you will see the following error,

  Traceback (most recent call last):

  File "factorial_v1.py", line 20, in <module>

    print fact( 5 - 4j )

  File "factorial_v1.py", line 11, in fact

    elif ( n < 0 ):

  TypeError: no ordering relation is defined for complex numbers

Clearly something is wrong with your definition of factorial function, and we need to disallow complex numbers to the input. To fix this, you can make another change you can make to solution is to error out against complex number input, you can add guards to check input as,

# recursive way of writing factorial

def fact( n ):

    if type(n) == complex:

        raise Exception("Cannot calculate factorial for complex numbers")

    if ( n == 0 ):

        return 1.0

    elif ( n < 0 ):

        raise Exception("Cannot calculate factorial for negative numbers")

    return fact( n - 1)*n;

Adding a Unit Test

Now let us focus on writing the test cases which exercise the function in various modes, and lock down the behavior. Python has a powerful unit test framework called the unittest module, which we will use to write unit tests to ensure our solution against possible regression.

Each test point lives in a function named as ‘test_’ and it exercises the various cases of the ‘fact’ and compares the results against the inbuilt math function ‘math.factorial’. Usually the comparison of function output with known results is done using the unit-test API methods in the next section.

The code listing for ‘factorial_test.py’ follows,

import unittest

import math

import factorial_v1

from test import test_support

class FactorialTest(unittest.TestCase):

    def setUp(self):

        print("setup")

    def tearDown(self):

        print("cleanup")

    def test_positives(self):

        for x in range(0,10+1):

            act = math.factorial( x )

            val = factorial_v1.fact( x )

            print("%d! = %g == %g"%(x,val,act))

            self.assertAlmostEqual( act, val, 1e-1 )

    def test_negative(self):

        passed = False

        try:

            factorial_v1.fact( -3 )

        except Exception as e:

            passed = True and (e.message.find("Cannot calculate")>= 0 )

        self.assertTrue( passed )

        ## alternate way

        with self.assertRaises( Exception ) as cm:

            factorial_v1.fact(-3)

if __name__ == "__main__":

    test_support.run_unittest(FactorialTest)

And your code should run and produce output like,

 ----------------------------------------------------------------------

 Ran 2 tests in 0.001s

 OK

Unit Test API

Unittest API has other assertion checks which are

'assertAlmostEqual', 'assertAlmostEquals', 'assertDictContainsSubset', 'assertDictEqual', 'assertEqual', 'assertEquals', 'assertFalse', 'assertGreater', 'assertGreaterEqual', 'assertIn', 'assertIs', 'assertIsInstance', 'assertIsNone', 'assertIsNot', 'assertIsNotNone', 'assertItemsEqual', 'assertLess', 'assertLessEqual', 'assertListEqual', 'assertMultiLineEqual', 'assertNotAlmostEqual', 'assertNotAlmostEquals', 'assertNotEqual', 'assertNotEquals', 'assertNotIn', 'assertNotIsInstance', 'assertNotRegexpMatches', 'assertRaises', 'assertRaisesRegexp', 'assertRegexpMatches', 'assertSequenceEqual', 'assertSetEqual', 'assertTrue', 'assertTupleEqual',