Regression Testing

Goal

Regression tests validate that a program still produces "correct" output after the program has been updated.  A common form of regression testing compares the System.out output of a run to a known correct output.  Many of our Java programs benefit from such tests.  JUnit is a standard for regression testing.

Problem

The problem is that JUnit is largely designed as a test harness to create input to a method and to capture the output of a method call and compare it to the correct results.  You need to do more when the output to be captured is via System.out.

Solution Regression Testing System.out

Here is how to create these tests. The instructions below are targeted for NetBeans, but can be used for other IDEs as well.  You will need the following JAR file and if you are interested, its Java code:

In NetBeans, create a JUnit file: 

Now, transform the test that was generated, which might look like this:

public void testMain() {
System.out.println("main");

String[] args = null;
Main.main(args);

// TODO review the generated test code and remove the default call to fail.
fail("The test case is a prototype.");
}

to:  

public void testMain() {
RegTest.Utility.redirectStdOut("out.txt"); // redirects standard out to file "out.txt" String[] args = null; Main.main(args); RegTest.Utility.validate("out.txt", "correctOut.txt", false); // test passes if files are equal }  

where "out.txt" is the name of the file to which all System.out is to be written, and "correctOut.txt" is the name of the file that contains the correct output against which the "out.txt" is compared.  "false" means don't sort the files before comparing -- sorting is useful if the order in which lines are output does not matter.

To execute a test program in NetBeans: Run-> TestProject.

Note: this jar file requires the use of a JUnit jar file, which you can get here.

Regression Testing Other Output Files

Sometimes programs produce file output, in addition to terminal output.  You want to check the contents of these files too in regression testing.  Use the same testing harness (routines above) except with different calls.  Simply use one or more calls to RegTest.Utility.validate() to compare outputted files with their correct counterparts.  You don't need to use RegTest.Utility.redirectStdOut() if you are not interested in capturing standard out output.  So the general format of a test is:

// this call is optional -- use only if you want to capture standard out
RegTest.Utility.redirectStdOut("stdout.txt"); (place output in file "stdout.txt")

{run program};

// now for every file that is produced, you can compare it to its correct output:
RegTest.Utility.validate("stdout.txt", "correctStdOut.txt", false); (false means don't sort before comparing files)
RegTest.Utility.validate("A.txt", "correctA.txt", true); (true means sort both files before comparing)
RegTest.Utility.validate("B.txt", "correctB.txt", true);

Advice

Generally, you may not know the correct output of your program.  What can do in such circumstances is to run the program, collect its output, and manually check to see if the output is correct. Once it is correct, you can simply rename the output file as "correct", and use it in your tests above.  Remember: when you are performing multiple tests, NetBeans will simply count the number of tests that have succeeded and failed.  So looking at any output files that may have been built along the way won't help you (as you won't know from which test the file was produced).  By developing and debugging tests one at a time, and finding their correct output, you'll save yourself a lot of headaches.