JUnit Regression Tests with System.out

Goal

Regression tests validate that "correct" output of a program is produced after the program has been updated.  A common form of regression testing compares the System.out output of a run to a known correct output.  Many of our Java programs benefit from such tests.  JUnit is a standard too for regression testing.

Problem

The problem is that JUnit is largely designed as a test harness to create input to a method and to capture the output of a method call and compare it to the correct results.  You need to do quite a bit more when the output to be captured is via System.out.

Solution

Here is how you can create these tests. The instructions below are targeted for NetBeans, but can be used for other IDEs as well.  You will need the following JAR file (and if you are interested, its Java code):

• regressionTest.jar

• source code (optional, FYI)

In NetBeans, you are to create a JUnit file for the main() method of your program: 

• right click on the Java file to create a test--> tools--> create JUnit Tests; OK.  This creates a JUnit test for the particular Java file you selected.

• right click on Test Libraries--> Add JAR/Folder --> select "regressionTest.jar".  This adds "regressionTest.jar" to the list of additional files you need to compile your test files.

Now, transform testMain that was generated:

public void testMain() {
   System.out.println("main");

   String[] args = null;
   Main.main(args);

   // TODO review the generated test code and remove the default call to fail.
   fail("The test case is a prototype.");
}

to:  

public void testMain() {
   System.out.println("main");

   RegTest.Diff.initTest("out.txt", "correct.txt");

   String[] args = null;
   Main.main(args);

   if (RegTest.Diff.finishTest()) fail("The test fails.");
}
 

where "out.txt" is the name of the file to which all System.out is to be written, and "correct.txt" is the name of the file that contains the correct output against which the "out.txt" is to be compared.

Note: initially there are no "out.txt" or "correct.txt" files.  The first time you run the test, "out.txt" file is created (and of course, the test will fail because there is no "correct.txt" file).  If you determine that "out.txt" is the correct output, simply rename "out.txt" to "correct.txt". Subsequent test executions will compare generated "out.txt" to "correct.txt" as required.

To execute a test program in NetBeans: Run-> TestProject.