Regression tests validate that a program still produces "correct" output after the program has been updated. A common form of regression testing compares the System.out output of a run to a known correct output. Many of our Java programs benefit from such tests. JUnit is a standard for regression testing.
The problem is that JUnit is largely designed as a test harness to create input to a method and to capture the output of a method call and compare it to the correct results. You need to do more when the output to be captured is via System.out.
Here is how to create these tests. The instructions below are targeted for NetBeans, but can be used for other IDEs as well. You will need the following JAR file and (if necessary) its Java code:
· JAR file
In NetBeans, create a JUnit file:
· right click on the Java file to create a test--> tools--> create JUnit Tests; OK. This creates a JUnit test for the particular Java file you selected.
· right click on Test Libraries--> Add JAR/Folder --> select "regressionTest.jar". This adds "regressionTest.jar" to the list of additional files you need to compile your test files.
Now, transform the test that was generated by netbeans, which might look like this:
where "out.txt" is the name of the file to which all System.out is to be written, and "correctOut.txt" is the name of the file that contains the correct output against which the "out.txt" is compared. "false" means don't sort the files before comparing -- sorting is useful if the order in which lines are output does not matter.
To execute a test program in NetBeans: Run-> TestProject.
Note: this jar file requires the use of a JUnit jar file, which you can get here.
Sometimes programs produce file output, in addition to terminal output. You want to check the contents of these files too in regression testing. Use the same testing harness (routines above) except with different calls. Simply use one or more calls to RegTest.Utility.validate() to compare outputted files with their correct counterparts. You don't need to use RegTest.Utility.redirectStdOut() if you are not interested in capturing standard out output. So the general format of a test is:
Generally, you may not know the correct output of your program. What can do in such circumstances is to run the program, collect its output, and manually check to see if the output is correct. Once it is correct, you can simply rename the output file as "correct", and use it in your tests above. Remember: when you are performing multiple tests, NetBeans will simply count the number of tests that have succeeded and failed. So looking at any output files that may have been built along the way won't help you (as you won't know from which test the file was produced). By developing and debugging tests one at a time, and finding their correct output, you'll save yourself a lot of headaches.