Unit Testing and approval tests

It’s been years now that unit testing frameworks and tools have grabbed our attention, made their way into our favorite IDE and sparked yet another wave of seemingly endless “my framework is better than yours” wars. And then there are the principal wars of whether TDD is better than Test After Development. And most excitingly the automated testing tools etc. Oh, and let’s not forget mocks! So we have all the tools we need – right? Well, kind of, no.

I recently attended a talk by Llewellyn Falco and Woody Zuill the other day, and they used a little library called Approval Tests (http://approvaltests.com/). I immediately fell in love with the concept and the price ($0). Llewellyn is one of the two developers of the product, hosted on http://sourceforge.net/projects/approvaltests/.

What does it do that years of frameworks don’t?

For me, a major nuisance is that you have a piece of code, and that piece of code produces a result (object, state – call it what you will). That result is commonly named “actual” in Visual Studio unit tests. The developer must then assert against that result to ensure that the code ran properly. In theory, unit tests are supposed to be small and only test one thing. In reality, functions which return complex objects rarely can be asserted with one assert. You’d typically find yourself asserting several properties or testing some behavior of the actual value. In corporate scenario, the unit tests might morph into some degree of integration tests and objects become more complex, requiring a slew of assert.this and assert.that().

What if you could just run the code, inspect the value returned, and – if it satisfies the requirements – set “that value” to be the comparison for all future runs? Wouldn’t that be nice? Good news: Approval Tests does just that. So now, instead of writing so many asserts against your returned value, you just call ApprovalTests.Approve() and the captured state is compared against your current runtime. Neat huh?

But wait, there’s more! What if your requirements are regarding a windows form, or a control? how do you write a unit test that asserts against a form? The answer is simple and genius: Capture the form as an image, and compare the image produced by your approved form (after you have carefully compared it to the form given to you by marketing – right?) and compare it to each subsequent run. ApprovalTests simply does a byte by byte comparison of the resultant picture of the form and tells you if it matches what you approved.

Let’s break down the steps for installation:

1. Add a reference to the ApprovalTests.dll assembly.

Yeah,it’s that simple.

Ok. Let’s break down the steps for adding an approval to your unit test

1. Write your unit test (arrange, act – no asserts) as usual.

2. Add a call to Approve().

Yeah, it’s that simple.

How do you tell it that the value received is “the right one”? Upon first run, the Approve() call will always fail. Why? Because it has no stored approved value to compare the actual it received against. When it fails, it will print out a message (console or test report – depends on your unit test runner). That message will contain a path for the file that it received, complaining about a lack of approved file. The value captured from that run (and any subsequent run) is stored in a file named something like “{your unit test file path}{your method}.received.xyz”. If you like the result of the run -the image matches what the form should look like, or the text value is what your object should contain etc – then you can rename it to “{your unit test file path}{your method}.approved.xyz”. You should check the approved file into your source control. After all, it’s content constitutes the basis of the assertion in your unit test!

Consider the very simple unit test code:

[Test]
        public void DateTime_Constructor_YMDSetDate()
        {
            DateTime actual;
            actual = new DateTime(2038, 4, 1);
            Approvals.Approve(actual.ToString());
        }

The function under test here is the constructor for System.DateTime which takes 3 parameters – month day and year. Upon first run, the test will fail with a message resembling

“_TestCase ‘MyProject.Tests.DateTimeTest.DateTime_Constructor_YMDSetDate’

failed: ApprovalTests.Core.Exceptions.ApprovalMissingException : Failed Approval: Approval File “C:\….\MyProject\DateTimeTest.DateTime_Constructor_YMDSetDate.approved.txt” Not Found.”_

That’s because this file was never created – it’s your first run. Using the standard approver, this file will actually now be created but will be empty. In that same directory you will find a file named {…}.received.txt. This is the capture from this run. You can open the received value file, and inspect the text to ensure that the value returned matches your expected value. If it does, simply rename that file to .approved.text or copy it’s content over and run the test again. This time, the Approve() method should succeed. If at any point in the future the method under test returns a different value, the approval will fail. If at any point in the future the specs change, you will need to re-capture a correct behavior and save it.

How do you easily compare the values from the last run and the saved approved value? The approved file is going to be a text file for string approvals, and an image file for WinForm approvals. As it turns out, you can instruct Approval Tests to launch a diff program with the 2 files (received, approved) automatically upon failure, or just open the received file upon failure or silently just fail the unit test. To control that behavior, you use a special attribute

[TestFixture]
[UseReporter(typeof(DiffReporter))]
public class ObjectWriterTest
{

The UserReporterAttribute allows you to specify one of 3 included reporters

1. DiffReporter – which opens tortoisediff for diffing the received and approved files if the comparison fails.

2. OpenReceivedFileReporter – which launches the received file using the application registered to it’s extension on your system if the comparison fails.

3. QuietReporter – which does not launch any program but only fails the unit test if the comparison fails.

When your have a CI server and run unit tests as part of your builds, you probably want to use the quiet reporter. For interactive sessions, one of the first 2 will probably be more suitable.

How are value comparisons performed? Under the standard built in methods, a file is written out and compared to byte by byte. If you wish to modify this or any behavior, you can implement your own approver or file writer or reporter by implementing simple interfaces. I ended up adding a general use object writer so that I can approve arbitrary objects. The effort was fairly painless and straightforward.

I did have to read the code to get the concept. If only my time machine worked: I could have read my own blog and saved myself 20 minutes. Yeah, right.

The project has some bells and whistles – plug ins for a few IDE’s and there’s a version for Java and Ruby. I have not reviewed these versions.

There you have it folks- a shiny new tool under your belt. I can see this saving my hours of mindless typing of Assert.* calls and I can go home early. Yeah, right.