//FIXME -- Jason Felice’s blog

Unit Testing in Haskell

QuickCheck is the gold standard for testing pure functional (side-effect free) code in Haskell. It allows you to assert properties of functions, such as “this function returns a non-negative result for any input.” QuickCheck generates input cases and checks the proposition automatically.

HUnit is the usual xUnit clone for Haskell. It deals with actions (Haskell’s term for functions having side-effects). You perform some action then assert the resulting state has been correctly mutated.

It is mighty convenient to use both for TDD on any non-trivial program.

Test Collection

Relying on the coder to remember to add tests to suites will eventually produce orphaned suites. Then comes the false confidence: I ran all the tests and they passed, and I know it doesn’t break assertion X, because there’s a test for that. I’ve seen it.

As OneSock has gotten a little bit bigger (it’s still small, don’t worry), I start to worry about this. I did a bit of research and found some test-collection tools:

quickcheck-script

This script will scan the source files supplied on the command-line for QuickCheck assertions and run them. It is very useful and easy to use. The drawback here is that it will not run my HUnit tests.

hstest

This program will scan for QuickCheck1 and HUnit tests in all sources in the current directory and run them. The drawback is its current lack of support for QuickCheck2. The philosophy is right for what I want to do – have a simple “brain idle” command for my TDD cycle that runs all the tests.

test-framework

This package provides a class with instances for HUnit, QuickCheck1, and QuickCheck2. For the most part, manual collection is required; however, there is a test-framework-th package which uses the TemplateHaskell extension to provide a meta-function that scans the current source file for tests and collects them.

The Decision

I’ve decided to go with hstest.

quickcheck-script doesn’t run HUnit tests.

While test-framework seems to have more mindshare and also seems to be well-thought-out, there is no script to automatically collect tests. The TemplateHaskell trick is nice; however, introducing another layer of complexity just for test collection is a bit overwhelming at this early stage—especially because it doesn’t solve the original problem: now a programmer must remember to register all source files rather than all tests in a test-runner program.

To work around the current lack of support for QuickCheck2, you can supply parameters to hide and expose packages. I’ve written this shell function:

function t() {
	hstest --expose-package=QuickCheck-1.2.0.0 --hide-package=QuickCheck-2.1.0.3
	rm -f hugsin
	return $?
}

I’ve found that my current few properties run unaltered under QuickCheck1.

I’ve also specified QuickCheck1 in my cabal file for the moment so that I don’t use features that hstest doesn’t like. This provokes a warning:

jfelice@flarp64:~/src/onesock$ cabal configure
Resolving dependencies...
Configuring OneSock-0.1...
Warning: This package indirectly depends on multiple versions of the same
package. This is highly likely to cause a compile failure.
package OneSock-0.1 requires QuickCheck-1.2.0.0
package Crypto-4.2.1 requires QuickCheck-2.1.0.3
jfelice@flarp64:~/src/onesock$

No build issues occur. Maybe the linker is eliminating all QuickCheck references (which are unreachable from main).

I work with message formats a lot. Being able to conjure up a hex dump on the spot is an invaluable skill. You’d think that you should have a library function for this; however, there’s often problems – it’s not in the path for this project, the built-in one doesn’t work on this JVM, the link settings aren’t right to use it, etc. So I’ve typed this hundreds of times:

  for (int i = 0; i < data.length; ++i) {
    if (i%16==0) System.err.println();
    System.err.printf("%02x ",data[i]);
  }
  System.err.println();

The C and C++ equivalents are almost identical.

A couple weeks ago, I met up Attila Domokos and did the LCD rubyquiz. Let me set the scene about the pattern in use at these Hackabous:

1. Pair up.
2. One test is written first.
3. Only as much functionality as is needed to make the test pass is written.
4. The next test is written…

…and so forth.

We moved through parsing the command-line easily, then coded the test for displaying a single digit. We noticed we were at a particular sort of impasse:

Implementing as little code as possible to pass this test could not produce code which could survive the next test. At least, not in a way that I can see.

What should you do? Should you write the code, debug it and pass the test, then add the next test and delete the code? This seems like an odd question to ask. The only reason I even ask it now is I’ve found some people who really advocate this.

Another example: I listened during the Roman Numerals kata as one team made first I, then II and III pass by coding:

  def convert(n)
    n.split("").size
  end

If I understand proponents’ arguments, in writing the code you are learning, and this learning will dominate the inefficiencies in this process over time.

I honestly think this is a misreading of agile. Agile was built in the context of an industry addicted to wasteful, heavy and rigid process. Should TDD become a wasteful, heavy and rigid process?

How do I proceed in these cases? First, I have nothing against writing more than one test up front. If I can think of several of the corner cases and potential pitfalls right now, I will write them down in xUnit. Sometimes switching from researching the requirements to coding is an expensive operation, and in these cases, I’ll lay out five or ten tests documenting the requirements. Sometimes, I have no idea how I’m going to implement something. Writing a batch of tests up front allows me to explore the corner cases and think more about designing a useful approach.

Given the multiple tests, I have no problem writing the more complicated algorithm first. We’ve already determined that we can’t (or perhaps just don’t know how) to evolve it, so we’re going to end up going for it anyway.

    public void testCallsRollbackIfSendingAuthResponseFails() throws Exception {
        final MockSender sender = new MockSender();
        final MockSession session = new MockSession() {
            @Override
            public DASAuthObj authorize(final DASAuthObj authObj) {
                mgr.dispatch( ... // AGHAGHAGH!
                return super.authorize(authObj);
            }
        };
        final DASSessionManager mgr = new DASSessionManager(session, sender);
        ...

[EDIT: OK, you can do it manually with Java arrays. Still icky.]

    public void testCallsRollbackIfSendingAuthResponseFails() throws Exception {
        final MockSender sender = new MockSender();
        final DASSessionManager[] mgrHolder = new DASSessionManager[1];
        final MockSession session = new MockSession() {
            @Override
            public DASAuthObj authorize(final DASAuthObj authObj) {
                mgrHolder[0].dispatch(new DASCloseAuthSinkPacket(1, true, 42));
                return super.authorize(authObj);
            }
        };
        final DASSessionManager mgr = new DASSessionManager(session, sender);
        mgrHolder[0] = mgr;

I’ve started a set of R utilities for software kanban at GitHub. I’ve been using the R environment more lately, and I really like how easy it makes visualizing and working with data. It has a bit of an “old-school” environment feel as well and it is functional and efficient. If you are an R guru (I’m not really, though I’m learning) and you have utilities, please contribute!

The utilities expect data to be stored in a data.frame. The first column must be named “Date” and have “Date” class. The second column is currently ignored; however, the intention is that it indicates the ticket category (usually represented on the board by ticket color), and so it should be a factor. Columns three onward (as many as you have flow steps) are tallies of each work flow step for the day, in reverse chronological order – starting with “Done” and ending with “Backlog”, in other words.

The easiest way to get this data into R is to store it in a CSV file and use read.csv to load it. Here is some sample data:

Date,Category,Done,QA,Development,Ready,Backlog
2009/10/20,Project,0,2,2,0,0
2009/10/20,Urgent,0,0,1,0,0
2009/10/22,Project,1,2,1,0,0
2009/10/22,Urgent,0,0,2,0,0
2009/10/23,Project,1,2,2,0,0
2009/10/23,Urgent,0,0,2,0,0

Load this into R with:

> my.data <- read.csv("my_data.csv")

An alternate method for maintaining your data within R is to use the fix() function on a daily basis, which is what I do. Two warnings, however:

1. Make sure to save your workspace so that you don't lose your stats.

2. fix() will strip the "Date" class from the Date column.

To re-add the Date class after using fix(), do this:

> my.data$Date <- as.Date(my.data$Date)

Plotting your CFD

Check out the rkanban sources into the "rkanban" directory under your R working directory and source the kanban.R file.

You can then plot your CFD from within R like so:

> source("rkanban/kanban.R")
> PlotCFD(my.data)

We’ve been Guicing up XPay for a couple weeks. For those not familiar with Guice, it is a dependency injection framework.

Guice in Ten Seconds

You annotate your classes types’ primary constructors, create “modules” which bind concrete types or singleton instances to interface and abstract types, then ask Guice to create you one of some type. Guice will automatically create all dependencies (constructor parameters) for that type before invoking the primary constructor, and if any of those types require dependencies create those first, and so forth. For example, if your class Foo needs some kind of PackagingStrategy (presumably an interface) passed to its constructor, and your module binds PackagingStrategy.class to BinaryPackagingStrategy.class, then when you ask Guice for an instance of Foo, it will create a new BinaryPackagingStrategy, “newing up” its constructor parameters recursively, and pass this to Foo’s constructor.

The benefits of this scheme are in the class of “hard to describe, but definitely there.” First and foremost, it improves system design, making interfaces clearer and more self-documenting by removing hidden dependencies on things like Singletons and Monostates. This improves testability because mucking with the global state encapsulated in Singleton and Monostate patterns requires a much deeper knowledge of classes under test and some often-shady practices. Without a good DI framework, passing a dependency down several layers of object graph is complicated and provides extra coupling, making the Singleton pattern much more attractive. With the DI framework, there’s really no inclination to make the dependency of a class anything other than a constructor parameter.

But, More Boilerplate?

There’s plenty of boilerplate that Guice removes, mostly in the “bean wiring” category, and this is good. Interestingly, there are areas in which I’ve found myself writing more boilerplate with Guice, and an instance of this is what I’d like to discuss today.

I work with some people who have become thoroughly disgusted with OOP and advocate for functional style with immutable data types. I appreciate functional style with immutable data types, but I must say that the Strategy pattern is something OOP does well and not something that FP does nearly as well. I tend to use strategy pattern quite a bit in our XPay (and in our C++ product, DAS). One reason to prefer Strategy pattern to switch statements is to have a single point of control over which strategy implementation to use. This use case usually conjures itself into being in a Factory method pattern which has the single switch statement which provides an instance of the concrete type based on input parameters.

In Guice, you can inject a Provider<Foo> and Guice will automatically create a type that produces instances of Foo. This is useful for dependencies. So our factory ends up looking something like this:

class FooFactory {
 
    private final Provider<TypeAFoo> typeAFooProvider;
    private final Provider<TypeBFoo> typeBFooProvider;
    private final Provider<TypeCAFoo> typeCAFooProvider;
    private final Provider<TypeCBFoo> typeCBFooProvider;
    private final Provider<TypeDFoo> typeDFooProvider;
    private final Provider<TypeEFoo> typeEFooProvider;
 
    @Inject
    public FooFactory(
            final Provider<TypeAFoo> typeAFooProvider,
            final Provider<TypeBFoo> typeBFooProvider,
            final Provider<TypeCAFoo> typeCAFooProvider,
            final Provider<TypeCBFoo> typeCBFooProvider,
            final Provider<TypeDFoo> typeDFooProvider,
            final Provider<TypeEFoo> typeEFooProvider
            )
    {
        this.typeAFooProvider = typeAFooProvider;
        this.typeBFooProvider = typeBFooProvider;
        this.typeCAFooProvider = typeCAFooProvider;
        this.typeCBFooProvider = typeCBFooProvider;
        this.typeDFooProvider = typeDFooProvider;
        this.typeEFooProvider = typeEFooProvider;
    }
 
    public Foo get(char type, boolean inverted) {
        switch (type) {
        case 'A':
            return typeAFooProvider.get();         
        case 'B':
            return typeBFooProvider.get();         
        case 'C':
            if (inverted)
                return typeCAFooProvider.get();
            else
                return typeCBFooProvider.get();
        case 'D':
            return typeBFooProvider.get();         
        case 'E':
            return typeBFooProvider.get();         
        default:
            throw new FooFactoryException("Can't determine type.");
        }
    }
};

Ugh!

This is “better” boilerplate than the bean wiring before in that it hints that we might be able to concoct a general solution; however, I haven’t yet found it. One solution which I’ve rejected is the “injecting an injector” solution:

class FooFactory {
 
    private final Injector injector;
 
    @Inject
    public FooFactory(final Injector injector) {
         this.injector = injector;
    }
    public Foo get(char type, boolean inverted) {
        switch (type) {
        case 'A':
            return injector.createInstance(TypeAFoo.class);
        case 'B':
            return injector.createInstance(TypeBFoo.class);
        case 'C':
            if (inverted)
                return injector.createInstance(TypeCAFoo.class);
            else
                return injector.createInstance(TypeCBFoo.class);
        case 'D':
            return injector.createInstance(TypeDFoo.class);
        case 'E':
            return injector.createInstance(TypeEFoo.class);
        default:
            throw new FooFactoryException("Can't determine type.");
        }
    }
};

The reason I’ve rejected this approach is that it prevents Guice from checking the entire dependency graph at boot – Guice doesn’t know which types you are going to create with the injector, and this has to defeat a lot of it’s validation magic.

I’ve been collecting these for a little while. Since we are now well under way in migrating to VS2005, now is a good time to post them.

.\Components/CommComponents/FDMSInterleaveTcpipComm/Test_FDMSInterleaveTcpipComm.h(106) : fatal error C1001: INTERNAL COMPILER ERROR
        (compiler file 'msc1.cpp', line 1794)
         Please choose the Technical Support command on the Visual C++
         Help menu, or open the Technical Support help file for more information

was caused by:

    template<typename VersionPolicy> class SVDotPacketProductionPolicy;
    class SVDotPacketProductionPolicy<SV24VersionPolicy>;
    friend class SVDotPacketProductionPolicy<SV24VersionPolicy>;
    class SVDotPacketProductionPolicy<SV40VersionPolicy>;
    friend class SVDotPacketProductionPolicy<SV40VersionPolicy>;

Include\boost/bind/arg.hpp(25) : fatal error C1001: INTERNAL COMPILER ERROR
        (compiler file 'msc1.cpp', line 1794)
         Please choose the Technical Support command on the Visual C++
         Help menu, or open the Technical Support help file for more information

This one started after I added

#include <boost/bind.hpp>

to a file and a simple, every-day use of boost::bind(). Moving the #include directive up before a number of other standard includes resolved the issue.

C:\projects\DASV2>scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
cl /FoComponents\AuthComponents\Credit\CCMps\Release\Tests.obj /c Components\AuthComponents\Credit\CCMps\Release\Tests.cpp /nologo /
TP /nologo /MT /W2 /GX /GR /O2 /Zi /Zm200 /D "WIN32" /D "_WINDOWS" /D "NDEBUG" /D "_MBCS" /FD /D "_WIN32_WINNT=0x0400" /D "_WIN32_DC
OM" /IWindowsSDK\Include /IInclude /Icryptlib /IXML\Xerces\src /I. /IComponents\AuthComponents\Credit\CCMps
Tests.cpp
Include\boost/function/function_template.hpp(514) : fatal error C1001: INTERNAL COMPILER ERROR
        (compiler file 'msc1.cpp', line 1794)
         Please choose the Technical Support command on the Visual C++
         Help menu, or open the Technical Support help file for more information
scons: *** [Components\AuthComponents\Credit\CCMps\Release\Tests.obj] Error 2
scons: building terminated because of errors.

06/01/2009 – This one was the result of passing a pointer to a static member function to boost::function1<>.

C:\projects\DASV2>scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
python.exe Utilities/CxxTest/cxxtestgen.py --runner=XmlPrinter --have-eh --have-std --check-memory -o CBase\ISOMsg\Debug\Tests.cpp CBase\ISOMsg\Test_Fields.h
cl /FoCBase\ISOMsg\Debug\Tests.obj /c CBase\ISOMsg\Debug\Tests.cpp /nologo /TP /nologo /MTd /Od /W2 /GX /GR /Zi /Zm200 /D "WIN32" /D "_WINDOWS" /D "_MBCS" /FD /D "_WIN32_WINNT=0x0400" /D "_WIN32_DCOM" /D "DEBUG" /D "_DEBUG" /IWindowsSDK\Include /IInclude /Icryptlib /IXML\Xerces\src /I. /ICBase\ISOMsg
Tests.cpp
.\CBase/ISOMsg/Test_Fields.h(157) : fatal error C1001: INTERNAL COMPILER ERROR
        (compiler file 'msc1.cpp', line 1794)
         Please choose the Technical Support command on the Visual C++
         Help menu, or open the Technical Support help file for more information
scons: *** [CBase\ISOMsg\Debug\Tests.obj] Error 2
scons: building terminated because of errors.

06/18/2009 – This one is the result of attempting to use a template member of template class. Here’s the (abbreviated) code:

template<typename C, C v>
struct ThreeArgConstructor
{
    template<class BaseT>
    struct Type : public BaseT
    {
        Type(int length)
            : BaseT(length, "TEST FIELD PACKAGER", v)
        {}
    };
};
 
template<
      BaseT
    , ConstructorT
    >
class Foo
{
    typedef typename ConstructorT::template Type<BaseT> TestPackager;
 
    void run() {
        TestPackager(10);
    }
};
 
int main() {
    Foo<Bar, ThreeArgConstructor<bool, false> >().run();
    return 0;
}

Context

A significant portion of your activities aren’t visible to your management, possibly because

• you have become the “product expert” which consults with others or
• your products’ production deployments require significant developer attention

Therefore,

Divide your whiteboard up into columns representing your work flow (e.g. “Backlog,” “Ready”, “Develop”, “QA”, “Done”) and obtain a large supply of Post-It(tm) notes. Use separate colors for the items assigned to you by your management and the items which come to you as the domain expert. If necessary, track enough information to build a cumulative flow diagram.

Discussion

There is something visceral about this approach that conveys understanding in a way which is not possible by explanation or discussion. It may be the case that your management knows what is happening but doesn’t quite “get it.”

This is a minimum useful subset of software kanban and an instance of “Information Radiator.”

A cumulative flow diagram can be created by simply counting the number of tickets in each bucket each morning and entering these into a spreadsheet in backwards order. You can read your average work-in-process (WIP) and cycle time from this diagram, making it an incredibly inexpensive way to measure the amount of effort spent on visible work versus other work.

Observed Instances

One

[to be filled in]

Context

• Each release of your product has some unique aspect to it, for example an intended customer-specific configuration.
• Releases of your product are frequently made which fail in some, though not necessarily always the same, downstream step.
• Verifying correct operation of each build empirically is time consuming or costly.

Therefore

1. Write a program which is the simplest possible wrapper for your release of the artifact consumed downstream.
2. Each time a failure causes a flow interruption, modify the program to detect this problem with the artifact and fail loudly, refusing to produce the artifact.

Discussion

This pattern operates on the general principle that a shorter feedback loop for failures will have less negative effect on effort hours and throughput. This is the same principle embodied by test-driven development applied in a slightly different context. To build on the analogy, consider instituting a policy of adding the automated release check to your artifact-builder before correcting the downstream problem, and verifying the release check by attempting to release a new build without correcting the original problem.

This is a kind of test-driven development which is fed by downstream regressions. It is an attempt to introduce a positive rachet effect.

If you do not have some unique aspect to each release, you could simply add unit tests which fail the build if in-repository artifacts are somehow inconsistent or incorrect. Even though we don’t usually think to add unit tests to verify, say, properties of configuration files stored in the repository, these unit tests provide a tighter feedback loop than this pattern and should be preferred.

Observed Instances

One

We have a product where frequent, customer-specific releases are made to an independent QA department. Two files are delivered, an installer exe and an ini file. The ini file is read by the installer exe and causes the installer to install the correct components and install the proper configuration into the registry. The install is a lengthy process (twenty minutes) on machines which have a lot of software installed, discouraging developers from running the installer. Also, verifying correct operation requires setting up a protocol emulator which may not be available.

The ini file layout is very brittle. It contains many options which must be consistent with other options. There are also many options which must be specified for every component in the configuration. Yet other options can cause some other options to be ignored entirely. The install itself rarely fails, it leaves the machine configured in an unusable state instead.

A significant portion of my time was spent in the QA lab troubleshooting failed installs. When I first started maintaining this software, it seemed like one third of a typical day might be spent in the QA lab. Since QA is on an isolated network, I went through many thumb drives, not remembering which machine I had last left it in.

I wrote a wrapper for building the install after reading the ini file. This also had the benefit of allowing us to remove unused components from an install. For each failure found in QA, I made it fail loudly and abort before building an install if the problem was detected with the ini file.

I haven’t been in the QA lab in more than four months. I have kept the same thumb drive for perhaps a year, though I’m no longer sure what’s on it.

This has had an unexpected compounding positive effect: The most likely causes of an install failing are now either procedural or due to configuration changes made for testing in QA. Previously, the most likely cause of an install failing was that it had been packaged incorrectly. As a result of this shift, development is no longer the first on the scene of a failed install. Instead, the techs responsible for deploying the software to the customer handle most QA installation issues.

It might seem that a more fundamental problem in this occurrence is the brittleness of the configuration format, that it admits self-inconsistency. This is probably true; however, I could not determine a way to re-engineering the format piecemeal, and customer commitments (as well as time logged in the QA lab!) made such a large project unlikely on a maintenance-mode product. Also, the current installer has all sorts of difficult-to-understand logic with undocumented intent. This solution was truly cheap to get running.

I’ve encountered a faulty assumption about how TCP works, actually in multiple code bases I’ve maintained. Interestingly, my colleague Jeff and perhaps soon-to-be colleague Dane have also encountered this more than once and have dubbed it “Super High-Performance Networking Code.”

In Super High-Performance Networking Code, the sending side might look something like this:

    void OnFooEvent(Event e)
    {
        char *ptr = e.GetVariableLengthPayloadDataAsBytes();
        int len = e.GetLengthOfPayloadData();
        int rc = send(m_socket, ptr, len, 0);
        // handle error.
    }

… and the receiving side might look like this …

    void ReceiveLoop()
    {
        while (!m_bDone)
        {
            char buf[REALLY_BIG];
            int len = recv(m_socket, buf, sizeof(buf), 0);
            if (len > 0)
            {
                HandleExactlyOneFooEventWithVariableLengthData(buf, len);
            }
        }
    }

The fatal assumption is that read sizes on the receiving end are coincident with the write sizes on the sending end.

TCP networking is a stream-based protocol and it makes NO guarantees that the read and write sizes are coincident. In fact, it has a few mechanisms which will actively thwart this, including Nagle’s algorithm. Also, all TCP networks have an MTU, and packets larger than this will be sliced up for delivery (although without affecting the data contained within or the guaranteed order of delivery). There is some handshaking to discover the minimum MTU along a route, though this is intended to allow client systems to adapt for maximum throughput and is not a guarantee that slicing won’t happen. It’s more like a guarantee that slicing won’t happen frequently. Additionally, loaded routers may well choose to split transmit units or collect and concatenate them in order to meet quality of service and throughput requirements.

Unfortunately, though, making this assumption will work often enough with small packet sizes to lull some developers into believing they’ve confirmed that assuming read and write sizes are coincident is sound.

If you are making this assumption, you need to redesign your wire protocol to include within it a mechanism for delineation of the data, such as sending the data size up front, or using some sort of end-of-data marker. You then need to collect the reads on the receiving side until you’ve reached the demarcation or expected data size.