Avoid the Mega-Setup

A well-written test clearly communicates the context, action and results for the behavior that it’s intended to preserve. Recently, I’ve noticed an anti-pattern that can be easy to fall into when writing automated tests. I call it the “mega-setup.” A mega-setup is when you have a large, complex test data structure that’s constructed once at the top of a file (or worse, hidden in a utility directory) and then reused throughout the test suite.

The Mega-Setup in Action

Let’s examine what a mega-setup looks like and how it might come to be by imagining we’re using test-driven development to write a getFollowers function. We might start out with a simple case:

describe('getFollowers', () => {
  test('simple case', () => {
    createTestUser({ name: 'A', follows: []});
    createTestUser({ name: 'B', follows: ['A']});
    expect(getFollowers('A')).toMatchObject(['B']);
  });
});

Next, we add a test for the empty followers case and refactor:

describe('getFollowers', () => {
  beforeEach(() => {
    createTestUser({ name: 'A', follows: []});
    createTestUser({ name: 'B', follows: ['A']});
  });
  test('simple case', () => {
    expect(getFollowers('A')).toMatchObject(['B'])
  });
  test('empty follows', () => {
    expect(getFollowers('B')).toMatchObject([]);
  });
});

Then, say we want to add an option to fetch followers-of-followers:

describe('getFollowers', () => {
  beforeEach(() => {
    createTestUser({ name: 'A', follows: []});
    createTestUser({ name: 'B', follows: ['A']});
    createTestUser({ name: 'C', follows: ['B']});
  });
  test('simple case', () => {
    expect(getFollowers('A')).toMatchObject(['B']);
  });
  test('empty follows', () => {
    expect(getFollowers('B')).toMatchObject([]); // this fails now
  });
  test('transitive followers', () => {
    expect(
      getFollowers('A', { transitive: true })
    ).toMatchObject(['B', 'C']);
  });
});

By naively adding to our global setup, we’ve now broken the second test’s assumptions. So, we fix it by adding more state to our setup:

describe('getFollowers', () => {
  beforeEach(() => {
    createTestUser({ name: 'A', follows: []});
    createTestUser({ name: 'B', follows: ['A']});
    createTestUser({ name: 'C', follows: ['A']});
    createTestUser({ name: 'D', follows: ['C']});
  });
  test('simple case', () => {
    expect(getFollowers('A')).toMatchObject(['B']);
  });
  test('empty follows', () => {
    expect(getFollowers('B')).toMatchObject([]);
  });
  test('transitive followers', () => {
    expect(
      getFollowers('A', { transitive: true })
    ).toMatchObject(['B', 'C', 'D']);
  });
});

Now, we’ve written three tests and it’s already becoming hard to understand the complexities of the fake world we’ve created to run our tests against. I’ll leave it to your imagination to extrapolate out to a mega-setup for a complex module that has data for each edge case that we want to exercise. The complexity of this mega setup works against development in a number of ways.

First, for any given test, it’s difficult to answer the question, “what details from the global setup are important for the behavior that this test is intended to preserve?” Thus, when trying to debug a failure in one of these tests, we have to filter out a lot of noise to find a root cause. This makes our tests more costly to work with, which means that they’ll increasingly stifle the forward progress we might want to make on a module.

Further, because of this low signal to noise ratio, a suite with a mega-setup provides poor documentation of the behavior of the module that we’re building. Imagine the poor programmer (your future self!) that comes along to read these tests in search of knowledge about one particular feature, but first has to grok the entire fake world. Instead, tests should help us move quickly by letting us avoid having to understand every single detail before using a module.

Finally, in the third step of our example we noticed a second issue with the mega-setup— because the fake world is intertwined, our tests now all implicitly rely on each other. When we want to change the conditions for one bit of behavior, there’s a fair chance it’ll violate the expectations of an unrelated test. These interdependencies make for a brittle system that will become increasingly difficult to change as the mega-setup grows.

Thus, the trouble with the mega-setup is that it drastically increases the difficulty of comprehending our tests and decreases our ability to make changes to tests independently.

The Alternative

So, how do we avoid falling into the trap of the mega-setup? As we write tests, we’ll need to keep the following principle in mind:

Each test should specify all of the relevant details that matter to it.

Let’s walk through test-driving the same example function as above, but this time, we’ll leave the setup in each test. We start with exactly the same first test:

describe('getFollowers', () => {
  test('simple case', () => {
    createTestUser({ name: 'A', follows: [] });
    createTestUser({ name: 'B', follows: ['A'] });
    expect(getFollowers('A')).toMatchObject(['B']);
  });
});

And we move on to the empty followers case:

describe('getFollowers', () => {
  test('simple case', () => {
    createTestUser({ name: 'A', follows: [] });
    createTestUser({ name: 'B', follows: ['A'] });
    expect(getFollowers('A')).toMatchObject(['B']);
  });
  test('empty followers', () => {
    createTestUser({ name: 'A', follows: [] });
    expect(getFollowers('A')).toMatchObject([]);
  });
});

This time, we don’t change the first test at all, even though they share a common first line. By keeping each test independent and unchanged, we can start ignoring the old tests when we add new ones. This, of course, is the whole point. So we add a test for the transitive property:

describe('getFollowers', () => {
  test('simple case', () => { ... });
  test('empty followers', () => { ... });
  test('transitive followers included', () => {
    createTestUser({ name: 'A', follows: [] });
    createTestUser({ name: 'B', follows: ['A'] });
    createTestUser({ name: 'C', follows: ['B'] });
    expect(
      getFollowers('A', { transitive: true })
    ).toMatchObject(['B', 'C']);
  });
});

This time, we’ve arrived at a place where each test is a little island of state and behavior, as opposed to the whole suite being a continent in the first example. As with many things, when you break down a problem into its discrete parts, each piece becomes much easier to understand and work with than the whole. To use another metaphor, in the same way that breaking up a monolith into microservices frees us to make progress on each piece in isolation, so too does breaking up the mega-setup free us to understand and work with each independent test.

A DRY Coda

You might be asking, “what about the principle of Don’t Repeat Yourself (DRY)?” In the final example above, it’s true that we could factor out all of the creation of user A into a beforeEach clause. However, I’d argue that the value in having each test case be self-contained greatly outweighs the cost of the repeated code. It would be a bit much to say that DRY doesn’t apply to tests — certainly, refactoring away common boilerplate can add to the clarity of a test (in the examples above, we’ve done that with the createTestUser function). But, when we refactor tests, it’s crucial to leave the important details within each test case rather than tuck them away in the name of brevity.