In unit testing, a lot has been written on OAPT, most commonly known as One Assert Per Test.
Given a basic unit test structured the way [WayBack] Martin Fowler: GivenWhenThen posted in 2013 (basically identical, though easier to remember for most people than the the 3A’s pattern Arrange-Act-Assert from the 2005 article [WayBack] thinking-out-loud: Principles for Test-Driven Development):
Feature: User trades stocks
Scenario: User requests a sell before close of trading
Given I have 100 shares of MSFT stock
And I have 150 shares of APPL stock
And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Then I should have 80 shares of MSFT stock
And I should have 150 shares of APPL stock
And a sell order for 20 shares of MSFT stock should have been executed
Neither Given–When–Then, nor Arrange–Act–Assert indicate there should be One Assert Per Test, however, having multiple asserts in the Then phase, might make it harder to figure out which asserts of a test passed, and which one(s) failed.
Other tests than unit tests
Note we are talking about unit tests here. There is a lot of confusion on what these are, and I have regular discussions with teams about calling tests unit tests, when in fact they are not.
More on this in the last section of this blog post, which quotes from these pages:
Asserts outside the Then phase
When putting asserts outside the Then phase, or even having multiple When-Then phases in one test, then you get into a whole different area of testing.
There are various ways to handle those situations, covered very well in [WayBack] Multiple Asserts Are OK – Industrial Logic: Some people suggest we should restrict ourselves to a single assertion per test. Are multiple asserts in a test ever OK?
- Run-on test
- Missing method or object
- De-duplication of similar tests
- Confirming setup
- Probe with multiple assertions
Here, “Run-on test” and “Confirming setup” definitely are bad, “Missing method or object” indicates some refactoring needs to take place, “De-duplication of tests” is OK if you factor out the set of assertions into one that provides a meaningful failure message, and “Probe with multiple assertions” can be OK, but should be closely inspected.
OAPT effects
So when applying OAPT, it is usually centered around phrasing the Then into one assert, or splitting the various Then parts that need to be true, in separate assertions that reside in separate methods.
Combining things in one assert, often is problematic, because upon test failure, most frameworks will not provide enough details to track back which part of the combined assert was violated.
Hence when following OAPT, tests are often split by introducing an extra method that holds the Given and When phases:
Method: Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Given I have 100 shares of MSFT stock
And I have 150 shares of APPL stock
And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, non-sold stock is unchanged
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then I should have 150 shares of APPL stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, sold stock is gone from portfolio
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then I should have 80 shares of MSFT stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, sell order has been executed
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then a sell order for 20 shares of MSFT stock should have been executed
Other test frameworks can make the distinction by for instance:
- automatically doing the split (and showing which actual clauses of the
Then part fail),
- have a short-hand to tag separate
Then clauses to a single scenario,
- allow for one assert to not skip the rest of the asserts (in essence allowing to report multiple failures per test)
- allow for deep comparisons (so instead of comparing many leaf attributes of two structures, you can compare the full structure in one go)
There is a lot to read about OAPT, some of which specify frameworks that make it easier to have one test method, having parameterised asserts, so the test runner can show the results of each assert in a separate result.
Some reading material:
Multiple asserts ain’t bad
There is an interesting study about having multiple asserts per test [WayBack] What Do the Asserts in a Unit Test Tell Us About Code Quality? (CSMR2013) which presents the results from the paper [WayBack] What Do The Asserts in a Unit Test Tell Us About Code Quality? A Study on Open Source and Industrial Projects
The conclusion is a two part one, of which both might surprise you:
- the number of asserts does not tell us anything about the production code quality;
- however, the number of asserted objects does.
What this basically means is that you should have one Unit-Under-Test.
This is why I would likely split the very first test that has 3 clauses under the Then part, into two tests, each centered around one Unit-Under-Test:
- The portfolio expectations
- The sell order expectation
OAPT, but still having multiple asserts in one method
One of the drawbacks of having multiple methods for one conceptual part pf a method, each having a different assert is the proliferation of methods.
What if you could have all these assertions in one method, but still run them as separate tests?
This is where a library lika
Repeat: we are talking unit-tests
If you are doing other tests than unit tests, then the above are still good guidelines, but might not all apply.
To get a better understanding on unit tests, read [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Definition of a Unit Test
A good unit test is:
- Able to be fully automated
- Has full control over all the pieces running (Use mocks or stubs to achieve this isolation when needed)
- Can be run in any order if part of many other tests
- Runs in memory (no DB or File access, for example)
- Consistently returns the same result (You always run the same test, so no random numbers, for example. save those for integration or range tests)
- Runs fast
- Tests a single logical concept in the system
- Readable
- Maintainable
- Trustworthy (when you see its result, you don’t need to debug the code just to be sure)
I consider any test that doesn’t live up to all these as an integration test and put it in its own “integration tests” project.
and [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Test Review Guidelines
How to do test reviews
Related: Definition of a unit test
Summary
Test Reviews (like code reviews, but on tests) can offer you the best process for teaching and improving the quality of your code and your unit tests while implementing unit testing into your organization. Review EVERY piece of unit testing code, and use the following points as a simple check list of things to watch out for. (you’ll find this is mainly useful when working in statically types languages such as Java or C#).
You can find these guidelines and more on the last page of the book. (but this page contains additions not found in the book)
Readability
- Make sure setup and teardown methods are not abused. It’s better to use factory methods for readability (p. 188, 214)
- Make sure the test tests one thing only (p. 179)
- Check for good and consistent naming conventions (p. 210-211)
- Make sure that only meaningful assert messages are used, or none at all (meaningful test names are better) (p. 212)
- Make sure asserts are separated from actions (different lines). (p. 214)
- Make sure tests don’t use magic strings and values as inputs. use the simplest inputs possible to prove your point.
- Make sure there is consistency in location of tests. make it easy to find related tests for a method, or a class, or a project.
Maintainability
- Make sure tests are isolated from each other and repeatable (p. 191)
- Make sure that testing private or protected methods is not the norm (public is always better) (P. 182)
- Make sure tests are not over-specified (p. 205)
- Make sure that state-based testing is preferred over using interaction testing (p. 83)
- Make sure strict mocks are used as little as possible (leads to over specification and fragile tests) (p. 106)
- Make sure there is no more than one mock per test (p. 94)
- Make sure tests do not mix mocks and regular asserts in the same test (testing multiple things)
- Make sure that tests ‘verify’ mock calls only on the single mock object in the test, and not on all the fake objects in the tests (the rest are stubs, this leads to over specification and fragile tests) (p.123)
- Make sure the test verifies only on a single call to a mock object. Verifying multiple calls on a mock object is either over specification or testing multiple things.
- Make sure that only in very rare cases a mock is also used as a stub to return a value in the same test (p. 84)
Trust
- Make sure the test does not contain logic or dynamic values (p. 178)
- Check coverage by playing with values (booleans or consts) (p. 180)
- Make sure unit tests are separated from integration tests (p. 180)
- Make sure tests don’t use things that keep changing in a unit test (like DateTime.Now ). Use fixed values.
- Make sure tests don’t assert with expected values that are created dynamically – you might be repeating production code.
–jeroen
The Wiert Corner - irregular stream of stuff 