Posted by jpluimers on 2021/03/17
Reminder to self: experiment with [WayBack] A DUnit Folder Iterator Extension – The Art of Delphi Programming.
This could be extended to virtual folders, allowing many integration tests to be run depending on some configuration hierarchy defined in a folder structure.
–jeroen
Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, Software Development, Unit Testing | Leave a Comment »
Posted by jpluimers on 2021/03/09
In unit testing, a lot has been written on OAPT, most commonly known as One Assert Per Test.
Given a basic unit test structured the way [WayBack] Martin Fowler: GivenWhenThen posted in 2013 (basically identical, though easier to remember for most people than the the 3A’s pattern Arrange-Act-Assert from the 2005 article [WayBack] thinking-out-loud: Principles for Test-Driven Development):
Feature: User trades stocks
Scenario: User requests a sell before close of trading
Given I have 100 shares of MSFT stock
And I have 150 shares of APPL stock
And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Then I should have 80 shares of MSFT stock
And I should have 150 shares of APPL stock
And a sell order for 20 shares of MSFT stock should have been executed
Neither Given–When–Then, nor Arrange–Act–Assert indicate there should be One Assert Per Test, however, having multiple asserts in the Then phase, might make it harder to figure out which asserts of a test passed, and which one(s) failed.
Note we are talking about unit tests here. There is a lot of confusion on what these are, and I have regular discussions with teams about calling tests unit tests, when in fact they are not.
More on this in the last section of this blog post, which quotes from these pages:
Then phaseWhen putting asserts outside the Then phase, or even having multiple When-Then phases in one test, then you get into a whole different area of testing.
There are various ways to handle those situations, covered very well in [WayBack] Multiple Asserts Are OK – Industrial Logic: Some people suggest we should restrict ourselves to a single assertion per test. Are multiple asserts in a test ever OK?
- Run-on test
- Missing method or object
- De-duplication of similar tests
- Confirming setup
- Probe with multiple assertions
Here, “Run-on test” and “Confirming setup” definitely are bad, “Missing method or object” indicates some refactoring needs to take place, “De-duplication of tests” is OK if you factor out the set of assertions into one that provides a meaningful failure message, and “Probe with multiple assertions” can be OK, but should be closely inspected.
So when applying OAPT, it is usually centered around phrasing the Then into one assert, or splitting the various Then parts that need to be true, in separate assertions that reside in separate methods.
Combining things in one assert, often is problematic, because upon test failure, most frameworks will not provide enough details to track back which part of the combined assert was violated.
Hence when following OAPT, tests are often split by introducing an extra method that holds the Given and When phases:
Method: Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Given I have 100 shares of MSFT stock
And I have 150 shares of APPL stock
And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, non-sold stock is unchanged
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then I should have 150 shares of APPL stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, sold stock is gone from portfolio
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then I should have 80 shares of MSFT stock
Feature: User trades stocks
Scenario: User requests a sell before close of trading, sell order has been executed
Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
Then a sell order for 20 shares of MSFT stock should have been executed
Other test frameworks can make the distinction by for instance:
Then part fail),Then clauses to a single scenario,There is a lot to read about OAPT, some of which specify frameworks that make it easier to have one test method, having parameterised asserts, so the test runner can show the results of each assert in a separate result.
Some reading material:
There is an interesting study about having multiple asserts per test [WayBack] What Do the Asserts in a Unit Test Tell Us About Code Quality? (CSMR2013) which presents the results from the paper [WayBack] What Do The Asserts in a Unit Test Tell Us About Code Quality? A Study on Open Source and Industrial Projects
The conclusion is a two part one, of which both might surprise you:
- the number of asserts does not tell us anything about the production code quality;
- however, the number of asserted objects does.
What this basically means is that you should have one Unit-Under-Test.
This is why I would likely split the very first test that has 3 clauses under the Then part, into two tests, each centered around one Unit-Under-Test:
One of the drawbacks of having multiple methods for one conceptual part pf a method, each having a different assert is the proliferation of methods.
What if you could have all these assertions in one method, but still run them as separate tests?
This is where a library lika
If you are doing other tests than unit tests, then the above are still good guidelines, but might not all apply.
To get a better understanding on unit tests, read [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Definition of a Unit Test
A good unit test is:
- Able to be fully automated
- Has full control over all the pieces running (Use mocks or stubs to achieve this isolation when needed)
- Can be run in any order if part of many other tests
- Runs in memory (no DB or File access, for example)
- Consistently returns the same result (You always run the same test, so no random numbers, for example. save those for integration or range tests)
- Runs fast
- Tests a single logical concept in the system
- Readable
- Maintainable
- Trustworthy (when you see its result, you don’t need to debug the code just to be sure)
I consider any test that doesn’t live up to all these as an integration test and put it in its own “integration tests” project.
and [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Test Review Guidelines
How to do test reviews
Related: Definition of a unit test
Summary
Test Reviews (like code reviews, but on tests) can offer you the best process for teaching and improving the quality of your code and your unit tests while implementing unit testing into your organization. Review EVERY piece of unit testing code, and use the following points as a simple check list of things to watch out for. (you’ll find this is mainly useful when working in statically types languages such as Java or C#).
You can find these guidelines and more on the last page of the book. (but this page contains additions not found in the book)
Readability
- Make sure setup and teardown methods are not abused. It’s better to use factory methods for readability (p. 188, 214)
- Make sure the test tests one thing only (p. 179)
- Check for good and consistent naming conventions (p. 210-211)
- Make sure that only meaningful assert messages are used, or none at all (meaningful test names are better) (p. 212)
- Make sure asserts are separated from actions (different lines). (p. 214)
- Make sure tests don’t use magic strings and values as inputs. use the simplest inputs possible to prove your point.
- Make sure there is consistency in location of tests. make it easy to find related tests for a method, or a class, or a project.
Maintainability
- Make sure tests are isolated from each other and repeatable (p. 191)
- Make sure that testing private or protected methods is not the norm (public is always better) (P. 182)
- Make sure tests are not over-specified (p. 205)
- Make sure that state-based testing is preferred over using interaction testing (p. 83)
- Make sure strict mocks are used as little as possible (leads to over specification and fragile tests) (p. 106)
- Make sure there is no more than one mock per test (p. 94)
- Make sure tests do not mix mocks and regular asserts in the same test (testing multiple things)
- Make sure that tests ‘verify’ mock calls only on the single mock object in the test, and not on all the fake objects in the tests (the rest are stubs, this leads to over specification and fragile tests) (p.123)
- Make sure the test verifies only on a single call to a mock object. Verifying multiple calls on a mock object is either over specification or testing multiple things.
- Make sure that only in very rare cases a mock is also used as a stub to return a value in the same test (p. 84)
Trust
- Make sure the test does not contain logic or dynamic values (p. 178)
- Check coverage by playing with values (booleans or consts) (p. 180)
- Make sure unit tests are separated from integration tests (p. 180)
- Make sure tests don’t use things that keep changing in a unit test (like DateTime.Now ). Use fixed values.
- Make sure tests don’t assert with expected values that are created dynamically – you might be repeating production code.
–jeroen
Posted in Agile, Conference Topics, Conferences, Development, Event, Software Development, Unit Testing | Leave a Comment »
Posted by jpluimers on 2021/02/17
A while back, I was writing some code to demonstrate a few inner workings of TInterfacedObject, interface reference counting and mixing object references with interface references.
One way to show this is through a test case that expects a certain exception to happen, but I forgot how to do that in DUnit. Luckily this pointed me on the right track: [WayBack] delphi – CheckException only accepts 0-parameter methods; how do I test that other methods throw exceptions? – Stack Overflow.
The solution shows that DUnit has had support for something similar as DUnitX: now has a WillRaiseAttribute to ease defining tests around code that should throw exceptions for a very long time (I think this was introduced around Delphi 2005).
You can do it in a property way:
unit InterfacedObjectTestCaseUnit; interface uses TestFramework; type TDebuggableInterfacedObjectTestCase = class(TTestCase) published procedure System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer(); end; implementation uses System.SysUtils, procedure TInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer(); var ObjectReference: System.TInterfacedObject; InterfaceReference: IInterface; begin ObjectReference := System.TInterfacedObject.Create(); InterfaceReference := ObjectReference; ExpectedException := System.SysUtils.EInvalidPointer; ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero // the below is optional; should not be reached. If it is reached, it will fail earlier than the encompassing `RunTest` method would ExpectedException := nil; // or `StopExpectingException();` end; end.
or in a method way for an exception that happens in the current method:
procedure TInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer(); var ObjectReference: System.TInterfacedObject; InterfaceReference: IInterface; begin ObjectReference := System.TInterfacedObject.Create(); InterfaceReference := ObjectReference; StartExpectingException(System.SysUtils.EInvalidPointer); ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero // the below is optional; should not be reached. If it is reached, it will fail earlier than the encompassing `RunTest` method would StopExpectingException(); end;
The alternative using CheckException that will raise earlier, but also tests the results of a complete method which also has to be parameterless:
procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount(); var ObjectReference: System.TInterfacedObject; InterfaceReference: IInterface; begin ObjectReference := System.TInterfacedObject.Create(); InterfaceReference := ObjectReference; ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero end; procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer_TTestMethod_Based(); begin CheckException(System_TInterfacedObject_Free_Before_RefCount, System.SysUtils.EInvalidPointer); end;
So I wrote a class helper based on TProc that allows you to test an anonymous method which usually has more fine grained testing potential.
Because of type compatibility, you have to call the inherited version of CheckException inside the new one:
unit TestCaseHelperUnit; interface uses System.SysUtils, TestFramework; type TTestCaseHelper = class helper for TTestCase public procedure CheckException(const AProc: TProc; const AExceptionClass: TClass; const msg: string = ''); end; implementation type TTestCaseInvoker = class strict private FProc: TProc; public constructor Create(const AProc: TProc); procedure Execute(); end; { TTestCaseInvoker } constructor TTestCaseInvoker.Create(const AProc: TProc); begin inherited Create(); FProc := AProc; end; procedure TTestCaseInvoker.Execute(); begin if Assigned(FProc) then FProc(); end; procedure TTestCaseHelper.CheckException(const AProc: TProc; const AExceptionClass: TClass; const msg: string = ''); var TestCaseInvoker: TTestCaseInvoker; begin TestCaseInvoker := TTestCaseInvoker.Create(AProc); try inherited CheckException(TestCaseInvoker.Execute, AExceptionClass, msg); // `inherited`, to avoid stack overflow because `TProc` is compatible with `TTestMethod` finally TestCaseInvoker.Free(); end; end; end.
The test then becomes this:
procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer_TProc_Based(); var ObjectReference: System.TInterfacedObject; InterfaceReference: IInterface; begin ObjectReference := System.TInterfacedObject.Create(); InterfaceReference := ObjectReference; CheckException(procedure () begin ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero end, System.SysUtils.EInvalidPointer); end;
Posted in Agile, Conference Topics, Conferences, Delphi, Development, Event, Software Development, Unit Testing | Leave a Comment »
Posted by jpluimers on 2020/02/13
Since Integration Tests have been around since the 1980s (yes, that long!), better ask what they mean in your teams.
If they are broad, you might want to re-consider and switch to narrowly scoped ones (but mind your pace).
A while ago, I landed another team, they were doing various kinds of test, but nobody had a good definition of which was what, so I dug up the below article.
Integration tests see if independently developed units of software work correctly when connected. Traditionally broad they are now better narrowly scoped.
Source: [WayBack] IntegrationTest
–jeroen
Posted in Agile, Development, Software Development, Unit Testing | Leave a Comment »
Posted by jpluimers on 2019/08/28
Must watch when you do unit testing (no matter the environment): [WayBack] Structure and Interpretation of Test Cases : Kevlin Henney on Vimeo which he presented at Javazone 2018 [Archive.is]
via [WayBack] Kevlin Henney – Google+
–jeroen
Posted in Agile, Conference Topics, Conferences, Development, Event, Software Development, Unit Testing | Leave a Comment »
