The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,466 other followers

Archive for the ‘Unit Testing’ Category

A DUnit Folder Iterator Extension – The Art of Delphi Programming

Posted by jpluimers on 2021/03/17

Reminder to self: experiment with [WayBack] A DUnit Folder Iterator Extension – The Art of Delphi Programming.

This could be extended to virtual folders, allowing many integration tests to be run depending on some configuration hierarchy defined in a folder structure.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, Software Development, Unit Testing | Leave a Comment »

Unittesting: why would you want OAPT, and how you might end up with a combined OAPT and OOPT.

Posted by jpluimers on 2021/03/09

In unit testing, a lot has been written on OAPT, most commonly known as One Assert Per Test.

Given a basic unit test structured the way [WayBack] Martin Fowler: GivenWhenThen posted in 2013 (basically identical, though easier to remember for most people than the the 3A’s pattern Arrange-Act-Assert from the 2005 article [WayBack] thinking-out-loud: Principles for Test-Driven Development):

Feature: User trades stocks
  Scenario: User requests a sell before close of trading
    Given I have 100 shares of MSFT stock
       And I have 150 shares of APPL stock
       And the time is before close of trading

    When I ask to sell 20 shares of MSFT stock
     
     Then I should have 80 shares of MSFT stock
      And I should have 150 shares of APPL stock
      And a sell order for 20 shares of MSFT stock should have been executed

Neither GivenWhenThen, nor ArrangeActAssert indicate there should be One Assert Per Test, however, having multiple asserts in the Then phase, might make it harder to figure out which asserts of a test passed, and which one(s) failed.

Other tests than unit tests

Note we are talking about unit tests here. There is a lot of confusion on what these are, and I have regular discussions with teams about calling tests unit tests, when in fact they are not.

More on this in the last section of this blog post, which quotes from these pages:

Asserts outside the Then phase

When putting asserts outside the Then phase, or even having multiple When-Then phases in one test, then you get into a whole different area of testing.

There are various ways to handle those situations, covered very well in [WayBack] Multiple Asserts Are OK – Industrial Logic: Some people suggest we should restrict ourselves to a single assertion per test. Are multiple asserts in a test ever OK?

  1. Run-on test
  2. Missing method or object
  3. De-duplication of similar tests
  4. Confirming setup
  5. Probe with multiple assertions

Here, “Run-on test” and “Confirming setup” definitely are bad, “Missing method or object” indicates some refactoring needs to take place, “De-duplication of tests” is OK if you factor out the set of assertions into one that provides a meaningful failure message, and “Probe with multiple assertions” can be OK, but should be closely inspected.

OAPT effects

So when applying OAPT, it is usually centered around phrasing the Then into one assert, or splitting the various Then parts that need to be true, in separate assertions that reside in separate methods.

Combining things in one assert, often is problematic, because upon test failure, most frameworks will not provide enough details to track back which part of the combined assert was violated.

Hence when following OAPT, tests are often split by introducing an extra method that holds the Given and When phases:

Method: Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
   Given I have 100 shares of MSFT stock 
      And I have 150 shares of APPL stock
      And the time is before close of trading
   When I ask to sell 20 shares of MSFT stock 

Feature: User trades stocks
  Scenario: User requests a sell before close of trading, non-sold stock is unchanged
    Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
    Then I should have 150 shares of APPL stock

Feature: User trades stocks
   Scenario: User requests a sell before close of trading, sold stock is gone from portfolio 
      Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
      Then I should have 80 shares of MSFT stock

Feature: User trades stocks
   Scenario: User requests a sell before close of trading, sell order has been executed
   Call method Given portfolio of 100 MSFT and 150 AAPL, When selling 20 MSFT
   Then a sell order for 20 shares of MSFT stock should have been executed

Other test frameworks can make the distinction by for instance:

  • automatically doing the split (and showing which actual clauses of the Then part fail),
  • have a short-hand to tag separate Then clauses to a single scenario,
  • allow for one assert to not skip the rest of the asserts (in essence allowing to report multiple failures per test)
  • allow for deep comparisons (so instead of comparing many leaf attributes of two structures, you can compare the full structure in one go)

There is a lot to read about OAPT, some of which specify frameworks that make it easier to have one test method, having parameterised asserts, so the test runner can show the results of each assert in a separate result.

Some reading material:

Multiple asserts ain’t bad

There is an interesting study about having multiple asserts per test [WayBack] What Do the Asserts in a Unit Test Tell Us About Code Quality? (CSMR2013) which presents the results from the paper [WayBack] What Do The Asserts in a Unit Test Tell Us About Code Quality? A Study on Open Source and Industrial Projects

The conclusion is a two part one, of which both might surprise you:

  1. the number of asserts does not tell us anything about the production code quality;
  2. however, the number of asserted objects does.

What this basically means is that you should have one Unit-Under-Test.

This is why I would likely split the very first test that has 3 clauses under the Then part, into two tests, each centered around one Unit-Under-Test:

  • The portfolio expectations
  • The sell order expectation

OAPT, but still having multiple asserts in one method

One of the drawbacks of having multiple methods for one conceptual part pf a method, each having a different assert is the proliferation of methods.

What if you could have all these assertions in one method, but still run them as separate tests?

This is where a library lika

Repeat: we are talking unit-tests

If you are doing other tests than unit tests, then the above are still good guidelines, but might not all apply.

To get a better understanding on unit tests, read [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Definition of a Unit Test

A good unit test is:

  • Able to be fully automated
  • Has full control over all the pieces running (Use mocks or stubs to achieve this isolation when needed)
  • Can be run in any order  if part of many other tests
  • Runs in memory (no DB or File access, for example)
  • Consistently returns the same result (You always run the same test, so no random numbers, for example. save those for integration or range tests)
  • Runs fast
  • Tests a single logical concept in the system
  • Readable
  • Maintainable
  • Trustworthy (when you see its result, you don’t need to debug the code just to be sure)

I consider any test that doesn’t live up to all these as an integration test and put it in its own “integration tests” project.

and [WayBack] Unit Testing Lessons in Ruby, Java and .NET – The Art Of Unit Testing – Test Review Guidelines

How to do test reviews

Related: Definition of a unit test

Summary

Test Reviews (like code reviews, but on tests) can offer you the best process for teaching and improving the quality of your code and your unit tests while implementing unit testing into your organization. Review EVERY piece of unit testing code, and use the following points as a simple check list of things to watch out for. (you’ll find this is mainly useful when working in statically types languages such as Java or C#).

You can find these guidelines and more on the last page of the book. (but this page contains additions not found in the book)

Readability

  • Make sure setup and teardown methods are not abused. It’s better to use factory methods for readability  (p. 188, 214)
  • Make sure the test tests one thing only (p. 179)
  • Check for good and consistent naming conventions (p. 210-211)
  • Make sure that only meaningful assert messages are used, or none at all (meaningful test names are better) (p. 212)
  • Make sure asserts are separated from actions (different lines). (p. 214)
  • Make sure tests don’t use magic strings and values as inputs. use the simplest inputs possible to prove your point.
  • Make sure there is consistency in location of tests. make it easy to find related tests for a method, or a class, or a project.

Maintainability

  • Make sure tests are isolated from each other and repeatable (p. 191)
  • Make sure that testing private or protected methods is not the norm (public is always better) (P. 182)
  • Make sure tests are not over-specified (p. 205)
  • Make sure that state-based testing is preferred over using interaction testing (p. 83)
  • Make sure strict mocks are used as little as possible (leads to over specification and fragile tests) (p. 106)
  • Make sure there is no more than one mock per test (p. 94)
  • Make sure tests do not mix mocks and regular asserts in the same test (testing multiple things)
  • Make sure that tests ‘verify’ mock calls only on the single mock object in the test, and not on all the fake objects in the tests (the rest are stubs, this leads to over specification and fragile tests) (p.123)
  • Make sure the test verifies only on a single call to a mock object. Verifying multiple calls on a mock object is either over specification or testing multiple things.
  • Make sure that only in very rare cases a mock is also used as a stub to return a value in the same test (p. 84)

Trust

  • Make sure the test does not contain logic or dynamic values (p. 178)
  • Check coverage by playing with values (booleans or consts) (p. 180)
  • Make sure unit tests are separated from integration tests (p. 180)
  • Make sure tests don’t use things that keep changing in a unit test (like DateTime.Now ). Use fixed values.
  • Make sure tests don’t assert with expected values that are created dynamically – you might be repeating production code.

–jeroen

Posted in Agile, Conference Topics, Conferences, Development, Event, Software Development, Unit Testing | Leave a Comment »

DUnit testing code that should raise a specific exception

Posted by jpluimers on 2021/02/17

A while back, I was writing some code to demonstrate a few inner workings of TInterfacedObject, interface reference counting and mixing object references with interface references.

One way to show this is through a test case that expects a certain exception to happen, but I forgot how to do that in DUnit. Luckily this pointed me on the right track: [WayBack] delphi – CheckException only accepts 0-parameter methods; how do I test that other methods throw exceptions? – Stack Overflow.

The solution shows that DUnit has had support for something similar as DUnitX: now has a WillRaiseAttribute to ease defining tests around code that should throw exceptions for a very long time (I think this was introduced around Delphi 2005).

You can do it in a property way:

unit InterfacedObjectTestCaseUnit;

interface

uses
  TestFramework;

type
  TDebuggableInterfacedObjectTestCase = class(TTestCase)
  published
    procedure System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer();
  end;

implementation

uses
  System.SysUtils,

procedure TInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer();
var
  ObjectReference:    System.TInterfacedObject;
  InterfaceReference: IInterface;
begin
  ObjectReference    := System.TInterfacedObject.Create();
  InterfaceReference := ObjectReference;
  ExpectedException  := System.SysUtils.EInvalidPointer;
  ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero
  // the below is optional; should not be reached. If it is reached, it will fail earlier than the encompassing `RunTest` method would
  ExpectedException := nil; // or `StopExpectingException();`
end;

end.

or in a method way for an exception that happens in the current method:

procedure TInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer();
var
  ObjectReference:    System.TInterfacedObject;
  InterfaceReference: IInterface;
begin
  ObjectReference    := System.TInterfacedObject.Create();
  InterfaceReference := ObjectReference;
  StartExpectingException(System.SysUtils.EInvalidPointer);
  ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero
  // the below is optional; should not be reached. If it is reached, it will fail earlier than the encompassing `RunTest` method would
  StopExpectingException();
end;

The alternative using CheckException that will raise earlier, but also tests the results of a complete method which also has to be parameterless:

procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount();
var
  ObjectReference:    System.TInterfacedObject;
  InterfaceReference: IInterface;
begin
  ObjectReference    := System.TInterfacedObject.Create();
  InterfaceReference := ObjectReference;
  ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero
end;

procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer_TTestMethod_Based();
begin
  CheckException(System_TInterfacedObject_Free_Before_RefCount, System.SysUtils.EInvalidPointer);
end;

So I wrote a class helper based on TProc that allows you to test an anonymous method which usually has more fine grained testing potential.

Because of type compatibility, you have to call the inherited version of CheckException inside the new one:

unit TestCaseHelperUnit;

interface

uses
  System.SysUtils,
  TestFramework;

type
  TTestCaseHelper = class helper for TTestCase
  public
    procedure CheckException(const AProc: TProc; const AExceptionClass: TClass; const msg: string = '');
  end;

implementation

type
  TTestCaseInvoker = class
  strict private
    FProc: TProc;
  public
    constructor Create(const AProc: TProc);
    procedure Execute();
  end;

{ TTestCaseInvoker }

constructor TTestCaseInvoker.Create(const AProc: TProc);
begin
  inherited Create();
  FProc := AProc;
end;

procedure TTestCaseInvoker.Execute();
begin
  if Assigned(FProc) then
    FProc();
end;

procedure TTestCaseHelper.CheckException(const AProc: TProc; const AExceptionClass: TClass; const msg: string = '');
var
  TestCaseInvoker: TTestCaseInvoker;
begin
  TestCaseInvoker := TTestCaseInvoker.Create(AProc);
  try
    inherited CheckException(TestCaseInvoker.Execute, AExceptionClass, msg); // `inherited`, to avoid stack overflow because `TProc` is compatible with `TTestMethod`
  finally
    TestCaseInvoker.Free();
  end;
end;

end.

The test then becomes this:

procedure TDebuggableInterfacedObjectTestCase.System_TInterfacedObject_Free_Before_RefCount_Should_Raise_EInvalidPointer_TProc_Based();
var
  ObjectReference:    System.TInterfacedObject;
  InterfaceReference: IInterface;
begin
  ObjectReference    := System.TInterfacedObject.Create();
  InterfaceReference := ObjectReference;
  CheckException(procedure ()
  begin
    ObjectReference.Free(); // this should raise an exception in System.TInterfacedObject.BeforeDestruction, as it checks the RefCount to be zero
  end,
  System.SysUtils.EInvalidPointer);
end;

DUnit code snippets

Read the rest of this entry »

Posted in Agile, Conference Topics, Conferences, Delphi, Development, Event, Software Development, Unit Testing | Leave a Comment »

When a team uses IntegrationTest – always ask what they mean with the term

Posted by jpluimers on 2020/02/13

Since Integration Tests have been around since the 1980s (yes, that long!), better ask what they mean in your teams.

If they are broad, you might want to re-consider and switch to narrowly scoped ones (but mind your pace).

A while ago, I landed another team, they were doing various kinds of test, but  nobody had a good definition of which was what, so I dug up the below article.

Integration tests see if independently developed units of software work correctly when connected. Traditionally broad they are now better narrowly scoped.

Source: [WayBack] IntegrationTest

–jeroen

Posted in Agile, Development, Software Development, Unit Testing | Leave a Comment »

Structure and Interpretation of Test Cases : Kevlin Henney on Vimeo

Posted by jpluimers on 2019/08/28

Must watch when you do unit testing (no matter the environment): [WayBackStructure and Interpretation of Test Cases : Kevlin Henney on Vimeo which he presented at Javazone 2018 [Archive.is]

via [WayBack] Kevlin Henney – Google+

–jeroen

Posted in Agile, Conference Topics, Conferences, Development, Event, Software Development, Unit Testing | Leave a Comment »

 
%d bloggers like this: