The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 4,262 other subscribers

Archive for the ‘FastMM’ Category

« Previous Entries

DUnit with FastMM: Detecting memory leaks in individual tests.

Posted by jpluimers on 2021/08/05

Steps:

  1. Add the conditional define FASTMM to your project
  2. Ensure you have $(BDS)\source\DUnit\src  in your search path in your project (as otherwise Delphi will pick the pre-built TestFramework.dcu file which was compiled without the FASTMM conditional define)
  3. Inside a test method, or the SetUp method of a class, set FailsOnMemoryLeak to True
  4. If the SetUp method of the class allocates memory, ensure the TearDown de-allocates it. Otherwise you will have leaks:
    • DUnit will check memory differences from the start of the SetUp until the end of the TearDown
    • DUnit will not take into account what is freed in the destructor or by automatic finalization after the destructor!
  5. Re-build your application (otherwise the DUnit TestFramework unit will not take into account the FASTMM conditional define)

Depending in your test framework, FailsOnMemoryLeak might be by default be False or True:

  • TestInsight by default has FailsIfMemoryLeaked set to True for the root test suite (which is then applied to FailsOnMemoryLeak of any test method).
    procedure RunRegisteredTests(const baseUrl: string);
var
  suite: ITestSuite;
  result: TTestResult;
  listener: ITestListener;
begin
  suite := RegisteredTests;
  if not Assigned(suite) then Exit;
  result := TTestResult.Create;
  result.FailsIfNoChecksExecuted := True;
  result.FailsIfMemoryLeaked := True;
  listener := TTestInsightListener.Create(baseUrl, suite.CountEnabledTestCases);
  result.AddListener(listener);
  try
    suite.Run(result);
  finally
    result.Free;
  end;
end;
  • Console DUnit runners (Text, or XML) by default have FailsIfMemoryLeaked set to False.
  • GUI DUnit runner has FailsIfMemoryLeaked depending on the options:

DUnit source differences

Note that recent Delphi versions (I think XE and up) ship with almost the same sources as https://sourceforge.net/code-snapshots/svn/d/du/dunit/svn/dunit-svn-r44-trunk.zip, with these Embarcadero changes:

  • all SVN timestamps are based on time zone -0400 instead of +0000:
    • $Date: 2008-04-24 07:59:47 -0400 (Thu, 24 Apr 2008) $
    • $Date: 2008-04-24 11:59:47 +0000 (Thu, 24 Apr 2008) $
  • Embarcadero removed:
    • all comment lines having TODO in them
    • all files of types .dof, .cfg, and.wmz
    • the files NGUITestRunner.nfm and NGUITestRunner.pas
    • the file /etc/usermap
    • the directory trees /private and /projects
  • Embarcadero changed
    • file /src/versioninfo.res from 9.3.0 to 9.2.1 (which is odd, as all files are from 9.3.0)
    • unit TextTestRunner to support:
      • CLR (used last in Delphi 2007)
      • FailureCount
      • ErrorCount
      • IndentLevel
      • PrefixChars
      • conditional defines ADDITIONAL_INFO, BASIC_INFO
      • output of UnitName
    • unit TestExtensions to support:
      • CLR (used last in Delphi 2007)
      • conditional defines ANDROID_FIXME and LINUX
      • compiler directive LEGACYIFEND
    • GUITestRunner.dfm to have ResultsView: TListView property Height value 45 instead of 39
    • the below methods to use ReturnAddress in stead of CallerAddr:
      • TGUITestCase.FindControl
      • TGUITestCase.Click
      • TGUITestCase.EnterKeyInto
      • TGUITestCase.EnterTextInto
      • TGUITestCase.Show
      • TGUITestCase.CheckTabTo overloads
      • TGUITestCase.CheckFocused overloads
      • TGUITestCase.CheckEnabled overloads
      • TGUITestCase.SetFocus overloads
      • TGUITestCase.CheckVisible overloads
    • method TGUITestRunner.RunTest
      • from
        class procedure TGUITestRunner.RunTest(test: ITest);
begin
  with TGUITestRunner.Create(nil) do
  begin
   try
    suite := test;
    ShowModal;
   finally
    Free;
   end;
  end;
end;
      • to
        class procedure TGUITestRunner.RunTest(test: ITest);
var
  GUI :TGUITestRunner;
begin
  Application.CreateForm(TGUITestRunner, GUI);
  with GUI do
  begin
   try
    suite := test;
    ShowModal;
   finally
    Free;
   end;
  end;
end;
    • unit GUITestRunner:
      • from
        procedure RunTest(test: ITest);
begin
  with TGUITestRunner.Create(nil) do
  begin
    try
      Suite := test;
      ShowModal;
    finally
      Free;
    end;
  end;
end;
      • to
        procedure RunTest(test: ITest);
var
  GUI :TGUITestRunner;
begin
  Application.CreateForm(TGUITestRunner, GUI);
  with GUI do
  begin
    try
      Suite := test;
      if Suite <> nil then
        ShowModal;
    finally
      Free;
    end;
  end;
end;
      • method TGUITestRunner.SetUp not to set SubItems[0] := ''; when there is no test suite.
      • method TGUITestRunner.FormCreate to use FormatSettings.TimeSeparator instead of TimeSeparator.
  • unit TestFramework to support:
    • HPPEMIT
    • LEGACYIFEND
    • CLR and _USE_SYSDEBUG_
    • AUTOREFCOUNT
    • NEXTGEN
    • ANDROID
    • the RunCountAttribute and RunCount support
    • a CheckEquals overload for uint64
    • a CheckNotEquals overload for TCharArray
    • CheckCircularRef
    • CompareFloatRelative
    • NotSameErrorMessage with WideString arguments instead of string
    • a TestDataDir function
    • ReturnAddress for older compilers
  • a new unit DUnitTestRunner
  • a new Makefile file
  • a new UnitTests.log file

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development, TestInsight | 1 Comment »

“Unexpected Memory Leak” after “MyTestApplication.exe: Memory Leak Detected”

Posted by jpluimers on 2021/07/28

Sometimes after a regular FastMM memory leak test report you get a dialog like this:

---------------------------
Unexpected Memory Leak
---------------------------
An unexpected memory leak has occurred. The unexpected small block leaks are:

61 - 68 bytes: Unknown x 1
---------------------------
OK   
---------------------------

The normal FastMM memory leak reporting dialog looks like this:

---------------------------
MyTestApplication.exe: Memory Leak Detected
---------------------------
...
Note: Memory leak detail is logged to a text file in the same folder as this application. To disable this memory leak check, undefine "EnableMemoryLeakReporting".
---------------------------
OK   
---------------------------

Searching for “Unexpected Memory Leak” in RTL or FastMM code did not reveal results, but that might be me not doing a proper search.

One big problem is that the regular memory leak dialog is being suppressed by setting SuppressMessageBoxes to True (see for instance my blog post Application shutdown: wait for all threads to terminate or not? or [WayBack] delphi – Generating a FASTMM report WITHOUT the shutdown dialog – Stack Overflow).

However the “Unexpected Memory Leak” message box is always shown.

Research

Location of the error message caption

  • GETMEM.INC has an identifier LeakMessageTitle:
    LeakMessageTitle: _PAnsiChr = 'Unexpected Memory Leak';
  • Unlike FastMM4.pas, GETMEM.INC does not take into account SuppressMessageBoxes.
  • Like FastMM4.pas, GETMEM.INC does take into account ReportMemoryLeaksOnShutdown.

More on GETMEM.INC: [WayBack] delphi – Reporting memory leaks on shutdown with a console application – Stack Overflow

Versioned FastMM4.pas.

Complicating the hunt

Not all allocations go via the memory manager. A complicating factor is that:

  • ALL TMonitor.Create allocations go through SysAllocMem:
class function TMonitor.Create: PMonitor;
begin
  if CacheLineSize = 0 then
    AtomicExchange(CacheLineSize, GetCacheLineSize);
  if (CPUCount > 1) and (FDefaultSpinCount = 0) then
    AtomicExchange(FDefaultSpinCount, 1000);
  if CacheLineSize > SizeOf(Result^) then
    Result := SysAllocMem(CacheLineSize)
  else
    Result := SysAllocMem(SizeOf(Result^));
  Result.FSpinCount := FDefaultSpinCount;
end;
  • ALL TMonitor.Destroy deallocations go through SysFreeMem:
procedure TMonitor.Destroy;
begin
  if (MonitorSupport <> nil) and (FLockEvent <> nil) then
    MonitorSupport.FreeSyncObject(FLockEvent);
  SysFreeMem(@Self);
end;

Debugging this is easiest to set a breakpoint in the FastMM4.pas finalization section enabling a breakpoint group that has breakpoints on these methods inside GETMEM.INC:

  • function SysGetMem(Size: NativeInt): Pointer;
  • function SysFreeMem(P: Pointer): Integer;
  • function SysReallocMem(P: Pointer; Size: NativeInt): Pointer;
  • function SysAllocMem(Size: NativeInt): Pointer;

For inspecting for instance an asm construct like TSmallPoolBlockPoolHeader[edx], use a conversion PSmallBlockPoolHeader(Pointer(EDX))^,r

Poor mans shotgun approach

This will hide all the SysAllocMem related leaks:

unit FastMM4WrapperUnit;

interface

{$ifdef UseFastMM4}
uses
  FastMM4,
  FastMM4Messages;
{$endif UseFastMM4}

implementation

initialization
//
finalization
// Disable GETMEM.INC reports that neglect SuppressMessageBoxes and NoErrMsg; this will effectively hide the leak as GETMEM.INC also does not write the leaks to a log file
{$ifdef UseFastMM4}
  if SuppressMessageBoxes then
  begin
    {$WARN SYMBOL_PLATFORM OFF} NoErrMsg := True {$WARN SYMBOL_PLATFORM ON}; // no ShowMessage from the RTL (except for memory leaks)
    ReportMemoryLeaksOnShutdown := False; // No ShowMessage from the RTL in the GETMEM.INC teardown
  end;
{$endif UseFastMM4}
end.

IsConsole: needs linker option

Note that I tried embedding this in the then portion, but enabling IsConsole fails:

    IsConsole := True; // can only force run-time error messages to be written to the console if the "Project/Options/Linking/Generate console application" is set.

The reason is that this does not allocate a console handle, so you really need to ensure the linker has allocated a console for you by enabling Project/Options/Linking/Generate console application (or option DCC_ConsoleTarget in the .dproj file)

For more details, see my post console – When is System.IsConsole true in Delphi? – Stack Overflow.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, Event, FastMM, Software Development | Leave a Comment »

Some useful FastMM related methods to track memory usage

Posted by jpluimers on 2021/07/21

Below, for my link archive, some searches and relevant posts on FastMM related method calls to track or report memory usage.

Searches:

  • LogMemoryManagerStateToFile
  • FastGetHeapStatus {Returns summarised information about the state of the memory manager. (For
    backward compatibility.)}
  • GetMemoryManagerState (InternalBlockSize, UseableBlockSize, AllocatedBlockCount, ReservedAddressSpace) {Returns statistics about the current state of the memory manager}GetMemoryManagerUsageSummary {Returns a summary of the information returned by GetMemoryManagerState}
  • GetMemoryMap {Non-POSIX only; Gets the state of every 64K block in the 4GB address space}
  • ScanMemoryPoolForCorruptions; {Scans the memory pool for any corruptions. If a corruption is encountered an “Out of Memory” exception is raised.}
    • It is very costly in CPU usage, but helps finding heap corruption quickly.
  • function GetCurrentAllocationGroup: Cardinal;
    • {Returns the current “allocation group”. Whenever a GetMem request is serviced
      in FullDebugMode, the current “allocation group” is stored in the block header.
      This may help with debugging. Note that if a block is subsequently reallocated
      that it keeps its original “allocation group” and “allocation number” (all
      allocations are also numbered sequentially).}
  • procedure PushAllocationGroup(ANewCurrentAllocationGroup: Cardinal);
    procedure PopAllocationGroup;

    • {Allocation groups work in a stack like fashion. Group numbers are pushed onto
      and popped off the stack. Note that the stack size is limited, so every push
      should have a matching pop.}
  • LogAllocatedBlocksToFile
    • {Logs detail about currently allocated memory blocks for the specified range of
      allocation groups. if ALastAllocationGroupToLog is less than
      AFirstAllocationGroupToLog or it is zero, then all allocation groups are
      logged. This routine also checks the memory pool for consistency at the same
      time, raising an “Out of Memory” error if the check fails.}
  • SetMMLogFileName
    • {Specify the full path and name for the filename to be used for logging memory
      errors, etc. If ALogFileName is nil or points to an empty string it will
      revert to the default log file name.}

Posts (note that not all of them get their calculations right):

These help you track leaks that do not appear as leaks during shutdown: memory allocations that will be released at the end of your application, but are mostly unused while your application is still alive.

A few things to take away from these:

  1. “Out of Memory” (or exception EOutOfMemor) could mean that the memory manager structures are hosed, but memory is still available.
  2. You can specify the FastMM log file used (for instance to include a PID or run-time ID in them so each run gets a separate filename)
  3. When carefully setting up allocation groups, you are able to zoom in at allocations

A gist with a MemoryManagerUnit showing a few of these calls is below.

An example of its usage is this:

procedure TMyTestClass.TestMethod();
begin
   TLogMemoryStatesHelper.DumpMemoryStatesBeforeAndAfter('TestMethod',
     TLogging.LogDir,
     TFileLogger.GetLogFileName,
     procedure (const AFormat: string; const Args: array of const)
     begin
       TLogging.LogEvent(ltInfoHigh, aFormat, Args);
     end,
     procedure()
     begin
       try
         // Given
         BuildSomeTestScenario();
         // When
         InitializeTestScenario();
         // Then
         CheckEquals(0, TestScenarioSummary());
       finally
         // Cleanup
         CleanUpTestScenario();
       end;
     end
  );
end;

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

Some links for detecting memory leaks from individual DUnit test methods

Posted by jpluimers on 2021/07/20

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development | Leave a Comment »

Delphi: migrating applications + DLLs that use ShareMem to using FastMM

Posted by jpluimers on 2021/04/22

Notes to myself:

I bumped into some legacy code with a windows process and DLLs both using ShareMem (now System.ShareMem) so that strings could be shared between the instances.

There were lots of memory leaks, so migrating to FastMM was important.

I followed these steps to get rid of ShareMem:

  1. Put FastMM4 at the top of the uses lists for both the application and DLL projects
  2. Remove ShareMem from these uses lists (in fact from any unit used)
  3. Follow the FAQ ensuring these defines are globally in all projects involved: ShareMM;ShareMMIfLibrary;AttemptToUseSharedMM in each project file or the below in a fork of the FastMM4 repository file FastMM4Options.inc
    {$define ShareMM}
{$define ShareMMIfLibrary}
{$define AttemptToUseSharedMM}
    • Q: How do I get my DLL and main application to share FastMM so I can safely pass long strings and dynamic arrays between them?
    • A: The easiest way is to define ShareMM, ShareMMIfLibrary and AttemptToUseSharedMM in FastMM4.pas and add FastMM4.pas to the top of the uses section of the .dpr for both the main application and the DLL.
  4. Resolve any error like [dcc32 Error] E2201 Need imported data reference ($G) to access 'IsMultiThread' from unit 'FastMM4': in projects that depend on run-time packages. Luckily, how to do that is in the FAQ too:
    • Q: I get the following error when I try to use FastMM with an application compiled to use packages: “[Error] Need imported data reference ($G) to access ‘IsMultiThread‘ from unit ‘FastMM4‘”. How do I get it to work?
    • A: Enable the “UseRuntimePackages” option in FastMM4Options.inc.

Related:

Note:

  • I did not use SimpleShareMem (now System.SimpleShareMem) as the source of it did not tell me anything about FastMM4 compatibility.
  • A long time ago, FastMM changed the EnableBackwardCompatibleMMSharing from the old EnableSharingWithDefaultMM conditional define.

–jeroen

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

« Previous Entries