The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,433 other followers

Archive for the ‘FastMM’ Category

“Unexpected Memory Leak” after “MyTestApplication.exe: Memory Leak Detected”

Posted by jpluimers on 2021/07/28

Sometimes after a regular FastMM memory leak test report you get a dialog like this:

---------------------------
Unexpected Memory Leak
---------------------------
An unexpected memory leak has occurred. The unexpected small block leaks are:

61 - 68 bytes: Unknown x 1
---------------------------
OK   
---------------------------

The normal FastMM memory leak reporting dialog looks like this:

---------------------------
MyTestApplication.exe: Memory Leak Detected
---------------------------
...
Note: Memory leak detail is logged to a text file in the same folder as this application. To disable this memory leak check, undefine "EnableMemoryLeakReporting".
---------------------------
OK   
---------------------------

Searching for “Unexpected Memory Leak” in RTL or FastMM code did not reveal results, but that might be me not doing a proper search.

One big problem is that the regular memory leak dialog is being suppressed by setting SuppressMessageBoxes to True (see for instance my blog post Application shutdown: wait for all threads to terminate or not? or [WayBack] delphi – Generating a FASTMM report WITHOUT the shutdown dialog – Stack Overflow).

However the “Unexpected Memory Leak” message box is always shown.

Research

Location of the error message caption

  • GETMEM.INC has an identifier LeakMessageTitle:
    LeakMessageTitle: _PAnsiChr = 'Unexpected Memory Leak';
  • Unlike FastMM4.pas, GETMEM.INC does not take into account SuppressMessageBoxes.
  • Like FastMM4.pas, GETMEM.INC does take into account ReportMemoryLeaksOnShutdown.

More on GETMEM.INC: [WayBack] delphi – Reporting memory leaks on shutdown with a console application – Stack Overflow

Versioned FastMM4.pas.

Complicating the hunt

Not all allocations go via the memory manager. A complicating factor is that:

  • ALL TMonitor.Create allocations go through SysAllocMem:
class function TMonitor.Create: PMonitor;
begin
  if CacheLineSize = 0 then
    AtomicExchange(CacheLineSize, GetCacheLineSize);
  if (CPUCount > 1) and (FDefaultSpinCount = 0) then
    AtomicExchange(FDefaultSpinCount, 1000);
  if CacheLineSize > SizeOf(Result^) then
    Result := SysAllocMem(CacheLineSize)
  else
    Result := SysAllocMem(SizeOf(Result^));
  Result.FSpinCount := FDefaultSpinCount;
end;
  • ALL TMonitor.Destroy deallocations go through SysFreeMem:
procedure TMonitor.Destroy;
begin
  if (MonitorSupport <> nil) and (FLockEvent <> nil) then
    MonitorSupport.FreeSyncObject(FLockEvent);
  SysFreeMem(@Self);
end;

Debugging this is easiest to set a breakpoint in the FastMM4.pas finalization section enabling a breakpoint group that has breakpoints on these methods inside GETMEM.INC:

  • function SysGetMem(Size: NativeInt): Pointer;
  • function SysFreeMem(P: Pointer): Integer;
  • function SysReallocMem(P: Pointer; Size: NativeInt): Pointer;
  • function SysAllocMem(Size: NativeInt): Pointer;

For inspecting for instance an asm construct like TSmallPoolBlockPoolHeader[edx], use a conversion PSmallBlockPoolHeader(Pointer(EDX))^,r

Poor mans shotgun approach

This will hide all the SysAllocMem related leaks:

unit FastMM4WrapperUnit;

interface

{$ifdef UseFastMM4}
uses
  FastMM4,
  FastMM4Messages;
{$endif UseFastMM4}

implementation

initialization
//
finalization
// Disable GETMEM.INC reports that neglect SuppressMessageBoxes and NoErrMsg; this will effectively hide the leak as GETMEM.INC also does not write the leaks to a log file
{$ifdef UseFastMM4}
  if SuppressMessageBoxes then
  begin
    {$WARN SYMBOL_PLATFORM OFF} NoErrMsg := True {$WARN SYMBOL_PLATFORM ON}; // no ShowMessage from the RTL (except for memory leaks)
    ReportMemoryLeaksOnShutdown := False; // No ShowMessage from the RTL in the GETMEM.INC teardown
  end;
{$endif UseFastMM4}
end.

IsConsole: needs linker option

Note that I tried embedding this in the then portion, but enabling IsConsole fails:

    IsConsole := True; // can only force run-time error messages to be written to the console if the "Project/Options/Linking/Generate console application" is set.

The reason is that this does not allocate a console handle, so you really need to ensure the linker has allocated a console for you by enabling Project/Options/Linking/Generate console application (or option DCC_ConsoleTarget in the .dproj file)

For more details, see my post console – When is System.IsConsole true in Delphi? – Stack Overflow.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, Event, FastMM, Software Development | Leave a Comment »

Some useful FastMM related methods to track memory usage

Posted by jpluimers on 2021/07/21

Below, for my link archive, some searches and relevant posts on FastMM related method calls to track or report memory usage.

Searches:

  • LogMemoryManagerStateToFile
  • FastGetHeapStatus {Returns summarised information about the state of the memory manager. (For
    backward compatibility.)}
  • GetMemoryManagerState (InternalBlockSize, UseableBlockSize, AllocatedBlockCount, ReservedAddressSpace) {Returns statistics about the current state of the memory manager}GetMemoryManagerUsageSummary {Returns a summary of the information returned by GetMemoryManagerState}
  • GetMemoryMap {Non-POSIX only; Gets the state of every 64K block in the 4GB address space}
  • ScanMemoryPoolForCorruptions; {Scans the memory pool for any corruptions. If a corruption is encountered an “Out of Memory” exception is raised.}
    • It is very costly in CPU usage, but helps finding heap corruption quickly.
  • function GetCurrentAllocationGroup: Cardinal;
    • {Returns the current “allocation group”. Whenever a GetMem request is serviced
      in FullDebugMode, the current “allocation group” is stored in the block header.
      This may help with debugging. Note that if a block is subsequently reallocated
      that it keeps its original “allocation group” and “allocation number” (all
      allocations are also numbered sequentially).}
  • procedure PushAllocationGroup(ANewCurrentAllocationGroup: Cardinal);
    procedure PopAllocationGroup;

    • {Allocation groups work in a stack like fashion. Group numbers are pushed onto
      and popped off the stack. Note that the stack size is limited, so every push
      should have a matching pop.}
  • LogAllocatedBlocksToFile
    • {Logs detail about currently allocated memory blocks for the specified range of
      allocation groups. if ALastAllocationGroupToLog is less than
      AFirstAllocationGroupToLog or it is zero, then all allocation groups are
      logged. This routine also checks the memory pool for consistency at the same
      time, raising an “Out of Memory” error if the check fails.}
  • SetMMLogFileName
    • {Specify the full path and name for the filename to be used for logging memory
      errors, etc. If ALogFileName is nil or points to an empty string it will
      revert to the default log file name.}

Posts (note that not all of them get their calculations right):

These help you track leaks that do not appear as leaks during shutdown: memory allocations that will be released at the end of your application, but are mostly unused while your application is still alive.

A few things to take away from these:

  1. “Out of Memory” (or exception EOutOfMemor) could mean that the memory manager structures are hosed, but memory is still available.
  2. You can specify the FastMM log file used (for instance to include a PID or run-time ID in them so each run gets a separate filename)
  3. When carefully setting up allocation groups, you are able to zoom in at allocations

A gist with a MemoryManagerUnit showing a few of these calls is below.

An example of its usage is this:

procedure TMyTestClass.TestMethod();
begin
   TLogMemoryStatesHelper.DumpMemoryStatesBeforeAndAfter('TestMethod',
     TLogging.LogDir,
     TFileLogger.GetLogFileName,
     procedure (const AFormat: string; const Args: array of const)
     begin
       TLogging.LogEvent(ltInfoHigh, aFormat, Args);
     end,
     procedure()
     begin
       try
         // Given
         BuildSomeTestScenario();
         // When
         InitializeTestScenario();
         // Then
         CheckEquals(0, TestScenarioSummary());
       finally
         // Cleanup
         CleanUpTestScenario();
       end;
     end
  );
end;

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

Some links for detecting memory leaks from individual DUnit test methods

Posted by jpluimers on 2021/07/20

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development | Leave a Comment »

Delphi: migrating applications + DLLs that use ShareMem to using FastMM

Posted by jpluimers on 2021/04/22

Notes to myself:

I bumped into some legacy code with a windows process and DLLs both using ShareMem (now System.ShareMem) so that strings could be shared between the instances.

There were lots of memory leaks, so migrating to FastMM was important.

I followed these steps to get rid of ShareMem:

  1. Put FastMM4 at the top of the uses lists for both the application and DLL projects
  2. Remove ShareMem from these uses lists (in fact from any unit used)
  3. Follow the FAQ ensuring these defines are globally in all projects involved: ShareMM;ShareMMIfLibrary;AttemptToUseSharedMM in each project file or the below in a fork of the FastMM4 repository file FastMM4Options.inc
    {$define ShareMM}
    {$define ShareMMIfLibrary}
    {$define AttemptToUseSharedMM}
    
    • Q: How do I get my DLL and main application to share FastMM so I can safely pass long strings and dynamic arrays between them?
    • A: The easiest way is to define ShareMM, ShareMMIfLibrary and AttemptToUseSharedMM in FastMM4.pas and add FastMM4.pas to the top of the uses section of the .dpr for both the main application and the DLL.
  4. Resolve any error like [dcc32 Error] E2201 Need imported data reference ($G) to access 'IsMultiThread' from unit 'FastMM4': in projects that depend on run-time packages. Luckily, how to do that is in the FAQ too:
    • Q: I get the following error when I try to use FastMM with an application compiled to use packages: “[Error] Need imported data reference ($G) to access ‘IsMultiThread‘ from unit ‘FastMM4‘”. How do I get it to work?
    • A: Enable the “UseRuntimePackages” option in FastMM4Options.inc.

Related:

Note:

  • I did not use SimpleShareMem (now System.SimpleShareMem) as the source of it did not tell me anything about FastMM4 compatibility.
  • A long time ago, FastMM changed the EnableBackwardCompatibleMMSharing from the old EnableSharingWithDefaultMM conditional define.

–jeroen

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

Delphi: quickly failing in use-after free scenarios

Posted by jpluimers on 2021/03/03

Two tricks that can help in use-after-free scenarios.

Call ScanMemoryPoolForCorruptions often

One of the scenarios of use after free, is that memory blocks get corrupted.

FastMM4 normall checks this at process end using the CheckBlocksOnShutdown method (in FastMM4.pas when writing this private at line 11156), but you can also do this process manually using the ScanMemoryPoolForCorruptions method (also in FastMM4.pas, but public at line L1356).

You can automate this process by setting the FullDebugModeScanMemoryPoolBeforeEveryOperation flag to True while in FullDebugMode as you see in the quoted code blocks below.

Note that calling ScanMemoryPoolForCorruptions between allocations might reveal wild pointer dereferences between allocations.

  - Added a global variable "FullDebugModeScanMemoryPoolBeforeEveryOperation".
    When this variable is set to true and FullDebugMode is enabled, then the
    entire memory pool is checked for consistency before every GetMem, FreeMem
    and ReallocMem operation. An "Out of Memory" error is raised if a
    corruption is found (and this variable is set to false to prevent recursive
    errors). This obviously incurs a massive performance hit, so enable it only
    when hunting for elusive memory corruption bugs. (Thanks to Marcus Mönnig.)

  {If this variable is set to true and FullDebugMode is enabled, then the
   entire memory pool is checked for consistency before every memory
   operation. Note that this incurs a massive performance hit on top of
   the already significant FullDebugMode overhead, so enable this option
   only when absolutely necessary.}
  FullDebugModeScanMemoryPoolBeforeEveryOperation: Boolean = False;

Call any virtual method on an instance reference

A quick way to test use-after free scenarios is to call a virtual method on an instance.

Virtual methods mean that the Virtual Method Table needs to be used as a starting point, so any nil pointer will get dereferenced.

Two simple methods that you can call, which have no side effects, except for referencing memory, and are virtual on [WayBack] TObject are [WayBack] GetHashCode and [WayBack] ToString. Both methods got added in Delphi 2009, and now support 64-bit and 32-bit compilers are below.

If you use use these in addition to FastMM4 clearing memory, and FastMM4 redirecting virtual methods of freed objects, you have a good chance of catching use-after free.

Without FastMM, they are also of good help, especially when the freed memory has since then been overwritten by new usage. FastMM4 is a lot more strict about this, so definitely recommended.

Calling these two methods help you to quickly fail with an EAccessViolation [WayBack] in use-after-free scenarios.

More on the virtual method table is for instance in [WayBack] Hallvard’s Blog: Method calls compiler implementation.

Read the rest of this entry »

Posted in Conference Topics, Conferences, Delphi, Development, Event, FastMM, Software Development, Undocumented Delphi | Leave a Comment »

 
%d bloggers like this: