The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,839 other subscribers

Archive for the ‘Delphi’ Category

« Previous Entries
Next Entries »

Delphi 10.2 Tokyo and up compiler issue: incompatible with Delphi 1.10.1 Berlin behaviour on unit path resolution

Posted by jpluimers on 2021/08/10

There is a very tricky compiler and IDE issue in Delphi 10.2 and up that on the compiler site behaves differently from Delphi 1.10.1 Berlin, and on the IDE side stays the same from Delphi 2005 on.

The issue baffled me, as lot’s features were not added in the Delphi compiler because of backward compatibility reasons.

The problem is that the compiler now favours the unit search path over the paths specified in the .dpr. This breaks compatibility with earlier versions and the IDE: the IDE still thinks pathnames in the .dpr files are the ones to follow.

There are some permutations on this problem, of which a few ones below.

Nonexisting unit in the project, but existing on the search path

  • .dpr file in a directory X
  • a unit inside a different directory A specified the .dpr file, but not on disk, and directory A not in the unit search path
  • a disk version of the unit in yet another directory B where directory B is in the unit search path

This results in Delphi 1..10.1 Berlin to error on compiling the project (because the unit is not found), but 10.2 and up succeeding (because the unit is found).

Existing unit in the project, but directory not on the search a path

Another permutation is this one:

  • .dpr file in a directory X
  • a unit inside a different directory A specified the .dpr file, and on disk, and directory A not in the unit search path

Two same named units, one on search path, other in project but not on search path

Yet another permutation:

  • .dpr file in a directory X
  • a unit inside a different directory A specified the .dpr file, on disk, and directory A not in the unit search path
  • a disk version of the unit in yet another directory B where directory B is in the unit search path
  • unit A and unit B have different content

This works the same in all versions of Delphi: the unit file referenced in the .dpr is compiled.

Via: [WayBack] Jeroen Pluimers on Twitter: “Delphi 10.2 Tokyo and 10.3 Rio compilers break Delphi 1…10.1 Berlin behaviour. Still reproducing and researching the consequences. It for instance means stuff suddenly compiles when it should not, or breaks when it should not. Preliminary reproduction at “

[WayBack] wiert.me / public / delphi / DelphiConsoleProjectWithMissingProjectUnitThatIsOnTheSearchPath · GitLab

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, Software Development | Leave a Comment »

DUnit with FastMM: Detecting memory leaks in individual tests.

Posted by jpluimers on 2021/08/05

Steps:

  1. Add the conditional define FASTMM to your project
  2. Ensure you have $(BDS)\source\DUnit\src  in your search path in your project (as otherwise Delphi will pick the pre-built TestFramework.dcu file which was compiled without the FASTMM conditional define)
  3. Inside a test method, or the SetUp method of a class, set FailsOnMemoryLeak to True
  4. If the SetUp method of the class allocates memory, ensure the TearDown de-allocates it. Otherwise you will have leaks:
    • DUnit will check memory differences from the start of the SetUp until the end of the TearDown
    • DUnit will not take into account what is freed in the destructor or by automatic finalization after the destructor!
  5. Re-build your application (otherwise the DUnit TestFramework unit will not take into account the FASTMM conditional define)

Depending in your test framework, FailsOnMemoryLeak might be by default be False or True:

  • TestInsight by default has FailsIfMemoryLeaked set to True for the root test suite (which is then applied to FailsOnMemoryLeak of any test method).
    procedure RunRegisteredTests(const baseUrl: string);
var
  suite: ITestSuite;
  result: TTestResult;
  listener: ITestListener;
begin
  suite := RegisteredTests;
  if not Assigned(suite) then Exit;
  result := TTestResult.Create;
  result.FailsIfNoChecksExecuted := True;
  result.FailsIfMemoryLeaked := True;
  listener := TTestInsightListener.Create(baseUrl, suite.CountEnabledTestCases);
  result.AddListener(listener);
  try
    suite.Run(result);
  finally
    result.Free;
  end;
end;
  • Console DUnit runners (Text, or XML) by default have FailsIfMemoryLeaked set to False.
  • GUI DUnit runner has FailsIfMemoryLeaked depending on the options:

DUnit source differences

Note that recent Delphi versions (I think XE and up) ship with almost the same sources as https://sourceforge.net/code-snapshots/svn/d/du/dunit/svn/dunit-svn-r44-trunk.zip, with these Embarcadero changes:

  • all SVN timestamps are based on time zone -0400 instead of +0000:
    • $Date: 2008-04-24 07:59:47 -0400 (Thu, 24 Apr 2008) $
    • $Date: 2008-04-24 11:59:47 +0000 (Thu, 24 Apr 2008) $
  • Embarcadero removed:
    • all comment lines having TODO in them
    • all files of types .dof, .cfg, and.wmz
    • the files NGUITestRunner.nfm and NGUITestRunner.pas
    • the file /etc/usermap
    • the directory trees /private and /projects
  • Embarcadero changed
    • file /src/versioninfo.res from 9.3.0 to 9.2.1 (which is odd, as all files are from 9.3.0)
    • unit TextTestRunner to support:
      • CLR (used last in Delphi 2007)
      • FailureCount
      • ErrorCount
      • IndentLevel
      • PrefixChars
      • conditional defines ADDITIONAL_INFO, BASIC_INFO
      • output of UnitName
    • unit TestExtensions to support:
      • CLR (used last in Delphi 2007)
      • conditional defines ANDROID_FIXME and LINUX
      • compiler directive LEGACYIFEND
    • GUITestRunner.dfm to have ResultsView: TListView property Height value 45 instead of 39
    • the below methods to use ReturnAddress in stead of CallerAddr:
      • TGUITestCase.FindControl
      • TGUITestCase.Click
      • TGUITestCase.EnterKeyInto
      • TGUITestCase.EnterTextInto
      • TGUITestCase.Show
      • TGUITestCase.CheckTabTo overloads
      • TGUITestCase.CheckFocused overloads
      • TGUITestCase.CheckEnabled overloads
      • TGUITestCase.SetFocus overloads
      • TGUITestCase.CheckVisible overloads
    • method TGUITestRunner.RunTest
      • from
        class procedure TGUITestRunner.RunTest(test: ITest);
begin
  with TGUITestRunner.Create(nil) do
  begin
   try
    suite := test;
    ShowModal;
   finally
    Free;
   end;
  end;
end;
      • to
        class procedure TGUITestRunner.RunTest(test: ITest);
var
  GUI :TGUITestRunner;
begin
  Application.CreateForm(TGUITestRunner, GUI);
  with GUI do
  begin
   try
    suite := test;
    ShowModal;
   finally
    Free;
   end;
  end;
end;
    • unit GUITestRunner:
      • from
        procedure RunTest(test: ITest);
begin
  with TGUITestRunner.Create(nil) do
  begin
    try
      Suite := test;
      ShowModal;
    finally
      Free;
    end;
  end;
end;
      • to
        procedure RunTest(test: ITest);
var
  GUI :TGUITestRunner;
begin
  Application.CreateForm(TGUITestRunner, GUI);
  with GUI do
  begin
    try
      Suite := test;
      if Suite <> nil then
        ShowModal;
    finally
      Free;
    end;
  end;
end;
      • method TGUITestRunner.SetUp not to set SubItems[0] := ''; when there is no test suite.
      • method TGUITestRunner.FormCreate to use FormatSettings.TimeSeparator instead of TimeSeparator.
  • unit TestFramework to support:
    • HPPEMIT
    • LEGACYIFEND
    • CLR and _USE_SYSDEBUG_
    • AUTOREFCOUNT
    • NEXTGEN
    • ANDROID
    • the RunCountAttribute and RunCount support
    • a CheckEquals overload for uint64
    • a CheckNotEquals overload for TCharArray
    • CheckCircularRef
    • CompareFloatRelative
    • NotSameErrorMessage with WideString arguments instead of string
    • a TestDataDir function
    • ReturnAddress for older compilers
  • a new unit DUnitTestRunner
  • a new Makefile file
  • a new UnitTests.log file

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development, TestInsight | 1 Comment »

TestInsight: when you think the math does not add up…

Posted by jpluimers on 2021/08/04

You’d think the math does not add up:

But it does: both “Success” and “Warnings” are counted as “passed”.

 

TL;DR: ensure the yellow “Warning” triangle is enabled

–jeroen

Posted in Delphi, Development, Software Development, TestInsight | Leave a Comment »

Parser generator toolset for Delphi (Yacc & Lex)

Posted by jpluimers on 2021/08/03

The most current Lex and Yacc for Delphi is [WayBack] GitHub – RomanYankovsky/ndyacclex: Parser generator toolset for Delphi (Yacc & Lex).

It came up during a funny Twitter thread, where one of the Delphi team members asked for it, despite – after decades of asking – there still being no official Delphi grammar available from the Delphi team, nor Google Search skills [WayBack].

Basically a parser generator is only as useful as the grammar you have for it.

There is no open source grammar for Delphi yet, so the best you can start with is from the same author: [WayBack] GitHub – RomanYankovsky/DelphiAST: Abstract syntax tree builder for Delphi

With DelphiAST you can take real Delphi code and get an abstract syntax tree. One unit at time and without a symbol table though.

Relevant tweets:

–jeroen

Posted in Delphi, Development, Software Development, Undocumented Delphi | 1 Comment »

Delphi run-time errors

Posted by jpluimers on 2021/07/29

Since I always forget where to get the full list: there is none in the documentation. Only parts.

Usually the mapping is from run-time errors to exceptions:

In addition, exceptions are converted to run-time errors when the exception handling mechanism in the SysUtils unit is not up.

This can happen early in start-up, or late un shut-down of an application.

The one I encountered most is the runtime error 216 during shutdown: it is an Access Violation (EAccessViolation).

Run-time errors (not changed since Delphi 2007)

Read the rest of this entry »

Posted in Delphi, Development, Software Development | Leave a Comment »

“Unexpected Memory Leak” after “MyTestApplication.exe: Memory Leak Detected”

Posted by jpluimers on 2021/07/28

Sometimes after a regular FastMM memory leak test report you get a dialog like this:

---------------------------
Unexpected Memory Leak
---------------------------
An unexpected memory leak has occurred. The unexpected small block leaks are:

61 - 68 bytes: Unknown x 1
---------------------------
OK   
---------------------------

The normal FastMM memory leak reporting dialog looks like this:

---------------------------
MyTestApplication.exe: Memory Leak Detected
---------------------------
...
Note: Memory leak detail is logged to a text file in the same folder as this application. To disable this memory leak check, undefine "EnableMemoryLeakReporting".
---------------------------
OK   
---------------------------

Searching for “Unexpected Memory Leak” in RTL or FastMM code did not reveal results, but that might be me not doing a proper search.

One big problem is that the regular memory leak dialog is being suppressed by setting SuppressMessageBoxes to True (see for instance my blog post Application shutdown: wait for all threads to terminate or not? or [WayBack] delphi – Generating a FASTMM report WITHOUT the shutdown dialog – Stack Overflow).

However the “Unexpected Memory Leak” message box is always shown.

Research

Location of the error message caption

  • GETMEM.INC has an identifier LeakMessageTitle:
    LeakMessageTitle: _PAnsiChr = 'Unexpected Memory Leak';
  • Unlike FastMM4.pas, GETMEM.INC does not take into account SuppressMessageBoxes.
  • Like FastMM4.pas, GETMEM.INC does take into account ReportMemoryLeaksOnShutdown.

More on GETMEM.INC: [WayBack] delphi – Reporting memory leaks on shutdown with a console application – Stack Overflow

Versioned FastMM4.pas.

Complicating the hunt

Not all allocations go via the memory manager. A complicating factor is that:

  • ALL TMonitor.Create allocations go through SysAllocMem:
class function TMonitor.Create: PMonitor;
begin
  if CacheLineSize = 0 then
    AtomicExchange(CacheLineSize, GetCacheLineSize);
  if (CPUCount > 1) and (FDefaultSpinCount = 0) then
    AtomicExchange(FDefaultSpinCount, 1000);
  if CacheLineSize > SizeOf(Result^) then
    Result := SysAllocMem(CacheLineSize)
  else
    Result := SysAllocMem(SizeOf(Result^));
  Result.FSpinCount := FDefaultSpinCount;
end;
  • ALL TMonitor.Destroy deallocations go through SysFreeMem:
procedure TMonitor.Destroy;
begin
  if (MonitorSupport <> nil) and (FLockEvent <> nil) then
    MonitorSupport.FreeSyncObject(FLockEvent);
  SysFreeMem(@Self);
end;

Debugging this is easiest to set a breakpoint in the FastMM4.pas finalization section enabling a breakpoint group that has breakpoints on these methods inside GETMEM.INC:

  • function SysGetMem(Size: NativeInt): Pointer;
  • function SysFreeMem(P: Pointer): Integer;
  • function SysReallocMem(P: Pointer; Size: NativeInt): Pointer;
  • function SysAllocMem(Size: NativeInt): Pointer;

For inspecting for instance an asm construct like TSmallPoolBlockPoolHeader[edx], use a conversion PSmallBlockPoolHeader(Pointer(EDX))^,r

Poor mans shotgun approach

This will hide all the SysAllocMem related leaks:

unit FastMM4WrapperUnit;

interface

{$ifdef UseFastMM4}
uses
  FastMM4,
  FastMM4Messages;
{$endif UseFastMM4}

implementation

initialization
//
finalization
// Disable GETMEM.INC reports that neglect SuppressMessageBoxes and NoErrMsg; this will effectively hide the leak as GETMEM.INC also does not write the leaks to a log file
{$ifdef UseFastMM4}
  if SuppressMessageBoxes then
  begin
    {$WARN SYMBOL_PLATFORM OFF} NoErrMsg := True {$WARN SYMBOL_PLATFORM ON}; // no ShowMessage from the RTL (except for memory leaks)
    ReportMemoryLeaksOnShutdown := False; // No ShowMessage from the RTL in the GETMEM.INC teardown
  end;
{$endif UseFastMM4}
end.

IsConsole: needs linker option

Note that I tried embedding this in the then portion, but enabling IsConsole fails:

    IsConsole := True; // can only force run-time error messages to be written to the console if the "Project/Options/Linking/Generate console application" is set.

The reason is that this does not allocate a console handle, so you really need to ensure the linker has allocated a console for you by enabling Project/Options/Linking/Generate console application (or option DCC_ConsoleTarget in the .dproj file)

For more details, see my post console – When is System.IsConsole true in Delphi? – Stack Overflow.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, Event, FastMM, Software Development | Leave a Comment »

Profiler and Memory Analysis Tools for Delphi – Stack Overflow

Posted by jpluimers on 2021/07/27

Lots of interesting profilers in [WayBack] Profiler and Memory Analysis Tools for Delphi – Stack Overflow:

--jeroen

Posted in Delphi, Development, Software Development | 2 Comments »

console – When is System.IsConsole true in Delphi? – Stack Overflow

Posted by jpluimers on 2021/07/22

Some highlights from [WayBack] console – When is System.IsConsole true in Delphi? – Stack Overflow:

  • Project/Options/Linking/Generate console application and {$APPTYPE CONSOLE} are two separate things. – [WayBack] Andreas Rejbrand.
  • The -cc commandline parameter for dcc32.exe (which could be in your ProjectX.cfg file!) is equivalent to “Generate console application):

    Found it: the executable has been created using dcc32.exe and the dpr / cfg file, the cfg contains a line

    -cc

    which creates a console application. – [WayBack] mjn

  • Note that in Delphi XE2, OSX applications can not use this variable as it is always true. See QC Entry 98956 and Why Does My OSX FireMonkey App Think It Is a Console App ? – [WayBack] mjn

A few notes I have found out myself

The linker option “Project/Options/Linking/Generate console application” is the same as “DCC_ConsoleTarget” in the .dproj file:

    <PropertyGroup Condition="'$(Base_Win32)'!=''">
        <DCC_ConsoleTarget>true</DCC_ConsoleTarget>
    </PropertyGroup>

Even setting IsConsole to True does not allocate a new console, so the STD_OUTPUT_HANDLE handle does not work!

The below method in the System unit, then does not show output:

procedure WriteErrorMessage;
{$IFDEF MSWINDOWS}
var
  Dummy: Cardinal;
begin
  if IsConsole then
  begin
    with TTextRec(Output) do
    begin
      if (Mode = fmOutput) and (BufPos > 0) then
        TTextIOFunc(InOutFunc)(TTextRec(Output));  // flush out text buffer
    end;
    // Leave #0 off end of runErrMsg
    WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), @runErrMsg, Sizeof(runErrMsg) - 1, Dummy, nil);
    WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), @sLineBreak[1], 2, Dummy, nil);
  end
  else if not NoErrMsg then
    MessageBoxA(0, runErrMsg, errCaption, 0);
end;

For console output, you have to enable the linker option DCC_ConsoleTarget allocates a console (if not yet allocated), enables the “STD_OUTPUT_HANDLE” handle, and sets IsConsole to True.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, Event, Software Development | Leave a Comment »

Some useful FastMM related methods to track memory usage

Posted by jpluimers on 2021/07/21

Below, for my link archive, some searches and relevant posts on FastMM related method calls to track or report memory usage.

Searches:

  • LogMemoryManagerStateToFile
  • FastGetHeapStatus {Returns summarised information about the state of the memory manager. (For
    backward compatibility.)}
  • GetMemoryManagerState (InternalBlockSize, UseableBlockSize, AllocatedBlockCount, ReservedAddressSpace) {Returns statistics about the current state of the memory manager}GetMemoryManagerUsageSummary {Returns a summary of the information returned by GetMemoryManagerState}
  • GetMemoryMap {Non-POSIX only; Gets the state of every 64K block in the 4GB address space}
  • ScanMemoryPoolForCorruptions; {Scans the memory pool for any corruptions. If a corruption is encountered an “Out of Memory” exception is raised.}
    • It is very costly in CPU usage, but helps finding heap corruption quickly.
  • function GetCurrentAllocationGroup: Cardinal;
    • {Returns the current “allocation group”. Whenever a GetMem request is serviced
      in FullDebugMode, the current “allocation group” is stored in the block header.
      This may help with debugging. Note that if a block is subsequently reallocated
      that it keeps its original “allocation group” and “allocation number” (all
      allocations are also numbered sequentially).}
  • procedure PushAllocationGroup(ANewCurrentAllocationGroup: Cardinal);
    procedure PopAllocationGroup;

    • {Allocation groups work in a stack like fashion. Group numbers are pushed onto
      and popped off the stack. Note that the stack size is limited, so every push
      should have a matching pop.}
  • LogAllocatedBlocksToFile
    • {Logs detail about currently allocated memory blocks for the specified range of
      allocation groups. if ALastAllocationGroupToLog is less than
      AFirstAllocationGroupToLog or it is zero, then all allocation groups are
      logged. This routine also checks the memory pool for consistency at the same
      time, raising an “Out of Memory” error if the check fails.}
  • SetMMLogFileName
    • {Specify the full path and name for the filename to be used for logging memory
      errors, etc. If ALogFileName is nil or points to an empty string it will
      revert to the default log file name.}

Posts (note that not all of them get their calculations right):

These help you track leaks that do not appear as leaks during shutdown: memory allocations that will be released at the end of your application, but are mostly unused while your application is still alive.

A few things to take away from these:

  1. “Out of Memory” (or exception EOutOfMemor) could mean that the memory manager structures are hosed, but memory is still available.
  2. You can specify the FastMM log file used (for instance to include a PID or run-time ID in them so each run gets a separate filename)
  3. When carefully setting up allocation groups, you are able to zoom in at allocations

A gist with a MemoryManagerUnit showing a few of these calls is below.

An example of its usage is this:

procedure TMyTestClass.TestMethod();
begin
   TLogMemoryStatesHelper.DumpMemoryStatesBeforeAndAfter('TestMethod',
     TLogging.LogDir,
     TFileLogger.GetLogFileName,
     procedure (const AFormat: string; const Args: array of const)
     begin
       TLogging.LogEvent(ltInfoHigh, aFormat, Args);
     end,
     procedure()
     begin
       try
         // Given
         BuildSomeTestScenario();
         // When
         InitializeTestScenario();
         // Then
         CheckEquals(0, TestScenarioSummary());
       finally
         // Cleanup
         CleanUpTestScenario();
       end;
     end
  );
end;

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

Some links for detecting memory leaks from individual DUnit test methods

Posted by jpluimers on 2021/07/20

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development | Leave a Comment »

« Previous Entries
Next Entries »