The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘Delphi’ Category

Some useful FastMM related methods to track memory usage

Posted by jpluimers on 2021/07/21

Below, for my link archive, some searches and relevant posts on FastMM related method calls to track or report memory usage.

Searches:

  • LogMemoryManagerStateToFile
  • FastGetHeapStatus {Returns summarised information about the state of the memory manager. (For
    backward compatibility.)}
  • GetMemoryManagerState (InternalBlockSize, UseableBlockSize, AllocatedBlockCount, ReservedAddressSpace) {Returns statistics about the current state of the memory manager}GetMemoryManagerUsageSummary {Returns a summary of the information returned by GetMemoryManagerState}
  • GetMemoryMap {Non-POSIX only; Gets the state of every 64K block in the 4GB address space}
  • ScanMemoryPoolForCorruptions; {Scans the memory pool for any corruptions. If a corruption is encountered an “Out of Memory” exception is raised.}
    • It is very costly in CPU usage, but helps finding heap corruption quickly.
  • function GetCurrentAllocationGroup: Cardinal;
    • {Returns the current “allocation group”. Whenever a GetMem request is serviced
      in FullDebugMode, the current “allocation group” is stored in the block header.
      This may help with debugging. Note that if a block is subsequently reallocated
      that it keeps its original “allocation group” and “allocation number” (all
      allocations are also numbered sequentially).}
  • procedure PushAllocationGroup(ANewCurrentAllocationGroup: Cardinal);
    procedure PopAllocationGroup;

    • {Allocation groups work in a stack like fashion. Group numbers are pushed onto
      and popped off the stack. Note that the stack size is limited, so every push
      should have a matching pop.}
  • LogAllocatedBlocksToFile
    • {Logs detail about currently allocated memory blocks for the specified range of
      allocation groups. if ALastAllocationGroupToLog is less than
      AFirstAllocationGroupToLog or it is zero, then all allocation groups are
      logged. This routine also checks the memory pool for consistency at the same
      time, raising an “Out of Memory” error if the check fails.}
  • SetMMLogFileName
    • {Specify the full path and name for the filename to be used for logging memory
      errors, etc. If ALogFileName is nil or points to an empty string it will
      revert to the default log file name.}

Posts (note that not all of them get their calculations right):

These help you track leaks that do not appear as leaks during shutdown: memory allocations that will be released at the end of your application, but are mostly unused while your application is still alive.

A few things to take away from these:

  1. “Out of Memory” (or exception EOutOfMemor) could mean that the memory manager structures are hosed, but memory is still available.
  2. You can specify the FastMM log file used (for instance to include a PID or run-time ID in them so each run gets a separate filename)
  3. When carefully setting up allocation groups, you are able to zoom in at allocations

A gist with a MemoryManagerUnit showing a few of these calls is below.

An example of its usage is this:

procedure TMyTestClass.TestMethod();
begin
   TLogMemoryStatesHelper.DumpMemoryStatesBeforeAndAfter('TestMethod',
     TLogging.LogDir,
     TFileLogger.GetLogFileName,
     procedure (const AFormat: string; const Args: array of const)
     begin
       TLogging.LogEvent(ltInfoHigh, aFormat, Args);
     end,
     procedure()
     begin
       try
         // Given
         BuildSomeTestScenario();
         // When
         InitializeTestScenario();
         // Then
         CheckEquals(0, TestScenarioSummary());
       finally
         // Cleanup
         CleanUpTestScenario();
       end;
     end
  );
end;

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, FastMM, Software Development | Leave a Comment »

Some links for detecting memory leaks from individual DUnit test methods

Posted by jpluimers on 2021/07/20

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, DUnit, Event, FastMM, Software Development | Leave a Comment »

inheritance – Delphi: How to call inherited inherited ancestor? – Stack Overflow

Posted by jpluimers on 2021/07/15

Officially, the answer to [WayBack] inheritance – Delphi: How to call inherited inherited ancestor? – Stack Overflow is that you can’t on the language level as I explained in my answer

You can’t in a regular language way, as this would break the object oriented aspects of the language.

You can fiddle around with pointers and clever casts to do this, but before even starting to answer that: is this really what you want?

As others mentioned: your need sounds like a serious “design smell” (which is similar to code smell, but more severe.

Edit:

Going down the pointer fiddling road might save you work in the short term, and cost you weeks of work in the long term.
This makes for some good reading on that: Upstream decisions, downstream costs.

If you really want, then there is a clever hack around this by [WayBack] User kludg – Stack Overflow.

His hack is centered around understanding what what the [WayBack] System.TMethod Record essentially is:

Read the rest of this entry »

Posted in Conference Topics, Conferences, Delphi, Development, Event, Software Development | Leave a Comment »

Some OmniThreadLibrary notes and links

Posted by jpluimers on 2021/07/14

For my link archive:

IOmniTimedTask and the Timed task name

You create a reference like this:

  TimedTask := Parallel.TimedTask.Execute(
    procedure(const task: IOmniTask)
    begin
      // ...
    end);

In the background, Parallel.TimedTask calls TOmniTimedTask.Create() which calls CreateTask(TOmniTimedTaskWorker.Create(), 'Timed task').Unobserved.Run

The problem is that TOmniTimedTaskWorker is private to the OtlParallel unit, which means you cannot take that call out without also copying that class.

There might be a workaround which I need to research based on the Apply method of IOmniTaskConfig, maybe through Parallel.ApplyConfig. These links might help:

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, Multi-Threading / Concurrency, Software Development | Leave a Comment »

Chromium Embedded Framework – some links

Posted by jpluimers on 2021/07/13

For my link archive:

–jeroen

Read the rest of this entry »

Posted in .NET, Delphi, Development, Software Development | Leave a Comment »

Delphi Tokyo binary BPLs not finding “@System@Net@Httpclient@Win@TCertificateStore@$bcctr$qqrv” means you likely have Delphi 10.2.2 installed

Posted by jpluimers on 2021/07/13

If you get errors loading BPLs that indicate this method cannot be found: “@System@Net@Httpclient@Win@TCertificateStore@$bcctr$qqrv” then it is likely you run Delphi 10.2.2.

Upgrade to Delphi 10.2.3 to resolve this problem.

This happens for instance with binary versions of these below libraries.

They cannot help, as Embarcadero removed the TCertificateStore class (present in Delphi 10.1.2) in 10.2.0, then re-added it in 10.2.3.

 

Libraries that I bumped into that were affected:

Note that some reports talk about “@System@Net@Httpclient@Win@TCertificateStore@$bcdtr$qqrv” in stead of “@System@Net@Httpclient@Win@TCertificateStore@$bcctr$qqrv“. I am not sure what causes this discrepancy yet.

–jeroen

Posted in Delphi, Delphi 10.2 Tokyo (Godzilla), Development, Software Development | Leave a Comment »

delphi – How to enable wirecompression on Firebird 3.0 – Stack Overflow

Posted by jpluimers on 2021/07/08

For my link archive:

–jeroen

Posted in Database Development, Delphi, Development, Firebird, Software Development | Leave a Comment »

E2213 Bad packaged unit format: ..\xxxxx.dcp.yyyy – Expected version: 32.0, Windows Unicode(x86) Found version: 105.101, Unk(CIL)

Posted by jpluimers on 2021/07/08

On my research list I have seen this happen with various libraries, but only libraries that deliver a single set of .bpl/.dcp files that you require for both debug and release builds.

The error occurs both ways:

  • building a “debug” build is fine, but after it a “release” build fails
  • building a “release” build is fine, but after it a “debug” build fails

This is with all separate DCU output directories for any permutation in any project: .\$(Platform)\$(Config)\$(SanitizedProjectName)

Example (where xxxx.pas is the first unit depending on Qrctrls, but I have seen it with other libraries that provide .bpl/.dcp combinations as well):

[dcc32 Fatal Error] xxxx.pas(14): E2213 Bad packaged unit format: ..\3rdParty\Quickrep506\19.0\Lib\Win32\Release\QR506RunDXE10_2.dcp.Qrctrls - Expected version: 32.0, Windows Unicode(x86) Found version: 105.101, Unk(CIL)

My best guess is that something is being cached in between builds, but not marked for the correct build configuration well enough.

–jeroen

Posted in Delphi, Delphi 10.2 Tokyo (Godzilla), Development, Software Development | Leave a Comment »

SetProcessWorkingSetSize: you hardly – if ever – need to call this from your process

Posted by jpluimers on 2021/07/07

There are quite a few posts that recommend using SetProcessWorkingSetSize to trim your process working set, usually in the SetProcessWorkingSetSize(ProcessHandle, -1, -1) form:

[WayBack] SetProcessWorkingSetSize function (winbase.h) | Microsoft Docs

Sets the minimum and maximum working set sizes for the specified process.

BOOL SetProcessWorkingSetSize(
  HANDLE hProcess,
  SIZE_T dwMinimumWorkingSetSize,
  SIZE_T dwMaximumWorkingSetSize );

The working set of the specified process can be emptied by specifying the value (SIZE_T)–1 for both the minimum and maximum working set sizes. This removes as many pages as possible from the working set. The [WayBack] EmptyWorkingSet function can also be used for this purpose.

In practice you hardly ever have to do this, mainly because this will write – regardless of (dis)usage – all of your memory to the pagefile, even the memory your frequently use.

Windows has way better heuristics to do that automatically for you, skipping pages you frequently use.

It basically makes sense in a few use cases, for instance when you know that most (like 90% or more) of that memory is never going to be used again.

Another use case (with specific memory sizes) is when you know that your program is going to use a defined range of memory, which is outside what Windows will heuristically expect from it.

A few more links that go into more details on this:

  • [WayBack] windows – Pros and Cons of using SetProcessWorkingSetSize – Stack Overflow answers by:
    • Hans Passant:

      SetProcessWorkingSetSize() controls the amount of RAM that your process uses, it doesn’t otherwise have any affect on the virtual memory size of your process. Windows is already quite good at dynamically controlling this, swapping memory pages out on demand when another process needs RAM.

      By doing this manually, you slow down your program a lot, causing a lot of page faults when Windows is forced to swap the memory pages back in.

      SetProcessWorkingSetSize is typically used to increase the amount of RAM allocated for a process. Or to force a trim when the app knows that it is going to be idle for a long time. Also done automatically by old Windows versions when you minimize the main window of the app.

    • Zack Yezek:

      The only good use case I’ve seen for this call is when you KNOW your process is going to hog a lot of the system’s RAM and you want to reserve it for the duration. You use it to tell the OS “Yes, I’m going to eat a lot of the system RAM during my entire run and don’t get in my way”.

    • Maxim Masiutin:

      We have found out that, for a GUI application written in Delphi for Win32/Win64 or written in a similar way that uses large and heavy libraries on top of the Win32 API (GDI, etc), it is worth calling SetProcessWorkingSetSize once.

      We call it with -1, -1 parameters, within a fraction of second after the application has fully opened and showed the main window to the user. In this case, the SetProcessWorkingSetSize(... -1, -1) releases lots of startup code that seem to not needed any more.

  • [WayBack] c# – How to set MinWorkingSet and MaxWorkingSet in a 64-bit .NET process? – Stack Overflow answer by Hans Passant:

    Don’t pinvoke this, just use the Process.CurrentProcess.MinWorkingSet property directly.

    Very high odds that this won’t make any difference. Soft paging faults are entirely normal and resolved very quickly if the machine has enough RAM. Takes ~0.7 microseconds on my laptop. You can’t avoid them, it is the behavior of a demand_paged virtual memory operating system like Windows. Very cheap, as long as there is a free page readily available.

    But if it “blips” you program performance then you need to consider the likelihood that it isn’t readily available and triggered a hard page fault in another process. The paging fault does get expensive if the RAM page must be stolen from another process, its content has to be stored in the paging file and has to be reset back to zero first. That can add up quickly, hundreds of microseconds isn’t unusual.

    The basic law of “there is no free lunch”, you need to run less processes or buy more RAM. With the latter option the sane choice, 8 gigabytes sets you back about 75 bucks today. Complete steal.

  • [WayBack] c++ – SetProcessWorkingSetSize usage – Stack Overflow answer by MSalters:

    I had an application which by default would close down entirely but keep listening for certain events. However, most of my code at that point would not be needed for a long time. To reduce the impact my process made, I called SetProcessWorkingSetSize(-1,-1);. This meant Windows could take back the physical RAM and give it to other apps. I’d get my RAM back when events did arrive.

    That’s of course unrelated to your situation, and I don’t think you’d benefit.

  • [WayBack] delphi – When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory) – Stack Overflow

    If your goal is for your application to use less memory you should look elsewhere. Look for leaks, look for heap fragmentations look for optimisations and if you think FastMM is keeping you from doing so you should try to find facts to support it. If your goal is to keep your workinset size small you could try to keep your memory access local. Maybe FastMM or another memory manager could help you with it, but it is a very different problem compared to using to much memory.

    you can check the FasttMM memory usage via FasttMM calls GetMemoryManagerState and GetMemoryManagerUsageSummary before and after calling API SetProcessWorkingSetSize.

    I don’t need to use SetProcessWorkingSetSize. FastMM will eventually release the RAM.


    To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I crated a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.

  • [WayBack] delphi – SetProcessWorkingSetSize – What’s the catch? – Stack Overflow answer by Rob Kennedy:

    Yes, it’s a bad thing. You’re telling the OS that you know more about memory management than it does, which probably isn’t true. You’re telling to to page all your inactive memory to disk. It obeys. The moment you touch any of that memory again, the OS has to page it back into RAM. You’re forcing disk I/O that you don’t actually know you need.

    If the OS needs more free RAM, it can figure out which memory hasn’t been used lately and page it out. That might be from your program, or it might be from some other program. But if the OS doesn’t need more free RAM, then you’ve just forced a bunch of disk I/O that nobody asked for.

    If you have memory that you know you don’t need anymore, free it. Don’t just page it to disk. If you have memory that the OS thinks you don’t need, it will page it for you automatically as the need arises.

    Also, it’s usually unwise to call Application.ProcessMessages unless you know there are messages that your main thread needs to process that it wouldn’t otherwise process by itself. The application automatically processes messages when there’s nothing else to do, so if you have nothing to do, just let the application run itself.

–jeroen

Posted in .NET, C, C++, Delphi, Development, Software Development, Windows Development | Leave a Comment »

Delphi: bumped into System.Classes.TLoginCredentialService while doing a “DefaultUsrPwDm” – Google Search

Posted by jpluimers on 2021/07/06

Doing a  “DefaultUsrPwDm” – Google Search, I bumped into the System.Classes.TLoginCredentialService class.

There is hardly any documentation.

It was introduced as [WayBack] System.Classes.TLoginCredentialService – XE2 API Documentation with the standard TObject template:

TObject is the ultimate ancestor of all objects and components.

System.Classes.TLoginCredentialService inherits from System.TObject. All content below this line refers to System.TObject.

TObject is the ultimate ancestor of all objects and components.

Although TObject is the based object of a component framework, not all objects are components. All component classes descend from TComponent.

Note: TObject is never directly instantiated. Although it does not use programming language features that prevent instantiation, TObject is an abstract class.

The documentation was updated in Delphi XE4 to [WayBack] System.Classes.TLoginCredentialService – RAD Studio API Documentation

TLoginCredentialService provides functionality for login action, regardless of framework.

TLoginCredentialService represents an extensible, framework agnostic, login credential service with support for callbacks (success, failure). It includes methods for:

  • Getting user credentials: DomainUsername and Password.
  • Registering and unregistering credentials from handlers.

See Also

That version also added [WayBack] System.Classes.ELoginCredendialError – RAD Studio API Documentation

ELoginCredendialError is the exception class for handling invalid login credentials.

ELoginCredendialError is raised when an application attempts to get specified, but unexisting, login credentials.

Nothing ever got added, no examples, no documentation, just look at for instance the Delphi Rio documentation:

Luckily, I found this for my research list (code at [WayBack] tlogincredentialservice.zip) : [WayBack] delphi – TLoginCredentialService usage example – Stack Overflow which uses globals, but that’s not the point of the demo (you can use different storage in your actual code)

I’ve just created a small demo of how to use it

Click here to download the code

In the following I’ll show some of the code:

First I need a record to hold Credentials, and a list of them :

Type
  TCredential = record
    Username, Password, Domain: string;
    constructor Create(const aUsername, aPassword, aDomain: string);
    function AreEqual(const aUsername, aPassword, aDomain: string): Boolean;
  end;

  TCredentialList = class(TList<TCredential>)
  public
    function IsValidCredential(const aUsername, aPassword, aDomain: string): Boolean;
  end;

then we need to define a context in wich we are calling this. Thats just a application unique string wich identifyes each Loginfunction

const
  Context = 'TForm1';

In Form create I create my list and add dummy data to it

procedure TForm1.FormCreate(Sender: TObject);
begin
  CredentialList := TCredentialList.Create;
  //Add Dummy data
  CredentialList.Add(TCredential.Create('AA', 'AA', 'DomainAA'));
  CredentialList.Add(TCredential.Create('BB', 'BB', 'DomainAA'));
  CredentialList.Add(TCredential.Create('CC', 'CC', 'DomainAA'));

  // Register your Login handler in a context.
  // This method is called when you try to login
  // by caling TLoginCredentialService.GetLoginCredentials();
  TLoginCredentialService.RegisterLoginHandler(Context, LoginCredentialEvent);
end;

I have placed a button on my form from wich I make my call to login :

procedure TForm1.Button1Click(Sender: TObject);
begin
  // The actual call to login
  // First param is the context
  // Second Parameres is a callback function given to the event handler.
  TLoginCredentialService.GetLoginCredentials(Context,
    function { LoginFunc } (const Username, Password, Domain: string): Boolean
    begin
      //The actual user validation
      Result := CredentialList.IsValidCredential(Username, Password, Domain);
    end);
end;

Finally I just need to implement my loginhandler:

//This is the "onLogin" event handler.
//This is called durring a login attempt
//The purpose of this event handler are to call tha callBack function with correct information
//and handle the result
procedure TForm1.LoginCredentialEvent(Sender: TObject; Callback: TLoginCredentialService.TLoginEvent; var Success: Boolean);
begin
  //Call the callback
  Callback(Sender, LabeledEdit1.Text, LabeledEdit2.Text, LabeledEdit3.Text, Success);

  //Handle the success.
  if Success then
    Label1.Caption := 'Yes'
  else
    Label1.Caption := 'No';
end;

Hope this answers the question.Dont forget to download the complete code here

–jeroen

Posted in Delphi, Development, Software Development | Leave a Comment »