The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,676 other followers

Delphi “Variant Records”, a few notes

Posted by jpluimers on 2013/03/14

Variant Records are a feature that has been in the Pascal language since Standard Pascal.

A cool page for historic perspective is R3R: Pascal Features in Popular Compilers, hopefully someone will update it to more modern versions of the mentioned compilers.

There is not much official documentation on the Delphi side on this, so below some parts of a case I used for a project that started in 1997 and is still in use to day.

The project involves communication between applications running on PCs (in 1997 there were many written in Borland Pascal 7 and a few in Delphi 3) and AS/400 (now called “System i” after an initial rename to “iSeries”) using CPI-C and APPC (both parts of SNA).

Back then, it was top notch for a few reasons:

  • Most of the LAN and WAN was already IP-based, but HOST (back then System/390, now called System z after an initial rename to “zSeries”), AS/400 and 3270 terminal emulation were still SNA based. Which meant using Microsoft SNA Server (meanwhile renamed to HIS or Host Integration Services).
  • Both an architect and the board were involved and because of strategic alliances, the solution had to make use of

    It was impossible to get this stable (the combination of these 3 requirements caused any cobination of high CPU usage, or undefined memory growth, or unreliable communications) which took 4 months to debug with overseas help form Microsoft, NetSoft and IBM to prove that direction – that worked in the labs of the 3 partners – was impractical in the real world.

  • After the initial failure, the AS/400 programmer and I sat together without the architect and the board to make up our minds.
    The AS/400 programmer was a very smart guy, and since we had a closet full of IBM Red Books, we decided to see what was available in the SNA protocol. Low and behold, it was very versatile, included authentication, automatic encryption over the wire (very useful for a huge financial), automatic release of resources used by broken sessions and much more.

The communications layer on the PC side is still in Delphi (as of mid 2012 in Delphi XE2) but in 2005 the DOS client software got rewritten in India by a company advertising to be CMM level 6 (impossible) and having deep experience with COOL:Gen (also named as IEF, Composer, COOL:Gen, Advantage Gen, AllFusion Gen, CA Gen) and C/C++.

In India, they had some C/C++ gurus that agreed it would be OK if the communications layer became a DLL and they would write a small C interface inbetween COOL:Gen and the DLL. It ended up that after half a week of me explaining how they should do that in Visual C++ 6 (modern as they were, they still opted to use the 1998 release while it was 2005), I wrote that portion for them as well.

The communication layer on the AS/400 side is still mainly COBOL with a bit of RPG IV.

In 2011, the whole communication part got rewritten on the PC side using a Delphi layer to MQ Series 5.3 (now called WebSphere MQ) that was officially already unsupported, and on the AS/400 side WebSphere MQ v6 with an interface layer still mostly in COBOL and RGP IV.

In retrospect that should have been done a lot earlier, as in 2006 the AS/400 got moved about a 1000 miles away, not a distance that the SNA protocol would easily cope with, not even with an Enterprise Extender. Back then I recommended rewriting the middle layer based on a solution sitting on top of TCP/IP, but they didn’t want to as “it worked fine”.

It took another 4 years for major interruptions to occur and go away spontaneously, and finally not to go away in 2011.

In 2012 another, but shorter, refactoring round was done to bring the PC side to Windows 7 and WebSphere MQ version 7.x, and the AS/400 side to e WebSphere MQ 7.x (and also move the AS/400 a lot closer).

Between 2011 and 2012, there have not been any outages apart from a few configuration issues (where someone forgot to make a couple of AS/400 subsystems automatically start upon AS/400 LPAR reboot).

The really cool thing about the SNA to WebSphere MQ transition is that the communications layer instantly became twice as fast for “normal” communications, an order of magnitude faster on heavy traffic, and orders better in case of communications trouble (now there are even reasonable error messages to the end users).

Back to Variant Records (:

The goal of Variant Records is to map different kinds of memory structures onto the same memory. It is similar to unions in C/C++.

The Delphi implementation is very similar to the Standard Pascal one.

Variant records are like absolute (where you define two variables to have the same memory location):

You use the a variation of case to distinguish the different structures to start at the same memory position with the record.
As with a regular case, the tags need to be unique.

Both support for RTTI and Windows messages relies heavily on variant records.

A record can start as a regular record, and towards the and can have one variant part in a record definition.

The cool thing is that Variant Records are not going away in the NEXTGEN compiler, just look at the documentation of the TTypeData variant record to see the NEXTGEN conditional defines.

Before looking at the example code, lets first list a few limitations.

  • You cannot used managed types inside variant records (but you can use them in the regular portion), as the compiler would not know which of the tagged cases it should initialize/finalize.
    So these are out: Strings, Open Dynamic Arrays, Interfaces, Anonymous methods, Variants or Records containing other managed types.
    If you still try to, you get error messages like E2154 or E2418, or E2569.
  • I think you can create a generic record as long as you apply a record constraint on the generic parameter.
  • In Delphi .NET (now defunct) you could get the E2417 error when the compiler could not determine the record size in the first compile pass.

The example code below starts out with a few simple things:

  1. single byte character type TChar (we are dealing with DOS and COOL:Gen, they don’t get Unicode yet)
  2. single byte character arrays (zero based called TChar###, 1-based called T1Char###).
  3. the TVariantData part (simplified from the communications code, but still shows enough detail on how and why)
    the trick here is that the record needs to be exactly 33 bytes long, and the “Contents” portion takes care of that.
  4. the TVariantKey part (shows how ID’s were done in the SNA implementation and are now done in the MQ implementation)
    with this one, we could run both the SNA version and the MQ version at the same time and do refactoring and regression testing.

This is basically all you need to know: look at how the case portions are formulated, you can use it with any ordinal (Boolean, Integer, enumeration) tag.

Have fun with it!

–jeroen

unit VariantRecordUnit;

interface

{ First a few basic types}

const
  TGuidStringSize = 38;

type
  TChar = AnsiChar; { single byte character, as it interfaces with DOS and CoolGen programs through C interface }
  TChar2    = array[0..   1] of TChar;
  TChar8    = array[0..   7] of TChar;
  TChar10   = array[0..   9] of TChar;
  TChar20   = array[0..  19] of TChar;
  T1Char33 = array[1..33] of TChar; { 1-based because the DOS CAS sources expect this }
  TGuidChar   = array[0..TGuidStringSize-1] of TChar;
  TMessageId = array[0..23] of Byte;

{ now the record types }

type
  TVariantData = record
  case Boolean of
    False: (
      ProgramName: TChar10;
      InterChangeFormat: TChar10;
      FunctionCode: TChar2;
      ReturnCode: TChar2;
      ErrorCode: TChar2;
      Zero: TChar2);
    True: (Contents: T1Char33);
  end; { total: 33 bytes }

  TId = packed record
    NetBiosName: TChar20; { historically, as DOS app defined it wrongly }
    TimeStamp: TChar8; { HHMMSShh because a DOS directory name can be no longer than 8 characters }
  end; { total: 28 bytes }

  TVariantKey = packed record
  case Integer of
    0: ( // SNA
      ConversationId: TId; { 28 bytes }
      GuidChars: TGuidChar); { 38 bytes }
    2: ( // MQ
      ConversationIdFiller: TId;
      MessageId: TMessageID); // 24 bytes
  end; { total: 66 bytes }

  TPacket = packed record
    EntryType : Byte;
    ReturnKey : TVariantKey;
    DataType  : Byte;
    Data      : TVariantData;
  end;

implementation

end.

–eof–

9 Responses to “Delphi “Variant Records”, a few notes”

  1. […] mentioned Delphi managed types is in Delphi “Variant Records”, a few notes indicating they are […]

  2. […] ObjectMethodology.co… on Delphi “Variant Records… […]

  3. Cool, that kinda’ fits under my everything class:

    TEverything = class
    TAnything;
    end;

    BTW, I’m looking for Delphi developers. Visit my website for contact information.

  4. abouchez said

    Note that Delphi XE3 changed internal memory alignment of variant records.

    So if the purpose of your record definition is to access external C API structures, be aware of this breaking change.
    See for instance what we have to do when accessing the http.sys API structures – http://synopse.info/fossil/info/5e5353438a

    I suspect also it is worth saying that “packet record” may not work with double variables, since x64/SSE2 expects them to be aligned, AFAIK.

  5. fcolibri said

    as I remember
    – the first Pascal compiler was written mid 1970
    – the P4 compiler was release in 1976 and extensively used the variant (pointed) records for the identifier and type structures
    – the ISO 7185 group started in 1977 and the spec came only around 1982. So the P4, with the variants, seem to predate this standard pascal version

    for a Pascal reference site, I may also recommend
    http://www.moorecad.com/standardpascal/

  6. woutervannifterick said

    Did my previous comment disappear, or what happened?

    Anyway, the link about Standard Pascal, to prove how old Variant Records are, have a timestamp of “Wed, 18 Jun 1902 08:00:00 GMT”. I don’t know if it was intentional, but it was funny :-)

  7. Wouter van Nifterick said

    Haha, the forum posts about standard pascal to prove how old variant records are have a timestamp of “Wed, 18 Jun 1902 08:00:00 GMT”. I don’t know if it was intentional, but it was funny. :-)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: