The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,839 other subscribers

Archive for the ‘XML/XSD’ Category

« Previous Entries
Next Entries »

Delphi, soap and wrapping values in cdata – Stack Overflow

Posted by jpluimers on 2020/09/15

For my link archive: [WayBack] Delphi, soap and wrapping values in cdata – Stack Overflow

–jeroen

Posted in Delphi, Development, SOAP/WebServices, Software Development, XML/XSD | Leave a Comment »

Not all XSD mappings to programming language constructs are possible

Posted by jpluimers on 2020/02/26

This post is a reminder to myself that not all mappings from XSD to programming languages are possible.

There are many impossible cases, so this is just a general reminder.

A Delphi specific case for instance is the mapping of enumerations: one reason is that XSD enumerations are case sensitive, but the Delphi language is not: [WayBack] Using XML Enumerations with Delphi XML Data Binding Wizard – Stack Overflow.

More generic examples from my answer to the above question:

  • In XSD you can derive from an existing type in two ways: extending it and limiting it. Object Oriented languages only allow you extend when deriving.
  • Delphi is not alone in these kinds of limitations. Generating wrappers from XSD schema’s is the field of specialized tools, even in the Java or .NET world.

I’ve seen horrible things with wildcards that are sort of mappable to Java, but not to C#. This could likely go on for much longer…

–jeroen

Posted in C#, Delphi, Development, Java, Java Platform, Software Development, XML, XML/XSD, XSD | 2 Comments »

Offline XML Pretty Print

Posted by jpluimers on 2019/12/20

I wrote about [WayBack] XML Pretty Print in Online XML Pretty Print many years ago.

That’s all fine for small XML file and on-line usage.

A while ago however, I had two cases where off-line XML pretty printing turned out to be much easier than online XML pretty printing:

  1. pretty-print many XML files in one go
  2. compare large (100 megabyte plus) XML files

So I went searching, especially for a solution that would be available for both nx based platforms (Linux/Mac OS X/etc) and Windows which got me a few options in [WayBack] unix – How to pretty print XML from the command line? – Stack Overflow.

I found [WayBack] tidy to be more practical than [WayBack] xmllint as tidy would split more lines which made it easier for  Beyond Compare (that is very much cross platform!) to spot and show the differences.

Not just easier in the sense that the (already excellent) diff engine from Beyond Compare (which really sets it apart from other differencing tools) but very much easier on the eyes as now differences where on lines below each other in stead of 1000s of character positions in one line you had to scroll through.

There is two important things to remember with tidy that both stem from its ancestry (it originates from an HTML):

  • you have to specify -xml
  • you have to specify the characterset (even though XML can specify it and without a definition, XML by default is UTF8, tidy does not know about that)

These were the command-lines I used on Windows to do the pretty printing:

tidy -xml -utf8 QCScaper.test@borland.com.cds.xml > QCScaper.test@borland.com.tidy.cds.xml
xmllint --format QCScaper.test@borland.com.cds.xml > QCScaper.test@borland.com.pretty-printed.cds.xml

Another trick is to write a small Delphi program centered around the FormatXMLData call which has been present since Delphi 2007 (see [WayBack] devnet.pdf), but documented since Delphi 2009 in [WayBack] XMLDoc.FormatXMLData Function.

Example code: [WayBack] delphi – How to reformat XML programmatically? – Stack Overflow.

An alternative is to use OmniXML: [WayBack] delphi – Nice bit of code to format an xml string – Stack Overflow

–jeroen

Related posts:

Read the rest of this entry »

Posted in Beyond Compare, Development, Power User, XML, XML/XSD | Leave a Comment »

… compare two JSON structures and pin-point … the differences – – Nicholas Ring – Google+

Posted by jpluimers on 2019/08/20

I’ve added a few WayBack/Archive.is links to the interesting comments by Zoë Peterson from Scooter Software (of Beyond Compare fame) at [WayBack] … compare two JSON structures and pin-point … the differences – – Nicholas Ring – Google+:

Beyond Compare 4 has an optional “JSON sorted” file format that uses jq to pretty print and sort JSON data before comparing it. It’s not included out of the box yet, but you can get a copy here:

If you’re interested in an actual algorithm and not just an app, I don’t have a suggestion handy, but could dig one up. Tree alignment is more complicated than sequence alignment and we did do research into it, but it was quite a few years ago and didn’t get incorporated into BC. XML alignment algorithms were being actively researched back in the aughts and they should trivially transfer to JSON.
…

It looks like our research mostly ended around 2002, and I wasn’t personally involved in it, so I don’t know how helpful this will be, but here’s what I have:

The general idea in the thread is that JSON – though not as formalised as XML – does have structure, so if you can normalise it, then XML ways of differencing should work.

Normalisation also means that you need to normalise any floating point, date time, escaping, quoting, etc. Maybe not for the faint of heart.

–jeroen

Posted in *nix, *nix-tools, Beyond Compare, Development, diff, JavaScript/ECMAScript, jq, JSON, Power User, Scripting, Software Development, XML, XML/XSD | Leave a Comment »

detect availability of node-set() function in xslt implementation of icecast server – Stack Overflow

Posted by jpluimers on 2019/03/07

Wow, I totally forgot I had done this xslt trick eons ago: [WayBack] detect availability of node-set() function in xslt implementation of icecast server – Stack Overflow

–jeroen

Posted in Development, Software Development, XML/XSD, XSLT | Leave a Comment »

Reminder to self: make a start for an XSD that validates Delphi dproj files

Posted by jpluimers on 2018/11/07

Below are a few interesting things from this thread [WayBack] Any idea why Delphi (At least since Seattle) does this to Dproj filesand… – Tommi Prami – Google+.

It’s likely that the WordPress system deleted parts between angle brackets because it thinks it’s invalid HTML and WordPress is PHP.

Anyhow: maybe by now I’ve found time for creating an XSD for .dproj files in addition to the one I mentioned yesterday for the .groupproj files: Validating a nested XML element with an empty namespace using XSD – Stack Overflow.

Oh and in the mean time for various Delphi versions, there is a (binary, because it uses some commercially licensed parts) [WayBack] Dproj changed or not changed? Normalize it! | The Art of Delphi Programming.

Files and directories I need to research:

In the mean time, QualityPortal indicates that:

Achim Kalwa:
That is a long living bug: https://quality.embarcadero.com/browse/RSP-11308
Still exists in Berlin Update 2.

Jeroen Wiert Pluimers
For more than a decade, the IDE rewriting dproj files in different order/indentation/spacing, properties appearing/disappearing in DFM files and writing DFM files despite no changes have driven me even more nuts than I was.

Walter Prins
+Jeroen Wiert Pluimers Yes. Given that .dproj is XML I’ve been almost annoyed enough to look at making some kind of pre-checkin script to force sort things to try and avoid the issue. Almost… Edit: Ah, just checked that RSP, which references this: https://github.com/joshkel/RadCli Seems someone has already done it.

Lübbe Onken
Did you try tidy_proj +Walter Prins? For me it fails with a “Unicode Encode Error” when I try to write an output file. In my case this is caused by German umlauts in the excluded bpl section descriptions. Changing the umlauts to something low-ascii allows tidy_proj to finish.

Walter Prins
+Lübbe Onken No, but I have just done so. Unicode bites again. To fix that error, change line 125 from:

print(proj.toxml())

to:

print(proj.toxml().encode(‘utf-8’))

Note: This will likely not display correctly if you allow the output to go to the console but will ensure it is correct if redirected back to file.

Edit: Ha, I was about to go report this issue ad suggest this fix, but I see you’ve already suggested essentially the same fix. ;)

Jeroen Wiert Pluimers’s profile photo
Jeroen Wiert Pluimers
+Walter Prins interesting tool! Do you know of a tool that can downgrade .dproj files so you can use them with older Delphi versions? (the opposite of what the IDE does)?

Or someone that has written more than my XSD on .dproj files? My one is so small that’s hardly a real attempt. https://wiert.me/2013/08/31/delphi-first-try-on-an-xsd-for-groupproj-files/

Walter Prins
+Jeroen Wiert Pluimers No, but interesting idea. I suppose you’re probably aware that the JEDI JCL/JVCL takes a somewhat similar (ish) related approach, in that it generates project and package files that are compatible with every desired version of Delphi from minimalist (XML) template files. Every so often I think it might be worth looking whether one can re-use the code from there easily but haven’t gotten around to it. Always something else to do first. :/

Jeroen Wiert Pluimers
+Walter Prins no I didn’t. Where should I start reading on that minimalist template approach?

Walter Prins
+Jeroen Wiert Pluimers Sorry I see I missed out the word installer in my comment: It’s the Jedi JCL/JVCL installers that uses xml templates etc.

But the point stands: I guess it should be (perhaps) possible to reuse its infrastructure for ones own projects, though as I say I keep meaning to look into this but haven’t really done so.

(BTW I’m assuming you are familiar with the JCL/JVCL and in in particular their installers that bootstrap from source code? If not then perhaps my comments may not be that useful?)

Anyway, not sure if or where there’s particular documentation about this (kind of doubt it), but if you have the JCL/JVCL installed somewhere, then first of all have a look at the “xml” folders e.g. “<jclroot>\jcl\packages\xmls” and “<jvclroot>\jvcl\packages\xml”.

These files appear to define projects and packages in a seemingly abstract/somewhat minimal way using XML. This seemingly is then used to automatically produce .dproj and .dpk files which are placed in e.g. “<jclroot>\jcl\packages\dXX” and “<jvclroot>\jvcl\packages\dXX” where dXX corresponds to a folder for each of the supported Delphi versions and compiled/used during installation.

Having just looked into this briefly a bit further as a result of this conversation: Key units here (in the case of JVCL) seems to be “<jvclroot>\devtools\PackagesGenerator\PackageGenerator.pas”, “<jvclroot>\install\JVCLInstall\PackageUtils.pas” and “<jvclroot>\devtools\common\PackageInformation.pas” (used predictably by <jvclroot>\JVCLInstall.dproj)

(Additionally, having looked at this a bit more closely, it also appears that the JCL does not actually replace all or even most of its .dproj and .dpk files after all, but ships version specific .dproj and .dpk files in most cases, though the JVCL does appear to mostly do so. )

Edit: The JCL also has this interesting sounding unit “jclroot>\source\windows\JclMsBuild.pas” which appears to be an MSBuild project file parser…

–jeroen

Posted in Conference Topics, Conferences, Continuous Integration, Delphi, Development, Event, msbuild, Software Development, XML, XML/XSD, XSD | Leave a Comment »

Validating a nested XML element with an empty namespace using XSD – Stack Overflow

Posted by jpluimers on 2018/11/06

In a quest for making Delphi Group Project XML files (with extension .groupproj) validate with XSD, I had to find out about [WayBack] Validating a nested XML element with an empty namespace using XSD – Stack Overflow.

I needed the XSD so I could import it in Delphi in order to write a good .groupproj file generator.

They are here:

And some more background posts are here:

–jeren

Posted in Delphi, Development, Software Development, XML, XML/XSD, XSD | 1 Comment »

Schema Central: XML vocabularies with all their XSD schema’s in a nice browsable overview including references, explanations and examples

Posted by jpluimers on 2018/09/11

Schema Central, by [WayBack] Datypic, Inc. (Priscilla Walmsley) is to golden that I quote the whole index page below.

The index refers to XML vocabularies. Each vocabulary page links the corresponding XSD pages. Each XSD page lists documentation, elements, etc. Each element page then documents the element, shows where it is referenced from.

Welcome to Schema Central, an interactive tool to traverse and search XML schemas. Please choose your XML vocabulary of interest to get started. Don’t see the XML vocabulary you use? Ask us to add it.

Core XML Technologies

Presentation-Oriented Standards

Office/Narrative Document Standards

Business Document Standards

NIEM-related Standards

Miscellaneous

Web Services Technologies

Security-Related Technologies

Site developed and hosted by Datypic, Inc.

Please report errors or comments about this site to contrib@functx.com

[WayBack]

If I ever need a long term XML/XSD/XSLT/XQuery hire, this is the “about” information I need:

Datypic provides consulting services and training, specializing in XML, content management and SOA/Web Services architecture and development. We are experts in XML-related technologies such as XML Schema, XSLT and XQuery, and have extensive experience with software development and implementation.

We participate in consulting projects ranging from one day to many months, anywhere in the world. We can arrange to work remotely or at your site, whichever you prefer.

For more information, please read about our services or our company, or contact us as pwalmsley@datypic.com.

Recent news from Priscilla Walmsley

  • I recently updated my book XQuery to match the final XQuery 3.1 recommendation. You can read more about it or order it from O’Reilly or Amazon.
  • I recently taught a course at the XML Summer School on Refactoring XSLT. Slides are available as a PDF.
  • More information about my recent books and articles is available on this site.

–jeroen

Posted in Development, Software Development, XML, XML/XSD, XPath, XSD, XSLT | Leave a Comment »

Delphi XE6 and up regression: “‘9999-12-31 23:59:59,1000’ is not a valid date and time” when passing a SOAP message with 9999-11-31T23:59:59.9999999; QC144171

Posted by jpluimers on 2018/09/06

A valid SOAP message with <urn:timeStamp>9999-11-31T23:59:59.9999999</urn:timeStamp> in a xs:dateTime field return '9999-12-31 23:59:59,1000' is not a valid date and time from a Delphi application with this SOAP response:

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
  <SOAP-ENV:Body>
    <SOAP-ENV:Fault>
      <faultcode>SOAP-ENV:Server</faultcode>
      <faultstring>'9999-12-31 23:59:59,1000' is not a valid date and time</faultstring>
      <faultactor/>
    </SOAP-ENV:Fault>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

The reason is this exception:

exception class EConvertError with message ''9999-12-31 23:59:59,1000' is not a valid date and time'.

This is from a .NET based test case passing in timeStamp = DateTime.MaxValuewhich is handled perfectly fine by other SOAP web services tested.

I know about different resolutions of time stamps, but would never expect the 999.9999 milliseconds to be rounded up to 1000 as it is always safer to truncated away from an upper limit.

A test using Soap UI [WayBack] with this parameter finally worked (max 3 digits second fraction):

<urn:timeStamp>9999-12-31T23:59:59.999</urn:timeStamp>

The true origin of problem is in this method in the Soap.XSBuiltIns unit which has been unchanged since at least Delphi 7:

function TXSBaseTime.GetMilliSecond: Word;
begin
  Result := Round(FractionalSeconds*1000);
end;

The problem exposed itself because as of Delphi XE6 the core of function TXSBaseCustomDateTime.GetAsDateTime piece was changed from

Result := EncodeDateTime(Year, Month, Day, Hour, Minute, Second, 0);

to

Result := EncodeDateTime(Year, Month, Day, Hour, Minute, Second, Millisecond);

A combination of lack of test cases and understanding XML specifications failed to reveal this bug.

The standards specify (among others):

  • '.' s+ (if present) represents the fractional seconds;
    The above is not limiting the amount of digits, not talking about milliseconds either.
  • All ·minimally conforming· processors ·must· support year values with a minimum of 4 digits (i.e., YYYY) and a minimum fractional second precision of milliseconds or three decimal digits (i.e. s.sss). However, ·minimally conforming· processors ·may· set an application-defined limit on the maximum number of digits they are prepared to support in these two cases, in which case that application-defined maximum number ·must· be clearly documented.
    Delphi not only limits the fractional second precission, it changes the limit over time and does not document the limit. Three strikes…
  • s -- represents a digit used in the time element "second". The two digits in a ss format can have values from 0 to 60. In the formats described in this specification the whole number of seconds ·may· be followed by decimal seconds to an arbitrary level of precision. This is represented in the picture by "ss.sss". A value of 60 or more is allowed only in the case of leap seconds.
    Given buggy the fractional second handling through milliseconds, the leap second handling is ripe for a test case as well.
    Strictly speaking, a value of 60 or more is not sensible unless the month and day could represent March 31, June 30, September 30, or December 31 in UTC. Because the leap second is added or subtracted as the last second of the day in UTC time, the long (or short) minute could occur at other times in local time. In cases where the leap second is used with an inappropriate month and day it, and any fractional seconds, should considered as added or subtracted from the following minute.

The reproduction is quite simple:

Read the rest of this entry »

Posted in .NET, C#, Conference Topics, Conferences, Delphi, Development, Event, SOAP/WebServices, Software Development, XML, XML/XSD | Leave a Comment »

Some search links on Delphi and C# WSDL imports I need to investigate further

Posted by jpluimers on 2018/09/05

Sometimes, the Delphi WSDL importer imports fine, but the generated code does not accept test cases sent by other tools.

Below are some links for messages and comment fragments that I want to investigate further.

I have included the .NET message, because my experience is that searching on those gives more accurate results for something that could be broken in more than one environment.

Based on those:

Some on-line tools prefer the WSDL to be in one document, but a lot of WSDL documents use import and or include features, so here are some links on that too:

Bruneau Babet correctly informed me that – though Delphi SOAP clients support both document literal and RPC encoded – Delphi SOAP servers cannot support document literal, as they can only support RPC encoded. Through that I found

  • [WayBack] Apache CXF — WSDLValidator
    • Check the WSDL document for XML well-formedness.
    • Validate the WSDL document against its XML schema.
    • Validate the WSDL document using some of the semantic rules defined in the WSDL specification.
    • Validate the WSDL document against custom validation rules, such as those defined by the Web Services Interoperability (WS-I) organization (i.e. WS-I Basic Profile rules).
    • Validate the WSDL against strange exceptions, incorrectly generated code and general bad WSDL issues.

Back on those days, the big plan was to move everything Delphi to the .NET platform which supports both document literal and RPC encoded.

All in all, document literal has been on the radar with the Delphi R&D team since at least 2009, and nothing has been done.

References:

I looks like a wsdl message request part entries need to be named parameters for some tooling to correctly infer document/literal in a wrapped way. Some links for further research on this:

When you are surely running SOAP over HTTP, you can use this small class to raise exceptions which automatically get translated into SOAP Faults having the right return code using a trick I bumped into a few years ago from [WayBack] web services – Accessing the original TWebRequest object in a Delphi SOAP Server – Stack Overflow:

unit SoapFaultWithHttpCodeExceptionUnit;

interface

uses
  System.SysUtils;

type
  ESoapFaultWithHttpCodeException = class(Exception)
  strict private
    FHttpStatusCode: Integer;
  public
    constructor Create(const AHttpStatusCode: Integer);
    property HttpStatusCode: Integer read FHttpStatusCode;
  end;

implementation

uses
  Winapi.WinHTTP,
  Soap.WebBrokerSOAP,
  Web.HTTPApp,
  IdCustomHTTPServer;

constructor ESoapFaultWithHttpCodeException.Create(const AHttpStatusCode: Integer);
var
  IdHTTPResponseInfo: TIdHTTPResponseInfo;
  ReasonString: string;
  WebDispatcher: IWebDispatcherAccess;
begin
  IdHTTPResponseInfo := TIdHTTPResponseInfo.Create(nil, nil, nil);
  try
    FHttpStatusCode := AHttpStatusCode;
    IdHTTPResponseInfo.ResponseNo := AHttpStatusCode;
    ReasonString := Format('%d: %s', [AHttpStatusCode, IdHTTPResponseInfo.ResponseText]);
    inherited Create(ReasonString);

    // https://stackoverflow.com/questions/10405227/accessing-the-original-twebrequest-object-in-a-delphi-soap-server
    if Supports(GetSOAPWebModule, IWebDispatcherAccess, WebDispatcher) then
    begin
      WebDispatcher.Response.StatusCode := HTTP_STATUS_SERVER_ERROR;
      WebDispatcher.Response.ReasonString := ReasonString;
    end

  finally
    IdHTTPResponseInfo.Free;
  end;
end;

end.

—jeroen

Read the rest of this entry »

Posted in .NET, C#, Conference Topics, Conferences, Delphi, Development, Event, SOAP/WebServices, Software Development, XML/XSD | Leave a Comment »

« Previous Entries
Next Entries »