The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,603 other followers

Archive for the ‘cURL’ Category

wget and curl: downloads that sometimes fail

Posted by jpluimers on 2018/10/19

For my archive somewhere between cURL 7.21.0 and 7.34.0 it does not like to be started from an RDP based tsclient share:

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.21.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  60805      0 --:--:--  0:00:01 --:--:-- 70012

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.34.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: web.archive.org

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www.xs4all.nl

C:\Users\jeroen\Downloads>copy \\tsclient\bin\curl.7.61.0.exe
        1 file(s) copied.

C:\Users\jeroen\Downloads>curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13    0    13    0     0     10      0 --:--:--  0:00:01 --:--:--    10

It fails the same way after net use B: \\tsclient\bin, so that does not matter.

The best link I could find until I got to the real problem was [WayBack] curl: (6) Could not resolve host: application РStack Overflow which shows a different problem: properly quoting.

In addition to remote-name, you can also grab the file name from the headers using --remote-header-name, and --remote-time use the remote file time. The --location follows 302-redirects. You can see that in the example below which I build based on

[WayBack] unix – Curl to grab remote filename after following location – Stack Overflow: The remote side sends the filename using the Content-Disposition header.curl 7.21.2 or newer does this automatically if you specify –remote-header-name / -J.curl -O -J -L $url

C:\Users\jeroen\Downloads>b:\curl.7.21.0.exe --location --remote-name --remote-time --remote-header-name "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  45748      0 --:--:--  0:00:01 --:--:-- 50772
curl: Saved to filename 'pkgWuppdiWP_DX102T_1-1-2.zip'

wget failed big time:

C:\Users\jeroen\Downloads>B:\wget.exe --no-check-certificate -v -v -v --content-disposition --restrict-file-names=windows "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
wget: Cannot read b:/.wgetrc (No such file or directory).
--2018-07-12 09:55:23--  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Resolving web.archive.org... 207.241.225.186
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Retrying.

...

--2018-07-12 09:55:23--  (try:20)  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Giving up.

This is not caused by the filename (Windows does not like the ?¬†question mark in output file names, so¬† – like¬†&¬†ampersand in file URLs – you have to quote the full URL, but also provide the --restrict-file-names=windows¬†parameter; see [WayBack] wget – I can’t download files with “?” – Super User).

–jeroen

Posted in *nix, *nix-tools, cURL, Power User, wget | Leave a Comment »

https://altd.embarcadero.com/ TLS certificate does not match domain name

Posted by jpluimers on 2018/09/07

One of the domains not yet monitored at embarcaderomonitoring.wiert.me, was the altd download server for ISOs and installers on http and https level. Ultimately you want https, as most of these are about installers, so you do not want any man-in-the-middle to fiddle with them.

TLS on altd fails

Upitmerobot is not yet smart enough to check validity of TLS certificates on https connections.

Chrome, Firefox, Safari, Internet Explorer, wget, curl and ssllabs however are.

altd hides as much from itself as possible

Uptimerobot did not like monitoring the plain http://altd.embarcadero.com/ and https://altd.embarcadero.com/ URLs, because the altd is not browsable, so it tries to hide most of its structure from access. This means they both return an odd response:

Those responses are actually 404 errors (note the - minus sign after curl --trace-ascii: it sends the trace to stdout):

$ wget http://altd.embarcadero.com/
--2018-09-05 10:44:23-- http://altd.embarcadero.com/
Resolving altd.embarcadero.com (altd.embarcadero.com)... 88.221.144.40, 88.221.144.10
Connecting to altd.embarcadero.com (altd.embarcadero.com)|88.221.144.40|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-09-05 10:44:23 ERROR 404: Not Found.

$ curl --verbose http://altd.embarcadero.com/
*   Trying 88.221.144.40...
* TCP_NODELAY set
* Connected to altd.embarcadero.com (88.221.144.40) port 80 (#0)
> GET / HTTP/1.1
> Host: altd.embarcadero.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Server: Apache
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 16
< Date: Wed, 05 Sep 2018 08:45:57 GMT
< Connection: keep-alive
< 
* Connection #0 to host altd.embarcadero.com left intact
File not found."

$ curl --trace-ascii - http://altd.embarcadero.com/
== Info:   Trying 88.221.144.40...
== Info: TCP_NODELAY set
== Info: Connected to altd.embarcadero.com (88.221.144.40) port 80 (#0)
=> Send header, 84 bytes (0x54)
0000: GET / HTTP/1.1
0010: Host: altd.embarcadero.com
002c: User-Agent: curl/7.54.0
0045: Accept: */*
0052: 
<= Recv header, 24 bytes (0x18)
0000: HTTP/1.1 404 Not Found
<= Recv header, 16 bytes (0x10)
0000: Server: Apache
<= Recv header, 45 bytes (0x2d)
0000: Content-Type: text/html; charset=iso-8859-1
<= Recv header, 20 bytes (0x14)
0000: Content-Length: 16
<= Recv header, 37 bytes (0x25)
0000: Date: Wed, 05 Sep 2018 08:47:19 GMT
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 2 bytes (0x2)
0000: 
<= Recv data, 16 bytes (0x10)
0000: File not found."
File not found."== Info: Connection #0 to host altd.embarcadero.com left intact

This is also the reason that WayBack does not want to archive that link, but it can be archived at [Archive.is] https://altd.embarcadero.com/.

Luckily, a Google search for site:altd.embarcadero.com revealed there is a non-installer file short enough (~72 kibibytes) for Uptime robot to check, so it now verifies it can access these:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, cURL, Encryption, HTTPS/TLS security, Monitoring, Power User, Security, Uptimerobot, wget | Leave a Comment »

How to tell if your site is served via CloudFlare | Igor’s Blog

Posted by jpluimers on 2018/08/10

Based on [Archive.is]¬†How to tell if your site is served via CloudFlare | Igor’s Blog, I’ve changed the script a little bit.

I’ve tested it with one of the domains from the Cloudbleed list (a pretty OK indication the site is using cloudflare) and a the example.org site¬†that does not:

# curl -sI https://feedly.com | grep "Server\|__cfduid\|CF-RAY"
Set-Cookie: __cfduid=d779ee6e244349cf06e2707771a9185e21492589239; expires=Thu, 19-Apr-18 08:07:19 GMT; path=/; domain=.feedly.com; HttpOnly
Server: cloudflare-nginx
CF-RAY: 351e5e9af8971497-AMS
# curl -sI https://example.org | grep "Server\|__cfduid\|CF-RAY"
Server: ECS (ewr/15BD)

Domain Source: pirate/sites-using-cloudflare: Archived list of domains using Cloudflare DNS at the time of the CloudBleed announcement

–jeroen

via: [WayBack] https://www.igorkromin.net/index.php/2017/04/18/how-to-tell-if-your-site-is-served-via-cloudflare/ РJoe C. Hecht РGoogle+

 

Posted in *nix, *nix-tools, cURL, Power User | Leave a Comment »

cURL – POST an XML file as a stream

Posted by jpluimers on 2017/10/25

I hope I’m not alone on this but I find the cURL documentation hard to follow and short on examples.

My goal was to mimic some HTTP XML posting traffic a server gets from IoT devices. Google Chrome Postman (or Postman REST Client) reproduction is very easy and will send.

TL;DR

  1. ensure you have an empty --header "Content-Type:"¬†header: this ensures that cURL¬†doesn’t add one and does not mess on how the content is being transferred.
  2. use the --data or --data-binary command with an @ to post a file as body.
  3. if you want --write-out then be sure you have a recent cURL version.

This is how the IoT or Postman will send.

  • Post headers like these:

Host:127.0.0.1:8080
Content-Length: 245
Connection:Keep-Alive

  • Content like this:

The data is being streamed to the HTTP server even with the very limited set of headers.

I’ve been unable to come up with exact cURL statement that exactly matches the headers and way the content is being transferred.

This is what I tried (in all examples, %1 is the IPv4 address of the HTTP 1.1 server):

  • POST¬†with the all the headers and the --data¬†command:

curl --request POST --header "Host: %1:8080" --header "Content-Length: 245" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will hang the connection: somehow cURL will never notify the upload is done and the HTTP server keeps waiting. When you put --verbose or --trace-ascii - on the command-line you will see something like this before hanging: * upload completely sent off: 245 out of 245 bytes.

Note the trick to emit the ASCII trace to stdout using --trace-ascii with the minus sign: thanks to [WayBack] Daniel Stenberg for answering [WayBack] How can I see the request headers made by curl when sending a request to the server? РStack Overflow.

You can do the same with --trace which dumps all characters (not only ASCII) including their HEX representation

  • POST¬†with the all but the Content-Length¬†headers and the --data¬†command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will automatically add a Content-Length: 245 header and complete the transfer. But it will also add a Content-Type: application/x-www-form-urlencoded header causing the content not being posted as a body.

  • POST¬†with a --form file=¬†command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --form file=@httpPostSample.xml http://%1:8080/target

This will automatically ad a Content-Length: xxx header (way longer than 245) because it converts the request into a Content-Type: multipart/form-data; boundary=------------------------e1c0d47bac806954 one (the hex at the end differs) which is totally unlike what Postman does.

It is also unlike to what the HTTP server accepts.

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data-binary @httpPostSample.xml http://%1:8080/target

curl –request POST –header “Host: %1:8080” –header “Connection: Keep-Alive” –data-binary @httpPostSample.xml http://%1:8080/target

It turns out that --data-ascii is exactly the same as --data and that --data-binary just skips some new-line conversion when compared to --data or --data-ascii. Contrary to the --data-raw documentation that suggest it is equivalent to --data-binary it seems --data-raw behaves exactly like --data and --data-ascii. Odd.

So these are all stuck with the Content-Type: application/x-www-form-urlencoded and I thought I was running out of options.

Then I found [WayBack] soundmonster had posted an answer at [WayBack] http РWhat is the cURL command-line syntax to do a POST request? РSuper User mentioning to add a Content-Type header.

So I changed the request to include the --header "Content-Type: text/xml; charset=UTF-8"  header:

  • curl --request POST --header "Content-Type: text/xml; charset=UTF-8"¬†--header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This works. But: the Content-Type header is not present in the original request.

Finally it occurred to me: What if cURL would not insert a Content-Type header if I add an empty Content-Type header?.

That works!

  • curl --request POST --header "Content-Type:"¬†--header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

It posts exactly the same content as the IoT devices and Postman do.

Phew!

 

I tried to combine this with the --write-out (a.k.a. -w) option, but for older versions of cURL (I could reproduce with 7.34) that forces cURL back in to Content-Type: application/x-www-form-urlencoded mode so watch your cURL version!

Later I will put more research in chuncked transfer. Links that might help me:

–jeroen

Some of the references:

Posted in *nix, bash, cURL, Development, Power User, Scripting, Software Development | Leave a Comment »

Windows/*n*x: Getting curl to output HTTP status code – Super User

Posted by jpluimers on 2017/10/24

The first trick works in Windowa and nx (thanks [WayBack] pvandenberk):

curl -s -o /dev/null -I -w "%{http_code}" http://www.example.org/

Inside a Windows batch file you need to escape the % to %% so you get this:

curl -s -o /dev/null -I -w "%%{http_code}" http://www.example.org/

The second is slick but only works on nx (thanks [WayBack] Heath Borders):

#creates a new file descriptor 3 that redirects to 1 (STDOUT)
exec 3>&1
# Run curl in a separate command, capturing output of -w "%{http_code}" into HTTP_STATUS
# and sending the content to this command's STDOUT with -o >(cat >&3)
HTTP_STATUS=$(curl -w "%{http_code}" -o >(cat >&3) 'http://example.com')

[WayBack] Getting curl to output HTTP status code? РSuper User

–jeroen

Posted in *nix, *nix-tools, bash, Batch-Files, cURL, Development, Power User, Scripting, Software Development | Leave a Comment »

 
%d bloggers like this: