The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My work

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,312 other followers

Archive for the ‘bash’ Category

Let’s stop copying C / fuzzy notepad

Posted by jpluimers on 2017/12/07

Ah, C. The best lingua franca we have… because we have no other lingua francas. Linguae franca. Surgeons general? C is fairly old — 44 years, now! — and comes from a time when there were possibly more architectures than programming languages. It works well for what it is, and what it is is a relatively simple layer of indirection atop assembly. Alas, the popularity of C has led to a number of programming languages’ taking significant cues from its design, and parts of its design are… slightly questionable. I’ve gone through some common features that probably should’ve stayed in C and my justification for saying so. The features are listed in rough order from (I hope) least to most controversial. The idea is that C fans will give up when I call it “weakly typed” and not even get to the part where I rag on braces. Wait, crap, I gave it away.

Great re-read towards the end of the year: [WayBackLet’s stop copying C / fuzzy notepad

Via: [WayBack] Old and busted: emacs vs vi. New and hot: Language war, everybody against everybody else. – Kristian Köhntopp – Google+

–jeroen

Posted in .NET, APL, Awk, bash, BASIC, C, C#, C++, COBOL, CoffeeScript, CommandLine, D, Delphi, Development, F#, Fortran, Go, Java, Java Platform, JavaScript/ECMAScript, Pascal, Perl, PHP, PowerShell, PowerShell, Python, Ruby, Scala, Scripting, Software Development, TypeScript, VB.NET, VBScript | 3 Comments »

bash: `printf` supports `\e` just like `echo -e` does

Posted by jpluimers on 2017/11/07

Learned a few things when modifying https://github.com/gkotian/gautam_linux/blob/master/scripts/colours.sh

Note: `printf` supports emitting `ESC` (ASCII character `\033` aka `27` aka `0x1B`)as `\e` the same way that `echo` does

https://linux.die.net/man/1/printf
https://linux.die.net/man/1/echo

Format strings are at https://linux.die.net/man/3/printf
%-10s means left adjusted (aligned) string of length 10

–jeroen

via:

I was investigating how the colour definitions on my OpenSuSE system actually work internally so I added some extra output: ${TYPE} and ${COLOUR}.

Source: Show type and colour definition in addition to the rendered colour. by jpluimers · Pull Request #5 · gkotian/gautam_linux

Posted in *nix, *nix-tools, bash, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux | Leave a Comment »

cURL – POST an XML file as a stream

Posted by jpluimers on 2017/10/25

I hope I’m not alone on this but I find the cURL documentation hard to follow and short on examples.

My goal was to mimic some HTTP XML posting traffic a server gets from IoT devices. Google Chrome Postman (or Postman REST Client) reproduction is very easy and will send.

TL;DR

  1. ensure you have an empty --header "Content-Type:" header: this ensures that cURL doesn’t add one and does not mess on how the content is being transferred.
  2. use the --data or --data-binary command with an @ to post a file as body.
  3. if you want --write-out then be sure you have a recent cURL version.

This is how the IoT or Postman will send.

  • Post headers like these:

Host:127.0.0.1:8080
Content-Length: 245
Connection:Keep-Alive

  • Content like this:

The data is being streamed to the HTTP server even with the very limited set of headers.

I’ve been unable to come up with exact cURL statement that exactly matches the headers and way the content is being transferred.

This is what I tried (in all examples, %1 is the IPv4 address of the HTTP 1.1 server):

  • POST with the all the headers and the --data command:

curl --request POST --header "Host: %1:8080" --header "Content-Length: 245" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will hang the connection: somehow cURL will never notify the upload is done and the HTTP server keeps waiting. When you put --verbose or --trace-ascii - on the command-line you will see something like this before hanging: * upload completely sent off: 245 out of 245 bytes.

Note the trick to emit the ASCII trace to stdout using --trace-ascii with the minus sign: thanks to [WayBack] Daniel Stenberg for answering [WayBackHow can I see the request headers made by curl when sending a request to the server? – Stack Overflow.

You can do the same with --trace which dumps all characters (not only ASCII) including their HEX representation

  • POST with the all but the Content-Length headers and the --data command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will automatically add a Content-Length: 245 header and complete the transfer. But it will also add a Content-Type: application/x-www-form-urlencoded header causing the content not being posted as a body.

  • POST with a --form file= command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --form file=@httpPostSample.xml http://%1:8080/target

This will automatically ad a Content-Length: xxx header (way longer than 245) because it converts the request into a Content-Type: multipart/form-data; boundary=------------------------e1c0d47bac806954 one (the hex at the end differs) which is totally unlike what Postman does.

It is also unlike to what the HTTP server accepts.

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data-binary @httpPostSample.xml http://%1:8080/target

curl –request POST –header “Host: %1:8080” –header “Connection: Keep-Alive” –data-binary @httpPostSample.xml http://%1:8080/target

It turns out that --data-ascii is exactly the same as --data and that --data-binary just skips some new-line conversion when compared to --data or --data-ascii. Contrary to the --data-raw documentation that suggest it is equivalent to --data-binary it seems --data-raw behaves exactly like --data and --data-ascii. Odd.

So these are all stuck with the Content-Type: application/x-www-form-urlencoded and I thought I was running out of options.

Then I found [WayBacksoundmonster had posted an answer at [WayBackhttp – What is the cURL command-line syntax to do a POST request? – Super User mentioning to add a Content-Type header.

So I changed the request to include the --header "Content-Type: text/xml; charset=UTF-8"  header:

  • curl --request POST --header "Content-Type: text/xml; charset=UTF-8" --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This works. But: the Content-Type header is not present in the original request.

Finally it occurred to me: What if cURL would not insert a Content-Type header if I add an empty Content-Type header?.

That works!

  • curl --request POST --header "Content-Type:" --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

It posts exactly the same content as the IoT devices and Postman do.

Phew!

 

I tried to combine this with the --write-out (a.k.a. -w) option, but for older versions of cURL (I could reproduce with 7.34) that forces cURL back in to Content-Type: application/x-www-form-urlencoded mode so watch your cURL version!

Later I will put more research in chuncked transfer. Links that might help me:

–jeroen

Some of the references:

Posted in *nix, bash, cURL, Development, Power User, Scripting, Software Development | Leave a Comment »

Windows/*n*x: Getting curl to output HTTP status code – Super User

Posted by jpluimers on 2017/10/24

The first trick works in Windowa and nx (thanks [WayBackpvandenberk):

curl -s -o /dev/null -I -w "%{http_code}" http://www.example.org/

Inside a Windows batch file you need to escape the % to %% so you get this:

curl -s -o /dev/null -I -w "%%{http_code}" http://www.example.org/

The second is slick but only works on nx (thanks [WayBackHeath Borders):

#creates a new file descriptor 3 that redirects to 1 (STDOUT)
exec 3>&1
# Run curl in a separate command, capturing output of -w "%{http_code}" into HTTP_STATUS
# and sending the content to this command's STDOUT with -o >(cat >&3)
HTTP_STATUS=$(curl -w "%{http_code}" -o >(cat >&3) 'http://example.com')

[WayBackGetting curl to output HTTP status code? – Super User

–jeroen

Posted in *nix, *nix-tools, bash, Batch-Files, cURL, Development, Power User, Scripting, Software Development | Leave a Comment »

nojhan/liquidprompt: A full-featured & carefully designed adaptive prompt for Bash & Zsh

Posted by jpluimers on 2017/10/05

Wow: nojhan/liquidprompt: A full-featured & carefully designed adaptive prompt for Bash & Zsh

This is really useful!

via:

Sort of tanslated from the first “via” (note that “mit Alles und Scharf” is hard to translate; it’s somewhere between “everything but the kitchen sink, but done right” and “right on the money”):

Bash Prompt Overkill: https://github.com/nojhan/liquidprompt is a Bash “Prompt doing it all right”-extension, which doesn’t care how much any feature costs as we have cores, gigabytes and SSD.

Liquid Prompt automagically recognises context and enables a plethora of features in the prompt when needed based on that context.

It’s like pixie dust for your prompt.

You can configure everything, but you don’t have to: the out of the box experience is already like pixie dust for your prompt.

It works on OS X too and is part of homebrew:

$ brew install liquidprompt
==> Using the sandbox
==> Downloading https://github.com/nojhan/liquidprompt/archive/v_1.11.tar.gz
==> Downloading from https://codeload.github.com/nojhan/liquidprompt/tar.gz/v_1.11
######################################################################## 100.0%
==> Caveats
Add the following lines to your bash or zsh config (e.g. ~/.bash_profile):
  if [ -f /usr/local/share/liquidprompt ]; then
    . /usr/local/share/liquidprompt
  fi
If you'd like to reconfigure options, you may do so in ~/.liquidpromptrc.
A sample file you may copy and modify has been installed to
  /usr/local/share/liquidpromptrc-dist
Don't modify the PROMPT_COMMAND variable elsewhere in your shell config;
that will break things.
==> Summary
🍺  /usr/local/Cellar/liquidprompt/1.11: 7 files, 125.6K, built in 3 seconds
[jeroenp:~/Versioned] 10s $

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, bash, bash, Development, OS X, Power User, Scripting, Software Development | Leave a Comment »

 
%d bloggers like this: