The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,570 other followers

Archive for October 19th, 2018

wget and curl: downloads that sometimes fail

Posted by jpluimers on 2018/10/19

For my archive somewhere between cURL 7.21.0 and 7.34.0 it does not like to be started from an RDP based tsclient share:

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.21.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  60805      0 --:--:--  0:00:01 --:--:-- 70012

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.34.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: web.archive.org

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www.xs4all.nl

C:\Users\jeroen\Downloads>copy \\tsclient\bin\curl.7.61.0.exe
        1 file(s) copied.

C:\Users\jeroen\Downloads>curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13    0    13    0     0     10      0 --:--:--  0:00:01 --:--:--    10

It fails the same way after net use B: \\tsclient\bin, so that does not matter.

The best link I could find until I got to the real problem was [WayBack] curl: (6) Could not resolve host: application РStack Overflow which shows a different problem: properly quoting.

In addition to remote-name, you can also grab the file name from the headers using --remote-header-name, and --remote-time use the remote file time. The --location follows 302-redirects. You can see that in the example below which I build based on

[WayBack] unix – Curl to grab remote filename after following location – Stack Overflow: The remote side sends the filename using the Content-Disposition header.curl 7.21.2 or newer does this automatically if you specify –remote-header-name / -J.curl -O -J -L $url

C:\Users\jeroen\Downloads>b:\curl.7.21.0.exe --location --remote-name --remote-time --remote-header-name "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  45748      0 --:--:--  0:00:01 --:--:-- 50772
curl: Saved to filename 'pkgWuppdiWP_DX102T_1-1-2.zip'

wget failed big time:

C:\Users\jeroen\Downloads>B:\wget.exe --no-check-certificate -v -v -v --content-disposition --restrict-file-names=windows "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
wget: Cannot read b:/.wgetrc (No such file or directory).
--2018-07-12 09:55:23--  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Resolving web.archive.org... 207.241.225.186
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Retrying.

...

--2018-07-12 09:55:23--  (try:20)  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Giving up.

This is not caused by the filename (Windows does not like the ?¬†question mark in output file names, so¬† – like¬†&¬†ampersand in file URLs – you have to quote the full URL, but also provide the --restrict-file-names=windows¬†parameter; see [WayBack] wget – I can’t download files with “?” – Super User).

–jeroen

Posted in *nix, *nix-tools, cURL, Power User, wget | Leave a Comment »

Euler’s identity – cool explanatory videos by Burkard Polster

Posted by jpluimers on 2018/10/19

[WayBack] e^iŌÄ = -1 – Daniela Osterhagen – Google+¬†reminded me of [WayBack]¬†Euler’s identity¬†(often called the most beautiful formula in the world) and these two great videos by Burkard Polster I bumped into a few years back below.

She points to a German link about “the most beautiful formula in the world”, to which I fully agree:

Gerade einmal sechs Symbole braucht es daf√ľr

[WayBack] Freistetters Formelwelt: Die schönste Formel der Welt РSpektrum der Wissenschaft

Read the rest of this entry »

Posted in Power User, science | Leave a Comment »

when btrfs-size shows a snapshot as 16777216.00TB or btrfs qgroup as 16.00EiB

Posted by jpluimers on 2018/10/19

A long time ago I wrote about the btrfs-size tool: [WayBack] A bash script to btrfs snapshot details like disk sizes (requires btrfs quota to be enabled).

One day, it showed a ridiculously large size for /tmp:

# ./btrfs-size.sh 
=============================================================================================================================================================================================================================================================
Snapshot / Subvolume                                               ID   Total    Exclusive Data
=============================================================================================================================================================================================================================================================
257 gen 855182 top level 5 path .snapshots                         257  4.30MB   4.30MB   
258 gen 856438 top level 257 path .snapshots/1/snapshot            258  1.84GB   193.01MB 
...
262 gen 856438 top level 5 path srv                                262  1.83GB   1.83GB   
263 gen 856438 top level 5 path tmp                                263  16777216.00TB16777216.00TB
264 gen 856438 top level 5 path usr/local                          264  260.00KB 260.00KB 
...
990 gen 849192 top level 257 path .snapshots/583/snapshot          990  1.83GB   8.23MB   
991 gen 849224 top level 257 path .snapshots/584/snapshot          991  2.09GB   62.66MB  
=============================================================================================================================================================================================================================================================
                                                                Exclusive Total: 3.26GB    

This tracks back to the output of this command, which I’ve shortened a bit:

# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5          16.00KiB     16.00KiB
0/257         4.30MiB      4.30MiB
...
0/262         1.83GiB      1.83GiB
0/263        16.00EiB     16.00EiB
0/264       260.00KiB    260.00KiB
...
255/274         0.00B        0.00B
255/797      16.00KiB     16.00KiB

This is a known issue as quotas in btrfs – though workable – aren’t fully stable yet:¬†[WayBack] Linux BTRFS Storage: Re: During a btrfs balance nearly all quotas of the subvolumes became exceeded

It also provides this simple solution:

Read the rest of this entry »

Posted in *nix, *nix-tools, btrfs, File-Systems, Power User | Leave a Comment »

git 2.19.1 on Windows: NullReferenceException during https pull/push/fetch

Posted by jpluimers on 2018/10/19

When using git 2.19.1 on Windows, you will likely get a NullReferenceException during pull/push/fetch operations on https connections.

Easiest workaround is to downgrade to 2.19.0: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

I am seeing the same thing using 2.19.1 against our internal Bitbucket server. Reverting back to 2.19.0 the problem no longer occurs.

Harder workaround is to install the Git Credential Manager 1.18.1: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub.

[WayBack] Release Git Credential Manager for Windows v1.18.1 · Microsoft/Git-Credential-Manager-for-Windows · GitHub

Bug Fixes:

  • Fixes Null Reference exceptions when parameters or contentType are not populated

To install the Git Credential Manager, download and double-click the GCMW-1.18.1.exe installer. It is that easy, it will even install Git for Windows 2.17.0 and the Microsoft .Net Framework for you if necessary.

Hopefully that one will integrated in a minor update to git for Windows soon: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

moderately soon, hopefully, there will be a minor version release of Git for Windows that will include the updated GCM v1.18.1. [just need more volunteer time for the PR and tests ;-)
That should eliminate the `Fatal: NullReferenceException” but appears not to fix the¬†#1874https://github.com/git-for-windows/git/issues/1874¬†delays. (still to hear from the OP if the issue is definitely the AV, or something elese)

If you really dare, you can try the nightly snapshot which already includes GCMW-1.18.1: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

the next official Git for Windows version will have a new Git Credential Manager version. Also, the current snapshot at https://wingit.blob.core.windows.net/files/index.html should have it already.

–jeroen

Posted in Development, DVCS - Distributed Version Control, git, Software Development, Source Code Management | Leave a Comment »

 
%d bloggers like this: