The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,947 other followers

Archive for the ‘Communications Development’ Category

« Previous Entries

OpenSSH scp has defaulted to the SFTP protocol for some 9 months now

Posted by jpluimers on 2022/05/25

Since I will be bitten by this someday, here the september 2021 observation that [Wayback] By default, scp(1) now uses SFTP protocol.

The original scp/rcp protocol remains available via the -O flag.

It refers to the august 2021 announcement in the [Wayback] OpenSSH: Release Notes – OpenSSH 8.7/8.7p1 (2021-08-20) that scp supported SFTP (like the sftp tool) and that it would become default soon:

Read the rest of this entry »

Posted in *nix, *nix-tools, Communications Development, Development, Internet protocol suite, OpenSSH, Power User, rsync, scp, SFTP, SSH, TCP | Leave a Comment »

Filezilla SFTP: figuring out the cause of “Connection timed out after 20 seconds of inactivity” part 2: about ((non-)?(interactive|login) ?){2} bash shells

Posted by jpluimers on 2022/05/12

Last year, I wrote about Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity” about sftp connection problems.

The solution there was to exclude part of bashrc with an if [Wayback] statement so bash would skip it during sftp, but not during ssh login:

[WayBack] linux – Use .bashrc without breaking sftp – Server Fault

  • From answer 1 (thanks [WayBack] Mike):

    Try doing this instead

    if [ "$SSH_TTY" ]
then
   source .bashc_real
fi
  • From Answer 2 (thanks [WayBack] Insyte):

    A good trick for testing the cleanliness of your login environment is to ssh in with a command, which simulates the same way scp/sftp connect. For example: ssh myhost /bin/true will show you exactly what scp/sftp sees when they connect.

That caused some scripts not to be run when switching user, for instance by doing sudo su -.

The reason for that was that I forgot to put enough research in part of Answer 2, so I quote a few bits more of it (highlights and code markup mine):

… it’s worth pointing out that you can accomplish this carefully selecting which startup files to put the verbose stuff in. From the bash man page:

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

The sftp/scp tools start an interactive non-login shell, so .bashrc will be sourced.

For further reading, there is the underlying bash manual as a PDF file [Wayback] and html document tree [Wayback]. Note it is large (the PDF is 190 pages).

I find the easiest way to navigate around bash documentation through these links:

Types of shell invocations: to login or to non-login and to interactive or to non-interactive

Basically, from the above answer there are [Archive.is] 4 types of shells (confirmed by these parts of the bash documentation: [Wayback] Section 6.1: Invoking-Bash and [Wayback] Section 6.2: Bash-Startup-Files):

  1. interactive login
  2. interactive non-login
  3. non-interactive login
  4. non-interactive non-login

And there are various means the shells can start (ssh, local console, …). The "$SSH_TTY" trick only checks interactive login via ssh, but fails to detect others.

So I did some digging for the correct information to log, which including the above are:

  • [Wayback] Section 4.3.1: The-Set-Builtin
    • -h Locate and remember (hash) commands as they are looked up for execution. This option is enabled by default.
    • -m Job control is enabled (see Job Control). All processes run in a separate process group. When a background job completes, the shell prints a line containing its exit status.
    • -B The shell will perform brace expansion (see Brace Expansion). This option is on by default.
    • -H Enable ‘!’ style history substitution (see History Interaction). This option is on by default for interactive shells.

    Note that in addition to this, there is the non-settable option i: The current shell is interactive (see the -i in section 6.1 below).

  • [Wayback] Section 4.3.2: The-Shopt-Builtin
    • login_shell The shell sets this option if it is started as a login shell (see Invoking Bash). The value may not be changed.
  • [Wayback] Section 6.1: Invoking-Bash

    There are several single-character options that may be supplied at invocation which are not available with the set builtin.

    • -i Force the shell to run interactively. Interactive shells are described in Interactive Shells.

    …

    A login shell is one whose first character of argument zero is ‘-’, or one invoked with the –login option.

  • [Wayback] Section 6.2: Bash-Startup-Files explains about these shell invocation types:
    • interactive login shell
    • interactive non-login shell
    • non-interactive shell
  • [Wayback] Section 6.3.1: Is-this-Shell-Interactive

    To determine within a startup script whether or not Bash is running interactively, test the value of the ‘-’ special parameter. It contains i when the shell is interactive. For example:

    case "$-" in
*i*)    echo This shell is interactive ;;
*)  echo This shell is not interactive ;;
esac

    Alternatively, startup scripts may examine the variable PS1; it is unset in non-interactive shells, and set in interactive shells. Thus:

    if [ -z "$PS1" ]; then
        echo This shell is not interactive
else
        echo This shell is interactive
fi

From theory to practice

After reading the above documentation links, I put the below code in the global .bashrc (which of course caused trouble with sftp, so I commented it out later):

echo "Option flags: '$-'"
echo "PS1: '$PS1'"
echo "shopt login_shell: '$(shopt login_shell)'"
echo "Parameter zero: '$0'"
[ "$SSH_TTY" ] ; echo "[ \"\$SSH_TTY\" ] outcome: $?"

And the output after these commands:

  1. ssh user@host
    Option flags: 'himBH'
PS1: '\u@\h:\w> '
shopt login_shell: 'login_shell     on'
Parameter zero: '-bash'
[ "$SSH_TTY" ] outcome: 0

    Verdict: interactive, login

  2. ssh user@host

    followed by

    sudo su -
    Option flags: 'himBH'
PS1: '\[\]\h:\w #\[\] '
shopt login_shell: 'login_shell     on'
Parameter zero: '-bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: interactive, login

  3. ssh user@host

    followed by

    bash
    Option flags: 'himBH'
PS1: '\u@\h:\w> '
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 0

    Verdict: interactive, non-login

  4. ssh user@host

    followed by

    sudo su -

    then by

    bash
    Option flags: 'himBH'
PS1: '\[\]\h:\w #\[\] '
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: interactive, non-login

  5. ssh user@host /bin/true
    Option flags: 'hBc'
PS1: ''
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: non-interactive, non-login

The final one is what for instance sftp will see. It excludes the non-interactive mark in the shopt option flags.

Modifications to my .bashrc file

Since the [Wayback] test for "$SSH_TTY" is inconsistent with the login being interactive, I modified the .bashrc section

if [ "$SSH_TTY" ]
then
   source .bashc_real
fi

to become

if [[ $- =~ i ]]
then
   # only during interactive login shells
   source .bashc_real
fi

I know the [[...]] over test shorthand [...] is a bashism, see [Wayback] if statement – Is double square brackets [[ ]] preferable over single square brackets [ ] in Bash? – Stack Overflow for why I like it.

More relevant documentation

I based the above changes not only on the mentioned StackOverflow post, but also doing some more Googling revealing these useful documentation and question/answer links:

  • [Wayback] Section 3.2.5.2: Conditional Constructs; [[ expression ]]

    [[…]]

    [[ expression ]]

    Return a status of 0 or 1 depending on the evaluation of the conditional expression expression. Expressions are composed of the primaries described below in Bash Conditional Expressions. Word splitting and filename expansion are not performed on the words between the [[ and ]]; tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, process substitution, and quote removal are performed. Conditional operators such as ‘-f’ must be unquoted to be recognized as primaries.

    …

    An additional binary operator, ‘=~’, is available, with the same precedence as ‘==’ and ‘!=’. When it is used, the string to the right of the operator is considered a POSIX extended regular expression and matched accordingly (using the POSIX regcomp and regexec interfaces usually described in regex(3)). The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is 2.

  • [Wayback] Section 4.1: Bourne Shell Builtins; test or [...] (Bash Reference Manual)
    test expr

    Evaluate a conditional expression expr and return a status of 0 (true) or 1 (false). Each operator and operand must be a separate argument. Expressions are composed of the primaries described below in Bash Conditional Expressions. test does not accept any options, nor does it accept and ignore an argument of -- as signifying the end of options.

    When the [ form is used, the last argument to the command must be a ].

  • [Wayback] bash – What are the special dollar sign shell variables? – Stack Overflow (thanks [Wayback] kojiro!):
    • $1, $2, $3, … are the positional parameters.
    • "$@" is an array-like construct of all positional parameters, {$1, $2, $3 ...}.
    • "$*" is the IFS expansion of all positional parameters, $1 $2 $3 ....
    • $# is the number of positional parameters.
    • $- current options set for the shell.
    • $$ pid of the current shell (not subshell).
    • $_ most recent parameter (or the abs path of the command to start the current shell immediately after startup).
    • $IFS is the (input) field separator.
    • $? is the most recent foreground pipeline exit status.
    • $! is the PID of the most recent background command.
    • $0 is the name of the shell or shell script.

    Most of the above can be found under Special Parameters in the Bash Reference Manual. There are all the environment variables set by the shell.

    For a comprehensive index, please see the Reference Manual Variable Index.

  • [Wayback] bash – Differentiate Interactive login and non-interactive non-login shell – Ask Ubuntu (thanks [Wayback] terdon)

    Briefly (see here for more details), with examples:

    • interactive login shell: You log into a remote computer via, for example ssh. Alternatively, you drop to a tty on your local machine (Ctrl+Alt+F1) and log in there.
    • interactive non-login shell: Open a new terminal.
    • non-interactive non-login shell: Run a script. All scripts run in their own subshell and this shell is not interactive. It only opens to execute the script and closes immediately once the script is finished.
    • non-interactive login shell: This is extremely rare, and you’re unlikey to encounter it. One way of launching one is echo command | ssh server. When ssh is launched without a command (so ssh instead of ssh command which will run command on the remote shell) it starts a login shell. If the stdin of the ssh is not a tty, it starts a non-interactive shell. This is why echo command | ssh server will launch a non-interactive login shell. You can also start one with bash -l -c command.

    If you want to play around with this, you can test for the various types of shell as follows:

    • Is this shell interactive?Check the contents of the $- variable. For interactive shells, it will include i: 
      ## Normal shell, just running a command in a terminal: interacive
$ echo $-
himBHs
## Non interactive shell
$ bash -c 'echo $-'
hBc
    • Is this a login shell?There is no portable way of checking this but, for bash, you can check if the login_shell option is set: 
      ## Normal shell, just running a command in a terminal: interacive
$ shopt login_shell 
login_shell     off
## Login shell; 
$ ssh localhost
$ shopt login_shell 
login_shell     on

    Putting all this together, here’s one of each possible type of shell:

    ## Interactive, non-login shell. Regular terminal
$ echo $-; shopt login_shell
himBHs
login_shell     off

## Interactive login shell
$ bash -l
$ echo $-; shopt login_shell
himBHs
login_shell     on

## Non-interactive, non-login shell
$ bash -c 'echo $-; shopt login_shell'
hBc
login_shell     off

## Non-interactive login shell
$ echo 'echo $-; shopt login_shell' | ssh localhost
Pseudo-terminal will not be allocated because stdin is not a terminal.
hBs
login_shell     on
  • [Wayback] Difference between Login Shell and Non-Login Shell? – Unix & Linux Stack Exchange

    A login shell is the first process that executes under your user ID when you log in for an interactive session. The login process tells the shell to behave as a login shell with a convention: passing argument 0, which is normally the name of the shell executable, with a - character prepended (e.g. -bash whereas it would normally be bash. Login shells typically read a file that does things like setting environment variables: /etc/profile and ~/.profile for the traditional Bourne shell, ~/.bash_profile additionally for bash†, /etc/zprofile and ~/.zprofile for zsh†, /etc/csh.login and ~/.login for csh, etc.

    When you log in on a text console, or through SSH, or with su -, you get an interactive login shell. When you log in in graphical mode (on an X display manager), you don’t get a login shell, instead you get a session manager or a window manager.

    It’s rare to run a non-interactive login shell, but some X settings do that when you log in with a display manager, so as to arrange to read the profile files. Other settings (this depends on the distribution and on the display manager) read /etc/profile and ~/.profile explicitly, or don’t read them. Another way to get a non-interactive login shell is to log in remotely with a command passed through standard input which is not a terminal, e.g. ssh example.com <my-script-which-is-stored-locally (as opposed to ssh example.com my-script-which-is-on-the-remote-machine, which runs a non-interactive, non-login shell).

    When you start a shell in a terminal in an existing session (screen, X terminal, Emacs terminal buffer, a shell inside another, etc.), you get an interactive, non-login shell. That shell might read a shell configuration file (~/.bashrc for bash invoked as bash, /etc/zshrc and ~/.zshrc for zsh, /etc/csh.cshrc and ~/.cshrc for csh, the file indicated by the ENV variable for POSIX/XSI-compliant shells such as dash, ksh, and bash when invoked as sh, $ENV if set and ~/.mkshrc for mksh, etc.).

    When a shell runs a script or a command passed on its command line, it’s a non-interactive, non-login shell. Such shells run all the time: it’s very common that when a program calls another program, it really runs a tiny script in a shell to invoke that other program. Some shells read a startup file in this case (bash runs the file indicated by the BASH_ENV variable, zsh runs /etc/zshenv and ~/.zshenv), but this is risky: the shell can be invoked in all sorts of contexts, and there’s hardly anything you can do that might not break something.

    † I’m simplifying a little, see the manual for the gory details.

If you want to avoid the [[...]] bashishm, then read [Wayback] Bashism: How to make bash scripts work in dash – Greg’s Wiki.

–jeroen

Posted in *nix, *nix-tools, ash/dash, bash, bash, Communications Development, Conference Topics, Conferences, Development, Event, Internet protocol suite, Power User, Scripting, SFTP, Software Development, SSH, TCP | Leave a Comment »

Setting up a GitLab project so it is served over https as a gitlab.io and a custom subdomain

Posted by jpluimers on 2022/05/05

Last week, I posted about Setting up a GitHub project so it is served over https as a custom github.io subdomain.

Today it’s the equivalent, but on GitLab.

Why GitLab? Two major reasons: unlike GitHub:

  1. it’s open source
  2. provides way more granular control over permissions
  3. allows a hierarchy of repositories on which you can specify that permission control

Already 2. and 3. combined are a huge advantage, though we will see that 3. also makes some of the subcases (hosting as user.gitlab.io from account gitlab.com/user where user is your username) is harder than the similar user.github.io, github.com/user combo.

So here we go, starting with a similar set of links:

The goal is to have

  1. page projects as or under wiert.gitlab.io (like wiert.gitlab.io/wiert)
  2. a gitlabstatus.wiert.me plain html (or maybe markdown) page project that eventually will show some status information (kind of like status.gitlab.com, but for different things).

The beauty of GitLab is that it supports hierarchies of repositories through groups and subgroups, so I already had these subgroups hoping they would cover both the first and second kind of page projects:

Steps I did

Since there are quite a few links above, here are the steps I took from my gitlab.com/wiert account and gitlab.com/wiert.me group.

Steps for wiert.gitlab.io/wiert

  1. For wiert.gitlab.io/wiert, try A (failed in part, and therefore interesting to understand why):
    1. Under leaf group gitlab.com/wiert.me/public/web/sites/gitlab.io, created a new GitLab repository
    2. Chose “Create from template”
    3. Chose the template “Pages/Plain HTML”
    4. Named the project “wiert” (with slug “wiert“) so it would appear at gitlab.com/wiert.me/public/web/sites/gitlab.io/wiert
    5. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    6. Now I got in a confusing situation as the page indicated “There are currently no pipelines.”, but an enabled blue “Run pipeline” button:
      By default there is no CI/CD pipeline, but there is an enabled blue "Run pipeline" button: confusing.

      By default there is no CI/CD pipeline, but there is an enabled blue “Run pipeline” button: confusing.

    7. Clicked the “Run pipeline” button nonetheless, and that created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    8. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job and deployed the page.
    9. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wiert.me.gitlab.io/public/web/sites/gitlab.io/wiert/ and https://wiert.me.gitlab.io/public/web/sites/gitlab.io/wiert/
      Warning: When using Pages under the general domain of a GitLab instance (gitlab.io), you cannot use HTTPS with sub-subdomains.

      Warning: When using Pages under the general domain of a GitLab instance (gitlab.io), you cannot use HTTPS with sub-subdomains.

      The sites do work (see the [Archive.is http version] and [Archive.is https version]), but the HTTPS fails because wiert.me.gitlab.io does not match the SANs (Subject Alternative Names) in the certificate: *.gitlab.io, gitlab.io

  2. For wiert.gitlab.io/wiert, try B (failed, and therefore interesting to understand why):
    1. In my my groups gitlab.com/dashboard/groups, added a new group wiert
    2. Added subgroups until the leaf gitlab.com/wiert/public/web/sites/gitlab.io which as URL is gitlab.com/wier1/public/web/sites/gitlab.io because user account wiert already occupies gitlab.com/wiert.
    3. Under leaf group gitlab.com/wier1/public/web/sites/gitlab.io, created a new GitLab repository
    4. Chose “Create from template”
    5. Chose the template “Pages/Plain HTML”
    6. Named the project “wiert” (with slug “wiert“) so it would appear at gitlab.com/wiert.me/public/web/sites/gitlab.io/wiert
    7. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    8. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    9. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    10. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    11. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wier1.gitlab.io/public/web/sites/gitlab.io/wiert and https://wier1.gitlab.io/public/web/sites/gitlab.io/wiert.
      Bummer: again not the wiert.gitlab.io/wiert I hoped for
      The sites do work (see the [Archive.is http version] and [Archive.is https version]). The HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

    12. If a user wiert exists and occupies gitlab.com/wiert, then a group named wiert cannot occupy gitlab.com/wiert, and therefore a project named wiert within that group won’t be deployed to wiert.gitlab.io/wiert.
      Maybe this can be shortened like “if there is a user wiert, then no group named wiert cannot be used to contain a project named wiert to host as wiert.gitlab.io/wiert“.
      Let’s find out!
  3. For  wiert.gitlab.io/wiert, try C (success, steps 1, 3, 4, 7 and 8 were the key ones):
    1. In my user gitlab.com/wiert, created a new GitLab repository
    2. Chose “Create from template”
    3. Chose the template “Pages/Plain HTML”
    4. Named the project “wiert” (with slug “wiert“) so it would appear at gitlab.com/wiert
    5. The odd but cool thing is that the actual project now ended up at gitlab.com/wiert/wiert:
    6. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    7. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    8. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    9. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    10. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wiert.gitlab.io/wiert/ and https://wiert.gitlab.io/wiert/.
      Success: finally the wiert.gitlab.io/wiert I hoped for:

      Success: published at https://wiert.gitlab.io/wiert/

      Success: published at https://wiert.gitlab.io/wiert/

      The sites do work fine (see the [Archive.is http version] and [Archive.is https version]). The HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

Steps for wiert.gitlab.io

  1. For wiert.gitlab.io, try A (failed, and therefore interesting to understand why):
    1. Under leaf group gitlab.com/wiert.me/public/web/sites/gitlab.io, created a new GitLab repository
    2. Chose “Create from template”
    3. Chose the template “Pages/Plain HTML”
    4. Named the project “wiert.gitlab.io” (with slug “wiert.gitlab.io“) so it would appear at gitlab.com/wiert.me/public/web/sites/gitlab.io/wiert.gitlab.io
    5. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    6. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    7. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    8. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    9. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wiert.me.gitlab.io/public/web/sites/gitlab.io/wiert.gitlab.io and https://wiert.me.gitlab.io/public/web/sites/gitlab.io/wiert.gitlab.io.
      Failure: not the wiert.gitlab.io I hoped for.

      The sites do work (see the [Archive.is http version] and [Archive.is https version]), but the HTTPS fails because wiert.me.gitlab.io does not match the SANs (Subject Alternative Names) in the certificate: *.gitlab.io, gitlab.io. The HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

  2. For wiert.gitlab.io, try B (failed, and therefore interesting to understand why):
    1. Under leaf group gitlab.com/wier1/public/web/sites/gitlab.io, created a new GitLab repository
    2. Chose “Create from template”
    3. Chose the template “Pages/Plain HTML”
    4. Named the project “wiert.gitlab.io” (with slug “wiert.gitlab.io“) so it would appear at gitlab.com/wier1/public/web/sites/gitlab.io/wiert.gitlab.io
    5. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    6. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    7. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    8. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    9. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wier1.gitlab.io/public/web/sites/gitlab.io/wiert.gitlab.io and https://wier1.gitlab.io/public/web/sites/gitlab.io/wiert.
      Bummer: again not the wiert.gitlab.io I hoped for
      The sites do work (see the [Archive.is http version] and [Archive.is https version]). The HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

    10. Try A and B were almost identical to wiert.gitlab.io/wiert try A and B, so let’s see if the solution C for that also works for us:
  3. For wiert.gitlab.io, try C (success, steps 1, 3, 4, 7 and 9 were the key ones)
    1. In my user gitlab.com/wiert, created a new GitLab repository
    2. Chose “Create from template”
    3. Chose the template “Pages/Plain HTML”
    4. Named the project “wiert.gitlab.io” (with slug “wiert.gitlab.io“) so it would appear at gitlab.com/wiert/wiert.gitlab.io.
    5. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    6. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    7. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    8. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    9. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wiert.gitlab.io/ and https://wiert.gitlab.io/.
      Success: finally the wiert.gitlab.io I hoped for with working sites (see the [Archive.is http version] and [Archive.is https version]).
    10. Note the HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

Steps for gitlabstatus.wiert.me

Having learned from the GitHub githubstatus.wiert.me procedure (where I had to wait a long time for the default *.wiert.me domain mapping timeout and the githubstatus.wiert.me DNS CNAME record to become effective), I started on the DNS CNAME record side which is documented at [Wayback] Custom domains and SSL/TLS certificates: Section 3. Set up DNS records for Pages: For subdomains | GitLab:

Subdomains (subdomain.example.com) require:

  • A DNS CNAME record pointing your subdomain to the Pages server.
  • A DNS TXT record to verify your domain’s ownership.
From DNS Record To
subdomain.example.com CNAME namespace.gitlab.io
_gitlab-pages-verification-code.subdomain.example.com TXT gitlab-pages-verification-code=00112233445566778899aabbccddeeff

Note that, whether it’s a user or a project website, the CNAME should point to your Pages domain (namespace.gitlab.io), without any /project-name.

DNS CNAME record pointing to GitLab.com project

The value for the TXT record is only known after you created the pages project, but the value for the CNAME record is known beforehand:

From DNS Record To
gitlabstatus.wiert.me CNAME namespace.gitlab.io

So let’s see if I can do this in one try, with these steps:

  1. For gitlabstatus.wiert.me, try A (success, steps 1, 3, 4, 7 and 9 were the key ones)
    1. In my DNS settings of the wiert.me domain, created a CNAME record from gitlabstatus.wiert.me to namespace.gitlab.io:

      gitlabstatus.wiert.me CNAME record pointing to namespace.gitlab.io

      gitlabstatus.wiert.me CNAME record pointing to namespace.gitlab.io

    2. Under leaf group gitlab.com/wiert.me/public/web/sites/wiert.me, created a new GitLab repository
    3. Chose “Create from template”
    4. Chose the template “Pages/Plain HTML”
    5. Named the project “gitlabstatus.wiert.me” (with slug “gitlabstatus.wiert.me“) so it would appear at gitlab.com/wiert.me/public/web/sites/wiert.me/gitlabstatus.wiert.me
    6. From the left sidebar, navigated to your project’s “CI/CD”, then “Pipelines”
    7. Again there was “There are currently no pipelines.”, but an enabled blue “Run pipeline” button, which I clicked
    8. That created [Wayback/Archive.is] a pipeline asking for parameters (that already had correct default values) and revealed a new blue “Run pipeline” button.
    9. Clicked that new “Run pipeline button” which created [Wayback/Archive.is] a job deployed the page.
    10. From the left sidebar, navigated to “Settings”, then “Pages” to get the links to the pages site: http://wiert.me.gitlab.io/public/web/sites/wiert.me/gitlabstatus.wiert.me and https://wiert.me.gitlab.io/public/web/sites/wiert.me/gitlabstatus.wiert.me.
      Intermediate success: working sites (see the [Archive.is http version] and [Archive.is https version]).
    11. Now it is time to get the DNS CNAME record from gitlabstatus.wiert.me to namespace.gitlab.io into operation by clicking the “New Domain” button:
      "New Domain" button in the "Pages" settings.

      “New Domain” button in the “Pages” settings.

    12. There I filled in the correct gitlabstatus.wiert.me domain name, then pressed the “Create New Domain” button:

      New domain becomes gitlabstatus.wiert.me

      New domain becomes gitlabstatus.wiert.me

    13. Then a page appeared voiding the DNS CNAME work I already did: the documentation is clearly wrong as these are the two DNS record entries to be made as shown by gitlab.com/wiert.me/public/web/sites/wiert.me/gitlabstatus.wiert.me/pages/domains/gitlabstatus.wiert.me:
      Correct instructions for the DNS records to get gitlabstatus.wiert.me working

      Correct instructions for the DNS records to get gitlabstatus.wiert.me working

      Subdomains (gitlabstatus.wiert.me) require:

      • A DNS CNAME record pointing your subdomain to the Pages server.
      • A DNS TXT record to verify your domain’s ownership.
      From DNS Record To
      gitlabstatus.wiert.me CNAME wiert.me.gitlab.io.
      _gitlab-pages-verification-code.gitlabstatus.wiert.me TXT gitlab-pages-verification-code=c5619988d386b1a36c253ce05db55dbb

      Basically the whole namespace.gitlab.io part of the documentation is a placeholder for the actual namespace that belongs to the leaf group the pages project is in (in my case wiert.me).

      So this is the new DNS entry, for which I had to wait until the DNS TTL to time out and effectuate:
      New DNS gitlabstatus.wiert.me CNAME record pointing to wiert.me.gitlab.io

      New DNS gitlabstatus.wiert.me CNAME record pointing to wiert.me.gitlab.io

      Note that this DNS administrative interface from WordPress.com does omit the final period of the CNAME destination (officially this would be wiert.me.gitlab.io.)

    14. After the CNAME DNS record, I also made the TXT DNS record:
      New DNS TXT record for verification of gitlabstatus.wiert.me

      New DNS TXT record for verification of gitlabstatus.wiert.me

      Then I waited a little for the DNS TXT record to be saved and try the verification of the TXT record.

    15. Even then, verification took some time. I had to click the refresh button a few times before verification succeeded:
      The DNS TXT record for gitlabstatus.wiert.me finally got verified

      The DNS TXT record for gitlabstatus.wiert.me finally got verified

    16. Now I could press blue “Save Changes” button below and waited for the CNAME record DNS TTL to expire so I could check the domain and – hopefully – the TLS certificate to be requested by Let’s Encrypt:
      After the gitlabstatus.wiert DNS TXT record got verified, I could save the domain information

      After the gitlabstatus.wiert DNS TXT record got verified, I could save the domain information

    17. After the old CNAME record DNS TTL expired and the new CNAME record came into effect, the domain became available as http://gitlabstatus.wiert.me/:
      Waiting for gitlabstatus.wiert.me to become active

      Waiting for gitlabstatus.wiert.me to become active

    18. After verification, the “Domains (1)” bit changed from this:
      Domain gitlabstatus.wiert.me information before verification

      Domain gitlabstatus.wiert.me information before verification

      to this:

      Domain gitlabstatus.wiert.me information after verification

      Domain gitlabstatus.wiert.me information after verification

    19. In the mean time, also the TLS certificate got issued by Let’s Encrypt, so the final sites now both worked: http://gitlabstatus.wiert.me/ and https://gitlabstatus.wiert.me/.
    20. Success: finally the gitlabstatus.wiert.me I hoped for with working sites (see the [Archive.is http version] and [Archive.is https version] for the wiert.me domain, and [Archive.is http version] and [Archive.is https version] for the wiert.me domain).
    21. Note the HTTP does not redirect to the HTTP version, as I did not tick the

      ☐ Force HTTPS (requires valid certificates)

In retrospect, this could have been shorter when I had done the DNS part later, which is contrary to how to do this with GitHub.

Conclusion

The conclusion seems this:

Gitlab Page repositories to be published as or under wiert.gitlab.io need to reside directly under user wiert. Having them reside under a different group like wiert or wiert.me won’t work.

Or in more generic terms:

When creating pages as user.gitlab.io you have to put your pages projects directly under your user account gitlab.com/user.

Putting them under groups or leaf groups fails, no matter if the (leaf) group is named user or otherwise.

In addition, you can add custom domains to any Gitlab repository (even one that never stated out as a GitLab Pages repository). It will work as soon as the domain DNS mapping is setup through both a CNAME mapping record and TXT verification record.

The steps for this in your GitLab repository are:

  1. Ensure you have a valid .gitlab-ci.yml file at the root of your repository; I used the [Wayback/Archive.is] one from [Wayback/Archive] GitLab Pages examples / plain-html · GitLab as my site is purely static
  2. Ensure you have a valid index.html file in the public directory of your repository, similar to [Wayback/Archive] GitLab Pages examples / plain-html · GitLab
  3. When both 1. and 2. are committed in your repository at GitLab, then it will automatically be deployed to a docker container on gitlab.io, which allows the outside world to visit your GitHub Pages sie, and the Let’s Encrypt Certificate to be generated (and prevents this error: [Wayback/Archive] GitLab Pages integration with Let’s Encrypt | GitLab: “Something went wrong while obtaining the Let’s Encrypt certificate”).
  4. Under “Settings” -> “Pages”, add a new domain name to the repository: now it automatically becomes a GitLab Pages repository.
  5. When adding the domain, the settings page will show both a DNS CNAME record and DNS TXT record; ensure both are applied on your primary DNS name server and replicated to all authoritative DNS name servers.
  6. Save the new page.
  7. Check if the page is available on the new domain you added.
  8. Optionally under “Settings” -> “Pages” enable the “Force HTTPS (requires valid certificates)” option and save.

TLS information

Note: I saved the TLS information – including certificates here:

More about the Let’s Encrypt certificates at [Wayback] Chain of Trust – Let’s Encrypt:

–jeroen

Read the rest of this entry »

Posted in Cloud, Communications Development, Development, DNS, Encryption, GitLab, Hosting, HTML, HTTPS/TLS security, Infrastructure, Internet, Internet protocol suite, Let's Encrypt (letsencrypt/certbot), Power User, Software Development, Source Code Management, TCP, TLS, Web Development | Leave a Comment »

capitaltg/thea: Certificate Checker and https://certchecker.app site

Posted by jpluimers on 2022/05/03

[Wayback/Archive.is] capitaltg/thea: Certificate Checker

Certificate Checker provides an easy-to-use solution to check certificates, certificate chains, and TLS configurations. To run Certificate Checker for publicly-accessible web sites you can go to: https://certchecker.app and enter in there a URL to check.
Users can easily run Certificate Checker in an internal network to validate or troubleshoot their TLS configuration. To run it on a local network you can run the Docker image as described below. You can also build the application and deploy it on an existing server.

It runs on [Wayback/Archive.is] Certificate Checker.

I used it to check various certificates and chains, including those for my GitHub Pages explained last week in Setting up a GitHub project so it is served over https as a custom subdomain.

–jeroen

Posted in Communications Development, Development, Encryption, HTTPS/TLS security, Internet protocol suite, Let's Encrypt (letsencrypt/certbot), Power User, Security, Software Development, TCP, TLS, Web Development | Leave a Comment »

Setting up a GitHub project so it is served over https as a github.io and a custom subdomain

Posted by jpluimers on 2022/04/27

Some links that helped me getting this working:

The goal is to have a githubstatus.wiert.me plain html (or maybe markdown) page that eventually will show some status information (kind of like githubstatus.com, but for different things).

Note that for free accounts, private repositories cannot publish pages: [Wayback] Troubleshooting custom domains and GitHub Pages – GitHub Docs:

GitHub Pages is available in public repositories with GitHub Free and GitHub Free for organizations, and in public and private repositories with GitHub Pro, GitHub Team, GitHub Enterprise Cloud, and GitHub Enterprise Server. For more information, see “GitHub’s products.”

[Wayback] GitHub’s products contradicts this by limiting GitHub Pages to only GitHub Pro and GitHub Team and higher levels.

Steps I did

Since there are quite a few links above, here are the steps I took from my github.com/jpluimers account:

  1. For  jpluimers.github.io/jpluimers (steps 1, 3, 4 and 5 were the key ones):
    1. Created a new GitHub repository github.com/jpluimers/jpluimers with license (in my case MIT license) and README.md (which by default is filled like this).
    2. Amended the README.md from my StackExchange profile.
    3. Enabled Pages publishing by switching the source from None to the main branch.
    4. GitHub automagically published it at jpluimers.github.io/jpluimers.
    5. Verified it was running at jpluimers.github.io/jpluimers (note there is no site at jpluimers.github.io yet – it shows a 404 error as in this archived version, see below why).
    6. Fixed the Twitter profile link so it is easier to find back my surviving rectal cancer story.
    7. Fixed the link to my www.race-checker.com running results as by now that is a Chinese Porn site because the underlying company 4Athletes Sports GmbH went belly up in 2019 (and still owns the race-checker trademark) and  updated the half marathon results as well.
    8. I verified the deployment actions at github.com/jpluimers/jpluimers/deployments/activity_log?environment=github-pages.
  2. For jpluimers.github.io (step 1 and 3 were the key ones):
    1. Created a new GitHub repository github.com/jpluimers/jpluimers.github.io with license (in my case MIT license) and README.md (which by default is filled like this: only one heading 1 line with the name of the repository).
    2. GitHub automagically set the source to the main branch, then published it at jpluimers.github.io.
    3. Verified it was running at jpluimers.github.io.
    4. I verified the deployment action at github.com/jpluimers/jpluimers.github.io/deployments/activity_log?environment=github-pages.
  3. For githubstatus.wiert.me :
    1. Created a new GitHub repository github.com/jpluimers/jpluimers.github.io with license (in my case MIT license) and README.md (which by default is filled like this: only one heading 1 line with the name of the repository).
    2. Enabled Pages publishing by switching the source from None to the main branch.
    3. I verified the jpluimers.github.io/githubstatus.wiert.me page existed (it will be gone soon).
    4. Set the custom domain to githubstatus.wiert.me.
    5. Now GitHub, after the DNS check, complained rightly that “githubstatus.wiert.me is improperly configured”, as it needs to be “set up with a correct CNAME record … We recommend you change this to a CNAME record pointing to jpluimers.github.io.”.
    6. So in the DNS settings panel of my wiert.me domain, I added a DNS record of type CNAME, with name githubstatus.wiert.me pointing to jpluimers.github.io.
      CNAME githubstatus.wiert.me Alias of jpluimers.github.io

      CNAME githubstatus.wiert.me Alias of jpluimers.github.io

      1. In retrospect, I should have reversed steps 6. and 4, as now this was the order of events, with a lot of waiting for the DNS to time-out.

        The DNS timeout was because githubstatus.wiert.me originally pointed via the DNS CNAME entry *.wiert.me to the blog at wiert.me, the timeouts were set by the domain provider (in this case WordPress.com), see the DNS nslookup information for *.wiert.me [Wayback/Archive.is].

        If I had set the DNS CNAME first, then the below list would have been much shorter.

        This was the order of events waiting for the DNS to timeout and the CNAME entry to take effect:

          1. Before entering the githubstatus.wiert.me custom domain “Your site is ready to be published at https://jpluimers.github.io/githubstatus.wiert.me/ “
          2. After entering the githubstatus.wiert.me custom domain:

            Your site is published at http://githubstatus.wiert.me/

            and a “Check Again” button preceded with:

            githubstatus.wiert.me is improperly configured
            Your site’s DNS settings are using a custom subdomain, githubstatus.wiert.me, that’s not set up with a correct CNAME record.
            …
            We recommend you change this to a CNAME record pointing to
            jpluimers.github.io.

            and an “☐ Enforce HTTPS” checkbox followed by:

            Unavailable for your site because your domain is not properly configured to support HTTPS (githubstatus.wiert.me)

          3. After configuring the DNS information, and pressing the “Check Again” button the text briefly shows

            githubstatus.wiert.me DNS check is in progress.
            Please wait for the DNS check to complete.

            and an “☐ Enforce HTTPS” checkbox followed by:

            Unavailable for your site because your domain is not properly configured to support HTTPS (githubstatus.wiert.me)

          4. After a few minutes at the top of the page:

            Domain githubstatus.wiert,me is not eligible for HTTPS at this time.

            followed by the same “Check Again” button preceded with:

            githubstatus.wiert.me is improperly configured
            Your site’s DNS settings are using a custom subdomain, githubstatus.wiert.me, that’s not set up with a correct CNAME record.
            …
            We recommend you change this to a CNAME record pointing to
            jpluimers.github.io.

          5. A few more minutes later:

            Requesting a certificate for githubstatus.wiert.me. It can take up to an hour to propagate.

            followed again by the above “Check Again” button.

          6. More than an hour later:

            Certificate already exists for githubstatus.wriert.me and is usable.

            followed again by the above “Check Again” button.

          7. The next morning, a green checkmark (✓) had appeared behind the githubstatus.wiert.me custom domain and the text following the “☐ Enforce HTTPS” had by:

            HTTPS provides a layer of encryption that prevents others from snooping on or tampering with traffic to your site.

          8. Both these URLs now function correctly (so I can test a page both with and without TLS):

        The above order is typical for DNS timeouts on a distributed computing system like GitHub: some parts of the system are waiting for the DNS time out and therefore list failure, while some other parts already have had the updated DNS CNAME entry and therefore list success

    7. After waiting for the DNS timeout (this was a long wait, I probably should have reversed steps 6. and 4.), verified that https://githubstatus.wiert.me/ was loading fine.
    8. I verified the deployment actions at github.com/jpluimers/githubstatus.wiert.me/deployments/activity_log?environment=github-pages

Note: I saved the TLS information – including certificates here:

–jeroen

Posted in Cloud, Communications Development, Development, Encryption, GitHub, HTML, HTTP, HTTPS/TLS security, Infrastructure, Internet protocol suite, Let's Encrypt (letsencrypt/certbot), Power User, Security, Software Development, Source Code Management, TCP, TLS, Web Development | Leave a Comment »

« Previous Entries
 
%d bloggers like this: