The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,351 other followers

Archive for the ‘Infrastructure’ Category

Running ArchiveTeam Warrior version 3.2 on ESXi

Posted by jpluimers on 2021/05/05

A while ago I wrote about Helping the WayBack ArchiveTeam team: running their Warrior virtual appliance on ESXi.

Since it was scheduled before my cancer treatment started and got posted when still recovering from it, I missed that version 3.2 of the [Wayback] ArchiveTeam Warrior appliance appeared in the [Wayback] Releases · ArchiveTeam/Ubuntu-Warrior at [Wayback] Release v3.2 · ArchiveTeam/Ubuntu-Warrior. You can download it form these places:

These two sites have not yet been updated, so they contain the older versions:

The source code now has been moved three times:

  1. [Wayback] ArchiveTeam/warrior-code
  2. [Wayback] ArchiveTeam/warrior-code2 · GitHub
  3. [Wayback] ArchiveTeam/Ubuntu-Warrior at master (this is version 3 and up)

The docker container

The new version of Archive Team Warrior now is basically a shell around [Wayback] Watchtower and the [Wayback] ArchiveTeam/warrior-dockerfile: A Dockerfile for the ArchiveTeam Warrior docker container. This makes updating the core way easier.

More on the docker container (in case you want to run it yourself) is at [Wayback] ArchiveTeam Warrior – Archiveteam – Installing and running with Docker:

You’ll need Docker (open source) and the Warrior Docker image.

  1. Download Docker from the link above and install it.
  2. Open your terminal. On Windows, you can use either Command Prompt (CMD) or PowerShell. On macOS and Linux you can use Terminal (Bash).
  3. Use the following command to start the Warrior as well as Watchtower, which will automatically keep your Warrior updated:
    docker run --detach --name watchtower --restart=on-failure --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable --cleanup --interval 3600 && docker run --detach --name archiveteam-warrior --label=com.centurylinklabs.watchtower.enable=true --restart=on-failure --publish 8001:8001 atdr.meo.ws/archiveteam/warrior-dockerfile

    (For a full explanation of this command, see items 3 and 4 here.)

  4. Using your regular web browser, visit http://localhost:8001/.

The virtual appliance

The virtual appliance is released as virtual appliance aimed by default at VirtualBox and steps to run with VMware: [Wayback] ArchiveTeam Warrior – Archiveteam.

Totally agreeing with Kristian Kohntopp, I do not understand why people use Virtualbox at all: I just run in too much issues like [Archive.is] Kristian Köhntopp on Twitter: “Hint: Wenn die Installation einer Linux-Distro in Virtualbox mit wechselnden, unbekannten Fehlern scheitert, hilft es, stattdessen einmal VMware Workstation oder kvm zu probieren. In meinem Fall hat es dann jedes einzelne Mal mit demselben Iso geklappt.”.

Inspecting the .ova file, which is basically a tar compressed file consisting of an OVF directory as per Open Virtualization Format:Design – Wikipedia

The entire directory can be distributed as an Open Virtual Appliance (OVA) package, which is a tar archive file with the OVF directory inside.

Inspecting the disk image inside the directory learned me that pure one-file binary VMDK disk images start with a KMDV signature in big-endian and KDMV in little-endian (first four bytes are 4b 44 4d 56). More on the VMDK file format can be found in these links (all via [Wayback] vmdk file format specification – Google Search):

So here are some steps to get the .ova image to run on ESXi. I think it should work for ESXI 5.1 and up, but I have tested only on ESXi 6.7:

Read the rest of this entry »

Posted in *nix, *nix-tools, Cloud, Containers, diff, Docker, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Infrastructure, Internet, InternetArchive, Kubernetes (k8n), patch, Power User, VirtualBox, Virtualization, VMware, VMware ESXi, VMware Workstation, WayBack machine | Leave a Comment »

Dockerfile with Bite Size Networking tools from b0rk

Posted by jpluimers on 2021/04/16

[WayBack] Ryan James Spencer on Twitter:

“I wrote a quick Dockerfile so people who purchase Bite Size Networking from  @b0rk can quickly have access to the tools. You can slim the image down to do debugging on docker networks once you get comfortable with which tools are most useful to you.”

He is planning to do more, so maybe a few of [WayBack] wizard zines get encapsulated into dockerfiles as well.

–jeroen

PS: [WayBack] Dockerfile | Docker Documentation

# N.B. The only tool missing here that is mentioned in the document is `zenmap`
# purely because this image is intended to be run via a CLI and `zenmap` is a GUI
# to `nmap` i.e. one can play around with the tools by running:
#
# $ docker build –name bite_size_networking:latest .
# $ docker run –rm -d –name bsn_test bite_size_networking:latest
# $ docker exec -it bsn_test bash
#
# Alternatively, one can change the `ENTRYPOINT` to `["bash"]` and run:
#
# $ docker run -it –name bsn_test bite_size_networking:latest
#
# then later (after exiting the shell):
#
# $ docker start bsn_test
# $ docker attach bsn_test
#
# One can also run this image on a docker network to capture packets and so
# forth for debugging purposes. Once you've found the tooling that best suits
# your needs, it may make sense to make a slimmed down version of this
# Dockerfile and, if wireguard isn't needed, base this image off
# `debian:stable` instead.
#
# Lastly, you can purchase Bite Size Networking or Julia's other fantastic
# zines over at https://wizardzines.com/
# We use `unstable` here since we install `wireguard` below
FROM debian:unstable
RUN apt update && \
DEBIAN_FRONTEND=noninteractive apt install -y \
dnsutils \
curl \
nmap \
tcpdump \
ngrep \
mitmproxy \
iptables \
ethtool \
stunnel \
hping3 \
lsof \
ipcalc \
p0f \
iperf \
apache2-utils \
wget \
python3 \
iftop \
nethogs \
iptraf \
httpie \
nload \
aria2 \
nftables \
tcpflow \
telnet \
openvpn \
links \
wireguard \
tshark
ENTRYPOINT ["sh", "-c", "while true; do sleep $(( 60 * 60 * 24 )); done"]

Posted in *nix, *nix-tools, Cloud, Containers, Docker, Infrastructure, Power User | Leave a Comment »

Helping the WayBack ArchiveTeam team: running their Warrior virtual appliance on ESXi

Posted by jpluimers on 2021/03/19

The [WayBack] Archiveteam helps the WayBack machine with feeding new content.

You can help that team by running one or more “warrior” virtual machine instances. The VM is distributed as a virtual appliance in an ova file according to the Open Virtualization Format.

That format sounds more generic than it actually is, so the (at the time of writing) archiveteam-warrior-v3-20171013.ova file at [WayBack] Index of /downloads/warrior3/ was created for VirtualBox.X

This meant running it on VMware ESXi or VMware vSphere takes a few steps for patching it, then uploading it to your VMware host.

Since I might want to run the appliance on multiple places or multiple instances, I wanted to have a ready-to-go solution, I created a git repository with both the patch instructions and the update at [WayBack] wiert.me / public / ova / archiveteam-warrior-v3-20171013.ESXi · GitLab.

Read the rest of this entry »

Posted in Cloud, Containers, Docker, Infrastructure, Internet, InternetArchive, Kubernetes (k8n), Power User, WayBack machine | Leave a Comment »

The user and the not so standard OSI 7 layer stack of compromises: xkcd Stack

Posted by jpluimers on 2021/02/26

Doing full stack development and operations requires one to think about compromises too. Randall Munroe did just that and created an OSI 7 layer like [WayBack] xkcd: Stack episode 2166:

Title text: Gotta feel kind of bad for nation-state hackers who spend years implanting and cultivating some hardware exploit, only to discover the entire target database is already exposed to anyone with a web browser.

And of course there is also a [WayBack] ExplainXKCD 2166

–jeroen

Posted in Development, DevOps, Software Development | Leave a Comment »

How not to do updates of your wiki site

Posted by jpluimers on 2021/02/17

If your company manages your own infrastructure, be sure you have monitoring on all levels.

It saves you from customers discovering issues like this: [WayBack] Thread by @jpluimers: “The @EmbarcaderoTech docwiki is down due to an error in duobook2.[…]”:

The @EmbarcaderoTech docwiki is down due to an error in duobook2. URLs pointing to wiki content fail, no matter the product. Examples for Rio and XE2 grabbed from docwiki.embarcadero.com/Libraries/Rio/… and docwiki.embarcadero.com/Libraries/XE2/…


This is the #1 reason for allowing archival of all your product documentation web-content in the @internetarchive, even for non-current products, as now only parts that have been allowed to save in the past are available.

Apart from nobody noticing the outage yet, which is bad in it’s own way, I hope the cause is not somebody fiddling with duobook (3 year old and unmaintained) without testing the consequences. As that would make the cause of the outage embarrassing.

[WayBack] https://github.com/ElectricVersion/DuoBook

Finally it is rather odd to get a HTTP 200 SUCCESS code on a failure. A HTTP 500 or 503 would be far more appropriate.

I wonder if that is a @mediawiki thing; maybe they could shed some light on that.

References en.wikipedia.org/wiki/List_of_H… and en.wikipedia.org/wiki/List_of_H….

The cool thing is that the stack traces teach you a lot about how a framework is structured.

Related:

  • [Archive.is/WayBack] RAD Studio API Documentation: Rio
    Exception encountered, of type "ArgumentCountError"
    [6a5b64d3a502a9acff148fe1] /Libraries/Rio/en/Main_Page ArgumentCountError from line 420 of /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php: Too few arguments to function DuoBook2Template::displayPrefs(), 0 passed in /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php on line 99 and exactly 1 expected
    Backtrace:
    #0 /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php(99): DuoBook2Template->displayPrefs()
    #1 /var/www/html/shared/BaseWiki27/includes/skins/SkinTemplate.php(248): DuoBook2Template->execute()
    #2 /var/www/html/shared/BaseWiki27/includes/OutputPage.php(2335): SkinTemplate->outputPage()
    #3 /var/www/html/shared/BaseWiki27/includes/MediaWiki.php(743): OutputPage->output()
    #4 /var/www/html/shared/BaseWiki27/includes/MediaWiki.php(509): MediaWiki->main()
    #5 /var/www/html/shared/BaseWiki27/index.php(43): MediaWiki->run()
    #6 {main}
  • [Archive.is/WayBack] XE2 API Documentation
    Exception encountered, of type "ArgumentCountError"
    [d3d353581c3915881b976ab6] /Libraries/XE2/en/Main_Page ArgumentCountError from line 420 of /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php: Too few arguments to function DuoBook2Template::displayPrefs(), 0 passed in /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php on line 99 and exactly 1 expected
    Backtrace:
    #0 /var/www/html/shared/BaseWiki27/skins/DuoBook2/DuoBook2.php(99): DuoBook2Template->displayPrefs()
    #1 /var/www/html/shared/BaseWiki27/includes/skins/SkinTemplate.php(248): DuoBook2Template->execute()
    #2 /var/www/html/shared/BaseWiki27/includes/OutputPage.php(2335): SkinTemplate->outputPage()
    #3 /var/www/html/shared/BaseWiki27/includes/MediaWiki.php(743): OutputPage->output()
    #4 /var/www/html/shared/BaseWiki27/includes/MediaWiki.php(509): MediaWiki->main()
    #5 /var/www/html/shared/BaseWiki27/index.php(43): MediaWiki->run()
    #6 {main}

–jeroen

Read the rest of this entry »

Posted in Development, DevOps, Infrastructure, Power User, Software Development, Web Development | Leave a Comment »

 
%d bloggers like this: