The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘ZFS’ Category

GitHub – oetiker/znapzend: zfs backup with remote capabilities and mbuffer integration.

Posted by jpluimers on 2024/10/14

I will likely need this one day: [Wayback/Archive] GitHub – oetiker/znapzend: zfs backup with remote capabilities and mbuffer integration.

Web-site: [Wayback/Archive] ZnapZend – open source ZFS backup with mbuffer and ssh support

--jeroen

Posted in *nix, *nix-tools, Power User, ZFS | Leave a Comment »

HTGWA: Create a ZFS RAIDZ1 zpool on a Raspberry Pi | Jeff Geerling

Posted by jpluimers on 2023/08/21

For my link archive: [Wayback/Archive] HTGWA: Create a ZFS RAIDZ1 zpool on a Raspberry Pi | Jeff Geerling

–jeroen

Posted in *nix, Power User, ZFS | Leave a Comment »

Backing up ZFS snapshots

Posted by jpluimers on 2018/12/06

An interesting idea at [WayBack] I was getting concerned about a backup, which had exceeded 1GB, when the data was only about 400MB. Once the job finished, I realized: Ahh, ZFS compres… – Dan Langille – Google+:

Here’s that script I use for creating/destroy the snaphots for a particular long dataset name. Then I backup from /mnt

[WayBack] gist.github.com/dlangille/480dbca509562eb03e76c2e1b576c6d2 is in sh, not even bash.

–jeroen

Read the rest of this entry »

Posted in Development, Power User, Scripting, sh, ZFS | Leave a Comment »

Nas4free – keeping an eye on the daily increase of ZFS disk usage

Posted by jpluimers on 2018/02/16

Assuming your nas4free is at https://nas4free, this page will give you the daily increase of ZFS usage:

https://nas4free/disks_zfs_snapshot.php

There you can estimate based on the snapshot history how soon you will run out of ZFS storage.

This is especially important when you have a retention configured using ZFS [WayBack] auto snapshots. Under the same assumption, those are configured at  https://nas4free/disks_zfs_snapshot_auto.php in order to support “Previous Versions” on CIFS shares using those ZFS volumes.

Note nas4free is way harder to configure in this respect than freenas, see for instance this article: [WayBack] The Ars NAS distribution shootout: FreeNAS vs NAS4Free | Ars Technica. Luckily there is this small guide to get you going [WayBack] Jason’s Notes » Blog Archive » Windows previous versions for ZFS backed Samba shares

In the logs at https://nas4free/diag_log.php you can view the snapshot management: when they are created and removed. When you have trouble with the snapshots, monitor your Nas4free version at https://nas4free/system_firmware.php.. For me, NAS4Free 10.3.* seem very stable.

If you for instance have a 5 gigabyte nightly backup written to such a snapshot volume and the volume has 500 gigabytes of free space, a retention of 100 days it will fill up after 100 days.

–jeroen

PS: background reading:

 

 

Posted in *nix, nas4free, Power User, ZFS | Leave a Comment »

Some real world ZFS on Linux experience by GridPP

Posted by jpluimers on 2016/04/22

GridPP – providing computing and storage facilities for grid computing in the UK – has published 3 nice articles on their use of ZFS on Linux and comparison against hardware RAID:

  1. GridPP storage news: ZFS vs Hardware Raid
  2. GridPP storage news: ZFS vs Hardware Raid System, Part II
  3. GridPP storage news: ZFS vs Hardware Raid System, Part III

Thanks Marcus Ebert for sharing these.

–jeroen

Posted in *nix, Power User, ZFS | Leave a Comment »

My ZFS question on G+: investigation for using a XW6600 based system with ZFS.

Posted by jpluimers on 2014/12/20

My ZFS question on G+:

Hi everyone. I’m a geek. Learned most of the stuff by doing, and keeping tracks of what I did on my tech-blog http://wiert.me

I want to start with ZFS on a pair of HP XW6600 machines having 32gigabyte of RAM.
Any help on that is much appreciated.

The idea is to have one of these here in a closet and the other remotely, and perform replication between them (I’ve a 50megabit fiber-to-the-home uplink which can grow to 100megabit plus, internally my network is gigabit).

My current data is on a Windows 2003 x64 server with dual RAID5 configurations that are synced every night (not optimal for various reasons) with about 12gigabyte of files having mostly read-only access patterns and these kinds of sizes:
– small files between 4kilobyte and a few megabytes
– photos between 5 and 20 megabytes
– ISO backups and 7zip archives of projects (operating system installers, etc) between 100megabytes-6gigabytes
plus an ESXi machine having about 4gigabyte of data (mostly sizes between 20 and 200 gigabyte).

New storage should initially be at least 16gigabyte with room for growth.

I’m having active experience with OpenSuSE, ESXi and Windows. Solaris experience is from a long time ago. Learning by doing is my way of quickly gaining knowledge.

My schedule is doing research until the end of January (partially overlapping with a holiday) then building and testing until the end of Q1, going live early Q2.

Current plan is to put a lot of Samsung M9T 2terabyte SATA drives (they are only 9.5millimeter high) into the XW6600 rigs.
Earlier this year I did some research on controllers and hard drives, and I wonder how much of it is still current: https://wiert.me/2014/03/12/lots-of-2-5-3-5-and-5-25-conversion-brackets-and-hot-swap-bays/
(A quick calculation shows I should be able to get at least 14 externally accessible M9T drives into this machine, plus room for internal SSDs, etc).

So: where should I get started?

Initial questions I have:
– how about rebuild time when drives are lost? (how does the process of cold/hot spares work, can this be automated, how fast is it?)
– I’m not happy about the RAID5 rebuild times, so are 2TB drives indeed the sizes to go for?
– how about configuring things like ZFS equivalents of stripe size, buffer sizes, etc?
– what SATA controllers to use (is mainboard OK, what in addition to the mainboard SATA?)
– how can ZFS be used as an iSCSI target? how well does that work? (That would be really nice to connect to it from ESXi, Windows and many Linuxes/Linii)
– what about compression and block-deduplication?
– what about ZIP and L2ARC? how to estimate their size?
– which ZFS implementation to use? ZoL? OpenSolaris? Nexenta? Others?
– can a ZFS volume grow by adding extra drives?
(14 drives would get ~20terrabyte based on Z-3: http://www.servethehome.com/raid-calculator/ or http://wintelguy.com/raidcalc.pl but I want to have room for growth)

Lots of 2.5, 3.5 and 5.25 conversion brackets, cages and hot-swap bays

–jeroen

via: Hi everyone. I’m a geek. Learned most of the stuff by doing, and keeping tracks….

Posted in *nix, Hardware, HP XW6600, Linux, openSuSE, Power User, SuSE Linux, ZFS | 2 Comments »

Best Buy Guides (BBGs) – mux’ blog – Tweakblogs – Tweakers

Posted by jpluimers on 2014/03/23

Interesting as it talks about both ZFS and ESXi servers with ECC memory: Best Buy Guides (BBGs) – mux’ blog – Tweakblogs – Tweakers.

Posted in *nix, ESXi5, ESXi5.1, ESXi5.5, Power User, VMware, VMware ESXi, ZFS | Leave a Comment »

Some ZFS links

Posted by jpluimers on 2013/09/23

I want to move my storage over from dual RAID5 to ZFS, so here are a few links for my research list:

SSD and supercapacitors

Best practices

Not so good practices

HBAs

Disks

Synchronous writes/ZIL

ZFS streams:

Replication

Reliability/Integrity

Howtos

Trivia

Recovery lessons

–jeroen

Posted in *nix, Power User, ZFS | Leave a Comment »