I was in a pinch.

I needed to make a full disk backup of a PC, but I had no external storage device with me to store it on. The local Wi-Fi network was also way too slow to transfer the disk over it.

All I had was my laptop with an Ethernet port, a Fedora Linux USB stick, and a short Ethernet cable.

I took the following steps:

• boot the target machine up with the Fedora Linux installer in a live environment
• modify the SSH configuration on the target machine to allow root user login with a password
  • it’s OK to do this on a temporary setup like this one, but don’t do it on an actual Linux server
• set a password for the root user on the target machine
  • only required because a live environment usually does not set one for root user
• connect both laptops with the Ethernet cable
• set static IPv4 addresses on both machines using network settings ¹
  • edit the “Wired” connection and open the IPv4 tab
  • example IP address on target: 192.168.100.5
  • example IP address on my laptop: 192.168.100.1
  • make sure to set the netmask to 255.255.255.0 on both!
• verify that the SSH connection works to the target machine
• back up the disk to your local machine using ssh and dd
  • example: ssh root@192.168.100.5 "dd if=/dev/sda" | dd of=disk-image-backup.iso status=progress
  • replace /dev/sda with the correct drive name!

And just like that, I backed up the 120 GB SSD at gigabit speeds to my laptop.

I’ve used a similar approach in the past when switching between laptops by running a live environment on both machines and copying the disk over with dd bit by bit.

You’ll also save time on not having to copy the data over twice, first to an external storage device, and then to the target device.

Just like most people out there, I have some files that are irreplaceable, such as cat pictures.

At one point I had a few single-board computers sitting idle, namely the Orange Pi Zero and the LattePanda V1, and a few 1TB SSD-s.

I hate idle hardware, so I did the most sensible thing and assembled a fleet of networked offsite backups for backing up the most important data.

My setup is based on various flavors of Linux, but the ideas will likely translate well onto other operating systems and solutions.

Networking

The most important part is the networking. The offsite backup endpoints connect together to my home server over a WireGuard network. The home server is, well, the server, and backup endpoints are clients.

I like this WireGuard Docker image a lot because it generates the server and client configurations automatically, but you can use plain WireGuard or a completely different networking solution to connect all the devices together. Some use Tailscale to make the setup process easier, but I like to keep things as self-hosted as possible.

I’m not a networking expert, but here’s how I’ve set up my network. For this example, the WireGuard network operates in the 10.13.69.0/24 range.¹

To only allow traffic between the devices and avoid tunneling everything through the home server, set the AllowedIPs setting to AllowedIPs = 10.13.69.0/24,10.13.69.1. We want to be able to access the backup endpoints, and nothing more.

All the devices have a static IP address in that network, such as 10.13.69.1 for the home server, 10.13.69.2 for a backup endpoint and so on.

The PersistentKeepalive = 25 option is present in the client configurations so that I don’t lose the ability to access the backup endpoints. With it, all the backup endpoints call back to the home server from time to time. The aforementioned Docker image automatically adds it to the generated configuration using the PERSISTENTKEEPALIVE_PEERS=all option. This setting is crucial. Without it, I sometimes ran into problems trying to connect from my home server to the backup endpoint, and that’s something you can’t easily alleviate without having physical access to the backup endpoints, which are offsite.

Remove the DNS configuration from generated WireGuard client configurations, as you don’t need it for this purpose.

Optionally, edit the /etc/hosts file for the home server and backup endpoints so that you can access your backup endpoints using simple hostnames, like orangepizero. Example row can look like this: 10.13.69.6 orangepizero.

if your WireGuard server operates in a network with a dynamic external IP address, as is common with many home internet connections, I recommend getting yourself a domain name that you can update whenever your IP address changes and using that in your WireGuard client configurations. Without this, an IP address change will result in your backup endpoints being inaccessible.

You’ll also likely need to set up port forwarding and/or traffic rules for your backup endpoints to be able to connect back to your WireGuard server.

Once you have the WireGuard connection set up and SSH running on the backup endpoints, you should be able to drop the backup endpoints into any network that you have permission for. Ask your friends and family, and sweeten the deal by offering free technical support or help in some other area in return. The cost of running a single-board computer 24/7 is minuscule with the typical power consumption being 1-3W, so that won’t be much of a concern.

Making backups

For making the actual backups themselves, you have all sorts of options.

I rely on rsync to copy the data over. It’s simple and it works, that’s all I expect from it. Example command: rsync -aAXvz /folder/to/back/up/ backupuser@backupendpoint:/backup/ --delete. The files will be compressed during transit with the -z option, and with --delete you’ll ensure that the target folder has all the files from the source, and nothing else.

The backup storage is specified in /etc/fstab with the nofail option present. This ensures that in case the disk dies, the backup endpoint will still boot properly, allowing me to access the machine to troubleshoot the issue and/or force a desperate reboot to try to fix things. A good alternative approach is to mount/unmount the remote disk manually as part of the backup script.

The backup storage uses the btrfs filesystem, and I use btrbk to take snapshots of the contents. If I accidentally delete all the files on the backup endpoint, then I can still recover from that situation because the data is still present in snapshots. 30 days is a good retention period: enough time to save the data in case of an accidental deletion, but short enough to avoid the backup disk getting full.

If you don’t want to use filesystem-level snapshots, then tools like restic are a good alternative. It can also operate over SSH and you can configure snapshot retention policies in your backup script. Just make sure to not lose the encryption password, and verify the backups once in a while.

Deployment

I manage my backup endpoints using some cobbled-together Ansible roles. I’ve perfected it to the point where the only manual steps are flashing the OS and setting up the storage, the rest is handled via Ansible.

I’d like to share my work here, but it will make Jeff Geerling cry. Maybe one day I’ll take the time to improve things…

Maintenance

All the backup endpoints update and reboot themselves regularly. It’s just the sensible thing to do.

Every 6-12 months I also do major OS version updates. It’s risky because of the whole offsite aspect of the solution, but so far I haven’t been burned yet.

Monitoring

Monitoring is an area where I have some room for improvement. So far, I’ve set up Prometheus node-exporter to all of the backup endpoints, and my home server keeps track of how the backup endpoints are doing.

This allows me to check once in a while if any of the backup endpoints has fallen off the network, or if the backup disk is getting full.

Issues I’ve faced

I’ve had this system running for a few years now, and it’s mostly stable! There have been some issues I’ve faced as well, though. Some are very specific to certain hardware, but I think there’s value in mentioning them.

I once blew up a backup server because of an Ansible configuration issue. That meant that I had to physically go pick up the server to re-image it.

The Orange Pi Zero was running quite hot, resulting in stability issues, so I put together a really janky cooling solution.

Hell, I did the same for the LattePanda as well.

It might look horrific but the extra cooling has fixed all the stability problems on both boards.

The lack of a real-time clock on the LattePanda has required me to make its backup script a bit special. I can’t rely on a systemd timer that automatically reboots the machine once in a while, so instead that part is present in the backup script. The issue is that the LattePanda boots up with the time being set in the past, and once it gets the actual time from the network, it will run all sorts of tasks because enough time has passed! This included the reboot timer as well.

At one point, the power supply on the LattePanda just died, and it was very visible on my graphs. That required a replacement.

Conclusion

That’s how I back up the most important data. I hope that this has given you inspiration to take your own backup approach to the next level!

I stumbled upon this Hardware Haven video about the Zimaboard recently.

I liked it a lot.

I finally bought one.

In short, Zimaboard is a small single-board computer that is relatively affordable and comes with an interesting selection of ports, which includes an exposed PCI Express port.

Before we get down to the build, here’s a list of aspects that I want to see in my dream home server:

• low power usage (2-15W typical power usage)
• 8GB of RAM or better
• enough performance to run my workloads, most of which are containerized
• 2x SATA or NVMe SSD slots, plus option for a third drive for the OS
• passively cooled and completely silent
• compact size
• gigabit Ethernet or better

You might be thinking, “Wait, that’s your dream setup? No clusters of machines, Threadrippers, 10 Gigabit networking, crazy number of disks?”. Well, yes. After years of trying all sorts of setups and learning about my home server usage patterns, this is the set of requirements that finds a balance between performance, efficiency and silence.

Basics

I recommend checking the product page to see the exact specifications. The configuration I bought was the 832 model: 8GB of RAM, 32GB of eMMC storage and a quad-core Intel Celeron N3450 CPU. It’s not the most powerful setup or even a recent one with the CPU being from 2016, but it’s just powerful enough to fit my needs.

This variant of the board costs 200 USD, but other configurations cost much less than that, I just needed the extra memory to be on the safe side. If you don’t care about the noise and size aspect of home servers, then you can get a better deal on the used market (see the TinyMiniMicro project for inspiration), but as you know by this point, I care about those aspects a lot.

The package

What sets the Zimaboard apart from other single board computers is how polished the product feels. Unlike a board like the Raspberry Pi, this one comes with a case and a cooling setup already attached to it. I suspect that a similarily configured Raspberry Pi 4/5 with all the accessories added on top would result in a price that’s quite similar to the cost of a top-of-the-line Zimaboard.

The heatsink looks great and is practical at the same time. Under the most torturous loads I could only see the CPU being around 72°C and due to it being passively cooled it made absolutely no noise. With the case being present, I do not have to worry about placing the board on my desk and scratching the table or shorting something out.

The board does not seem to have a power button, but by default it’s configured to power on as soon as you connect the power supply, which is great if you’re going to use this as a home server.

The box that the board was shipped survived and overall I’d say that the packaging is good. The board comes with some stickers and a single SATA data+power cable. The included power adapter comes with EU, US and UK plugs all included.

Shipping to Estonia was quite fast, taking just 10 days. The shipping costs were 18 USD.

Storage

The included 32GB eMMC storage is fine for hosting your operating system. The read speeds cap out at around 175 MB/s and the typical write speeds I observed were around 50-100 MB/s. The storage is identified as mmc-BJTD4R_0xc7d04e40 under /dev/disk/by-id/, and searching online suggests that it’s a Samsung chip.

Because this board offers two SATA ports, I also added a SATA Y-cable to my order (4 USD) so that I can take my existing Samsung 870 QVO 4TB SATA drives and move my home server setup to this board.

Allegedly this cable can also be used to drive two 3.5" hard drives powered by the board itself according to the shop page for the Y-cable, but I suspect that at that point you’ll be pushing the limits of the 12V/3A power adapter.

There is also a white drive activity LED placed near the SATA power connector on the board. Perhaps not for everyone, but I like the aesthetic and the sight of the server doing server things.

The performance of the SATA ports is what you would expect. When performing read operations on both SSD-s I saw the maximum total transfer rates hover around 900-950MB/s, which is pretty close to the SATA III transfer speed limit.

There is no native way to mount the two SATA drives to the Zimaboard. The creators of the board do sell a metal bracket, but it doesn’t seem to integrate that well to the board. However, there exists a 3D printable design that houses two 2.5" drives, even 15mm ones, and that’s how I ended up using a 3D printer for the first time in my life. The print was done using a Voron v2 Afterburner printer hosted at k-space and it came out pretty well.

The SSD mounting holes were a bit finicky due to slight printing errors, and during my first installation attempt I forgot to put in the plastic middle layer of the case that’s between the PCB and the stock bottom cover, but other than that the installation was a breeze.

The drives never hit above 64°C, which is absolutely okay for these SSD-s. I’m not sure how actual spinning hard drives might fare in this environment, but I don’t think that it’s going to be that serious because 2.5" hard drives should not generate as much heat in the first place.

Power consumption

What I love about this board is the power usage. At this point my home server is not even the most power hungry component of my setup, that honor now goes to the ISP-provided router/modem combo box that always draws at least 12W, even when it’s in bridge mode.

Here are my power usage measurements (measured with a simple power meter):

• idle: 2.5W
• no drives attached, CPU stress test (stress -c 4): 9.0W
• one Samsung 870 QVO 4TB SSD attached, CPU stress test: 13.2W
• 2 SSD-s attached, typical power draw in my setup (~10-40% CPU usage): ~8-9W
• 2 SSD-s attached, max load on SSD-s and CPU: ~14W

The Zimaboard, ISP modem/router box, my TP-Link router/Wi-Fi AP and my CyberPower UPS all together use at most around 34W, all combined. That’s even less than what my Dell monitor uses at reasonably low brightness levels!

These tests are not scientifically accurate, but they should give you an idea on what power consumption numbers to expect when running this setup.

Performance

I use a simple Prometheus Node Exporter + Grafana setup to view how much resources my various servers use. My home server has lately either been an ASRock Deskmini X300 based setup or an old ThinkPad T430, and something that both had in common was that the CPU usage was generally very low, mostly at or below the 10% mark. There would be bursty loads from time to time and backup processes running that bump that up, but not significantly.

Memory usage of my setup was also quite low, with all my services and containers fitting into 2GB during typical usage. With this information and some CPU performance comparisons done, I knew that the Zimaboard will likely be able to handle my home server tasks.

This board is not very powerful, but if you mainly rely on containerized workloads and can rely on Intel QuickSync to accelerate media transcodes, then you’ll be just fine.

I was actually impressed with how well the GPU-accelerated transcode worked on this machine once you set everything up properly. If you’re running Linux and Jellyfin, run vainfo to get a list of supported codecs and make sure that you have hardware decoding selected for those in Jellyfin settings. Also enable hardware encoding. I checked with intel_gpu_top to see if work was offloaded to the GPU and saw activity there, which means that hardware acceleration for Jellyfin worked out great! Check the Jellyfin hardware acceleration for more details on other requirements that have to be met for all of this to work.

Caveats

The Zimaboard does ship with a Linux-based OS called CasaOS. However, I had no intention of using it as I know what my requirements are and my infrastructure is already decently documented and set up via Ansible, which is why I can’t say how good that experience is. I also did not test Windows 10 or 11.

During testing, I noticed that the USB 3 ports can act a bit weird. The Fedora Server USB stick would not boot properly if connected to one of the USB ports, and when doing other tests with external USB storage I noticed hiccups from time to time. Not sure what might be the cause, but I’m writing it down nevertheless. It’s not a dealbreaker for me luckily since I don’t rely much on the USB ports in typical use anyway.

I did not test the PCI Express port, simply because I don’t need it, yet. It’s a great addition, though, and opens up plenty of modding capabilities in the future.

Fedora Server and btrfs

The Zimaboard gave me an opportunity to start fresh with my home server.

I made a leap: I’ve ditched ZFS and am now running Fedora Server with my storage being on a btrfs RAID1 setup, snapshotted and backed up using btrbk. It was a bit tricky to migrate and set everything up regarding backups and snapshotting, but I got it working.

ZFS is great, but it has always felt like an unwanted guest in the Linux ecosystem. The kernel developers don’t care much for maintaining compatibility with ZFS since it’s not in the kernel due to licensing issues, and Ubuntu has been one of the few distros that actually ships a kernel that includes ZFS built in. I didn’t want to be tied to Ubuntu forever, especially because of how they try to make snap a thing. ZFS DKMS builds are generally okay on distros like Debian, but on others you might find yourself not being able to access your data after a reboot because of a kernel update.

btrfs has had some issues in the past, especially with the RAID5/6 setup, but in my single and dual disk setups it has been solid for years, except for that one time around 2018-2019 when I ran btrfs RAID1 over USB storage. To be fair to btrfs, that was a pretty stupid setup.

Before committing to btrfs, I used two USB sticks to create a RAID1 setup and created real torture test scenarios. Tests looked something like this:

• write a file to the filesystem
• use md5sum to calculate a hash of it for verification purposes
• completely wipe one USB drive with dd
• run md5sum to calculate the hash again (it matched every time)
• run btrfs scrub on the filesystem to fix all errors
• rinse and repeat with variations to this setup

After doing all that, I was quite confident that this was going to work.

One thing to note with btrfs: if you’re running any type of multi-disk and redundant setup like RAID1 or RAID10, then make sure to include the mount option degraded in /etc/fstab so that you can still mount your filesystem if one or more of your drives fail. If you lose too many drives then it’s still probably going to fail to boot.

Here’s my /etc/fstab setup to serve as an example:

LABEL=turbo /turbo      btrfs subvol=turbo,compress-force=zstd:1,ssd,degraded,nofail            0 0

When doing the migration I kept all the paths the same, and each ZFS dataset was recreated using btrfs subvolumes, which is why this filesystem is mounted on a top-level folder.

Alternatives

My notes for the next dream home server setup included a few candidates:

• the Zimaboard

• Asustor Flashstor 6 or Flashstor 12

• a TinyMiniMicro style machine with at least two NVMe SSD slots

The first two fall nicely into my requirements for the dream home server. I picked the Zimaboard mainly because it’s much cheaper and I was already running two SATA SSD-s for my home server storage, so migrating would be really easy and I would not have to buy any new drives.

If large capacity SATA SSD-s start becoming less common and NVMe SSD-s become even cheaper than they already are, then I’ll have to look into something like a Flashstor.

Conclusion

Overall, I’m very happy with this purchase. The performance is just enough for my services to work reasonably fast, the board uses very little power and it’s silent, and it looks good on my wall.

My setup is also quite flexible, so in case I need more resources, I can get more Zimaboards and make them serve different purposes. However, it’s more likely that I’m going to discover a new toy to experiment and play with by the time I run out of resources on this one.

2024-08-20 update

I ran the Zimaboard for over half a year, but decided to switch back to the ASRock Deskmini X300.

I still love the low power consumption and the expandability, but I ended up putting more load on the system than I initially anticipated CPU-wise (running over 20 different Docker containers, some heavy). Should’ve guessed that it would happen eventually.

If it came with an embedded AMD Ryzen APU, then that would probably change things up quite a bit.

Zimaboard: still the closest thing to my dream home server, but it’s that last little bit of performance being missing that gets me. If your needs are moderate or low, then it’s still a fantastic option, if the price is right.

My eyes are now on the Asustor Flashstor Gen 2 machines. Up to 12 NVMe SSD-s plus 4 cores of AMD Ryzen goodness sounds too good to be true. Let’s see what the reviews will say once it’s out.

It all started with me getting a Steam Deck.

Background

After getting familiar with the Steam Deck and how the Proton compatibility layer works, I decided to write a backup script that would back up everything in the home folder, excluding the Steam games themselves due to the sheer girth of modern games (how the hell has GTA V ballooned up to 100+GB???).

Among the folders that I backed up was compatdata, a folder that contains files that Proton needs to make Windows games run. If you browse the folder, the contents look like a mini-Windows installation, and among those files you can also find your savegames. It made sense to me to back up this folder, and I was quite happy to know that no matter what happens, my game saves would be safe as long as I made a copy of this folder.

Fast forward a few weeks: I’m doing some changes to my self-hosting infrastructure and decide to redeploy changes to a backup server that I have. Nothing fancy, just Ansible roles that make sure that the backup server has some configuration present and that the backup folder has the right permissions.

The step looks something like this:

- name: Fix permissions
  ansible.builtin.file:
    name: /path/to/backups
    state: directory
    owner: user
    group: user
    recurse: yes

Everything was okay until I saw this.

After trying to recover the backup server and giving up as soon as I found that I cannot log in as root or use sudo, I gave up on doing that over the network and decided to get physical acccess to the server. One reinstall and Ansible run later, the server is okay again.

Investigation

How did we end up here though?

The backups from the Steam Deck are made to my home server using rsync. The Steam Deck is just a Linux machine, after all, and it made the most sense to me as I could make a backup that preserves all the permissions and links. Should I ever screw something up, I can run the same script in reverse and have everything working as it used to. I would hate setting up all my games and customizations again.

The script looked something like this.

rsync -aAXzv --delete-before /home/deck/ backupuser@myserver:/path/to/steamdeck/ \
  --exclude .local/share/Steam/steamapps/common \
  --exclude .local/share/Steam/steamapps/downloading

sudo shutdown now

Free tech tip: you can create a desktop entry for any script, add it to Steam, and run it within the Steam Deck UI when you’re done playing. You’ll have a backup and the system will shut down automatically once it’s done!

Nothing suspicious about the script, right?

There’s one problem with it: rsync also sends over links. These come in various types (symlinks, hard links) and act as a pointer to another file or folder. Turns out that Proton (or Wine) loves using symlinks. Most usages I saw were pointing towards common distributions of Proton, which makes perfect sense since it saves disk space.

There are also links that point to the root folder /. The use case for these seems also makes sense: present the game with a drive like Z:, and point it to / on Linux side, and now the user can easily navigate their whole file system within the context of the game they’re running. Maybe you want to install the game or its add-ons into a different folder, or maybe you want to load a save game that you have somewhere on your Steam Deck.

Just one problem with this: what happens when you have this symlink on another machine, such as my backup server, and you use Ansible to set the permissions for a folder containing this symlink, recursively? And what if the follow setting in Ansible is on by default since Ansible 2.5?

Turns out that the answer is that you’re going to mess up the file permissions on the whole machine, and most things stop working at that point. I could still login to one user on that box, but I could not do anything that would help recover the state of the machine at that point.

The fix (?)

There are many options for avoiding this problem, or at least working around it. I’m not sure what the perfect fix looks like. In case you know one, let me know.

One thing I added to my backup script was --no-links, which instructs rsync to not copy symlinks. That setting has to appear after the first batch of arguments (rsync -aAXzv) because otherwise it will be overridden by the -a parameter.

I also updated my Ansible setup to avoid setting the permissions for the Steam Deck backup folder as a precaution.

The major downside with my tweaks is that I don’t really have a “full” backup of my Steam Deck anymore. I do have backups of my game saves, but recovering from the backup will become a bit of a hassle due to me having to find and copy the game saves manually to the new compatdata folders since the backed up ones are missing all the symlinks to Proton/Wine-managed dependencies.

Alternatively I can consider using a backup tool like restic that should preserve symlinks. I’ll just need to also test recovering from that backup method.

Either way I can’t really blame Valve, Wine or Proton developers for my own fuck-up. They used symlinks in a way that allowed them to save disk space and give the user easier access to their files in-game. It’s just unfortunate that I learned about this setup the hard way.

Completely unrelated to my issue, but I can’t help but remember that one person whose machine got wiped by Steam.

You have probably heard about the Raspberry Pi. It’s a nice little affordable single-board computer with a huge community using it for all sorts of projects.

I got my first Raspberry Pi, the Model B+, during my first year at university, which was around the winter of 2014/2015. The idea of a super tiny PC that could actually do useful things was just very fascinating to me, and I loved the way that green PCB looked.

Over the years, I’ve used that Pi and newer revisions of them in all sorts of use cases: a simple Kodi media box, a Minecraft server, a web server, retro game emulation box, and much more.

Because the goal of the Raspberry Pi was to be an affordable platform for experimentation and learning for everyone, they had to keep the costs down. This meant that the computing power that the Raspberry Pi 1 packed wasn’t much. With the Model B+, you get one ARM CPU core that you can overclock to 1 GHz, and 512MB of RAM. It wasn’t much even at the time of release, and it definitely isn’t great in 2022.

Due to the very limited performance, the use cases for a Raspberry Pi 1 are quite limited in 2022. Use cases that don’t require much computational power are often better solved by other platforms. Use cases that are actually useful to me and solve a problem I have are too much for the Pi 1. At the same time, I absolutely hate it when I have computing equipment sitting around on a shelf doing nothing. I can’t even sell this Pi, because it is not worth much any more and it has some sentimental value for me.

Taking all that into account, and looking at my parts box, I decided to give one idea a go: let’s build the slowest damn Syncthing backup endpoint imaginable.

The build

Here’s a quick rundown of the parts:

• SBC: Raspberry Pi 1 Model B+
• Storage (microSD): SanDisk 8GB that I had lying around
• Storage (USB): Crucial BX500 1TB SATA 2.5" SSD, in an IcyBox USB-SATA enclosure
• Power: official Raspberry Pi microUSB power adapter
• Networking: TP-Link TL-WN722N USB WiFI adapter
• OS: Raspberry Pi OS 32-bit (lite)
• Acoustic foam: yes.

Use case: networked offsite backup of files that I cannot afford to lose, powered by Syncthing, btrfs as the filesystem, and btrbk as the snapshotting solution.

I set the OS up using the Raspberry Pi imager due to the useful options it offered, such as automatically setting my SSH public key authentication up. Once booted, I logged in and ran sudo apt update -y. I knew what to expect regarding performance, and was still surprised at how slow things have become. Updating the system and deploying the configuration using Ansible took hours.

The 1TB SSD is formatted as a btrfs file system and mounted to /storage. For the Syncthing service, I created a separate user syncthing with the home folder on /storage/home/syncthing. The synced data will live on /storage/syncthing. Keeping the home folder on the SSD is intentional: Syncthing keeps track of application state in a database and storing it on the microSD card will wear it out much faster. The database can also be quite big if you have a lot of files to sync.

I’m not going to go into the btrbk setup in much detail, but if you’re interested in that, then I do have a write-up about btrbk. Long story short: snapshots are cool, and they help prevent data loss in case of accidental deletions.

I set up Syncthing to allow access to the GUI from over the network by changing the ~/.config/syncthing/config.xml file while the service was stopped. Just change the GUI listen address to 0.0.0.0:8384, that will make the GUI accept connections not only from localhost, but from all machines on the same network. The next step was to set a password to the GUI and check the “Use HTTPS for GUI” box, because you probably don’t want any rando messing your machine up.

The choice to go with the WiFi for networking is related to the environment this system will eventually be deployed to. Ethernet isn’t an option there.

Once the system was all set up and files were syncing extremely slowly, I kept an eye on the Pi, especially the red power LED. I had set the CPU to run at 1 GHz using raspi-config, and with both the WiFi adapter and SSD connected, I had concerns that the Pi could not supply enough power. The power limitations of the Pi are a common source for issues and that has come up in the past as well, especially with external storage connected to the Pi. However, with this particular setup this has not been an issue so far. The fact that Pi is so underpowered that it cannot even make full use of the SSD is probably a contributing factor to the overall stability.

The file sync was very slow, but at least it was working. The typical transfer speeds hovered around 1.5 MB/s. Not great, but given the fact that this setup will operate in an environment where the download speeds are typically capped around 15-20 Mbit/s, this will be more than enough.

How well will this setup work over a longer time period? Only time will tell. Just like with other disastrous tests that I have done, I’ll try to update this post whenever anything noteworthy happens.

Does this setup make any sense if you need a performant Syncthing endpoint? Hell no. However, I’m very happy that this little Pi finally has a purpose.

Alternatives

This project was about finding a purpose for an SBC I already had lying around.

If the price and availability of Raspberry Pi-s wasn’t a complete shitshow in 2022, I would go for a Raspberry Pi 4 in a heartbeat. I can’t, and that makes me and most of the Raspberry Pi community very sad. The more performant Pi-s open up so many possibilities.

An Orange Pi Zero running Armbian was something I also considered, since I had one lying around, but it was a bit too competent for the job, and I might find a better use for it eventually.

If you wanted performance, then any x86-based PC will do just fine. My recommendation is to go for an used PC, be it a laptop or a very small and efficient PC. Just keep in mind that the power usage is typically higher on these setups compared to a small ARM-based single-board computer. If a machine runs 24/7, then the difference between 2-3 W and 10-20 W will start to matter, especially when energy prices are quite high in your region.

As some of you might own, I’m still rocking a ThinkPad T430, a laptop model originally released in 2012. It’s not the fastest laptop out there, but it is plenty fast for a number of tasks, even most software development work.

I also try to keep my machines up to date and properly backed up. On my laptop, this means having around a couple of scheduled tasks, such as restic backups. They don’t really take up a lot of time, but if an intensive scheduled task starts while you’re having a video call over Google Meet while also sharing the screen, things can get a bit slow, ruining the experience.

After a couple of these instances, I started thinking about finding a solution.

Scheduled tasks

Here’s a quick rundown of regular tasks that I like to run on my laptop.

Backups using restic: hourly backups to my home server. The file scanning process of the backup is quite intense, eating up a lot of my CPU time.

System updates using dnf and flatpak: usually run daily, because I’m too lazy to install the updates using a GUI, and at the same time I cannot afford to run out of date software.

Backups of critical files from my server to my laptop: there are some files I just cannot afford to lose. For this reason, I make a daily backup of the files to my laptop.

Filesystem maintenance: I don’t trust my storage devices (and neither should you), which is why I run a weekly scrub on my SSD that’s running the btrfs filesystem. If there are errors detected, I will know as soon as possible.

These tasks can be quite intensive, especially on hardware that’s a bit older.

With some tasks, such as system updates, you may also run into random issues. Updating Mozilla Firefox while it’s actively running results in the browser requiring a restart immediately after opening a new tab. This is very annoying and inconvenient when you’re in the middle of your work.

The “time-to-sleep” script

To avoid intensive scheduled tasks running while I’m doing actual work, I tried rescheduling these tasks to a more quiet time. All I needed to do was to edit the systemd timers so that the jobs start after midnight and leave the laptop running.

This solution has quite an obvious downside: my laptop will be powered on pretty much 24/7, and since electricity is super expensive and laptop maintenance a bit of a pain, I’d be wasting resources, time and eventually money.

I could make the last scheduled task make the machine shut down, but what if I have a long-running file copy operation running? It would be interrupted, unless I specifically disabled the scheduled task.

Then it clicked for me.

I know when I’m planning on going to sleep. Sleeping also has a regular cycle as it happens daily. Why not just put all the scheduled daily tasks into one script that I can run whenever I intend to shut down my laptop?

And thus, the time-to-sleep script was born. It’s a simple Bash script that lives in /root/.local/bin/time-to-sleep. To make sure I can also reasonably observe the behaviour of the script, I’ve implemented it as a simple systemd service that lives in /etc/systemd/system/time-to-sleep.service. It’s disabled by default, but can be started manually.

[Unit]
Description=Run maintenance tasks

[Service]
ExecStart=/root/.local/bin/time-to-sleep

I have an alias in ~/.bashrc that I use to start the service.

alias time-to-sleep="sudo systemctl start time-to-sleep.service"

Whenever the time comes to go to sleep, or when I’m just going to do something else and can leave the laptop running at my desk, I just open a terminal window, type time-to-sleep and I’m good to go. If I’m somewhere with a very poor internet connection or need to conserve battery power, I can opt to not run that script.

Going with this approach also saves me a lot of trouble on the technical side as well. I don’t need to think of a magical way to automatically detect if I’m working at the moment, or if I’m running on battery, or if I’m behind a limited internet connection. Sometimes your job becomes so much easier if you make some assumptions about when and where the script runs. You’re in control of choosing when to run it, after all.

In case something goes wrong, I can rely on the set -e line in my script to fail hard. When I come back to my machine, it will be quite noticeable that something has gone wrong, as the laptop will still be running. Who needs a fancy monitoring setup when you have visual evidence of failure?

As for the hourly backups, I solved that problem by only making backups of more important files. It takes much fewer resources compared to the full /home folder backup that runs as part of the time-to-sleep script.

Conclusion

After all this work, I don’t have to worry about any scheduled tasks hogging up all the resources while working. At the same time, I still get all the benefits that the scheduled tasks bring.

Example

In case you need it: here’s an example of the script that I’m running, with all the personal details scrubbed out. Feel free to use it as inspiration for your own time-to-sleep script!

#!/bin/bash

set -e

echo "Restic full home folder backup."
export RESTIC_REPOSITORY='sftp:backupuser@backupserver.lan:/path/to/repo/'
export RESTIC_PASSWORD='someverysecurepasswordgoeshere'

restic unlock

restic backup --verbose --exclude-caches --cleanup-cache \
  --iexclude=/home/*/downloads \
  --exclude=/home/*/.cache \
  --exclude=/home/*/.gradle \
  --exclude=/home/*/.local/share/Trash \
  /home

restic forget --prune \
  --keep-hourly 0 \
  --keep-daily 7 \
  --keep-monthly 0

echo "Restic full home folder backup done."

echo "Backing up important data from the server."

rsync -az --delete-before \
  user@myserver.lan:/path/to/nextcloud/ \
  /storage/backups/nextcloud/

chown -R localuser:localuser /storage/backups/nextcloud/

echo "Backing up important data from the server done."

echo "Updating system."

dnf update -y

flatpak update -y && flatpak uninstall --unused -y

echo "Updating system done."

echo "Scrubbing disk."
btrfs scrub start -B /
echo "Scrubbing disk done."

echo "All done! Going to sleep."

shutdown now

I like having a safety net whenever I’m doing something potentially destructive, which is why I use the btrfs file system for my operating system and my data. Snapshots are one half of my “whoops, there goes all my work” strategy (backups are the other half).

I’ve written about how I use snapshots on btrfs using snapper, but lately I’ve become annoyed with it.

Shortcomings of snapper

snapper is great while you’re on the happy path, but when you wander off of it, it gets a bit frustrating. This is 100% my own personal experience and I cannot rule out any PEBCAK scenarios, but it’s how I felt using the tool.

The snapshots are on the same subvolume, such as /home/.snapshots, which means that I have to specifically exclude the .snapshots folder in every tool and script that I use for making backups. Without that change, the tools ended up scanning the folder and eating up a lot of resources, mainly the CPU and storage.

Backing up snapshots to something like an external backup SSD wasn’t something that snapper supported out of the box as well, meaning that for offline backups I still had to resort to a good old rsync copy. It’s inconvenient and I won’t have a backup that also contained recent snapshots, but it was manageable.

I also reinstalled Fedora Linux on my ThinkPad recently to see how well my Ansible playbooks hold up (quite decently!) and found that setting up snapper is a bit of a pain. I tried to set up snapper again, but it wasn’t as straightforward as I was hoping for, with it not properly picking up the configuration file changes. At the end, it just ran into weird errors when I tried it to prune some snapshots according to the policy I set up. Since I was running out of disk space, I just gave up, uninstalled snapper and destroyed the snapshots manually.

After all that (and a recommendation by a friend) I decided to bite the bullet and try out btrbk.

btrbk

Setting up btrbk on Fedora Linux 36 was a breeze. Install it via dnf install btrbk, copy your configuration to /etc/btrbk/, make sure that the directory used for snapshots exists with mkdir /btrbk_snapshots and you’re good to go! btrbk is just a huge Perl script, so no fancy dependencies are needed.

My btrfs filesystem has three subvolumes: /, /home and /storage. Data stored on these subvolumes has different levels of importance, which is also reflected in the snapshot retention policies.

The configuration on my setup is the following:

timestamp_format        long

volume /
  snapshot_dir /btrbk_snapshots
  subvolume /
    snapshot_preserve_min   12h
    snapshot_preserve       24h
  subvolume /home
    snapshot_preserve_min   48h
    snapshot_preserve       7d
  subvolume /storage
    snapshot_preserve_min   6h
    snapshot_preserve       7d

To take the snapshots, I’ve set up a systemd timer that runs every 5 minutes and runs the following command:

btrbk -c /etc/btrbk/btrbk.conf run --progress

A successful run will have output like this:

--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.1)
    Date:   Sat Jul  9 14:00:00 2022
    Config: /etc/btrbk/btrbk.conf
Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/
+++ /btrbk_snapshots/ROOT.20220709T1400
--- /btrbk_snapshots/ROOT.20220708T1300
/home
+++ /btrbk_snapshots/home.20220709T1400
/storage
+++ /btrbk_snapshots/storage.20220709T1400
--- /btrbk_snapshots/storage.20220709T0745
--- /btrbk_snapshots/storage.20220709T0750
--- /btrbk_snapshots/storage.20220709T0755

I love how the output is very concise about what it just did.

To make a backup of my drive to an external SSD that’s also a btrfs filesystem, I created a separate config at /etc/btrbk/backup-to-ext-drive.conf:

timestamp_format        long

target_preserve_min    no
target_preserve        7d

volume /
  snapshot_dir /btrbk_snapshots
  target /mnt/backup/T430
  subvolume /
  subvolume /home
  subvolume /storage

In this example, /mnt/backup/T430 is a btrfs subvolume itself.

Instead of running rsync, my backup script contains the line:

btrbk -c /etc/btrbk/backup-to-ext-drive.conf run --progress

I love how the default behaviour in btrbk is to keep snapshots separate from the subvolumes themselves. Making a full copy of /home or any other filesystem doesn’t require me to always exclude the .snapshots directory, which is nice.

The snapshot names themselves are also human readable. If I accidentally messed up and my changes from an hour ago were lost, I’d just have to go to /btrbk_snapshots, look up a snapshot from an hour ago, and copy whatever I need from the snapshot. This has already saved my butt at least once.

Conclusion

It’s not often that a piece of software gets me that excited, but btrbk has somehow managed to do it. It’s a single script that has all the example use cases documented in its README and the functionality does exactly what I want.

That’s all I really need from software.

Huge thanks to Digital Integrity GmbH for building, maintaining and sharing this tool with the world!

Update: I’ve switched to btrbk for managing btrfs snapshots due to certain deficiencies that I had with snapper. Those may have been fixed since then, but btrbk has served me very well, and I’m sticking with it in the foreseeable future.

Friday afternoon. You’re trying out a script that you wrote to mass-rename and move some files around. You finish the script and test it out.

Oops.

All the files now have all the wrong names, and some have been randomly moved 10 folders deep. It’s a mess.

And you didn’t make a backup of your files right before this step because you thought that you wouldn’t need it. It’s just a simple script, after all…

If you’ve found yourself in a similar situation and want to know how to avoid problems in the future, then this article is for you

btrfs: overview

btrfs (butter-fs, bee-tee-arr-eff-ess) is a modern filesystem present in the Linux kernel that has a lot of great features:

• different RAID levels: RAID0, RAID1, RAID10…
• data integrity guarantees
• snapshots
• and many more!

btrfs does have some downsides and has had its fair share of issues and controversy in the past, but at least during the last couple of years it’s been fine in my simple RAID1 + snapshots use cases.

In this post, we’ll focus on the snapshots feature that btrfs offers.

If you’re not sure what snapshots are, then you can think of them as a still image of the state of your filesystem at a certain point in time. You can create a snapshot of the filesystem, delete all your files, and still be able to get your files back just like you left them.

If you’re a developer who has used git and doesn’t mind oversimplified (and probably incorrect) explanations: you can think of snapshots like git commits. You can take snapshots (commit), roll back to a previous snapshot (reset/revert) or view the contents of a previous snapshot (checkout commit).

One thing to keep in mind with snapshots is that they will take up space, especially if you make a lot of them and move files around a lot.

There are many tools available to manage btrfs snapshots. The one I’ve opted for is snapper.

Snapper quick-start

Here’s a quick guide on how to get started with snapshotting your btrfs filesystem. The example is based on a system with a btrfs filesystem mounted on /home

• Install snapper
  • Ubuntu/Debian: sudo apt install snapper
  • Fedora: sudo dnf install snapper
• Setup the configuration: sudo snapper -c home create-config /home
• /etc/snapper/configs now has a text file named home that you can configure to your liking. My personal recommendation is to use the TIMELINE snapshots configuration to configure the maximum number of hourly and daily snapshots.
• snapper ships with two systemd timers that automatically take new snapshots and remove old ones. Make sure that they are enabled:
  • sudo systemctl enable --now snapper-timeline.timer snapper-cleanup.timer
• After a while, you should be able to see snapshots appear in the output of sudo snapper -c home list.
• If you want to browse the files for the old snapshots, go to /home/.snapshots.

Example output of sudo snapper -c home list on my machine:

> sudo snapper -c home list

    # | Type   | Pre # | Date                     | User | Cleanup  | Description | Userdata
------+--------+-------+--------------------------+------+----------+-------------+---------
   0  | single |       |                          | root |          | current     |         
   1  | single |       | K 16 märts 2022 08:12:01 | root | timeline |             |         
 242  | single |       | K 16 märts 2022 13:00:16 | root | timeline |             |         
 302  | single |       | K 16 märts 2022 14:00:05 | root | timeline |             |         
 362  | single |       | K 16 märts 2022 15:00:10 | root | timeline |             |         
 422  | single |       | K 16 märts 2022 16:00:19 | root | timeline |             |         
 461  | single |       | K 16 märts 2022 17:20:29 | root | timeline |             |         
 501  | single |       | K 16 märts 2022 18:00:02 | root | timeline |             |
...         

That’s it!

Snapper tech tips

If you’re new to snapshots, then I recommend keeping an eye on your disk usage and tuning the snapper timeline related configurations to your liking. For my /home folder, my configuration looks like this:

# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="300"
TIMELINE_LIMIT_HOURLY="48"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="0"
TIMELINE_LIMIT_YEARLY="0"

With this configuration, I can undo my screw-ups for up to 7 days, with more granular control for changes done within the last 48 hours.

If you have data that changes more often than other data on the same filesystem, then I recommend utilizing btrfs subvolumes and creating separate snapper configurations for each.

For example: a folder containing gigabytes of cat pictures is unlikely to change all that often, so a longer snapshot retention policy of 24 hourly/30 daily snapshots is a sensible one.

However, the contents of my downloads folder changes more frequently and may contain bigger files that I need temporarily, such as the ISO file for the latest Fedora/Debian release. In a situation like that, having a 6 hourly/7 daily snapshots policy will make more sense, especially if I don’t want to run out of space due to excessive number of snapshots taken for that filesystem.

Further reading: ZFS snapshots

ZFS is a fantastic filesystem that offers many of the features boasted by btrfs, and some that btrfs doesn’t have. ZFS also supports snapshots. To set up automatic snapshot creation and pruning on ZFS, I can recommend sanoid.

Maybe I am unlucky with computer hardware.

Maybe I have the opposite of the golden touch of Midas.

Or maybe it’s just the fact that I use my hardware for things they were never meant to be used for.

Anyway, I have acquired two Seagate 4TB external hard drives for doing some testing with ZFS. I shucked them (took them out of their enclosure) and started using them as internal drives in many different configurations.

Funnily enough, even though the two external drives had different branding (Backup Plus, Expansion) and enclosure design, the hard drives in them were identical.

Some info about the drives:

• Model: ST4000LM024-2AN17V
• Recording type: SMR (shingled magnetic recording)
• Form factor: 2.5"
• Interface: SATA
• Thickness: 15 mm

Before I get to the stupid stuff, I would like to mention what these drives are good for, and where they suffer heavily.

Good:

• WORM (write once, read many) type of workloads, such as a Steam library, your media collection etc.

Bad:

• random read/write workloads (performance is absolutely rubbish and makes the system unusable)
• write heavy workloads (SMR requires some clean-up activities that are managed by the drive, and you cannot do anything while those take place)

The stupid stuff

I have done some pretty dumb things with these, such as using them in an InWin Chopin computer case and letting them hit 60C under load. Fun fact: these drives actually fit almost perfectly in the case, as long as you bend one small metal section for one of the drive cages. Not so fun fact: they will overheat if you have a high CPU load and you cannot do much about that.

Currently these drives are sitting in my main desktop. Initially I tried using them in a mirrored configuration under Windows, because having your whole Steam library downloaded with disk space to spare is pretty cool. Unfortunately, one of the disks started having issues and it got kicked out of the mirror. It kept doing this every time it accumulated bad sectors.

Now these two sit in a ZFS mirror configuration inside a VM, acting as some sort of a temporary scratch disk. Another free fun fact: kvm (or qemu, or libvirt, one of those) will pause the VM if it detects that one of your disks has encountered I/O errors, and seems that the only “fix” is to force a restart of the VM (could not unpause the VM).

It has become some sort of a fun pastime for me to keep an eye on one of the failing drives and the status of the ZFS mirror.

Will ZFS detect some errors?

How large can the bad sector count get before the drive dies?

When will it actually die?

Why is S.M.A.R.T still reporting that the disk is OK?

So far, so good.

Expect this list to grow with time.