Yesterday brought exciting news on the ZFS and Ubuntu fronts — experimental ZFS root support in the installer for the upcoming interim release of Ubuntu, Eoan Ermine. An accessory appear on 2019-10-09 Eoan’s daily build—not in the regular beta release and, in fact, not even in the “current daily” when we first went to download it. It’s that new! (Readers wishing to play with the new work can find it in crocodile daily write, come Here.)
For ZFS newbies
If you’re new to the ZFS hype train, you might be wondering why the new file system option in the OS installer is such a big deal. So here’s a quick explanation: ZFS is a copy-on-write file system, which can take atomic snapshots of entire file systems. This is just like magic if you’re not used to it—a snapshot of a 10TB file system can be taken instantly without interrupting any system processes in the slightest. Once the snapshot is taken, it is an immutable record of the exact, unblocked state of the file system at the time the picture was taken.
When an image is first taken, it takes up no additional disk space. As time goes by and changes are made to the file system, the space required to store the photography grows by the amount of data that has been deleted or changed. So let’s say you take a snapshot of a 10TB file system: the snapshot completes immediately, no additional room is required. Then you delete a 5MB JPEG file—now the snapshot takes up 5MB of disk space, because it still contains the deleted JPEG. Then you convert 5MB of data into a database, and photography takes 10MB — 5MB for the deleted JPEG and another 5MB for the converted data in the database.
That’s an awesome ZFS feature. There’s also the ability to manage multiple disks in a native RAID-like system, inline compression with optional algorithms, fast asynchronous replication, and more. But we’ll focus mostly on visuals here, for one’s sake other what you can do with an image is rotate it.
Although there isn’t any support built into Eoan’s apt package manager for taking snapshots automatically yet, we can introduce snapshot—oops—reversing time manually. In the above gallery, first we draw a ZFS image. Eoan has split our root file system into tons of small data files (more on that later), so we use -r option for zfs snapshot to take pictures regularly throughout the whole tree.
Now that we have insured ourselves against mistakes, we do something we will regret. For the purposes of this demo, we just removed Firefox—but we can really recover from anything up to and including a feature rm -rf --no-preserve-root / This method involves a little extra legwork. After uninstalling Firefox, we need to change our snapshots to restore the system to its original state.
Since the root filesystem is dispersed by a widow of individual datasets, we need to convert them all individually. Although this is a pain for the ordinary user without an additional toolkit, it makes it possible to perform more granular restoration operations if we feel the choice—like changing the root file system without changing it back. /home. Undoubtedly Ubuntu will have a toolkit to make this easier, but for now, we do a bit of sysadmin-fu and pipe zfs list to grep to awk to xargsoh my
The command line acrobatics may have been obnoxious, but the switch itself is instant, Firefox is back. It still doesn’t work quite right, though, because orphaned filehandles-we change the live root file system, which is kind of a cowboy thing to do. To do things perfectly, a reboot is necessary—but after the reboot, everything is the way it was before, and without the need to wait through any lengthy Windows Recovery Mode. groveling on the file system.
For ZFS enthusiasts
In this section, we’ll take a look at how Ubuntu mounts file systems in the Eoan experimental installer. The version in our daily build is 0.8.1, so this is great news for the ZFS fans among us, even without the experimental root installer — assuming the final version of Eoan follows this, we’ll get encryption native, TRIM, device removal, and zpool checkpoints. These features are included in ZFS on Linux lord since 0.8, but this is the first time they have appeared in Ubuntu’s native ZFS.
So far—remember, this is alpha software in a daily build—the installer doesn’t give you any control over how to mount the disk when you choose to install ZFS; just do what you want to do. The Eoan VM I created has a single 20GB virtual disk. The Eoan installer places this into one primary and two logical partitions—a small UEFI boot partition and partitions for two separate ZFS storage pools, named bpool we had rpool.
Bpool is pretty boring; it’s just where the system/boot directory is mounted. Eoan makes this pool 2GB, which is double what is conservative /boot are normally provided to; This is likely to allow room to maintain a fairly deep archive of images in the future. rpool n has all the remaining disk space after UEFI and bpool divisions are created; it’s where all the stuff goes, including your root filesystem, home directory, etc.
Underneath rpool, you’ll find a surprising number of small databases, all of which fit into specific areas in what would normally be a single root file system. This appears to us to be an inherited BSD-ism — most Linux distributions abandoned the concept of multi-partitioned disks with multiple file systems over twenty years ago, but FreeBSD — which already has root ZFS options in the installer it for many years now — is a lot. more confused about it.
The good thing about uploading the root file system into several separate databases is that you can image and convert them individually. In some cases, this is great—it’s an obvious, clear, and useful difference between changing the entire root file system and changing your own home directory, for example. Most users—even very capable sysadmin types—will be much more confused about how and why you might want to roll back. /usr without changing, say, /var/lib/AccountServices, though. It’s good that you might if you really want, but we are not sure that the power outweighs the need of the simple method.
Looking a little deeper, we can see that Eoan doesn’t set any special database properties on all these separate databases. It is setting compression = lz4 on the entire pool, though. This is a good thing—many people worry that compressed file systems are slow file systems, but LZ4 stream compression is so lightweight that it’s “free.” We have done extensive testing over the years of ZFS experience and have never seen a situation where LZ4 is not a good idea. Even a $50 tinkertoy APU from several years ago can compress and decompress LZ4 faster than a pair of fast SSDs can keep up, with no significant CPU utilization.
We spot a bug quickly while looking at pools and databases. Both (Edit: a Canonical core admin explain that this is not a bug; bpool body and bpool/BOOT/ubuntu_oalrlu (assuming that the string of characters is apparently intended to be a unique system identifier) is /boot set as their mountpoint. This is clearly not causing any major problems now, and we are sure it will get the metal out well before Eoan lives.bpool is arranged canmount=no. The reason for bpool‘s unmountable mountpoint is so that any new datasets are created under bpool will automatically move in /boot/newdatasetit is not /bpool/newdataset.)
Although Eoan creates datasets automatically for both my real user accounts homedir and root’s, the adduser command does not create one for the new test user. This is something we also hope to be ironed out fairly quickly—even if it is adduser itself doesn’t take those steps, a GUI for adding new users will be created, if it doesn’t already exist. This is also easy to do by hand; in the above example, where the new user test it’s not login, we can test upgrade to zfs dataset homedir like this:
root@eoan:~$ zfs create rpool/ROOT/ubuntu_oal4lu/test_twm547 root@eoan:~$ rsync -ha /home/test/ /rpool/ROOT/ubuntu_oal4lu/test_twm547/ root@eoan:~$ rm -r /home/test root@eoan:~$ chown test.test /rpool/ROOT/ubuntu_oal4lu/test_twm547 root@eoan:~$ zfs set mountpoint=/home/test rpool/ROOT/ubuntu_oal4lu/test_twm547 root@eoan:~$ zfs mount rpool/ROOT/ubuntu_oal4lu/test_twm547
… and that would be it.
The next big thing we came up with was a device for taking photos automatically. You can’t roll back to a picture you never took, so a secure ZFS system should always take beautiful pictures. There’s nothing in Eoan to take snapshots for you yet—the only standard cron job is one that flushes the pool once per month—but there are some general purpose ZFS snapshot orchestration tools readily available; these as well zfs-auto-photo and my own sanoid.
Alpha software is alpha!
Finally, we want to remind readers that while ZFS itself is very stable, the installer of ZFS of Ubuntu and its use as a root file system is still available. alpha quality We don’t recommend that you try to use the new ZFS developer on systems you care deeply about until the developer has made it past alpha, past beta, and all the way to full release quality. This also means that you should be sympathetic about any bugs you see playing with you at first—again, this is alpha Software and bugs are not only possible, they should be expected.
With all that said, we’re very excited about ZFS on root making progress appearing in Ubuntu—and we hope these features and more will make it into the much-anticipated end-of-month release of Eoan Ermine.
Image listing by Jim Salter