systemd really, really sucks

systemd is controversial, but it’s obvious that distributions love it, and many users do, too. It’s bad for both objective and subjective reasons, and I’ll try to explain why as objectively as possible.

Linux is the second-most popular descendent of Unix, the first being OSX. In some ways, OSX is more faithful to the Unix heritage, having a BSD-derived userspace and a Mach-based kernel. However, Apple long ago gave up on the Unix Philosophy[Coined by Ken Tompson, the philosophy advocates minimalist, modular systems with the motto: do one thing, do it well.], which is perhaps the most important tenet of Unix. It’s important to recognize that this philosophy of small, composable resources is also considered critical to good software architecture, design and programming.

Linux itself is a monolithic kernel, which is already fudging the Unix philosophy. Unix kernels have traditionally been monolithic, mainly for performance reasons, and when Linus Torvalds developed Linux he chose a monolithic kernel architecture; the userspace for Linux, however, has always followed Unix Philosophy, until systemd. Broadly speaking, that meant that Linux distributions were composed of:

An init system, responsible for starting and keeping system services running. This is often referred to as “pid 0”. Historically, this was a small program with simple rules, often merely executing shell scripts in a clearly defined order designed to ensure that – for example – the network was started before the network file system was launched. These are traditionally long running or one-time jobs.
A logging facility, responsible for storing messages from the various software systems.
A timed job control daemon, responsible for running jobs on a schedule. These tasks were the ones that are run periodically, such as cleaning up old log files, or updating the system time from a network source.
A network management system, which could be composed of several different pieces of software – one for handling ethernet, one for handling WiFi, one for performing DNS lookups, one for setting up VPNs.

These systems are interchangeable. Users could mix and match, changing cron systems for ones with different features. The init system, in particular, went throuh a period of experimentation with different systems having less shell forking, better dependency resolution, and more concurrency. Linux boot times saw a dramatic decrease, and the init process often got less complex and more compact. Systemd started as one of these.

But systemd did not stop with init. Systemd slowly grew to absorb more and more subsystems: logging (journald), cron (integrated into systemd), session management (logind), network management (systemd-networkd), domain name resolution (systemd-resolvd), even home directory mounting (systemd-homed). And while systemd advocates will argue that systemd is not monolithic, it is. You can prove this to yourself by trying to use systemd for init without using journald and the logging. You can install a cron system alongside systemd and choose to not use the systemd timed jobs, but the cron capability is hard-baked into systemd. elogind is logind decoupled from systemd, and this separation was notoriously difficult and now maintains no link to systemd. Again, the proof is that systemd depends on many of these subsystems, and none run without systemd. If you want to use a different subsystem alongside systemd, you generally still have all of the monolithic encrustation, you’re merely not using it.

This lack of modularity – where “modularity” means free choice in interchanging components for other, unrelated projects – is where most of the criticism of systemd stems. It’s the purest, and most important, criticism of systemd, but my personal grievances against systemd go further. Because, in my experience, systemd isn’t good software.

journald is particularly bad. The binary storage format means that logs are no longer easily accessible without using systemd tooling – with journald you can no longer use the standard POSIX toolset of cat, grep, and tail to view logs. Instead specialized journalctl syntax must be memorized, and above all, it’s slow. On a VPS where logs are infrequently viewed, it can take tens of seconds for journald to unpack and decode the most recent log entry.

networkd and resolvd are also poorly implemented, with complex and (again) levels of indirection to accomplish simple things. Getting IPv6 working is no longer a matter of a single sysctl command, but with systemd requires changing multiple configuration files and indeed installing a new service script! Changing DNS resolution, which used to be relatively straightforward and requiring no more than man pages, now require decyphering the order in which resolvd gathers configuration from multiple files to compile and generate /etc/resolv.conf.

We can evaluate the “systemd isn’t monolithic” claim based on how it’s packaged and distributed. systemd now encompases:

All of these are packaged, on Arch, in the single systemd package. You can not install only part of the systemd ecosystem under Arch. Debian breaks these up into boot, homed, and resolved; everything else in the list is in one of these three. Of all of these, homed is interesting as it’s (a) something systemd function without, and (b) also the one systemd service few distributions use.

It’s also notable that systemd, in its own words, is incompatible with the Linux kernel user session keyring. This is a highly secure keyring that increases protection broadly, and provides the only real protection against root or the user account being compromized. Because systemd is incompatible with it, Poettering himself says:

I really think that keys should be shared among sessions of the same user, and hence should be attached to the user, and not any session⁰

This is an absurd security stance to take: “because I can’t make my software work with a more secure model as a result of architectural decisions, I think user’s should use less secure models.”

The Arch systemd and systemd-libs packages weigh in at 36MB. In comparison, to get the same functionality without systemd, a distribution could use:

job	service	notes
init	systemd	PID 0, the init system, and the job scheduling
log	journald	System logging
resolv	systemd-resolved	Domain resolution
/home	systemd-homed	Mounts /home, handling things like encrypted home directories
session mgmt	systemd-logind
power ctrl	systemd-hibernate/suspend/poweroff
set hostname	systemd-hostnamed
VM/continer/session	systemd-machined
NTP	systemd-timesyncd
udev	systemd-udevd

job	service	size
init	dinit⁰	741 KiB
log	cronie⁰	227 KiB
resolv	metalog⁰	47 KiB
/home	N/A (/etc/fstab)	0
session mgmt	seatd⁰	112 KiB
power ctrl	N/A (dinit)	0
set hostname	N/A (/etc/hostname)	0
VM/continer/session	Various¹	/
NTP	ntp⁰	4 MiB
udev	mdevd⁰	456 KiB
––––––––––:	———————:
Total	<	5.74 MiB

systemd-machined is another kitchen sink service that controls VMs, containers, and connecting to user sessions. For auto-starting VMs and containers, you’d use dinit; for controlling them you would use the standard commands, e.g. podman. systemd-machined provides convenient wrapping to connect to user session DBUS, which is necessary for managing user session containers; this can also be done with a two-line bash script. ⁰: https://github.com/davmac314/dinit ⁰: https://github.com/cronie-crond/cronie/ ⁰: http://metalog.sourceforge.net ⁰: https://sr.ht/~kennylevinsen/seatd/ ⁰: https://www.ntp.org/ ⁰: https://skarnet.org/software/mdevd/ ⁰: https://github.com/systemd/systemd/issues/1299

Note that, for each component, there are usually several projects that can provide the capability, depending on the user’s specifications: small size and simplicity vs broad capability and complexity is the common trade-off. For example, Alpine uses eudev instead of mdev, and while Artix has no default init, it leans toward openrc. There are at least 3 mature, established cron projects, and many more more obscure ones providing different features, e.g. tasker. Many of these packages are older, more mature, and better tested than the systemd components.

It would not be hyperbolic to suggest that, at some point in the future, Linux distributions will become 1. the Linux kernel, 2. the GNU userspace, and 3. systemd. There will be no other services. There will be systemd-sshd, systemd- ftpd, systemd-displaymanager, systemd-polkit, systemd-firewalld, and so on.

Why, then, have so many distributions adopted it? For one, because systemd scope creep was something that happened over time. systemd started as merely another PID 0 tool. And then it took over cron, and logging, and as it ages it absorbs more and more service capability. Once committed, and because systemd components tend to not function correctly if they do not have access to other systemd components, distributions simply switched to the systemd services. The other reason is more compelling: systemd is monolithic: there is one way of doing things. This makes it easier for administrators to move between Linux systems, because all commands are exactly the same on all distributions. There is no variation, and no diversity; nothing new to learn if you install a different distribution. Why, once in this state, anyone would want to switch distributions is another question, but it is unquestionably more simple for administrators. But also, it means that desktop environments have an easier time integrating with underlying services. Diversity certainly makes writing configuration GUIs harder, and if your objective is to make the terminal obsolete, systemd is a good development.

If Linux distros continue to standardize on systemd, Linux will eventually resemble Windows, with user choice a thing of the past. So, not only is systemd monolithic and removes user choice, but it’s 7x larger to provide equivalent capability. Is this the direction Linux wants to go?

The situation isn’t entirely bleak. Several distributions eschew systemd, and many allow users to select which software they want to use for subsystems – often choosing between minimal and lightweight, or full-featured but bulkier tooling. Artix (Arch-based), Alpine, Void, ChimeraOS are just a few shipping with clever, straightforward, competent, and lightweight init systems like dinit; or with much more complex but versitile systems like s6. Despite systemd’s adoption by so many distributions, init innovation has continued, and the Linux ecosystem is richer for the diverisity.

Systemd was an improvement on SysV init, but the ecosystem has become bloated, monolithic, and tightly coupled, and it’s negatively impacted innovation of core Linux services. This leaves a gap for new operating systems such as Redox, which lay the groundwork in modularity and composability in the very kernel. Between the optimized, efficient, pro-choice Linux distributions and the new playing fields provided by Redox, users will continue to have options and be able to compose their computing to meet their needs.

At least, we can hope.↩︎︎