The problem with the anti systemd attitude

If you spend any time in the Suckless community, you inevitably encounter the anti-systemd folks, which leads you to articles criticizing systemd. There’s a rot lurking in that community – the brotherhood of systemd-hatred, and that’s what I’m talking about today.

Maybe, after reading all of those criticisms of systemd, you’ve been tempted to switch to a non-systemd distribution; maybe Devuan, or Void, or Alpine, or any number of other distributions – there are even lists of “best non-systemd Linux distributions.” For me, it was Artix.

Artix was an obvious choice since I was already running Arch, and Artix has an actually pretty good migration page. I’ve written about why I decided to migrate in detail, but the bottom line was that – while I like systemd – it started failing for me.

The migration was not easy. The first issue I encountered was due entirely to my choice of init systems. I had chosen s6, and that was a mistake. A horrible, horrible mistake. s6 does nothing like any other init systems, so you have to first learn a boroque ecosystem of commands which (for me) are not clearly delineated in responsibility. It’s a horrible choice for migrating a distribution; when things don’t work (and they won’t), before you can fix anything you must first learn s6, and that’s really hard when your network is down – because the author of s6 refuses to write, or include, man pages. So s6 is an extremely complex system with no offline documentation. That’s right: it’s an init system that excludes the standard document system used in Unix systems since the 70’s. I’m not bitter, though.

After only four days of working to get my laptop to where it’d boot into a state where everything was working (X, network, services, etc), and mainly with fighting with s6, I replaced it with dinit and things got immediately easier. For one, dinit uses the same CLI interface every other init system (including the much-maligned systemd) uses: start, stop, enable, disable. Service files are embarrassingly simple; the simplest (and, unusually, most common) is two lines: a type, for dinit’s purposes, and the command. And – bonus! – dinit doesn’t demand to be PID 1! It runs just fine as a non-root user process, which means the other thing systmed does well – user jobs – is even easier with dinit. And it has beautiful, readable, and complete man pages – I really can’t complement Davin McCall enough, because dinit is a wonderful example of Unix philosophy tooling.

But I digress: I came not to praise dinit – we’re talking about systemd, and the (very) loose community of anti-systemd. If the init system were all there was to it, there’d be no controversy. If you’re a masochist, you choose s6; if you’re not, use dinit, or openrc, or runit – or, just stay with systemd. Systemd’s init is super simple; service files are easy to understand, the command line follows convention (start, stop, enable, disable), it has a powerful dependency system that considers devices, it’s fast, it’s easy to understand, and it just works. And that’s my problem with the anti-systemd crowd, because a some of the tooling that systemd has absorbed – the non-init bloat – aren’t desktop-user-friendly.

Systemd is user-friendly. Want to manage services? systemctl. Want to manage cron jobs? systemctl. Want to see logs? journalctl. That’s pretty much what you have to learn; resolved, homed, all that extra stuff is optional (although, admittedly, at this rate, it soon won’t be). Want to manage cron jobs? Still systemctl. Want to see the output of your cron jobs? Still journalctl. Worried about rotating logs? Trick question! With systemd, you don’t have to!

There is a wiki page called “Criticisms of Linux.” It’s quite interesting to read, but one of the quotes is particularly relevant for this essay. It’s from Con Kolvas, a Linux kernel ex-developer, and it says:

If there is any one big problem with kernel development and Linux it is the complete disconnection of the development process from normal users. You know, the ones who constitute 99.9% of the Linux user base. The Linux kernel mailing list is the way to communicate with the kernel developers. To put it mildly, the Linux kernel mailing list (lkml) is about as scary a communication forum as they come. Most people are absolutely terrified of mailing the list lest they get flamed for their inexperience, an inappropriate bug report, being stupid or whatever. … I think the kernel developers at large haven’t got the faintest idea just how big the problems in userspace are.

Systemd addresses this issue, because systemd is easy. It’s a one-stop-shop for all your service needs. It’s exactly one configuration syntax to learn, three commands to learn, and it’s the same on every system. There are GUI tools built around it. It’s largely controllable via dbus, so it’s eminently scriptable.

I won’t lie: one of the things that drove me away from systemd was journald. Logs all in one place is a great thing, but there are issues. The first is that you have to access logs through the command, and the command can be slow. It’s unmarshalling from disk, or uncompressing, or something, but I’ve had times when I’ve tried to read logs on one of my server and journald took tens of seconds to restore and show me the output. This sort of thing is never a problem when logs are stored in /var/log; what is a problem is that you never really know which log file to look in. Again, I’m speaking about desktop users, the ones Con is talking about above, not sysadmins who live in this every day. Where do they start looking for errors when something goes wrong? In /var/log/messages? /var/log/daemon, /var/log/user, /var/log/kernel, or /var/log/errors? If they’re having trouble logging in to X, do they look in /var/log/Xorg.0.log, /var/log/lightdm/x-0.log, /vor/log/lightdm/current, /var/log/lightdm/lightdm.log, or /var/log/lightdm/seat0-greater.log??? All of them? Since I migrated, I can start my desktop by logging into the console and running startx, but I can’t log in through lightdm: it just cycles back to the login, with no message, and no useful logged reason in any file that I can find. If this was journalctl, I’d just list everything and incrementally search, and I’m certain I’d be able to figure out what this issue is if only by looking at the timestamps of every logged message for every subsystem, in one place. Systemd’s log management has problems, but it got the userspace part right.

One of the things I’m missing most is systemd’s cron management. Non-systemd cron offerings suck. dcron is nice and light, but it insists on forking itself, which doesn’t play well with some init servers and violates one of the DAEMON(7) recommendations: make background forking optional¹. cronie is better, but both dcron and cronie both suffer from the inherited “email-as-logging” issue. Both expect to have a working MTA to email the output of jobs to someone. If a user wants to get any output from their cron jobs, to do debugging or whatever, they need to run an MTA service (cronie accepts an -s argument, to log to syslog). It’s inherited behavior from legacy cron, but it only considers servers, and it demonstrates that Con Kolvas’ criticism of Linux kernel community applies equally well to the userspace. Note, again, that systemd doesn’t have this problem. If a user wants to set up a user job, they make a .timer file with an OnCalendar entry. There’s some syntax to learn, but it’s no more convoluted than crontab’s, and systemd jobs log all output to the system log. It doesn’t email log output.

Systemd has problems. Its development process has problems; its anti-Unix-philosophy and monolithic architecture is a problem, and many of its tools are arguably broken (journalctl and resolved). But one thing it does do right is its focus on user-friendliness. It’s easy to write UI tools around; it’s consistent and convenient, and it makes Linux friendlier. And I don’t see that in the anti-systemd space; dinit is a joy, but tools for other behoviors that systemd has absorbed are still designed for sysadmins, not end-users, and this needs to change.