systemd Fundamentals

systemd is the init system and service manager running as PID 1 on nearly every modern Linux distribution (Debian, Ubuntu, RHEL-family, Arch, openSUSE, Fedora). It replaced SysVinit’s “run a shell script per service, sequentially” model with a parallelised, dependency-aware, declarative unit system. Love it or hate it — you interact with it dozens of times a day on any Linux box.

What systemd actually does

systemd is not just an init. It’s a constellation of components with one common config language. The pieces you’ll encounter:

Component	Role
systemd (PID 1)	Start the system, supervise services, reap zombies
journald	Capture stdout/stderr + structured log events
logind	Track user sessions and seats
udevd	React to hardware hotplug events
resolved	Local DNS stub resolver
networkd	Network configuration (alternative to NetworkManager)
timesyncd	SNTP client
timers	Modern cron replacement
nspawn	Lightweight container runtime
homed	Portable user homes

You’re not forced to use all of them — most distros pick a subset. Debian, for instance, still uses NetworkManager on desktops but systemd-networkd on servers.

Units — the central abstraction

A unit is systemd’s name for “a thing it manages.” Everything is a unit. Files live in /etc/systemd/system/ (admin-provided, wins) and /usr/lib/systemd/system/ (package-provided).

Type	Suffix	What it represents
service	.service	A daemon / one-shot command
socket	.socket	A listening socket; launches a service on connection
timer	.timer	Schedule triggering another unit (cron replacement)
mount	.mount	A filesystem mount point
automount	.automount	Mount on first access
path	.path	React to filesystem changes
target	.target	A grouping / milestone (replaces runlevels)
device	.device	A device (managed by udev events)
slice	.slice	A cgroup for resource control
scope	.scope	An externally-created group (e.g. a user session)

You can systemctl cat any unit to see the full resolved file.

A minimal service unit

# /etc/systemd/system/myapp.service
[Unit]
Description=My App
After=network-online.target
Wants=network-online.target
 
[Service]
Type=simple
ExecStart=/usr/local/bin/myapp --config /etc/myapp.yaml
Restart=on-failure
RestartSec=5s
User=myapp
Group=myapp
Environment=LOG_LEVEL=info
WorkingDirectory=/var/lib/myapp
 
[Install]
WantedBy=multi-user.target

Three sections, each doing one thing:

• [Unit] — metadata and ordering (After, Before, Requires, Wants, Conflicts)
• [Service] — how to run the process
• [Install] — where it attaches when enabled (what “enables” it to start at boot)

After editing, always systemctl daemon-reload.

Service types

Type= changes how systemd considers the service “started”:

Type	Meaning
simple (default)	ExecStart runs, and that’s the main process
forking	Old-school daemon that fork()s and parent exits; systemd waits for the fork
oneshot	Run and exit; often paired with RemainAfterExit=yes for “did this thing happen” targets
notify	Service uses sd_notify() to tell systemd when ready — best for proper readiness semantics
exec	Like simple, but ready = execve() returned
idle	Wait until all other jobs finished before starting (nice for last-step stuff)

For modern apps, prefer simple (most apps) or notify (anything that does real startup work).

Dependencies and ordering — the two-axes model

This trips everyone up once. There are two independent axes:

Requirement (if / whether)

• Requires=foo.service — foo must start; if foo fails, we fail too.
• Wants=foo.service — foo is nice to have; we start anyway if foo fails. Prefer this.
• Requisite=foo.service — foo must already be running; don’t start it, just check.
• Conflicts=foo.service — starting us stops foo (and vice-versa).

Ordering (before / after)

• After=foo.service — if both are scheduled, start us after foo is active.
• Before=foo.service — ditto but reversed.

Key insight: Requires= without After= means “start them together, in parallel.” If you need “start foo, then start me,” you need both: Requires=foo.service and After=foo.service. Missing the ordering edge is the #1 systemd dependency bug.

Targets — the “runlevel” replacement

A target is a grouping with no action of its own. Reaching a target means: “every unit that wants this target is active.”

Target	Role
multi-user.target	Console, networking, services — normal server boot
graphical.target	multi-user + display manager — desktop boot
rescue.target	Single-user, minimal services
emergency.target	Even less — root shell only, read-only rootfs
reboot.target, poweroff.target, halt.target	Shutdown variants
network-online.target	Network is configured and reachable (use Wants= + After=, not just After=network.target)

Set the default: systemctl set-default multi-user.target.

journald — structured logging

Every unit’s stdout + stderr is captured by systemd and sent to journald. Logs include a pile of metadata: unit, PID, UID, MESSAGE_ID, priority, cursor, and custom FIELD=value pairs. That’s why journalctl can filter so precisely.

journalctl -u nginx                     # one unit
journalctl -u nginx -f                  # follow (like tail -f)
journalctl --since "2 hours ago"
journalctl --since today --until "1 hour ago"
journalctl -p err                        # priority err or worse (0..7)
journalctl -k                            # kernel log (dmesg)
journalctl -b                            # current boot
journalctl -b -1                         # previous boot
journalctl _PID=1234                     # by structured field
journalctl -u nginx -o json-pretty       # full structured output

Persistence: by default journald keeps logs until a boot or quota fills. Make them persistent:

mkdir -p /var/log/journal
systemctl restart systemd-journald

Or keep them in RAM (Storage=volatile in /etc/systemd/journald.conf) if disks are slow — but you lose logs across reboots.

Caps: SystemMaxUse=1G, SystemKeepFree=500M, etc. in journald.conf.

Timers — better cron

A timer unit triggers a service unit. No more wrapping commands in weird cron escape rules.

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup
 
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true      # run at next boot if missed
RandomizedDelaySec=15m
 
[Install]
WantedBy=timers.target

Pairs with backup.service (oneshot). Enable: systemctl enable --now backup.timer.

Why timers > cron:

• Full journald logging with journalctl -u backup.service
• Dependency ordering (After=network-online.target)
• Persistent=true recovers missed runs across reboots/downtime
• Resource limits (cgroup controls apply)
• systemctl list-timers shows schedule + last/next run
• Calendar spec is richer: OnCalendar=Mon..Fri 09:00, OnCalendar=weekly, OnBootSec=15m

Socket activation

A powerful but underused feature. Define a .socket unit that listens; systemd starts the .service on first connection. Two wins:

1. Lazy start — service only runs when needed.
2. Zero-downtime restarts — systemd holds the socket; the service can restart without dropping in-flight connections.

# /etc/systemd/system/myapp.socket
[Socket]
ListenStream=8080
Accept=no
 
[Install]
WantedBy=sockets.target

The service unit must understand sd_listen_fds() (most Go / Rust / Python / C apps can).

Daily commands (memorise these)

systemctl start|stop|restart|reload <unit>
systemctl enable|disable <unit>              # boot-time start
systemctl enable --now <unit>                 # start + enable in one go
systemctl status <unit>
systemctl is-active <unit>                    # script-friendly
systemctl is-enabled <unit>
systemctl cat <unit>                          # full resolved unit file
systemctl edit <unit>                         # create a drop-in override
systemctl edit --full <unit>                  # edit the unit directly
systemctl daemon-reload                       # reload after editing
systemctl list-units --failed
systemctl list-units --type=service
systemctl list-timers
systemctl list-dependencies <unit>
 
journalctl -xe                                # last logs with explanations
journalctl -u <unit> -f
journalctl -p err -b

Drop-ins — the right way to modify a unit

Don’t edit the file in /usr/lib/systemd/system/ — your next package upgrade wipes it. Instead:

systemctl edit nginx.service

Creates /etc/systemd/system/nginx.service.d/override.conf — a drop-in that overlays the original. Use it to tweak Environment=, ExecStart= (prefix with empty ExecStart= to reset), Restart=, limits, etc.

Resource control (cgroups)

Every unit runs in a cgroup. Systemd exposes cgroup v2 controls directly in unit files:

[Service]
CPUQuota=50%
MemoryMax=512M
IOReadBandwidthMax=/var/lib/myapp 50M
TasksMax=200

Useful for “this service is a known CPU hog; contain it” without touching Docker.

Security hardening directives

systemd ships a lot of sandboxing knobs. Worth enabling on any service that doesn’t need unrestricted root:

[Service]
NoNewPrivileges=true
ProtectSystem=strict          # /usr, /boot read-only
ProtectHome=true              # /home, /root hidden
PrivateTmp=true               # private /tmp namespace
PrivateDevices=true           # no /dev except essentials
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

Check what’s enforced with systemd-analyze security <unit> — it grades each unit.

Boot analysis

systemd-analyze                           # boot time breakdown
systemd-analyze blame                      # slowest units
systemd-analyze critical-chain             # longest dependency chain
systemd-analyze plot > boot.svg            # visual timeline

Gold for “why is my server so slow to boot?”

Pitfalls I see repeatedly

1. Forgetting daemon-reload after editing a unit → systemd still runs the old version.
2. Requires= without After= — dependency starts but not before.
3. After=network.target when you meant After=network-online.target — network.target fires before routes are up.
4. Editing the package unit in /usr/lib/systemd/system/ instead of a drop-in → lost on upgrade.
5. Type=simple on a forking daemon — systemd thinks it started, but the real daemon is a grandchild it doesn’t own. Use Type=forking with PIDFile= or convert the app to Type=notify.
6. Logging to a file from inside the service → duplicates with journald. Just use stdout/stderr.

See also

• Linux Fundamentals
• Linux Permissions and Users
• Observability
• 🐧 Linux MOC