Logs and journald

Why it matters

When something breaks, the log usually says why. A service that will not start, a disk that is full, a failed login, a crashed process: each leaves a message with a timestamp. The skill that separates a confident administrator from a guesser is going straight to the right log, filtering it down to the few relevant lines, and reading them carefully.

Two places logs live

1. The systemd journal, read with journalctl. systemd captures the output of every service it runs, plus kernel and boot messages, and stores them in a structured, indexed form.
2. Text files in /var/log, written by rsyslog and by applications themselves: syslog, auth.log, kern.log, dpkg.log, nginx/error.log and so on. On the Red Hat family, the main file is /var/log/messages and logins go to /var/log/secure.

Many systems have both: services log to the journal and rsyslog copies messages into text files. Reading most system logs needs sudo or membership of the adm (Debian/Ubuntu) or systemd-journal group.

journalctl essentials

journalctl -u nginx                 # everything from one service
journalctl -u nginx -n 50           # the last 50 lines
journalctl -u nginx -f              # follow new lines live (Ctrl+C to stop)
journalctl -u nginx --since "1 hour ago"
journalctl --since "2026-10-23 09:00" --until "2026-10-23 10:00"
journalctl -p err -b                # errors and worse, since this boot
journalctl -b -1                    # the previous boot (if the journal is persistent)
journalctl -k                       # kernel messages (like dmesg)
journalctl --disk-usage             # how much space the journal uses

Priorities go from 0 (emerg) to 7 (debug): emerg, alert, crit, err, warning, notice, info, debug. -p err shows err and everything more severe, which is a great first filter.

Combine filters freely: journalctl -u ssh -p warning --since today.

Text logs essentials

sudo tail -n 50 /var/log/syslog          # the end of the file
sudo tail -f /var/log/auth.log           # follow live
sudo less +G /var/log/syslog             # open at the end; / to search, q to quit
sudo grep -i "error" /var/log/syslog | tail
sudo grep "Failed password" /var/log/auth.log | wc -l
zgrep "Failed password" /var/log/auth.log.2.gz   # search a compressed old log

Old logs are rotated by logrotate: auth.log becomes auth.log.1, then auth.log.2.gz, and so on, and the oldest are deleted. That keeps /var/log from filling the disk. Rules live in /etc/logrotate.conf and /etc/logrotate.d/.

Reading a log line

Oct 23 10:42:17 web1 sshd[2211]: Failed password for invalid user admin from 203.0.113.50 port 51122 ssh2

Worked example: why will the service not start?

You break nginx on purpose by adding a typo to its configuration, then restart it:

sudo systemctl restart nginx
# Job for nginx.service failed because the control process exited with error code.
systemctl status nginx          # shows "failed" and the last few lines
journalctl -u nginx -n 20 --no-pager

The journal shows a line like:

nginx: [emerg] unknown directive "server_nme" in /etc/nginx/sites-enabled/default:22

That message names the file, the line number and the mistake. Fix the typo, check the config with the program's own test (sudo nginx -t), then restart. Many services have a similar check: sshd -t, apachectl configtest, named-checkconf.

A method that works

1. Reproduce the problem and note the exact time.
2. Narrow by unit and time: journalctl -u <service> --since "5 min ago".
3. Filter by severity: -p err, or grep -iE "error|fail|denied".
4. Read the first error, not the last. Later errors are often consequences of the first one.
5. Search the exact message in the official docs if it is unfamiliar.

Keeping the journal

On some systems the journal lives only in memory (/run/log/journal) and is lost at reboot. If /var/log/journal exists, it is persistent. Limit its size in /etc/systemd/journald.conf with SystemMaxUse=, or clean up old entries with sudo journalctl --vacuum-time=2weeks.

Common mistakes

• Reading only the last line. The useful error is often several lines up.
• Ignoring time zones, so you look at the wrong hour.
• Deleting log files to free space. Use logrotate and journalctl --vacuum-* instead.
• Forgetting sudo and concluding a log is empty when you simply cannot read it.
• Posting logs publicly without removing IP addresses, usernames, hostnames or tokens.

Recap

• Logs live in the systemd journal (journalctl) and in text files under /var/log.
• journalctl -u <unit>, -f, --since, -p err and -b narrow the journal quickly.
• tail, less, grep and zgrep read text and rotated logs.
• Find the first error, read the whole message, and use the service's own config test before restarting.

Try it

The Do tab has you break a service on your practice system on purpose and find the error in the logs.