Run a Linux server with confidence: users and permissions, services, packages, logs, SSH keys, bash scripts and scheduled jobs.

Linux Administration

Run `ps aux | head` and `ps -ef --forest | head -30`. Find PID 1 and note which program it is.

Open `top`, press `M` to sort by memory and `P` to sort by CPU, note the load average and compare it with `nproc`. Press `q` to quit.

Start a harmless background process with `sleep 600 &`, find it with `pgrep -a sleep`, stop it with `kill <PID>` and confirm it is gone.

Install a web server: `sudo apt update && sudo apt install -y nginx` (Red Hat family: `sudo dnf install -y nginx`, then `sudo systemctl enable --now nginx`).

Run `systemctl status nginx` and read the Loaded and Active lines. Test it with `curl -I http://localhost`.

Stop it with `sudo systemctl stop nginx`, run curl again and note the error, then start it again with `sudo systemctl start nginx`.

Check boot behaviour with `systemctl is-enabled nginx`, then disable and re-enable it with `sudo systemctl disable nginx` and `sudo systemctl enable nginx`.

View the unit file with `systemctl cat nginx` and find its ExecStart line.

List failed units with `systemctl --failed`. When you are done, leave nginx installed for the logs lesson or remove it with `sudo apt remove nginx`.

## Why it matters

A server exists to run programs: a web server, a database, an API, a monitoring agent. An administrator needs to answer "is it running?", "why is the machine slow?" and "how do I restart it safely?" many times a day. Those questions are answered with process tools and with **systemd**, the service manager on almost every modern distro.

## Processes

Every running program is a **process** with a numeric **process ID (PID)**, an owner (a user) and a parent process that started it.

```bash
ps aux | head            # every process: user, PID, CPU%, MEM%, command
ps aux | grep sshd       # find one by name
pgrep -a ssh             # PIDs and command lines matching a name
ps -ef --forest | less   # show the parent/child tree
```

PID 1 is special: it is the first process the kernel starts, and on modern distros it is `systemd`. Everything else descends from it.

### Watching live: top and htop

```bash
top
```

`top` updates every few seconds. The header shows **load average** (how many processes wanted CPU over the last 1, 5 and 15 minutes; compare it with the number of CPUs from `nproc`), memory and swap use. Inside `top`, press `P` to sort by CPU, `M` to sort by memory and `q` to quit. `htop` (install it with your package manager) is a friendlier version with colours and scrolling.

### Stopping a process: signals

You stop a process by sending it a **signal**:

```bash
kill 4321          # sends SIGTERM (15): "please exit cleanly"
kill -9 4321       # sends SIGKILL (9): the kernel ends it immediately
pkill -f myscript  # signal by name instead of PID
```

Always try plain `kill` first. `SIGTERM` lets the program save data and close files. `kill -9` cannot be caught, so the program gets no chance to clean up; use it only when a process ignores `SIGTERM`. You can signal your own processes freely; other users' processes need `sudo`.

### Foreground and background jobs

In an interactive shell, `Ctrl+C` interrupts the foreground program, `Ctrl+Z` pauses it, `bg` resumes it in the background and `fg` brings it back. Adding `&` starts a command in the background. These jobs end when you log out, which is why real services are run by systemd instead.

## systemd and units

**systemd** starts the system at boot and supervises long-running services. Each thing it manages is a **unit**, described by a unit file. Service units end in `.service`, for example `ssh.service` or `cron.service`. Vendor unit files live in `/usr/lib/systemd/system/`; your local overrides go in `/etc/systemd/system/`.

The everyday commands:

```bash
systemctl status ssh             # running? since when? main PID? last log lines
sudo systemctl start nginx       # start now
sudo systemctl stop nginx        # stop now
sudo systemctl restart nginx     # stop then start (drops connections)
sudo systemctl reload nginx      # re-read config without a full restart (if supported)
sudo systemctl enable nginx      # start automatically at boot
sudo systemctl disable nginx     # do not start at boot
systemctl is-active nginx        # prints active / inactive
systemctl is-enabled nginx       # prints enabled / disabled
systemctl list-units --type=service --state=running
systemctl --failed               # units that crashed or failed to start
```

Two ideas people mix up: **start/stop** changes what is running *right now*; **enable/disable** changes what happens *at the next boot*. `sudo systemctl enable --now nginx` does both.

### Reading `systemctl status`

```text
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-10-22 09:14:03 UTC; 2h ago
   Main PID: 812 (nginx)
      Tasks: 3 (limit: 4558)
     Memory: 4.1M
```

`Loaded` tells you where the unit file is and whether it is enabled. `Active` tells you whether it is running and since when. If a service failed, the last few log lines appear underneath, which is often enough to see why.

### A peek at a unit file

```bash
systemctl cat cron
```

```ini
[Unit]
Description=Regular background program processing daemon

[Service]
ExecStart=/usr/sbin/cron -f $EXTRA_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

`ExecStart` is the command systemd runs, `Restart=on-failure` brings it back if it crashes, and `WantedBy` is what `enable` hooks it into. After editing any unit file, run `sudo systemctl daemon-reload` so systemd notices.

## Worked example: install and manage a web server

```bash
sudo apt install nginx            # Ubuntu enables and starts it on install
systemctl status nginx
curl -I http://localhost          # HTTP/1.1 200 OK
sudo systemctl stop nginx
curl -I http://localhost          # connection refused
sudo systemctl start nginx
```

## Common mistakes

- **Reaching for `kill -9` first.** Try `systemctl stop` for services and plain `kill` for processes.
- **Killing a service's process instead of stopping the service.** With `Restart=on-failure` or `always`, systemd may just start it again. Use `systemctl stop`.
- **Enabling but not starting** (or starting but not enabling), then being surprised after a reboot.
- **Editing a unit file and forgetting `daemon-reload`.**
- **Restarting when reload would do,** which drops users' connections on a busy server.

## Recap

- Every program is a process with a PID and an owner; `ps`, `pgrep` and `top` inspect them.
- `kill` sends SIGTERM (clean exit); `kill -9` is a last resort.
- systemd (PID 1) manages services as units; `systemctl start/stop/restart/reload/status` controls them now, `enable/disable` controls boot.
- `systemctl --failed` and `systemctl status` are the first stops when something is down.

## Try it

The Do tab walks you through installing a small web server on your practice system and controlling it with systemctl.


Inspect running processes with ps and top, stop them with kill, and manage services with systemctl.

Processes and systemd

Linux Crash Course - systemd: Using the systemctl command

Linux Crash Course - The ps Command

Refresh the package lists with `sudo apt update` and read the last line: how many packages can be upgraded?

Run `apt list --upgradable` and pick one package from the list. Look it up with `apt show <name>`.

Local VM: take a snapshot. Then run `sudo apt upgrade` and read the summary before typing Y. Note whether anything would be removed.

Check whether a reboot is needed with `[ -f /var/run/reboot-required ] && cat /var/run/reboot-required || echo no reboot needed`. Reboot the practice system if it is.

Search for and install a small tool: `apt search tree`, then `sudo apt install tree`. Run `tree -L 1 /etc | head`.

Inspect it: list its files with `dpkg -L tree` and find which package owns `/usr/bin/tree` with `dpkg -S /usr/bin/tree`.

See where your packages come from: `ls /etc/apt/sources.list.d/` and `apt policy tree`.

Check automatic updates with `systemctl status unattended-upgrades` and `cat /etc/apt/apt.conf.d/20auto-upgrades`.

Remove the tool and clean up: `sudo apt purge tree` then `sudo apt autoremove`.

## Why it matters

On Linux you rarely download an installer from a website. Software comes from **repositories** through a **package manager**, which installs programs together with everything they depend on, keeps them updated, and removes them cleanly. Keeping packages patched is also one of the most important security jobs on any server: many real breaches exploit a vulnerability that already had a fix available.

## Packages, repositories and dependencies

- A **package** is an archive with a program, its files and metadata: version, description and the other packages it needs (its **dependencies**).
- A **repository** is a server that hosts packages. Your system's list of repositories lives in `/etc/apt/sources.list` and `/etc/apt/sources.list.d/` on Debian/Ubuntu, or `/etc/yum.repos.d/` on the Red Hat family.
- Repositories **sign** their package lists with a cryptographic key. The package manager refuses packages whose signature does not match, which protects you from tampered downloads.

## apt on Debian and Ubuntu

```bash
sudo apt update                 # refresh the list of available packages (installs nothing)
apt list --upgradable           # what could be updated
sudo apt upgrade                # install available updates
sudo apt install tree htop      # install packages and their dependencies
apt search json                 # search names and descriptions
apt show htop                   # details: version, size, dependencies
sudo apt remove htop            # remove the program, keep its config files
sudo apt purge htop             # remove the program and its config
sudo apt autoremove             # remove dependencies nothing needs any more
```

The most common beginner confusion: **`apt update` does not update software**. It downloads the latest package *lists*. `apt upgrade` installs the new versions. You normally run them together: `sudo apt update && sudo apt upgrade`.

Underneath, `apt` uses **dpkg**, which handles single `.deb` files:

```bash
dpkg -l | grep nginx            # is it installed, and which version?
dpkg -L tree                    # which files did this package install?
dpkg -S /usr/bin/htop           # which package owns this file?
```

## dnf on the Red Hat family

The ideas are identical; the command names differ slightly.

| Task | Debian/Ubuntu | RHEL/Fedora/Rocky |
|---|---|---|
| Refresh lists | `sudo apt update` | (automatic; `sudo dnf check-update`) |
| Install | `sudo apt install pkg` | `sudo dnf install pkg` |
| Update all | `sudo apt upgrade` | `sudo dnf upgrade` |
| Remove | `sudo apt remove pkg` | `sudo dnf remove pkg` |
| Search | `apt search word` | `dnf search word` |
| Owner of a file | `dpkg -S /path` | `rpm -qf /path` |

## Installing software safely

1. **Prefer the distro's own repositories.** Their packages are tested together and receive security updates automatically.
2. **Read what apt is about to do.** Before you answer `Y`, apt lists what it will install, upgrade and *remove*. If an install wants to remove many packages, stop and find out why.
3. **Add third-party repositories only from the vendor's official instructions** (for example PostgreSQL, Docker or Node.js docs), and understand that you are trusting their signing key with root on your machine.
4. **Be wary of `curl ... | sudo bash`.** It runs a script from the internet as root without letting you read it. Download it, read it, then decide.
5. **Do not mix in random `.deb` or `.rpm` files** from file-sharing sites.
6. **Snapshot first** on your practice VM before big upgrades, and on real servers follow the change process (test environment first, a maintenance window, a rollback plan).

## Automatic security updates

Ubuntu servers usually run **unattended-upgrades**, which installs security updates daily:

```bash
systemctl status unattended-upgrades
cat /etc/apt/apt.conf.d/20auto-upgrades
```

Some updates (for example a new kernel) only take effect after a reboot. Ubuntu creates `/var/run/reboot-required` when that is the case:

```bash
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required
```

## Other formats you will meet

- **Snap** and **Flatpak** bundle an app with its own libraries; common on desktops.
- **Language package managers** (`pip`, `npm`, `cargo`) install libraries for a programming language. Use them inside a virtual environment or project folder, not with `sudo` into the system Python, so they do not fight with the OS package manager.
- **Containers** package a whole application environment; you will meet them in later cloud and DevOps courses.

## Common mistakes

- **Running `apt upgrade` without `apt update` first**, so nothing new is found.
- **Answering `Y` without reading the list** of packages to be removed.
- **`sudo pip install` into the system Python**, which can break OS tools that depend on it.
- **Never rebooting after kernel updates**, leaving the vulnerable kernel running.
- **Adding a random PPA or repository** from a blog post to get a newer version.

## Recap

- Software comes from signed repositories through a package manager that resolves dependencies.
- `apt update` refreshes lists; `apt upgrade` installs updates; `apt install/remove/purge/autoremove` manage packages (dnf on the Red Hat family).
- `dpkg -L` and `dpkg -S` connect packages and files.
- Stay with official repositories, read before confirming, avoid piping scripts into root, and reboot when updates require it.

## Try it

In the Do tab you will update your practice system and install, inspect and remove a package.


Install, update and remove software safely with apt (and dnf), and understand repositories and signed packages.

Package management

Linux Crash Course - The apt Command

The apt Package Manager | Top Docs with Jay LaCroix

Back up the config first: `sudo cp /etc/nginx/sites-available/default ~/default.backup`.

Look at recent logs: `journalctl -u nginx -n 20 --no-pager` and `sudo tail -n 20 /var/log/syslog` (Red Hat family: `/var/log/messages`).

Open the config with `sudo nano /etc/nginx/sites-available/default`, change one `server_name` to `server_nme`, and save.

Restart with `sudo systemctl restart nginx` and note the time and the error printed.

Find the cause with `journalctl -u nginx --since "5 min ago" --no-pager` and write down the file and line number it names.

Confirm with the program's own check: `sudo nginx -t`.

Restore the config with `sudo cp ~/default.backup /etc/nginx/sites-available/default`, run `sudo nginx -t` again, then `sudo systemctl restart nginx` and `systemctl status nginx`.

List this boot's errors with `journalctl -p err -b --no-pager | tail -20`, and check the journal's size with `journalctl --disk-usage`.

Count failed SSH logins (may be zero on a local VM): `sudo grep -c "Failed password" /var/log/auth.log`.

## Why it matters

When something breaks, the log usually says why. A service that will not start, a disk that is full, a failed login, a crashed process: each leaves a message with a timestamp. The skill that separates a confident administrator from a guesser is going straight to the right log, filtering it down to the few relevant lines, and reading them carefully.

## Two places logs live

1. **The systemd journal**, read with `journalctl`. systemd captures the output of every service it runs, plus kernel and boot messages, and stores them in a structured, indexed form.
2. **Text files in `/var/log`**, written by `rsyslog` and by applications themselves: `syslog`, `auth.log`, `kern.log`, `dpkg.log`, `nginx/error.log` and so on. On the Red Hat family, the main file is `/var/log/messages` and logins go to `/var/log/secure`.

Many systems have both: services log to the journal and rsyslog copies messages into text files. Reading most system logs needs `sudo` or membership of the `adm` (Debian/Ubuntu) or `systemd-journal` group.

## journalctl essentials

```bash
journalctl -u nginx                 # everything from one service
journalctl -u nginx -n 50           # the last 50 lines
journalctl -u nginx -f              # follow new lines live (Ctrl+C to stop)
journalctl -u nginx --since "1 hour ago"
journalctl --since "2026-10-23 09:00" --until "2026-10-23 10:00"
journalctl -p err -b                # errors and worse, since this boot
journalctl -b -1                    # the previous boot (if the journal is persistent)
journalctl -k                       # kernel messages (like dmesg)
journalctl --disk-usage             # how much space the journal uses
```

**Priorities** go from 0 (emerg) to 7 (debug): emerg, alert, crit, **err**, **warning**, notice, info, debug. `-p err` shows `err` and everything more severe, which is a great first filter.

Combine filters freely: `journalctl -u ssh -p warning --since today`.

## Text logs essentials

```bash
sudo tail -n 50 /var/log/syslog          # the end of the file
sudo tail -f /var/log/auth.log           # follow live
sudo less +G /var/log/syslog             # open at the end; / to search, q to quit
sudo grep -i "error" /var/log/syslog | tail
sudo grep "Failed password" /var/log/auth.log | wc -l
zgrep "Failed password" /var/log/auth.log.2.gz   # search a compressed old log
```

Old logs are **rotated** by `logrotate`: `auth.log` becomes `auth.log.1`, then `auth.log.2.gz`, and so on, and the oldest are deleted. That keeps `/var/log` from filling the disk. Rules live in `/etc/logrotate.conf` and `/etc/logrotate.d/`.

## Reading a log line

```text
Oct 23 10:42:17 web1 sshd[2211]: Failed password for invalid user admin from 203.0.113.50 port 51122 ssh2
```

| Part | Meaning |
|---|---|
| `Oct 23 10:42:17` | when (check the server's time zone with `timedatectl`) |
| `web1` | which host |
| `sshd[2211]` | which program and its PID |
| the rest | the message: a login attempt for a user that does not exist, from that IP |

## Worked example: why will the service not start?

You break nginx on purpose by adding a typo to its configuration, then restart it:

```bash
sudo systemctl restart nginx
# Job for nginx.service failed because the control process exited with error code.
systemctl status nginx          # shows "failed" and the last few lines
journalctl -u nginx -n 20 --no-pager
```

The journal shows a line like:

```text
nginx: [emerg] unknown directive "server_nme" in /etc/nginx/sites-enabled/default:22
```

That message names the file, the line number and the mistake. Fix the typo, check the config with the program's own test (`sudo nginx -t`), then restart. Many services have a similar check: `sshd -t`, `apachectl configtest`, `named-checkconf`.

### A method that works

1. **Reproduce** the problem and note the exact time.
2. **Narrow** by unit and time: `journalctl -u <service> --since "5 min ago"`.
3. **Filter by severity:** `-p err`, or `grep -iE "error|fail|denied"`.
4. **Read the first error**, not the last. Later errors are often consequences of the first one.
5. **Search the exact message** in the official docs if it is unfamiliar.

## Keeping the journal

On some systems the journal lives only in memory (`/run/log/journal`) and is lost at reboot. If `/var/log/journal` exists, it is persistent. Limit its size in `/etc/systemd/journald.conf` with `SystemMaxUse=`, or clean up old entries with `sudo journalctl --vacuum-time=2weeks`.

## Common mistakes

- **Reading only the last line.** The useful error is often several lines up.
- **Ignoring time zones,** so you look at the wrong hour.
- **Deleting log files to free space.** Use logrotate and `journalctl --vacuum-*` instead.
- **Forgetting `sudo`** and concluding a log is empty when you simply cannot read it.
- **Posting logs publicly** without removing IP addresses, usernames, hostnames or tokens.

## Recap

- Logs live in the systemd journal (`journalctl`) and in text files under `/var/log`.
- `journalctl -u <unit>`, `-f`, `--since`, `-p err` and `-b` narrow the journal quickly.
- `tail`, `less`, `grep` and `zgrep` read text and rotated logs.
- Find the first error, read the whole message, and use the service's own config test before restarting.

## Try it

The Do tab has you break a service on your practice system on purpose and find the error in the logs.


Find the cause of an error with journalctl, /var/log and a few filtering commands.

Logs and journald

journalctl Basics: How to Easily Check Your Linux Logs

Understanding Linux Logging: Var Log, Journal, and Logger

Manage processes and systemd services, install and patch software safely, and find errors in the logs.