Cloud engineers, database administrators, cybersecurity engineers and machine learning engineers all work on Linux servers that have no desktop, only a terminal. Building on Git & the Command Line and Networking Fundamentals, this intermediate course moves from using a shell to running a system. You start by setting up a safe practice system (a local virtual machine, WSL, or a free-tier cloud VM with a budget alert and a cleanup plan), learn the filesystem hierarchy, and control access with users, groups, chmod and chown. The second module covers what keeps a server running: processes and systemd services, installing and patching software with apt and dnf, and finding the cause of a failure in journald and /var/log. The last module covers remote access and automation: SSH key logins and safe hardening, a bash backup script that checks its inputs and verifies its output, and scheduling it with cron or a systemd timer. Every hands-on task runs on your own practice system, never on a work or production machine.

Run a Linux server with confidence: users and permissions, services, packages, logs, SSH keys, bash scripts and scheduled jobs.

Linux Administration

Choose your option: a local VM (recommended), WSL on Windows (`wsl --install` in an administrator PowerShell, then restart), or a free-tier cloud VM.

Local VM: download the Ubuntu Server LTS ISO from ubuntu.com and create a VM with 2 CPUs, 2–4 GB RAM and a 20 GB or larger disk. Cloud VM: create a billing budget alert first, then launch the smallest free-tier eligible Ubuntu LTS instance.

Install or launch Ubuntu and create a normal user account with a password you will remember.

Log in and run `cat /etc/os-release`, `uname -r` and `hostnamectl`. Note the distro version and kernel version.

Run `free -h` and `df -h`. Note how much memory and free disk space your system has.

Run `sudo apt update` and confirm sudo asks for your password and the package lists download without errors.

Local VM: take a snapshot named "clean install" so you can roll back later. WSL: note that `wsl --export` can back up the distro. Cloud VM: write down every resource you created (instance, disk, public IP).

Cloud VM only: when you finish each study session, stop the instance; when you finish the course, delete the instance, its disk and its public IP, and confirm the billing page shows nothing running.

## Why it matters

Most of the internet runs on Linux. Web servers, databases, cloud virtual machines, containers, network appliances and the GPU machines that train machine learning models are almost all Linux systems. Cloud engineers, database administrators, security engineers and ML engineers spend a large part of their day logged in to Linux machines that have no desktop at all, only a terminal.

You already know the basics of the shell from Git & the Command Line. This course moves from "using a terminal" to "running a system": users and permissions, services, packages, logs, remote access and automation. To practise that safely you need a Linux system of your own that you are allowed to break.

## What a distribution is

Linux itself is only the **kernel**: the core program that talks to the hardware, schedules processes and manages memory. A **distribution** (distro) bundles the kernel with everything else you need: a shell, core tools, a package manager, a service manager and default settings.

Distros come in families that share a package format and most commands:

| Family | Examples | Package tool | Where you meet it |
|---|---|---|---|
| Debian | Debian, Ubuntu, Linux Mint | `apt` (`.deb`) | Cloud VMs, laptops, most tutorials |
| Red Hat | RHEL, Fedora, Rocky Linux, AlmaLinux | `dnf` (`.rpm`) | Enterprise servers, banks, government |
| Others | Arch, Alpine, SUSE | `pacman`, `apk`, `zypper` | Hobbyists, tiny containers, some enterprises |

This course uses **Ubuntu Server LTS** for its examples. "LTS" means long-term support: about five years of security updates, which is why it is so common on servers. Almost everything you learn transfers to the other families; where a command differs (for example `apt` versus `dnf`), the lesson says so.

## Three safe ways to get a practice system

Do all of the work in this course on a practice system, **never on a work laptop, a school machine or any production server**. Administration commands can delete data, lock you out or stop services that other people rely on. On your own throwaway system, a mistake only costs you a reinstall.

1. **A local virtual machine (recommended).** A hypervisor such as VirtualBox (Windows, Linux, Intel Macs) or UTM (Apple silicon Macs) runs a whole computer inside a window. Download the Ubuntu Server ISO from ubuntu.com, create a VM with 2 CPUs, 2–4 GB RAM and a 20–25 GB disk, and install. The big advantage is **snapshots**: save the VM's state before a risky step and roll back if it goes wrong.
2. **WSL on Windows.** The Windows Subsystem for Linux runs a real Ubuntu userland beside Windows. In an administrator PowerShell run `wsl --install`, restart, and create a Linux username and password. WSL is quick to set up. Recent versions support systemd, but a few service lessons behave differently from a full VM, so a VM is still the better choice if your computer can run one.
3. **A free-tier cloud VM.** AWS, Google Cloud, Azure and Oracle Cloud offer small free-tier instances. This is the closest to real work, but it is a machine on the public internet and it can cost money if you pick the wrong size or forget to delete it. If you go this way, set a budget alert first, choose a size labelled "free tier eligible", and delete the instance (and its disk and public IP) when you finish.

## Your first look around

After installing, log in and confirm what you are running:

```bash
cat /etc/os-release     # distro name and version
uname -r                # kernel version
hostnamectl             # hostname, OS, kernel, architecture
whoami                  # your username
free -h                 # memory in human-readable units
df -h                   # disk usage per filesystem
```

On Ubuntu, `/etc/os-release` shows something like `PRETTY_NAME="Ubuntu 24.04.1 LTS"`. The kernel version from `uname -r` is separate from the distro version, because the kernel is a separate project.

### What sudo is

Your user is a normal account. Administrative commands need root privileges, which you get one command at a time with `sudo`:

```bash
sudo apt update
```

`sudo` asks for **your** password, logs the command and runs it as root. Use it only when a command needs it, and only on your practice system. A good habit: read the full command before pressing Enter whenever it starts with `sudo`.

## Common mistakes

- **Practising on a real machine.** Even "harmless" changes to permissions or services can break a shared or work system. Use the VM.
- **Skipping snapshots.** Take one right after a clean install, and before any lesson that changes SSH or services. Rolling back takes seconds.
- **Too little disk.** A 10 GB disk fills quickly once you install packages. Give the VM at least 20 GB.
- **Forgetting the cloud VM.** A stopped instance can still bill for its disk and public IP. Delete what you created.
- **Running everything as root.** Logging in as root, or typing `sudo -i` and staying there, removes your safety net. Stay a normal user and use `sudo` per command.

## Recap

- Linux is the kernel; a distribution adds tools, a package manager and defaults.
- Debian/Ubuntu (`apt`) and Red Hat family (`dnf`) cover most servers you will meet.
- Practise on a local VM (best), WSL, or a free-tier cloud VM with a budget alert and a cleanup plan.
- `cat /etc/os-release`, `uname -r`, `free -h` and `df -h` tell you what you are working on.
- `sudo` runs one command as root; use it deliberately and only on your practice system.

## Try it

Open the Do tab and set up the practice system you will use for the rest of the course.


Choose a Linux distribution and set up a safe practice system in a virtual machine, WSL or a free-tier cloud VM.

Distros and installing Linux

How to Install Ubuntu 24.04 LTS on VirtualBox in Windows 11

WSL 2: Getting started

Run `ls /` and match each top-level folder you see to its role from the lesson table.

Find where the `ls`, `python3` and `systemctl` programs live with `which ls python3 systemctl`.

List the SSH configuration folder with `ls /etc/ssh/` and view the start of the hosts file with `cat /etc/hosts`.

Find every file named `*.conf` directly in `/etc` with `find /etc -maxdepth 1 -name "*.conf" 2>/dev/null` and count them by adding `| wc -l`.

Run `ls -lh /var/log` and identify the log that records logins and sudo use (on Ubuntu, `auth.log`).

Read two kernel-generated files: `head -5 /proc/meminfo` and `grep -c processor /proc/cpuinfo`. Compare the numbers with `free -h` and `nproc`.

Check disk use with `df -h /` and find the size of the log folder with `sudo du -sh /var/log`.

Find the three biggest folders inside `/var` with `sudo du -sh /var/* 2>/dev/null | sort -h | tail -3`.

## Why it matters

When a service misbehaves on a server, the questions are always the same: where is its configuration, where are its logs, and which program is actually running? On Linux the answers follow a shared map called the **Filesystem Hierarchy Standard (FHS)**. Once you know the map, you can find your way around almost any Linux machine, including one you have never seen before.

## One tree, starting at /

Windows has drive letters (`C:\`, `D:\`). Linux has a single tree that starts at the **root directory**, written `/`. Extra disks, USB drives and network shares are **mounted** somewhere inside that tree instead of getting their own letter. Run `ls /` and you will see the top level.

| Directory | What it holds | Example |
|---|---|---|
| `/etc` | System-wide configuration, mostly plain text | `/etc/ssh/sshd_config`, `/etc/hosts` |
| `/var` | Data that changes while the system runs | `/var/log/` (logs), `/var/lib/` (databases, package state) |
| `/usr` | Installed programs and their read-only files | `/usr/bin/python3`, `/usr/share/doc/` |
| `/bin`, `/sbin` | Essential commands (on modern distros, links into `/usr`) | `/bin/ls` |
| `/home` | One folder per normal user | `/home/alex` |
| `/root` | The root user's home folder | |
| `/tmp` | Temporary files, often cleared on reboot | |
| `/opt` | Optional, self-contained third-party software | `/opt/someapp/` |
| `/dev` | Device files for disks, terminals and more | `/dev/sda`, `/dev/null` |
| `/proc`, `/sys` | Virtual files the kernel generates about processes and hardware | `/proc/cpuinfo` |
| `/boot` | Kernel and bootloader files | |
| `/mnt`, `/media` | Mount points for extra or removable drives | |

A useful rule of thumb: **config in `/etc`, changing data and logs in `/var`, programs in `/usr`, people in `/home`.**

## "Everything is a file"

Linux exposes a lot of the system as files. Your disk is `/dev/sda` (or `/dev/vda` in many VMs). `/dev/null` is a black hole that discards whatever is written to it. `/proc/meminfo` is not stored on disk at all: the kernel generates its contents when you read it.

```bash
cat /proc/cpuinfo | grep "model name" | head -1
cat /proc/meminfo | head -3
```

This is why so many admin tasks come down to reading or editing a text file.

## Finding things

### Where is a program?

```bash
which python3          # path of the command your shell would run
type ls                # also shows aliases and shell built-ins
```

`which python3` prints something like `/usr/bin/python3`.

### Where is a config file?

Configuration for a program is usually in `/etc/<program>` or `/etc/<program>.conf`:

```bash
ls /etc/ssh/
ls /etc | head -30
```

When you do not know the name, search by name with `find`. Searching system folders produces "Permission denied" noise for folders you cannot read, so send errors to `/dev/null`:

```bash
find /etc -name "*.conf" 2>/dev/null | head
find / -name "sshd_config" 2>/dev/null
```

### Where are the logs?

Traditional text logs live in `/var/log`:

```bash
ls -lh /var/log
```

On Ubuntu you will see `syslog`, `auth.log` (logins and sudo), `dpkg.log` (package installs) and folders for individual services. Many systems also keep logs in the systemd journal, which a later lesson covers.

### What is using my disk?

```bash
df -h                       # free space per mounted filesystem
du -sh /var/log             # total size of one folder
sudo du -sh /var/* 2>/dev/null | sort -h | tail -5
```

`df` answers "how full is each disk?" and `du` answers "which folder is big?". A full disk is one of the most common causes of a broken server, and `/var/log` is a frequent culprit.

## Absolute and relative paths, again

An **absolute path** starts at `/` and works from anywhere: `/etc/hosts`. A **relative path** starts from your current folder: `../notes.txt`. In scripts and config files, prefer absolute paths, because the "current folder" of a script run by a scheduler is often not what you expect.

## Common mistakes

- **Editing files in `/usr`.** Package updates overwrite them. Put local configuration in `/etc` and your own scripts in `/usr/local/bin` or your home folder.
- **Deleting files in `/var/log` to free space** while a service has them open. The space is not freed until the service restarts. Use the log rotation and journal tools you will meet later instead.
- **Treating `/tmp` as storage.** It can be wiped at reboot.
- **Running `find /` without silencing errors**, then missing the real result in a wall of "Permission denied".

## Recap

- Linux has one tree rooted at `/`; extra drives are mounted into it.
- Config lives in `/etc`, logs and changing data in `/var`, programs in `/usr`, users in `/home`.
- `/proc`, `/sys` and `/dev` expose the kernel and devices as files.
- `which`, `find`, `ls /var/log`, `df -h` and `du -sh` find programs, configs, logs and disk hogs.

## Try it

The Do tab has a scavenger hunt through your practice system's filesystem.


Learn what lives in /etc, /var, /usr, /home and the other top-level folders, and find any config, log or program.

The filesystem hierarchy

Linux Crash Course: Navigating the Linux Filesystem

Linux Directories Explained in 100 Seconds

Run `id` and `ls -l ~` and read your uid, groups and the permission string of one file.

Create a test user and a group: `sudo adduser maria` (any password), then `sudo groupadd webteam`.

Add both users to the group with `sudo usermod -aG webteam maria` and `sudo usermod -aG webteam $USER`, then confirm with `groups maria`.

Create the shared folder: `sudo mkdir -p /srv/webteam`, `sudo chown root:webteam /srv/webteam`, `sudo chmod 2770 /srv/webteam`. Check it with `ls -ld /srv/webteam`.

Log out and back in (or run `newgrp webteam`), then create a file with `touch /srv/webteam/plan.txt` and confirm with `ls -l` that its group is webteam.

Make the file readable by the group but writable only by you: `chmod 640 /srv/webteam/plan.txt`. Verify the string reads `-rw-r-----`.

Switch to the test user with `su - maria`, try `cat /srv/webteam/plan.txt` (should work) and `echo hi >> /srv/webteam/plan.txt` (should fail). Type `exit` to return.

Create a script with `printf '#!/bin/bash\necho hello\n' > ~/hello.sh`, try `./hello.sh` (permission denied), then `chmod u+x ~/hello.sh` and run it again.

Clean up: `sudo deluser --remove-home maria`, `sudo groupdel webteam` and `sudo rm -r /srv/webteam`.

## Why it matters

Linux was built for many users sharing one machine, and that design is still what keeps servers safe. A web server process should be able to read its web files but not your SSH keys. A database's data folder should be writable by the database and nobody else. Most "permission denied" errors you will debug, and many security findings, come down to the ideas in this lesson.

## Users and groups

Every process runs as a **user**, and every user belongs to one or more **groups**.

```bash
whoami          # your username
id              # your user id (uid), main group and all groups
getent passwd alex
getent group sudo
```

- **root** (uid 0) can do anything. Normal users get admin rights through `sudo`, which on Ubuntu is granted by membership of the `sudo` group (`wheel` on the Red Hat family).
- **System users** such as `www-data`, `mysql` or `sshd` have no login and exist so that each service runs with only its own rights.
- User accounts are listed in `/etc/passwd`; password hashes are in `/etc/shadow`, which only root can read.

Creating users and groups (on your practice system):

```bash
sudo adduser maria                 # interactive on Debian/Ubuntu (useradd -m on others)
sudo groupadd webteam
sudo usermod -aG webteam maria     # -a = append; without it you REPLACE her groups
groups maria
```

## Reading permissions

`ls -l` shows the permissions of each file:

```text
-rw-r----- 1 alex webteam 1204 Oct 23 09:12 report.txt
drwxr-x--- 2 alex webteam 4096 Oct 23 09:10 shared
```

Break the first column into four parts:

| Part | `report.txt` | Meaning |
|---|---|---|
| Type | `-` | `-` file, `d` directory, `l` symlink |
| Owner (u) | `rw-` | alex can read and write |
| Group (g) | `r--` | members of webteam can read |
| Others (o) | `---` | everyone else gets nothing |

The letters mean different things for files and directories:

| | File | Directory |
|---|---|---|
| `r` | read the contents | list the names inside |
| `w` | change the contents | create, rename or delete entries inside |
| `x` | run it as a program | enter it (`cd`) and reach files inside |

The directory `x` surprises many people: without it you cannot open a file inside, even if the file itself is readable.

## chmod: changing permissions

**Symbolic mode** adds (`+`), removes (`-`) or sets (`=`) letters for `u`, `g`, `o` or `a` (all):

```bash
chmod u+x backup.sh        # owner may run it
chmod g+w report.txt       # group may edit it
chmod o-r report.txt       # others may no longer read it
chmod u=rwx,g=rx,o= shared # exact set
```

**Octal mode** sets everything at once. Each digit is a sum: read 4, write 2, execute 1.

| Octal | Letters | Typical use |
|---|---|---|
| `644` | `rw-r--r--` | normal files |
| `640` | `rw-r-----` | files only a group may read |
| `600` | `rw-------` | private files, SSH private keys |
| `755` | `rwxr-xr-x` | programs, scripts and normal folders |
| `750` | `rwxr-x---` | a team folder |

```bash
chmod 640 report.txt
chmod 750 shared
chmod -R g+rX shared       # recursive; capital X adds x only to folders
```

## chown: changing owner and group

Only root can give a file to another user, so `chown` usually needs `sudo`:

```bash
sudo chown maria report.txt            # new owner
sudo chown maria:webteam report.txt    # owner and group
sudo chgrp webteam shared              # group only
sudo chown -R www-data:www-data /var/www/site
```

## Worked example: a shared team folder

Goal: alex and maria can both edit files in `/srv/webteam`; nobody else can see inside.

```bash
sudo mkdir -p /srv/webteam
sudo chown root:webteam /srv/webteam
sudo chmod 2770 /srv/webteam      # rwx for owner and group, nothing for others
sudo usermod -aG webteam alex
```

The leading `2` sets the **setgid** bit on the folder, so new files created inside inherit the `webteam` group instead of the creator's personal group. Group changes apply at your next login, so log out and back in (or run `newgrp webteam`) before testing.

## Default permissions: umask

New files do not start at `777`. The **umask** removes bits from the default. With the common umask `022`, new files get `644` and new folders `755`. Run `umask` to see yours.

## Common mistakes

- **`chmod 777` to "fix" a permission error.** It lets every user and every compromised service change the file. Find out which user needs access and grant exactly that.
- **`usermod -G` without `-a`**, which silently removes the user from every other group, sometimes including `sudo`.
- **Forgetting the directory `x` bit**, so a readable file inside an unreadable folder still cannot be opened.
- **Recursive `chmod -R 644` on a folder**, which strips `x` from its subfolders so nobody can enter them. Use `X` or `find -type f`/`-type d`.
- **Testing group changes without logging in again.**

## Recap

- Every process runs as a user; groups share access between users.
- `ls -l` shows type, then owner/group/others permissions as `rwx`.
- `chmod` changes permissions (symbolic or octal: r=4, w=2, x=1); `chown` changes owner and group.
- Grant the least access that works; never reach for `777`.

## Try it

Use the Do tab to build a shared team folder on your practice system and prove who can and cannot get in.


Read rwx permissions, create users and groups, and control access with chmod and chown.

Users, groups, and permissions

Linux Crash Course - Understanding File & Directory Permissions

How to use chmod | Manage File Permissions in Linux

Set up a safe practice system, find your way around the filesystem, and control access with users, groups and permissions.

The Linux System

Run `ps aux | head` and `ps -ef --forest | head -30`. Find PID 1 and note which program it is.

Open `top`, press `M` to sort by memory and `P` to sort by CPU, note the load average and compare it with `nproc`. Press `q` to quit.

Start a harmless background process with `sleep 600 &`, find it with `pgrep -a sleep`, stop it with `kill <PID>` and confirm it is gone.

Install a web server: `sudo apt update && sudo apt install -y nginx` (Red Hat family: `sudo dnf install -y nginx`, then `sudo systemctl enable --now nginx`).

Run `systemctl status nginx` and read the Loaded and Active lines. Test it with `curl -I http://localhost`.

Stop it with `sudo systemctl stop nginx`, run curl again and note the error, then start it again with `sudo systemctl start nginx`.

Check boot behaviour with `systemctl is-enabled nginx`, then disable and re-enable it with `sudo systemctl disable nginx` and `sudo systemctl enable nginx`.

View the unit file with `systemctl cat nginx` and find its ExecStart line.

List failed units with `systemctl --failed`. When you are done, leave nginx installed for the logs lesson or remove it with `sudo apt remove nginx`.

## Why it matters

A server exists to run programs: a web server, a database, an API, a monitoring agent. An administrator needs to answer "is it running?", "why is the machine slow?" and "how do I restart it safely?" many times a day. Those questions are answered with process tools and with **systemd**, the service manager on almost every modern distro.

## Processes

Every running program is a **process** with a numeric **process ID (PID)**, an owner (a user) and a parent process that started it.

```bash
ps aux | head            # every process: user, PID, CPU%, MEM%, command
ps aux | grep sshd       # find one by name
pgrep -a ssh             # PIDs and command lines matching a name
ps -ef --forest | less   # show the parent/child tree
```

PID 1 is special: it is the first process the kernel starts, and on modern distros it is `systemd`. Everything else descends from it.

### Watching live: top and htop

```bash
top
```

`top` updates every few seconds. The header shows **load average** (how many processes wanted CPU over the last 1, 5 and 15 minutes; compare it with the number of CPUs from `nproc`), memory and swap use. Inside `top`, press `P` to sort by CPU, `M` to sort by memory and `q` to quit. `htop` (install it with your package manager) is a friendlier version with colours and scrolling.

### Stopping a process: signals

You stop a process by sending it a **signal**:

```bash
kill 4321          # sends SIGTERM (15): "please exit cleanly"
kill -9 4321       # sends SIGKILL (9): the kernel ends it immediately
pkill -f myscript  # signal by name instead of PID
```

Always try plain `kill` first. `SIGTERM` lets the program save data and close files. `kill -9` cannot be caught, so the program gets no chance to clean up; use it only when a process ignores `SIGTERM`. You can signal your own processes freely; other users' processes need `sudo`.

### Foreground and background jobs

In an interactive shell, `Ctrl+C` interrupts the foreground program, `Ctrl+Z` pauses it, `bg` resumes it in the background and `fg` brings it back. Adding `&` starts a command in the background. These jobs end when you log out, which is why real services are run by systemd instead.

## systemd and units

**systemd** starts the system at boot and supervises long-running services. Each thing it manages is a **unit**, described by a unit file. Service units end in `.service`, for example `ssh.service` or `cron.service`. Vendor unit files live in `/usr/lib/systemd/system/`; your local overrides go in `/etc/systemd/system/`.

The everyday commands:

```bash
systemctl status ssh             # running? since when? main PID? last log lines
sudo systemctl start nginx       # start now
sudo systemctl stop nginx        # stop now
sudo systemctl restart nginx     # stop then start (drops connections)
sudo systemctl reload nginx      # re-read config without a full restart (if supported)
sudo systemctl enable nginx      # start automatically at boot
sudo systemctl disable nginx     # do not start at boot
systemctl is-active nginx        # prints active / inactive
systemctl is-enabled nginx       # prints enabled / disabled
systemctl list-units --type=service --state=running
systemctl --failed               # units that crashed or failed to start
```

Two ideas people mix up: **start/stop** changes what is running *right now*; **enable/disable** changes what happens *at the next boot*. `sudo systemctl enable --now nginx` does both.

### Reading `systemctl status`

```text
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-10-22 09:14:03 UTC; 2h ago
   Main PID: 812 (nginx)
      Tasks: 3 (limit: 4558)
     Memory: 4.1M
```

`Loaded` tells you where the unit file is and whether it is enabled. `Active` tells you whether it is running and since when. If a service failed, the last few log lines appear underneath, which is often enough to see why.

### A peek at a unit file

```bash
systemctl cat cron
```

```ini
[Unit]
Description=Regular background program processing daemon

[Service]
ExecStart=/usr/sbin/cron -f $EXTRA_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

`ExecStart` is the command systemd runs, `Restart=on-failure` brings it back if it crashes, and `WantedBy` is what `enable` hooks it into. After editing any unit file, run `sudo systemctl daemon-reload` so systemd notices.

## Worked example: install and manage a web server

```bash
sudo apt install nginx            # Ubuntu enables and starts it on install
systemctl status nginx
curl -I http://localhost          # HTTP/1.1 200 OK
sudo systemctl stop nginx
curl -I http://localhost          # connection refused
sudo systemctl start nginx
```

## Common mistakes

- **Reaching for `kill -9` first.** Try `systemctl stop` for services and plain `kill` for processes.
- **Killing a service's process instead of stopping the service.** With `Restart=on-failure` or `always`, systemd may just start it again. Use `systemctl stop`.
- **Enabling but not starting** (or starting but not enabling), then being surprised after a reboot.
- **Editing a unit file and forgetting `daemon-reload`.**
- **Restarting when reload would do,** which drops users' connections on a busy server.

## Recap

- Every program is a process with a PID and an owner; `ps`, `pgrep` and `top` inspect them.
- `kill` sends SIGTERM (clean exit); `kill -9` is a last resort.
- systemd (PID 1) manages services as units; `systemctl start/stop/restart/reload/status` controls them now, `enable/disable` controls boot.
- `systemctl --failed` and `systemctl status` are the first stops when something is down.

## Try it

The Do tab walks you through installing a small web server on your practice system and controlling it with systemctl.


Inspect running processes with ps and top, stop them with kill, and manage services with systemctl.

Processes and systemd

Linux Crash Course - systemd: Using the systemctl command

Linux Crash Course - The ps Command

Refresh the package lists with `sudo apt update` and read the last line: how many packages can be upgraded?

Run `apt list --upgradable` and pick one package from the list. Look it up with `apt show <name>`.

Local VM: take a snapshot. Then run `sudo apt upgrade` and read the summary before typing Y. Note whether anything would be removed.

Check whether a reboot is needed with `[ -f /var/run/reboot-required ] && cat /var/run/reboot-required || echo no reboot needed`. Reboot the practice system if it is.

Search for and install a small tool: `apt search tree`, then `sudo apt install tree`. Run `tree -L 1 /etc | head`.

Inspect it: list its files with `dpkg -L tree` and find which package owns `/usr/bin/tree` with `dpkg -S /usr/bin/tree`.

See where your packages come from: `ls /etc/apt/sources.list.d/` and `apt policy tree`.

Check automatic updates with `systemctl status unattended-upgrades` and `cat /etc/apt/apt.conf.d/20auto-upgrades`.

Remove the tool and clean up: `sudo apt purge tree` then `sudo apt autoremove`.

## Why it matters

On Linux you rarely download an installer from a website. Software comes from **repositories** through a **package manager**, which installs programs together with everything they depend on, keeps them updated, and removes them cleanly. Keeping packages patched is also one of the most important security jobs on any server: many real breaches exploit a vulnerability that already had a fix available.

## Packages, repositories and dependencies

- A **package** is an archive with a program, its files and metadata: version, description and the other packages it needs (its **dependencies**).
- A **repository** is a server that hosts packages. Your system's list of repositories lives in `/etc/apt/sources.list` and `/etc/apt/sources.list.d/` on Debian/Ubuntu, or `/etc/yum.repos.d/` on the Red Hat family.
- Repositories **sign** their package lists with a cryptographic key. The package manager refuses packages whose signature does not match, which protects you from tampered downloads.

## apt on Debian and Ubuntu

```bash
sudo apt update                 # refresh the list of available packages (installs nothing)
apt list --upgradable           # what could be updated
sudo apt upgrade                # install available updates
sudo apt install tree htop      # install packages and their dependencies
apt search json                 # search names and descriptions
apt show htop                   # details: version, size, dependencies
sudo apt remove htop            # remove the program, keep its config files
sudo apt purge htop             # remove the program and its config
sudo apt autoremove             # remove dependencies nothing needs any more
```

The most common beginner confusion: **`apt update` does not update software**. It downloads the latest package *lists*. `apt upgrade` installs the new versions. You normally run them together: `sudo apt update && sudo apt upgrade`.

Underneath, `apt` uses **dpkg**, which handles single `.deb` files:

```bash
dpkg -l | grep nginx            # is it installed, and which version?
dpkg -L tree                    # which files did this package install?
dpkg -S /usr/bin/htop           # which package owns this file?
```

## dnf on the Red Hat family

The ideas are identical; the command names differ slightly.

| Task | Debian/Ubuntu | RHEL/Fedora/Rocky |
|---|---|---|
| Refresh lists | `sudo apt update` | (automatic; `sudo dnf check-update`) |
| Install | `sudo apt install pkg` | `sudo dnf install pkg` |
| Update all | `sudo apt upgrade` | `sudo dnf upgrade` |
| Remove | `sudo apt remove pkg` | `sudo dnf remove pkg` |
| Search | `apt search word` | `dnf search word` |
| Owner of a file | `dpkg -S /path` | `rpm -qf /path` |

## Installing software safely

1. **Prefer the distro's own repositories.** Their packages are tested together and receive security updates automatically.
2. **Read what apt is about to do.** Before you answer `Y`, apt lists what it will install, upgrade and *remove*. If an install wants to remove many packages, stop and find out why.
3. **Add third-party repositories only from the vendor's official instructions** (for example PostgreSQL, Docker or Node.js docs), and understand that you are trusting their signing key with root on your machine.
4. **Be wary of `curl ... | sudo bash`.** It runs a script from the internet as root without letting you read it. Download it, read it, then decide.
5. **Do not mix in random `.deb` or `.rpm` files** from file-sharing sites.
6. **Snapshot first** on your practice VM before big upgrades, and on real servers follow the change process (test environment first, a maintenance window, a rollback plan).

## Automatic security updates

Ubuntu servers usually run **unattended-upgrades**, which installs security updates daily:

```bash
systemctl status unattended-upgrades
cat /etc/apt/apt.conf.d/20auto-upgrades
```

Some updates (for example a new kernel) only take effect after a reboot. Ubuntu creates `/var/run/reboot-required` when that is the case:

```bash
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required
```

## Other formats you will meet

- **Snap** and **Flatpak** bundle an app with its own libraries; common on desktops.
- **Language package managers** (`pip`, `npm`, `cargo`) install libraries for a programming language. Use them inside a virtual environment or project folder, not with `sudo` into the system Python, so they do not fight with the OS package manager.
- **Containers** package a whole application environment; you will meet them in later cloud and DevOps courses.

## Common mistakes

- **Running `apt upgrade` without `apt update` first**, so nothing new is found.
- **Answering `Y` without reading the list** of packages to be removed.
- **`sudo pip install` into the system Python**, which can break OS tools that depend on it.
- **Never rebooting after kernel updates**, leaving the vulnerable kernel running.
- **Adding a random PPA or repository** from a blog post to get a newer version.

## Recap

- Software comes from signed repositories through a package manager that resolves dependencies.
- `apt update` refreshes lists; `apt upgrade` installs updates; `apt install/remove/purge/autoremove` manage packages (dnf on the Red Hat family).
- `dpkg -L` and `dpkg -S` connect packages and files.
- Stay with official repositories, read before confirming, avoid piping scripts into root, and reboot when updates require it.

## Try it

In the Do tab you will update your practice system and install, inspect and remove a package.


Install, update and remove software safely with apt (and dnf), and understand repositories and signed packages.

Package management

Linux Crash Course - The apt Command

The apt Package Manager | Top Docs with Jay LaCroix

Back up the config first: `sudo cp /etc/nginx/sites-available/default ~/default.backup`.

Look at recent logs: `journalctl -u nginx -n 20 --no-pager` and `sudo tail -n 20 /var/log/syslog` (Red Hat family: `/var/log/messages`).

Open the config with `sudo nano /etc/nginx/sites-available/default`, change one `server_name` to `server_nme`, and save.

Restart with `sudo systemctl restart nginx` and note the time and the error printed.

Find the cause with `journalctl -u nginx --since "5 min ago" --no-pager` and write down the file and line number it names.

Confirm with the program's own check: `sudo nginx -t`.

Restore the config with `sudo cp ~/default.backup /etc/nginx/sites-available/default`, run `sudo nginx -t` again, then `sudo systemctl restart nginx` and `systemctl status nginx`.

List this boot's errors with `journalctl -p err -b --no-pager | tail -20`, and check the journal's size with `journalctl --disk-usage`.

Count failed SSH logins (may be zero on a local VM): `sudo grep -c "Failed password" /var/log/auth.log`.

## Why it matters

When something breaks, the log usually says why. A service that will not start, a disk that is full, a failed login, a crashed process: each leaves a message with a timestamp. The skill that separates a confident administrator from a guesser is going straight to the right log, filtering it down to the few relevant lines, and reading them carefully.

## Two places logs live

1. **The systemd journal**, read with `journalctl`. systemd captures the output of every service it runs, plus kernel and boot messages, and stores them in a structured, indexed form.
2. **Text files in `/var/log`**, written by `rsyslog` and by applications themselves: `syslog`, `auth.log`, `kern.log`, `dpkg.log`, `nginx/error.log` and so on. On the Red Hat family, the main file is `/var/log/messages` and logins go to `/var/log/secure`.

Many systems have both: services log to the journal and rsyslog copies messages into text files. Reading most system logs needs `sudo` or membership of the `adm` (Debian/Ubuntu) or `systemd-journal` group.

## journalctl essentials

```bash
journalctl -u nginx                 # everything from one service
journalctl -u nginx -n 50           # the last 50 lines
journalctl -u nginx -f              # follow new lines live (Ctrl+C to stop)
journalctl -u nginx --since "1 hour ago"
journalctl --since "2026-10-23 09:00" --until "2026-10-23 10:00"
journalctl -p err -b                # errors and worse, since this boot
journalctl -b -1                    # the previous boot (if the journal is persistent)
journalctl -k                       # kernel messages (like dmesg)
journalctl --disk-usage             # how much space the journal uses
```

**Priorities** go from 0 (emerg) to 7 (debug): emerg, alert, crit, **err**, **warning**, notice, info, debug. `-p err` shows `err` and everything more severe, which is a great first filter.

Combine filters freely: `journalctl -u ssh -p warning --since today`.

## Text logs essentials

```bash
sudo tail -n 50 /var/log/syslog          # the end of the file
sudo tail -f /var/log/auth.log           # follow live
sudo less +G /var/log/syslog             # open at the end; / to search, q to quit
sudo grep -i "error" /var/log/syslog | tail
sudo grep "Failed password" /var/log/auth.log | wc -l
zgrep "Failed password" /var/log/auth.log.2.gz   # search a compressed old log
```

Old logs are **rotated** by `logrotate`: `auth.log` becomes `auth.log.1`, then `auth.log.2.gz`, and so on, and the oldest are deleted. That keeps `/var/log` from filling the disk. Rules live in `/etc/logrotate.conf` and `/etc/logrotate.d/`.

## Reading a log line

```text
Oct 23 10:42:17 web1 sshd[2211]: Failed password for invalid user admin from 203.0.113.50 port 51122 ssh2
```

| Part | Meaning |
|---|---|
| `Oct 23 10:42:17` | when (check the server's time zone with `timedatectl`) |
| `web1` | which host |
| `sshd[2211]` | which program and its PID |
| the rest | the message: a login attempt for a user that does not exist, from that IP |

## Worked example: why will the service not start?

You break nginx on purpose by adding a typo to its configuration, then restart it:

```bash
sudo systemctl restart nginx
# Job for nginx.service failed because the control process exited with error code.
systemctl status nginx          # shows "failed" and the last few lines
journalctl -u nginx -n 20 --no-pager
```

The journal shows a line like:

```text
nginx: [emerg] unknown directive "server_nme" in /etc/nginx/sites-enabled/default:22
```

That message names the file, the line number and the mistake. Fix the typo, check the config with the program's own test (`sudo nginx -t`), then restart. Many services have a similar check: `sshd -t`, `apachectl configtest`, `named-checkconf`.

### A method that works

1. **Reproduce** the problem and note the exact time.
2. **Narrow** by unit and time: `journalctl -u <service> --since "5 min ago"`.
3. **Filter by severity:** `-p err`, or `grep -iE "error|fail|denied"`.
4. **Read the first error**, not the last. Later errors are often consequences of the first one.
5. **Search the exact message** in the official docs if it is unfamiliar.

## Keeping the journal

On some systems the journal lives only in memory (`/run/log/journal`) and is lost at reboot. If `/var/log/journal` exists, it is persistent. Limit its size in `/etc/systemd/journald.conf` with `SystemMaxUse=`, or clean up old entries with `sudo journalctl --vacuum-time=2weeks`.

## Common mistakes

- **Reading only the last line.** The useful error is often several lines up.
- **Ignoring time zones,** so you look at the wrong hour.
- **Deleting log files to free space.** Use logrotate and `journalctl --vacuum-*` instead.
- **Forgetting `sudo`** and concluding a log is empty when you simply cannot read it.
- **Posting logs publicly** without removing IP addresses, usernames, hostnames or tokens.

## Recap

- Logs live in the systemd journal (`journalctl`) and in text files under `/var/log`.
- `journalctl -u <unit>`, `-f`, `--since`, `-p err` and `-b` narrow the journal quickly.
- `tail`, `less`, `grep` and `zgrep` read text and rotated logs.
- Find the first error, read the whole message, and use the service's own config test before restarting.

## Try it

The Do tab has you break a service on your practice system on purpose and find the error in the logs.


Find the cause of an error with journalctl, /var/log and a few filtering commands.

Logs and journald

journalctl Basics: How to Easily Check Your Linux Logs

Understanding Linux Logging: Var Log, Journal, and Logger

Manage processes and systemd services, install and patch software safely, and find errors in the logs.

Running Services

Make sure the server is installed and running: `sudo apt install -y openssh-server`, then `systemctl status ssh`. On a local VM, take a snapshot now.

Decide where you will connect from: your host computer into the VM (VirtualBox: add a NAT port-forwarding rule from host port 2222 to guest port 22, then connect with `-p 2222` to 127.0.0.1), a cloud VM's public IP, or simply `localhost` from inside the practice system.

On the client, create a key with a passphrase: `ssh-keygen -t ed25519 -C "practice key"`.

Copy the public key to the server with `ssh-copy-id` (for example `ssh-copy-id -p 2222 youruser@127.0.0.1`). Cloud VMs often install your key at launch; check `~/.ssh/authorized_keys` there.

Log in with the key in a new terminal and confirm you are not asked for the account password. Run `ls -la ~/.ssh` on the server and check the permissions are 700 for the folder and 600 for authorized_keys.

Keeping that session open, create `/etc/ssh/sshd_config.d/01-hardening.conf` with `sudo nano` containing `PermitRootLogin no`, `PasswordAuthentication no` and `KbdInteractiveAuthentication no` on separate lines.

Test the config with `sudo sshd -t`, then confirm the effective values with `sudo sshd -T | grep -iE "passwordauth|permitroot"`.

Restart with `sudo systemctl restart ssh`, then open a NEW terminal and log in again with the key before closing the old session.

Prove passwords are refused: `ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password youruser@host` should fail with Permission denied (publickey).

Add a `Host practice` entry to `~/.ssh/config` on the client and log in with `ssh practice`.

## Why it matters

SSH (Secure Shell) is how administrators reach almost every Linux server: cloud VMs, database hosts, network devices, build machines. It encrypts everything between you and the server. It is also one of the most attacked services on the internet: any server with SSH open to the world sees automated password-guessing attempts within minutes. Logging in with **keys** and turning off password logins removes that whole class of attack.

## How SSH works, briefly

- The **SSH server** (`sshd`, in the `openssh-server` package) listens on TCP port 22.
- The **client** (`ssh`) connects: `ssh alex@192.0.2.10`.
- On first connection the server proves its identity with a **host key**. Your client shows its fingerprint and asks whether to trust it, then remembers it in `~/.ssh/known_hosts`. If the host key later changes, the client warns loudly: that can mean the server was rebuilt, or that someone is intercepting the connection. Find out which before continuing.
- You then authenticate, with a password or with a key.

## Key pairs

A key pair is two linked files:

- The **private key** (`~/.ssh/id_ed25519`) stays on your computer. Never copy it to a server, email it, paste it into chat or commit it to Git.
- The **public key** (`~/.ssh/id_ed25519.pub`) can be shared freely. Putting it in a server's `~/.ssh/authorized_keys` means "whoever holds the matching private key may log in as this user".

When you connect, the server sends a challenge that only the private key can answer. The private key never leaves your machine.

### Create a key

```bash
ssh-keygen -t ed25519 -C "alex laptop 2026"
```

Accept the default path and **set a passphrase**. The passphrase encrypts the private key file, so a stolen laptop does not mean stolen server access. `ssh-agent` remembers the unlocked key for your session so you do not type the passphrase every time:

```bash
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
```

### Install the public key on the server

```bash
ssh-copy-id alex@192.0.2.10
```

This appends your public key to `~/.ssh/authorized_keys` on the server and sets the right permissions. Without `ssh-copy-id`, append the `.pub` file's single line manually. SSH is strict about permissions and silently ignores keys if they are too open:

```bash
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys ~/.ssh/id_ed25519
```

Now `ssh alex@192.0.2.10` should log you in without asking for the account password (it may ask for the key's passphrase).

### A client config file

Save typing with `~/.ssh/config` on your own computer:

```text
Host practice
    HostName 192.0.2.10
    User alex
    Port 22
    IdentityFile ~/.ssh/id_ed25519
```

Then `ssh practice` is enough. The same file works for `scp` and Git over SSH.

## Hardening the server

Server settings are in `/etc/ssh/sshd_config` and in files under `/etc/ssh/sshd_config.d/`. For most settings sshd uses the **first** value it reads, and the `.d` files are read first in name order, so put your changes in a file with a low number, for example `/etc/ssh/sshd_config.d/01-hardening.conf`:

```text
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers alex
```

- **`PermitRootLogin no`:** log in as yourself and use `sudo`, so every action is tied to a person.
- **`PasswordAuthentication no`:** password guessing becomes useless.
- **`AllowUsers`:** only the named accounts may log in over SSH.

### Apply changes without locking yourself out

This is the part that matters most. A mistake here can leave you unable to reach a remote server.

1. **Confirm key login works first,** in a separate terminal.
2. **Keep your current session open** while you change anything.
3. **Test the config:** `sudo sshd -t` prints nothing when it is valid.
4. **Check the effective values:** `sudo sshd -T | grep -iE "passwordauth|permitroot"`.
5. **Restart:** `sudo systemctl restart ssh` (the unit is `sshd` on the Red Hat family).
6. **Open a new terminal and log in again** before closing the old session.

On a cloud VM, also know where the provider's web console is, so you can get in if SSH breaks. A firewall such as `ufw` (`sudo ufw allow OpenSSH` *before* `sudo ufw enable`) and a tool like fail2ban add further layers, but keys plus no passwords is the big win.

## Copying files over SSH

```bash
scp report.txt practice:/tmp/
scp practice:/var/log/syslog ./syslog-copy
rsync -av ./site/ practice:~/site/      # copies only what changed
```

## Common mistakes

- **Sharing or committing the private key.** If it leaks, remove its public key from every `authorized_keys` and make a new pair.
- **No passphrase on the key.**
- **Wrong permissions on `~/.ssh`,** so the server ignores the key and falls back to a password.
- **Disabling passwords before testing key login,** or closing the only session before testing the new config.
- **Typing `yes` to a changed host key warning without checking why.**

## Recap

- SSH encrypts remote logins; the server proves itself with a host key stored in `known_hosts`.
- `ssh-keygen -t ed25519` makes a key pair; the private key stays with you, the public key goes in `authorized_keys`.
- Harden with `PermitRootLogin no` and `PasswordAuthentication no`, tested with `sshd -t`, while keeping a session open.
- `~/.ssh/config`, `scp` and `rsync` make daily work faster.

## Try it

In the Do tab you will set up key login to your practice system and switch off password logins safely.


Log in to a Linux machine with SSH keys instead of passwords, and apply a few safe hardening settings.

SSH and keys

SSH Key Authentication | How to Create SSH Key Pairs

5 Must Have Tweaks to Secure OpenSSH

Create practice data: `mkdir -p ~/projects/demo && echo "hello" > ~/projects/demo/notes.txt`.

Create `~/bin/backup.sh` (`mkdir -p ~/bin`) with the shebang, `set -euo pipefail`, and `log` and `die` functions from the lesson.

Add argument handling: the source folder is required (`${1:-}`), the destination defaults to `$HOME/backups`, and the script exits with a usage message and code 1 when the source is missing or not a directory.

Add the tar command that writes a dated `.tar.gz`, followed by the two verification checks (`-s` and `tar -tzf`).

Make it executable with `chmod u+x ~/bin/backup.sh` and run `~/bin/backup.sh ~/projects`. Confirm an archive appears in `~/backups`.

Test the checks: run it with no arguments and with `/does/not/exist`, and print `echo $?` after each. Both should print a clear error and exit code 1.

Test a restore: extract the archive into `/tmp/restore-test` and compare with `diff -r ~/projects /tmp/restore-test/projects`.

Add the keep-the-newest-7 clean-up line, run the script 9 times (`for i in $(seq 9); do ~/bin/backup.sh ~/projects; sleep 1; done`), and confirm only 7 archives remain.

Install and run ShellCheck (`sudo apt install -y shellcheck`, then `shellcheck ~/bin/backup.sh`) and fix anything it reports.

## Why it matters

Anything you type more than twice is a candidate for a script. Administrators script backups, user setup, log clean-up, health checks and deployments. A good script does the same thing the same way every time, which is more reliable than a person following notes at 2 a.m. A bad script, one that keeps going after an error, can do real damage quickly. This lesson focuses on writing scripts that **check before they act**.

## Anatomy of a script

```bash
#!/usr/bin/env bash
# backup.sh: archive a folder into a dated tar.gz
set -euo pipefail

echo "Hello from $0"
```

- The first line, the **shebang**, tells the system which interpreter runs the file.
- `set -e` stops the script when a command fails. `set -u` treats unset variables as errors (so a typo cannot expand to an empty string). `set -o pipefail` makes a pipeline fail if any command in it fails.
- Make it executable and run it: `chmod u+x backup.sh`, then `./backup.sh`.

## The building blocks

### Variables and quoting

```bash
name="web logs"
echo "Backing up $name"          # double quotes expand variables
echo 'Literal $name'             # single quotes do not
today=$(date +%F)                # command substitution: 2026-10-23
```

No spaces around `=`. **Always quote variables** (`"$file"`), or a value containing a space becomes two arguments.

### Arguments

`$1`, `$2`... are the script's arguments, `$#` is how many there are, and `"$@"` is all of them.

```bash
src="${1:-}"                     # first argument, or empty if missing
dest="${2:-$HOME/backups}"       # default value when not given
```

### Tests and if

```bash
if [[ -d "$src" ]]; then echo "folder exists"; fi
if [[ ! -r "$src" ]]; then echo "cannot read $src" >&2; exit 1; fi
```

Useful tests: `-d` directory, `-f` regular file, `-e` exists, `-r` readable, `-w` writable, `-z` empty string, `-n` non-empty, and `-eq`/`-lt`/`-gt` for numbers. `>&2` sends a message to standard error, where error messages belong.

### Exit codes

Every command returns an **exit code**: `0` means success, anything else means failure. `$?` holds the last one. Your scripts should `exit 0` on success and a non-zero code on failure, so other tools (and cron, next lesson) can tell what happened.

### Loops and functions

```bash
for f in "$dest"/*.tar.gz; do
  echo "found $f"
done

log() { echo "$(date '+%F %T') $*"; }
log "starting"
```

## Worked example: a backup script with checks

```bash
#!/usr/bin/env bash
# backup.sh SRC [DEST] - archive SRC into DEST/name-YYYY-mm-dd-HHMMSS.tar.gz
set -euo pipefail

log() { echo "$(date '+%F %T') $*"; }
die() { echo "ERROR: $*" >&2; exit 1; }

src="${1:-}"
dest="${2:-$HOME/backups}"
keep=7

[[ -n "$src" ]]  || die "usage: $0 SRC [DEST]"
[[ -d "$src" ]]  || die "source folder not found: $src"
[[ -r "$src" ]]  || die "cannot read: $src"
mkdir -p "$dest"
[[ -w "$dest" ]] || die "cannot write to: $dest"

name="$(basename "$src")-$(date +%F-%H%M%S).tar.gz"
archive="$dest/$name"

log "archiving $src to $archive"
tar -czf "$archive" -C "$(dirname "$src")" "$(basename "$src")"

# verify: the archive must exist, be non-empty and be readable by tar
[[ -s "$archive" ]] || die "archive is empty: $archive"
tar -tzf "$archive" > /dev/null || die "archive is corrupt: $archive"

# keep only the newest $keep archives for this source
ls -1t "$dest"/"$(basename "$src")"-*.tar.gz | tail -n +$((keep + 1)) | xargs -r rm --
log "done: $(du -h "$archive" | cut -f1)"
```

What makes it trustworthy:

1. **Inputs are checked** before anything happens, with clear messages and a non-zero exit.
2. **`set -euo pipefail`** stops on the first unexpected failure.
3. **The result is verified:** the archive exists, is not empty and can be listed by `tar`. A backup you have not tested restoring is only a hope.
4. **Deletion is narrow:** it only removes old archives matching this source's name pattern, in the destination folder.

Run it:

```bash
./backup.sh ~/projects ~/backups
./backup.sh /does/not/exist ; echo "exit code: $?"    # ERROR ... exit code: 1
```

## Testing a restore

```bash
mkdir -p /tmp/restore-test
tar -xzf ~/backups/projects-2026-10-23-104500.tar.gz -C /tmp/restore-test
diff -r ~/projects /tmp/restore-test/projects && echo "restore matches"
```

## Check your script with ShellCheck

`shellcheck` (`sudo apt install shellcheck`) reads a script and points out unquoted variables, common traps and portability problems. Run `shellcheck backup.sh` before you trust any script.

## Common mistakes

- **Unquoted variables**, which break on spaces and can turn `rm -r $dir/` into `rm -r /` if `$dir` is empty. Quote everything and use `set -u`.
- **No checks before destructive commands** such as `rm`, `mv` or overwriting with `>`.
- **Relative paths in scripts** that later run from a different folder. Use absolute paths or paths built from arguments.
- **Ignoring exit codes**, so a failed backup still prints "done".
- **Backups stored on the same disk as the data** in real life. For practice it is fine; at work, copy them elsewhere.

## Recap

- Start scripts with a shebang and `set -euo pipefail`.
- Quote variables, check arguments with `[[ ... ]]` tests, and send errors to stderr with a non-zero exit.
- Verify what the script produced, and test restores.
- Run `shellcheck` on everything you write.

## Try it

The Do tab has you write, break and verify your own backup script on your practice system.


Write a bash backup script that checks its inputs, stops on errors and verifies its own result.

Bash scripting

Bash in 100 Seconds

Bash Scripting Tutorial for Beginners

Confirm cron is running with `systemctl status cron` (Red Hat family: `crond`), and save any existing jobs with `crontab -l > ~/crontab.backup 2>/dev/null || true`.

Check your script works with absolute paths: `/home/$USER/bin/backup.sh /home/$USER/projects` (use the real path printed by `echo $HOME`).

Run `crontab -e` and add a test job that runs every 2 minutes with absolute paths and `>> /home/youruser/backup.log 2>&1` at the end.

Wait at least 4 minutes, then check cron started it: `grep CRON /var/log/syslog | tail -5` or `journalctl -u cron --since "10 min ago"`.

Check it succeeded: `tail /home/youruser/backup.log` and `ls -lt ~/backups | head -3`.

Break it on purpose: change the source folder in the job to one that does not exist, wait 2 minutes and confirm the ERROR line appears in the log. Then fix it.

Change the schedule to a real nightly time, such as `30 2 * * *`, and confirm with `crontab -l`. Write in plain words what your schedule means.

Look at system jobs: `ls /etc/cron.daily/` and `systemctl list-timers --all | head`.

Clean up when you are done with the course: remove the job with `crontab -e` (do not use `crontab -r` if you have other jobs) and delete `~/backups` if you no longer need it.

## Why it matters

A backup script that someone has to remember to run will eventually not be run. Servers rely on scheduled jobs for backups, log clean-up, certificate renewal, report generation and health checks. **cron** is the classic Linux scheduler and is available on nearly every system; **systemd timers** are the modern alternative. Knowing both, and knowing how to tell whether a scheduled job actually ran, is a core admin skill.

## How cron works

The `cron` service wakes up every minute, reads the schedules (**crontabs**) and runs any job whose time matches. Each user has their own crontab, and jobs run as that user.

```bash
crontab -l        # list your jobs
crontab -e        # edit your jobs (opens an editor; choose nano if asked)
crontab -r        # remove ALL your jobs: be careful, there is no undo
```

System-wide jobs live in `/etc/crontab`, `/etc/cron.d/`, and the folders `/etc/cron.daily/`, `/etc/cron.weekly/` and so on, where packages drop their own maintenance scripts.

## The five time fields

```text
┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12)
│ │ │ │ ┌─ day of week (0-7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *  command to run
```

`*` means "every". `,` lists values, `-` makes a range and `/` sets a step.

| Schedule | Meaning |
|---|---|
| `30 2 * * *` | every day at 02:30 |
| `*/15 * * * *` | every 15 minutes |
| `0 9 * * 1-5` | 09:00 Monday to Friday |
| `0 0 1 * *` | midnight on the first of each month |
| `@reboot` | once, when the system starts |
| `@daily` | once a day (same as `0 0 * * *`) |

Times use the server's time zone (check with `timedatectl`), and many servers run in UTC. A site such as crontab.guru explains any expression in plain English if you want to double-check one.

## The cron environment trap

cron does **not** run your jobs in the same environment as your terminal. Its `PATH` is minimal (often `/usr/bin:/bin`), your shell aliases and profile are not loaded, and the working directory is your home folder. That is why "it works when I run it, but not from cron" is so common. The fixes:

- Use **absolute paths** for the script and for anything it calls.
- Set variables at the top of the crontab if needed: `PATH=/usr/local/bin:/usr/bin:/bin`.
- Make the script itself robust (shebang, `set -euo pipefail`, checks), as in the previous lesson.

## Capture the output

By default cron tries to email a job's output to the user, which usually goes nowhere on a server. Redirect it to a log file instead:

```cron
30 2 * * * /home/alex/bin/backup.sh /home/alex/projects /home/alex/backups >> /home/alex/backup.log 2>&1
```

`>>` appends standard output to the log, and `2>&1` sends errors to the same place. Because the script exits non-zero on failure and prints "ERROR: ...", the log tells you exactly what went wrong.

## Did it run?

```bash
grep CRON /var/log/syslog | tail          # Ubuntu: cron records each job it starts
journalctl -u cron --since today          # the same via the journal (unit "crond" on Red Hat)
tail /home/alex/backup.log                # your job's own output
ls -lt /home/alex/backups | head -3       # and the result itself
```

The syslog entry proves cron *started* the job. Only your log and the result prove it *worked*. For important jobs, monitor the result: alert if the newest backup is older than a day.

## Worked example: test with a short interval first

Do not wait until 02:30 to find out a job is broken. Schedule it every two minutes, watch it run, then change it to the real time.

```cron
*/2 * * * * /home/alex/bin/backup.sh /home/alex/projects /home/alex/backups >> /home/alex/backup.log 2>&1
```

After a few minutes, check the log and the backups folder. When it works, `crontab -e` again and change `*/2 * * * *` to `30 2 * * *`.

### Avoid overlapping runs

If a job can take longer than its interval, two copies can run at once. `flock` prevents that:

```cron
*/5 * * * * /usr/bin/flock -n /tmp/backup.lock /home/alex/bin/backup.sh /home/alex/projects >> /home/alex/backup.log 2>&1
```

## systemd timers, in brief

A **timer** unit triggers a **service** unit on a schedule. Advantages: output goes to the journal automatically (`journalctl -u backup`), `Persistent=true` runs a job that was missed while the machine was off, and `systemctl list-timers` shows when each timer last ran and will next run.

```ini
# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target
```

It needs a matching `backup.service` with `ExecStart=` pointing at the script, then `sudo systemctl daemon-reload` and `sudo systemctl enable --now backup.timer`. Use cron for simple per-user jobs; reach for timers when you want journald logging, missed-run catch-up or dependencies on other services.

## Common mistakes

- **Relative paths or commands not on cron's PATH.**
- **No output redirection,** so failures vanish silently.
- **Confusing the fields,** for example `* 2 * * *`, which runs every minute from 02:00 to 02:59.
- **Forgetting the time zone** of the server.
- **Running `crontab -r` when you meant `crontab -e`.** Keep a copy: `crontab -l > ~/crontab.backup`.
- **Only checking that cron started the job,** never that it succeeded.

## Recap

- cron runs jobs from each user's crontab on a five-field schedule (`minute hour day month weekday`).
- Use absolute paths, redirect output with `>> log 2>&1`, and test with a short interval first.
- Check `grep CRON /var/log/syslog` or `journalctl -u cron`, your log file and the result.
- systemd timers add journald logging, catch-up of missed runs and `systemctl list-timers`.

## Try it

The Do tab has you schedule your backup script, prove it ran, and move it to a real nightly time.


Schedule your backup script with cron, log its output, and know when a systemd timer is the better choice.

Scheduling jobs

Linux Crash Course - Scheduling Tasks with Cron

Linux/Mac Tutorial: Cron Jobs - How to Schedule Commands with crontab

Log in with SSH keys and harden SSH, write a bash backup script with checks, and schedule it with cron.

Linux Administration

About this course

Course content

The Linux System

Running Services

Automation and Access

Where this course leads

Next course on the path

Careers that use it