Run a Linux server with confidence: users and permissions, services, packages, logs, SSH keys, bash scripts and scheduled jobs.

Linux Administration

Make sure the server is installed and running: `sudo apt install -y openssh-server`, then `systemctl status ssh`. On a local VM, take a snapshot now.

Decide where you will connect from: your host computer into the VM (VirtualBox: add a NAT port-forwarding rule from host port 2222 to guest port 22, then connect with `-p 2222` to 127.0.0.1), a cloud VM's public IP, or simply `localhost` from inside the practice system.

On the client, create a key with a passphrase: `ssh-keygen -t ed25519 -C "practice key"`.

Copy the public key to the server with `ssh-copy-id` (for example `ssh-copy-id -p 2222 youruser@127.0.0.1`). Cloud VMs often install your key at launch; check `~/.ssh/authorized_keys` there.

Log in with the key in a new terminal and confirm you are not asked for the account password. Run `ls -la ~/.ssh` on the server and check the permissions are 700 for the folder and 600 for authorized_keys.

Keeping that session open, create `/etc/ssh/sshd_config.d/01-hardening.conf` with `sudo nano` containing `PermitRootLogin no`, `PasswordAuthentication no` and `KbdInteractiveAuthentication no` on separate lines.

Test the config with `sudo sshd -t`, then confirm the effective values with `sudo sshd -T | grep -iE "passwordauth|permitroot"`.

Restart with `sudo systemctl restart ssh`, then open a NEW terminal and log in again with the key before closing the old session.

Prove passwords are refused: `ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password youruser@host` should fail with Permission denied (publickey).

Add a `Host practice` entry to `~/.ssh/config` on the client and log in with `ssh practice`.

## Why it matters

SSH (Secure Shell) is how administrators reach almost every Linux server: cloud VMs, database hosts, network devices, build machines. It encrypts everything between you and the server. It is also one of the most attacked services on the internet: any server with SSH open to the world sees automated password-guessing attempts within minutes. Logging in with **keys** and turning off password logins removes that whole class of attack.

## How SSH works, briefly

- The **SSH server** (`sshd`, in the `openssh-server` package) listens on TCP port 22.
- The **client** (`ssh`) connects: `ssh alex@192.0.2.10`.
- On first connection the server proves its identity with a **host key**. Your client shows its fingerprint and asks whether to trust it, then remembers it in `~/.ssh/known_hosts`. If the host key later changes, the client warns loudly: that can mean the server was rebuilt, or that someone is intercepting the connection. Find out which before continuing.
- You then authenticate, with a password or with a key.

## Key pairs

A key pair is two linked files:

- The **private key** (`~/.ssh/id_ed25519`) stays on your computer. Never copy it to a server, email it, paste it into chat or commit it to Git.
- The **public key** (`~/.ssh/id_ed25519.pub`) can be shared freely. Putting it in a server's `~/.ssh/authorized_keys` means "whoever holds the matching private key may log in as this user".

When you connect, the server sends a challenge that only the private key can answer. The private key never leaves your machine.

### Create a key

```bash
ssh-keygen -t ed25519 -C "alex laptop 2026"
```

Accept the default path and **set a passphrase**. The passphrase encrypts the private key file, so a stolen laptop does not mean stolen server access. `ssh-agent` remembers the unlocked key for your session so you do not type the passphrase every time:

```bash
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
```

### Install the public key on the server

```bash
ssh-copy-id alex@192.0.2.10
```

This appends your public key to `~/.ssh/authorized_keys` on the server and sets the right permissions. Without `ssh-copy-id`, append the `.pub` file's single line manually. SSH is strict about permissions and silently ignores keys if they are too open:

```bash
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys ~/.ssh/id_ed25519
```

Now `ssh alex@192.0.2.10` should log you in without asking for the account password (it may ask for the key's passphrase).

### A client config file

Save typing with `~/.ssh/config` on your own computer:

```text
Host practice
    HostName 192.0.2.10
    User alex
    Port 22
    IdentityFile ~/.ssh/id_ed25519
```

Then `ssh practice` is enough. The same file works for `scp` and Git over SSH.

## Hardening the server

Server settings are in `/etc/ssh/sshd_config` and in files under `/etc/ssh/sshd_config.d/`. For most settings sshd uses the **first** value it reads, and the `.d` files are read first in name order, so put your changes in a file with a low number, for example `/etc/ssh/sshd_config.d/01-hardening.conf`:

```text
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers alex
```

- **`PermitRootLogin no`:** log in as yourself and use `sudo`, so every action is tied to a person.
- **`PasswordAuthentication no`:** password guessing becomes useless.
- **`AllowUsers`:** only the named accounts may log in over SSH.

### Apply changes without locking yourself out

This is the part that matters most. A mistake here can leave you unable to reach a remote server.

1. **Confirm key login works first,** in a separate terminal.
2. **Keep your current session open** while you change anything.
3. **Test the config:** `sudo sshd -t` prints nothing when it is valid.
4. **Check the effective values:** `sudo sshd -T | grep -iE "passwordauth|permitroot"`.
5. **Restart:** `sudo systemctl restart ssh` (the unit is `sshd` on the Red Hat family).
6. **Open a new terminal and log in again** before closing the old session.

On a cloud VM, also know where the provider's web console is, so you can get in if SSH breaks. A firewall such as `ufw` (`sudo ufw allow OpenSSH` *before* `sudo ufw enable`) and a tool like fail2ban add further layers, but keys plus no passwords is the big win.

## Copying files over SSH

```bash
scp report.txt practice:/tmp/
scp practice:/var/log/syslog ./syslog-copy
rsync -av ./site/ practice:~/site/      # copies only what changed
```

## Common mistakes

- **Sharing or committing the private key.** If it leaks, remove its public key from every `authorized_keys` and make a new pair.
- **No passphrase on the key.**
- **Wrong permissions on `~/.ssh`,** so the server ignores the key and falls back to a password.
- **Disabling passwords before testing key login,** or closing the only session before testing the new config.
- **Typing `yes` to a changed host key warning without checking why.**

## Recap

- SSH encrypts remote logins; the server proves itself with a host key stored in `known_hosts`.
- `ssh-keygen -t ed25519` makes a key pair; the private key stays with you, the public key goes in `authorized_keys`.
- Harden with `PermitRootLogin no` and `PasswordAuthentication no`, tested with `sshd -t`, while keeping a session open.
- `~/.ssh/config`, `scp` and `rsync` make daily work faster.

## Try it

In the Do tab you will set up key login to your practice system and switch off password logins safely.


Log in to a Linux machine with SSH keys instead of passwords, and apply a few safe hardening settings.

SSH and keys

SSH Key Authentication | How to Create SSH Key Pairs

5 Must Have Tweaks to Secure OpenSSH

Create practice data: `mkdir -p ~/projects/demo && echo "hello" > ~/projects/demo/notes.txt`.

Create `~/bin/backup.sh` (`mkdir -p ~/bin`) with the shebang, `set -euo pipefail`, and `log` and `die` functions from the lesson.

Add argument handling: the source folder is required (`${1:-}`), the destination defaults to `$HOME/backups`, and the script exits with a usage message and code 1 when the source is missing or not a directory.

Add the tar command that writes a dated `.tar.gz`, followed by the two verification checks (`-s` and `tar -tzf`).

Make it executable with `chmod u+x ~/bin/backup.sh` and run `~/bin/backup.sh ~/projects`. Confirm an archive appears in `~/backups`.

Test the checks: run it with no arguments and with `/does/not/exist`, and print `echo $?` after each. Both should print a clear error and exit code 1.

Test a restore: extract the archive into `/tmp/restore-test` and compare with `diff -r ~/projects /tmp/restore-test/projects`.

Add the keep-the-newest-7 clean-up line, run the script 9 times (`for i in $(seq 9); do ~/bin/backup.sh ~/projects; sleep 1; done`), and confirm only 7 archives remain.

Install and run ShellCheck (`sudo apt install -y shellcheck`, then `shellcheck ~/bin/backup.sh`) and fix anything it reports.

## Why it matters

Anything you type more than twice is a candidate for a script. Administrators script backups, user setup, log clean-up, health checks and deployments. A good script does the same thing the same way every time, which is more reliable than a person following notes at 2 a.m. A bad script, one that keeps going after an error, can do real damage quickly. This lesson focuses on writing scripts that **check before they act**.

## Anatomy of a script

```bash
#!/usr/bin/env bash
# backup.sh: archive a folder into a dated tar.gz
set -euo pipefail

echo "Hello from $0"
```

- The first line, the **shebang**, tells the system which interpreter runs the file.
- `set -e` stops the script when a command fails. `set -u` treats unset variables as errors (so a typo cannot expand to an empty string). `set -o pipefail` makes a pipeline fail if any command in it fails.
- Make it executable and run it: `chmod u+x backup.sh`, then `./backup.sh`.

## The building blocks

### Variables and quoting

```bash
name="web logs"
echo "Backing up $name"          # double quotes expand variables
echo 'Literal $name'             # single quotes do not
today=$(date +%F)                # command substitution: 2026-10-23
```

No spaces around `=`. **Always quote variables** (`"$file"`), or a value containing a space becomes two arguments.

### Arguments

`$1`, `$2`... are the script's arguments, `$#` is how many there are, and `"$@"` is all of them.

```bash
src="${1:-}"                     # first argument, or empty if missing
dest="${2:-$HOME/backups}"       # default value when not given
```

### Tests and if

```bash
if [[ -d "$src" ]]; then echo "folder exists"; fi
if [[ ! -r "$src" ]]; then echo "cannot read $src" >&2; exit 1; fi
```

Useful tests: `-d` directory, `-f` regular file, `-e` exists, `-r` readable, `-w` writable, `-z` empty string, `-n` non-empty, and `-eq`/`-lt`/`-gt` for numbers. `>&2` sends a message to standard error, where error messages belong.

### Exit codes

Every command returns an **exit code**: `0` means success, anything else means failure. `$?` holds the last one. Your scripts should `exit 0` on success and a non-zero code on failure, so other tools (and cron, next lesson) can tell what happened.

### Loops and functions

```bash
for f in "$dest"/*.tar.gz; do
  echo "found $f"
done

log() { echo "$(date '+%F %T') $*"; }
log "starting"
```

## Worked example: a backup script with checks

```bash
#!/usr/bin/env bash
# backup.sh SRC [DEST] - archive SRC into DEST/name-YYYY-mm-dd-HHMMSS.tar.gz
set -euo pipefail

log() { echo "$(date '+%F %T') $*"; }
die() { echo "ERROR: $*" >&2; exit 1; }

src="${1:-}"
dest="${2:-$HOME/backups}"
keep=7

[[ -n "$src" ]]  || die "usage: $0 SRC [DEST]"
[[ -d "$src" ]]  || die "source folder not found: $src"
[[ -r "$src" ]]  || die "cannot read: $src"
mkdir -p "$dest"
[[ -w "$dest" ]] || die "cannot write to: $dest"

name="$(basename "$src")-$(date +%F-%H%M%S).tar.gz"
archive="$dest/$name"

log "archiving $src to $archive"
tar -czf "$archive" -C "$(dirname "$src")" "$(basename "$src")"

# verify: the archive must exist, be non-empty and be readable by tar
[[ -s "$archive" ]] || die "archive is empty: $archive"
tar -tzf "$archive" > /dev/null || die "archive is corrupt: $archive"

# keep only the newest $keep archives for this source
ls -1t "$dest"/"$(basename "$src")"-*.tar.gz | tail -n +$((keep + 1)) | xargs -r rm --
log "done: $(du -h "$archive" | cut -f1)"
```

What makes it trustworthy:

1. **Inputs are checked** before anything happens, with clear messages and a non-zero exit.
2. **`set -euo pipefail`** stops on the first unexpected failure.
3. **The result is verified:** the archive exists, is not empty and can be listed by `tar`. A backup you have not tested restoring is only a hope.
4. **Deletion is narrow:** it only removes old archives matching this source's name pattern, in the destination folder.

Run it:

```bash
./backup.sh ~/projects ~/backups
./backup.sh /does/not/exist ; echo "exit code: $?"    # ERROR ... exit code: 1
```

## Testing a restore

```bash
mkdir -p /tmp/restore-test
tar -xzf ~/backups/projects-2026-10-23-104500.tar.gz -C /tmp/restore-test
diff -r ~/projects /tmp/restore-test/projects && echo "restore matches"
```

## Check your script with ShellCheck

`shellcheck` (`sudo apt install shellcheck`) reads a script and points out unquoted variables, common traps and portability problems. Run `shellcheck backup.sh` before you trust any script.

## Common mistakes

- **Unquoted variables**, which break on spaces and can turn `rm -r $dir/` into `rm -r /` if `$dir` is empty. Quote everything and use `set -u`.
- **No checks before destructive commands** such as `rm`, `mv` or overwriting with `>`.
- **Relative paths in scripts** that later run from a different folder. Use absolute paths or paths built from arguments.
- **Ignoring exit codes**, so a failed backup still prints "done".
- **Backups stored on the same disk as the data** in real life. For practice it is fine; at work, copy them elsewhere.

## Recap

- Start scripts with a shebang and `set -euo pipefail`.
- Quote variables, check arguments with `[[ ... ]]` tests, and send errors to stderr with a non-zero exit.
- Verify what the script produced, and test restores.
- Run `shellcheck` on everything you write.

## Try it

The Do tab has you write, break and verify your own backup script on your practice system.


Write a bash backup script that checks its inputs, stops on errors and verifies its own result.

Bash scripting

Bash in 100 Seconds

Bash Scripting Tutorial for Beginners

Confirm cron is running with `systemctl status cron` (Red Hat family: `crond`), and save any existing jobs with `crontab -l > ~/crontab.backup 2>/dev/null || true`.

Check your script works with absolute paths: `/home/$USER/bin/backup.sh /home/$USER/projects` (use the real path printed by `echo $HOME`).

Run `crontab -e` and add a test job that runs every 2 minutes with absolute paths and `>> /home/youruser/backup.log 2>&1` at the end.

Wait at least 4 minutes, then check cron started it: `grep CRON /var/log/syslog | tail -5` or `journalctl -u cron --since "10 min ago"`.

Check it succeeded: `tail /home/youruser/backup.log` and `ls -lt ~/backups | head -3`.

Break it on purpose: change the source folder in the job to one that does not exist, wait 2 minutes and confirm the ERROR line appears in the log. Then fix it.

Change the schedule to a real nightly time, such as `30 2 * * *`, and confirm with `crontab -l`. Write in plain words what your schedule means.

Look at system jobs: `ls /etc/cron.daily/` and `systemctl list-timers --all | head`.

Clean up when you are done with the course: remove the job with `crontab -e` (do not use `crontab -r` if you have other jobs) and delete `~/backups` if you no longer need it.

## Why it matters

A backup script that someone has to remember to run will eventually not be run. Servers rely on scheduled jobs for backups, log clean-up, certificate renewal, report generation and health checks. **cron** is the classic Linux scheduler and is available on nearly every system; **systemd timers** are the modern alternative. Knowing both, and knowing how to tell whether a scheduled job actually ran, is a core admin skill.

## How cron works

The `cron` service wakes up every minute, reads the schedules (**crontabs**) and runs any job whose time matches. Each user has their own crontab, and jobs run as that user.

```bash
crontab -l        # list your jobs
crontab -e        # edit your jobs (opens an editor; choose nano if asked)
crontab -r        # remove ALL your jobs: be careful, there is no undo
```

System-wide jobs live in `/etc/crontab`, `/etc/cron.d/`, and the folders `/etc/cron.daily/`, `/etc/cron.weekly/` and so on, where packages drop their own maintenance scripts.

## The five time fields

```text
┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12)
│ │ │ │ ┌─ day of week (0-7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *  command to run
```

`*` means "every". `,` lists values, `-` makes a range and `/` sets a step.

| Schedule | Meaning |
|---|---|
| `30 2 * * *` | every day at 02:30 |
| `*/15 * * * *` | every 15 minutes |
| `0 9 * * 1-5` | 09:00 Monday to Friday |
| `0 0 1 * *` | midnight on the first of each month |
| `@reboot` | once, when the system starts |
| `@daily` | once a day (same as `0 0 * * *`) |

Times use the server's time zone (check with `timedatectl`), and many servers run in UTC. A site such as crontab.guru explains any expression in plain English if you want to double-check one.

## The cron environment trap

cron does **not** run your jobs in the same environment as your terminal. Its `PATH` is minimal (often `/usr/bin:/bin`), your shell aliases and profile are not loaded, and the working directory is your home folder. That is why "it works when I run it, but not from cron" is so common. The fixes:

- Use **absolute paths** for the script and for anything it calls.
- Set variables at the top of the crontab if needed: `PATH=/usr/local/bin:/usr/bin:/bin`.
- Make the script itself robust (shebang, `set -euo pipefail`, checks), as in the previous lesson.

## Capture the output

By default cron tries to email a job's output to the user, which usually goes nowhere on a server. Redirect it to a log file instead:

```cron
30 2 * * * /home/alex/bin/backup.sh /home/alex/projects /home/alex/backups >> /home/alex/backup.log 2>&1
```

`>>` appends standard output to the log, and `2>&1` sends errors to the same place. Because the script exits non-zero on failure and prints "ERROR: ...", the log tells you exactly what went wrong.

## Did it run?

```bash
grep CRON /var/log/syslog | tail          # Ubuntu: cron records each job it starts
journalctl -u cron --since today          # the same via the journal (unit "crond" on Red Hat)
tail /home/alex/backup.log                # your job's own output
ls -lt /home/alex/backups | head -3       # and the result itself
```

The syslog entry proves cron *started* the job. Only your log and the result prove it *worked*. For important jobs, monitor the result: alert if the newest backup is older than a day.

## Worked example: test with a short interval first

Do not wait until 02:30 to find out a job is broken. Schedule it every two minutes, watch it run, then change it to the real time.

```cron
*/2 * * * * /home/alex/bin/backup.sh /home/alex/projects /home/alex/backups >> /home/alex/backup.log 2>&1
```

After a few minutes, check the log and the backups folder. When it works, `crontab -e` again and change `*/2 * * * *` to `30 2 * * *`.

### Avoid overlapping runs

If a job can take longer than its interval, two copies can run at once. `flock` prevents that:

```cron
*/5 * * * * /usr/bin/flock -n /tmp/backup.lock /home/alex/bin/backup.sh /home/alex/projects >> /home/alex/backup.log 2>&1
```

## systemd timers, in brief

A **timer** unit triggers a **service** unit on a schedule. Advantages: output goes to the journal automatically (`journalctl -u backup`), `Persistent=true` runs a job that was missed while the machine was off, and `systemctl list-timers` shows when each timer last ran and will next run.

```ini
# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target
```

It needs a matching `backup.service` with `ExecStart=` pointing at the script, then `sudo systemctl daemon-reload` and `sudo systemctl enable --now backup.timer`. Use cron for simple per-user jobs; reach for timers when you want journald logging, missed-run catch-up or dependencies on other services.

## Common mistakes

- **Relative paths or commands not on cron's PATH.**
- **No output redirection,** so failures vanish silently.
- **Confusing the fields,** for example `* 2 * * *`, which runs every minute from 02:00 to 02:59.
- **Forgetting the time zone** of the server.
- **Running `crontab -r` when you meant `crontab -e`.** Keep a copy: `crontab -l > ~/crontab.backup`.
- **Only checking that cron started the job,** never that it succeeded.

## Recap

- cron runs jobs from each user's crontab on a five-field schedule (`minute hour day month weekday`).
- Use absolute paths, redirect output with `>> log 2>&1`, and test with a short interval first.
- Check `grep CRON /var/log/syslog` or `journalctl -u cron`, your log file and the result.
- systemd timers add journald logging, catch-up of missed runs and `systemctl list-timers`.

## Try it

The Do tab has you schedule your backup script, prove it ran, and move it to a real nightly time.


Schedule your backup script with cron, log its output, and know when a systemd timer is the better choice.

Scheduling jobs

Linux Crash Course - Scheduling Tasks with Cron

Linux/Mac Tutorial: Cron Jobs - How to Schedule Commands with crontab

Log in with SSH keys and harden SSH, write a bash backup script with checks, and schedule it with cron.