Pipes, grep, and redirection

Why it matters

The real power of the command line is combining small tools. One line can find every error in a log, count them by type and save a report, on a file far too big to open in an editor. Developers use this to debug, data scientists to peek at datasets, and security engineers to sift logs for attacks.

Streams: input, output, errors

Every command has three streams:

• standard input (stdin): what it reads, by default the keyboard,
• standard output (stdout): normal results, by default the screen,
• standard error (stderr): error messages, also the screen by default.

Redirection and pipes reconnect these streams.

Redirection: send output to a file

$ ls > files.txt          # stdout to a file (creates or OVERWRITES it)
$ echo "more" >> files.txt  # append to the end instead
$ ls nothere 2> errors.txt  # stderr to a file
$ sort < names.txt          # read stdin from a file

> silently replaces the file's contents. Use >> when you want to add. Never redirect into the file you are reading: sort data.txt > data.txt empties data.txt before sort reads it.

A practice log

Create this file in ~/cli-practice (the <<'EOF' part is a "here document": everything up to EOF becomes the file):

cat > app.log <<'EOF'
2026-10-01 09:00:01 INFO server started
2026-10-01 09:02:13 WARN disk 80% full
2026-10-01 09:05:44 ERROR db timeout
2026-10-01 09:06:02 INFO request /home 200
2026-10-01 09:07:30 ERROR db timeout
2026-10-01 09:09:12 ERROR auth failed
2026-10-01 09:10:00 INFO request /login 200
2026-10-01 09:11:45 WARN slow request /search
EOF

grep: find lines that match

$ grep ERROR app.log         # lines containing ERROR
$ grep -i error app.log      # ignore case
$ grep -n auth app.log       # show line numbers
6:2026-10-01 09:09:12 ERROR auth failed
$ grep -c ERROR app.log      # count matching lines
3
$ grep -v INFO app.log       # lines that do NOT match (5 lines)
$ grep -r "TODO" project/    # search every file in a folder

Put patterns with spaces or special characters in quotes: grep "db timeout" app.log.

Pipes: output of one command into the next

The | (pipe) sends one command's stdout into the next command's stdin:

$ grep ERROR app.log | wc -l     # count error lines: 3
$ ls -l | less                   # page through a long listing
$ history | grep git             # find earlier git commands

Useful pipeline tools:

Worked example: which errors happen most?

Build the pipeline one step at a time and look at the output after each step:

$ grep ERROR app.log
2026-10-01 09:05:44 ERROR db timeout
2026-10-01 09:07:30 ERROR db timeout
2026-10-01 09:09:12 ERROR auth failed
$ grep ERROR app.log | cut -d' ' -f4-
db timeout
db timeout
auth failed
$ grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn
      2 db timeout
      1 auth failed
$ grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn > error-report.txt

-f4- means "field 4 to the end". The last line saves the report instead of printing it. The same pipeline works on a million-line log.

Common mistakes

• > when you meant >>, overwriting a file you wanted to keep.
• uniq without sort first, so non-adjacent duplicates are not merged.
• Forgetting quotes around patterns with spaces: grep db timeout app.log searches for db in two files named timeout and app.log.
• Writing one giant pipeline at once. Build it step by step.

Quick recap

• > writes (overwrites), >> appends, 2> captures errors, < reads from a file.
• grep finds lines: -i, -n, -c, -v, -r.
• | chains commands; sort | uniq -c | sort -rn is the classic "count and rank" pattern.

Try it

Open the Do tab to analyse the practice log with pipelines.