Storage

Why it matters

Almost every cloud system stores files somewhere: user uploads, logs, backups, website assets, training datasets and model files. Object storage is the cheapest and most durable place to keep them, and it is also where many of the largest data leaks have happened, through buckets that were accidentally made public. Knowing how to store data and how to lock it down is a core skill for both cloud and ML engineers.

Three kinds of cloud storage

How object storage works

In Amazon S3 you create a bucket (a container with a globally unique name) in a region. Each object has a key (its name, which can contain slashes so it looks like a path), the data itself, and metadata:

s3://acme-ml-datasets-2026/raw/2026/10/sales.csv
     bucket                key

There are no real folders; raw/2026/10/ is just part of the key, and the console displays it as folders for convenience. S3 Standard stores each object across several availability zones and is designed for extremely high durability, so losing an object to hardware failure is very unlikely. Losing it to human error is much more likely, which is why versioning (keeping old versions when an object is overwritten or deleted) is worth turning on for important buckets.

Storage classes trade retrieval speed for price: Standard for frequently used data, Infrequent Access for data read a few times a year, and Glacier classes for archives. Lifecycle rules move or delete objects automatically, for example "move logs to Glacier after 30 days, delete after 365".

Access control

By default a new S3 bucket is private, and Block Public Access is on. Access is then granted in layers:

• IAM policies say what a user or role can do (next module).
• Bucket policies are JSON documents attached to the bucket that say who can do what with its objects.
• Pre-signed URLs give one person temporary access to one object without making anything public.

A bucket policy that allows one IAM role to read objects under reports/ looks like this:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::111122223333:role/report-reader" },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::acme-reports-2026/reports/*"
  }]
}

Encryption at rest is applied by default to new S3 objects, and you can choose provider-managed or customer-managed keys.

Worked example with the CLI

If you have the AWS CLI configured (with a user that is not root), the everyday commands are short:

aws s3 mb s3://yourname-practice-2026-10 --region eu-west-1   # make bucket
echo "hello cloud" > hello.txt
aws s3 cp hello.txt s3://yourname-practice-2026-10/notes/hello.txt
aws s3 ls s3://yourname-practice-2026-10/notes/
aws s3 presign s3://yourname-practice-2026-10/notes/hello.txt --expires-in 300
aws s3 rb s3://yourname-practice-2026-10 --force                # delete bucket and contents

The presign command prints a URL that works for 300 seconds. Anyone with the link can download the file during that time, so treat it like a password.

Common mistakes

• Turning off Block Public Access to "make sharing easier". Use pre-signed URLs or a CDN instead.
• Putting secrets or personal data in a bucket without knowing who can read it.
• No versioning on important data, so an accidental delete is permanent.
• Storing millions of tiny files with no lifecycle rule and paying for them forever.
• Forgetting request and data-transfer charges when a public dataset gets popular.

Quick recap

• Block storage is a VM's disk, file storage is a shared folder, object storage holds objects in buckets reached over HTTPS.
• Keys look like paths but are just names; turn on versioning and lifecycle rules where they matter.
• Buckets are private by default; grant access with IAM and bucket policies, and share single files with pre-signed URLs.

Try it

In the Do tab you create a private bucket, upload a file, share it with a pre-signed link and delete everything.