Text and embeddings

Why this matters

Support tickets, product reviews, clinical notes, contracts: a large share of business data is text. Models cannot read words directly; they need numbers. EmbeddingsVectors that represent text so that similar meanings land close together. turn text into vectors so that similar meanings land close together. Once text is a vector, everything you already know applies: nearest neighbours, logistic regression, clustering, search. Most practical text classification today is "embed the text, then train a small model on top".

From words to vectors

The oldest approach is bag of wordsA representation with one column per vocabulary word counting how often it appears, often weighted by TF-IDF.: one column per vocabulary word, counting how often it appears (often weighted by TF-IDF so common words count less). It works well for simple tasks but treats "refund" and "reimbursement" as unrelated columns.

Word embeddings (word2vec, GloVe) fix that. Each word gets a dense vector of, say, 300 numbers, learned from billions of sentences so that words used in similar contexts get similar vectors. Famous example: vector("king") - vector("man") + vector("woman") lands near vector("queen").

Sentence embeddingsOne vector for a whole sentence, produced by a transformer that reads the words in order. go one step further: a transformer model reads the whole sentence, word order and all, and outputs one vector for it. These are what you will mostly use.

Measuring similarity: cosine

To compare two vectors we use cosine similarityThe cosine of the angle between two vectors, from 1 (same direction) to -1 (opposite); it ignores length., the cosine of the angle between them:

cosine(a, b) = (a . b) / (|a| * |b|)

It ranges from 1 (same direction) through 0 (unrelated) to -1 (opposite). Cosine ignores vector length, which is what you want: a long message and a short one about the same topic should still match.

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

cosine(np.array([1.0, 2.0]), np.array([2.0, 4.0]))   # 1.0, same direction

Worked example: routing support tickets

You want to route messages into billing, delivery, access or praise. A simple but strong recipe:

1. Embed every training message. With word vectors, average the vectors of the words you know.
2. Compute a centroidThe mean embedding of all training texts with one label. per label: the mean embedding of its training messages.
3. Predict the label whose centroid has the highest cosine similarity to the new message.

"Please refund the payment"  -> embed -> closest centroid: billing
"The package never arrived"  -> embed -> closest centroid: delivery

This nearest-centroid classifier needs only a handful of examples per class, which makes it a useful baseline and a good fit when labelled data is scarce. With more labels, train logistic regression on the embeddings instead.

The tools you will use

sentence-transformers gives you a pretrained sentence embedding model in three lines (free, runs on a laptop CPU or in Colab):

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

encoder = SentenceTransformer("all-MiniLM-L6-v2")      # small, 384-dimensional
X_train = encoder.encode(train_texts, normalize_embeddings=True)
X_test = encoder.encode(test_texts, normalize_embeddings=True)

clf = LogisticRegression(max_iter=1000).fit(X_train, train_labels)
print(clf.score(X_test, test_labels))

With normalize_embeddings=True every vector has length 1, so a plain dot product equals cosine similarity.

For a classic baseline, scikit-learn's TfidfVectorizer plus LogisticRegression is fast and surprisingly hard to beat on keyword-driven tasks. Always run it first: if TF-IDF already scores 95%, a bigger model may not be worth the cost.

For the best accuracy with thousands of labelled examples, fine-tuneUpdating a pretrained model's own weights on your labelled data for better accuracy. a small transformer (for example DistilBERT) with the Hugging Face transformers library. That updates the encoder itself for your labels, at the cost of a GPU and more care.

Practical details

• Preprocessing: with modern embeddings, keep the text mostly as written. Lowercasing and stripping punctuation help bag-of-words, but transformer models were trained on natural text.
• Long texts: embedding models have a maximum input length (often 256–512 tokens). Truncate, or split into chunks and average.
• Language and domain: a general English model may do poorly on legal, medical or non-English text. Check a domain or multilingual model.
• Class imbalance: if 90% of tickets are billing, accuracy is misleading. Report per-class precision and recall (from Machine Learning Foundations).
• Privacy: if you send text to a hosted embedding API, you are sharing it with a third party. Local models avoid that.

Common mistakes

• Comparing vectors from different models. Embeddings from two models live in different spaces; never mix them.
• Euclidean distance on unnormalised embeddings when the model was trained for cosine.
• Leakage through duplicates. Near-identical tickets in train and test inflate the score. Deduplicate before splitting.
• Skipping the baseline. Always compare with TF-IDF + logistic regression and with "predict the most common class".

Quick recap

• Embeddings map text to vectors where similar meaning means a small angle.
• Cosine similarity compares direction and ignores length.
• Embed, then classify: nearest centroid for few labels, logistic regression for more, fine-tuning for the most.
• Start with a TF-IDF baseline and check per-class metrics.

Try it

Open the Do tab. Using a small embedding table, you will write embed, cosine, fit_centroids and predict, and route five support messages to the right team.