Text and embeddings

Why this matters

Support tickets, product reviews, clinical notes, contracts: a large share of business data is text. Models cannot read words directly; they need numbers. Embeddings turn text into vectors so that similar meanings land close together. Once text is a vector, everything you already know applies: nearest neighbours, logistic regression, clustering, search. Most practical text classification today is "embed the text, then train a small model on top".

From words to vectors

The oldest approach is bag of words: one column per vocabulary word, counting how often it appears (often weighted by TF-IDF so common words count less). It works well for simple tasks but treats "refund" and "reimbursement" as unrelated columns.

Word embeddings (word2vec, GloVe) fix that. Each word gets a dense vector of, say, 300 numbers, learned from billions of sentences so that words used in similar contexts get similar vectors. Famous example: vector("king") - vector("man") + vector("woman") lands near vector("queen").

Sentence embeddings go one step further: a transformer model reads the whole sentence, word order and all, and outputs one vector for it. These are what you will mostly use.

Measuring similarity: cosine

To compare two vectors we use cosine similarity, the cosine of the angle between them:

cosine(a, b) = (a . b) / (|a| * |b|)

It ranges from 1 (same direction) through 0 (unrelated) to -1 (opposite). Cosine ignores vector length, which is what you want: a long message and a short one about the same topic should still match.

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

cosine(np.array([1.0, 2.0]), np.array([2.0, 4.0]))   # 1.0, same direction

Worked example: routing support tickets

You want to route messages into billing, delivery, access or praise. A simple but strong recipe:

1. Embed every training message. With word vectors, average the vectors of the words you know.
2. Compute a centroid per label: the mean embedding of its training messages.
3. Predict the label whose centroid has the highest cosine similarity to the new message.

"Please refund the payment"  -> embed -> closest centroid: billing
"The package never arrived"  -> embed -> closest centroid: delivery

This nearest-centroid classifier needs only a handful of examples per class, which makes it a useful baseline and a good fit when labelled data is scarce. With more labels, train logistic regression on the embeddings instead.

The tools you will use

sentence-transformers gives you a pretrained sentence embedding model in three lines (free, runs on a laptop CPU or in Colab):

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

encoder = SentenceTransformer("all-MiniLM-L6-v2")      # small, 384-dimensional
X_train = encoder.encode(train_texts, normalize_embeddings=True)
X_test = encoder.encode(test_texts, normalize_embeddings=True)

clf = LogisticRegression(max_iter=1000).fit(X_train, train_labels)
print(clf.score(X_test, test_labels))

With normalize_embeddings=True every vector has length 1, so a plain dot product equals cosine similarity.

For a classic baseline, scikit-learn's TfidfVectorizer plus LogisticRegression is fast and surprisingly hard to beat on keyword-driven tasks. Always run it first: if TF-IDF already scores 95%, a bigger model may not be worth the cost.

For the best accuracy with thousands of labelled examples, fine-tune a small transformer (for example DistilBERT) with the Hugging Face transformers library. That updates the encoder itself for your labels, at the cost of a GPU and more care.

Practical details

• Preprocessing: with modern embeddings, keep the text mostly as written. Lowercasing and stripping punctuation help bag-of-words, but transformer models were trained on natural text.
• Long texts: embedding models have a maximum input length (often 256–512 tokens). Truncate, or split into chunks and average.
• Language and domain: a general English model may do poorly on legal, medical or non-English text. Check a domain or multilingual model.
• Class imbalance: if 90% of tickets are billing, accuracy is misleading. Report per-class precision and recall (from Machine Learning Foundations).
• Privacy: if you send text to a hosted embedding API, you are sharing it with a third party. Local models avoid that.

Common mistakes

• Comparing vectors from different models. Embeddings from two models live in different spaces; never mix them.
• Euclidean distance on unnormalised embeddings when the model was trained for cosine.
• Leakage through duplicates. Near-identical tickets in train and test inflate the score. Deduplicate before splitting.
• Skipping the baseline. Always compare with TF-IDF + logistic regression and with "predict the most common class".

Quick recap

• Embeddings map text to vectors where similar meaning means a small angle.
• Cosine similarity compares direction and ignores length.
• Embed, then classify: nearest centroid for few labels, logistic regression for more, fine-tuning for the most.
• Start with a TF-IDF baseline and check per-class metrics.

Try it

Open the Do tab. Using a small embedding table, you will write embed, cosine, fit_centroids and predict, and route five support messages to the right team.