Introduction
Every day you use natural language processing (NLP) without noticing: the spam filter in your email, the suggestions on your phone keyboard, search engines, translation apps, voice assistants and chatbots. NLP is the part of computer science that lets programs work with human language: read it, sort it, pull facts out of it, and write it.
This course builds the core ideas of NLP in plain Python, with no libraries, so you can see every step. You will tokenize text, count words, weight them with TF-IDF, search documents, classify text with Naive Bayes, build language models, tag parts of speech with an HMM, find names, correct spelling, compare word embeddings, train a byte-pair-encoding tokenizer and compute attention, the heart of modern large language models.
How it works
Common NLP tasks include:
- Classification: is this email spam? Is this review positive or negative?
- Information extraction: which people, places and dates does this article mention?
- Search and similarity: which documents best answer this query?
- Tagging and parsing: which words are nouns and verbs, and how do they connect?
- Generation: translate, summarize, answer a question or continue a text.
Language is hard for computers for three main reasons:
- Ambiguity: "I saw her duck" can be about a bird or about ducking. "Bank" can be a river bank or a money bank.
- Variety: the same idea has many forms ("buy", "bought", "purchase"), and people misspell, abbreviate and use slang.
- Context: "it" or "they" refers to something said earlier; "not bad" means good.
Almost every NLP system, from a ten-line script to a large language model, follows a pipeline:
- Clean the raw text (remove markup, fix encodings).
- Split it into sentences and then into tokens.
- Normalize the tokens: lowercase them, maybe remove very common words or cut words down to their stems.
- Turn tokens into numbers: counts, TF-IDF weights or embedding vectors.
- Model: classify, search, tag or generate.
A collection of texts is called a corpus. The number of tokens in it is its length; the number of distinct tokens is its vocabulary size. Their ratio, the type-token ratio, tells you how varied the language is. Word frequencies follow Zipf's law: a few words (the, of, and) are extremely common, and most words are rare. That one fact shapes many techniques in this course.
Example
text = "The cat sat on the mat. The dog sat too! Did the cat see the dog?"
for mark in "!?":
text = text.replace(mark, ".")
sentences = [s.strip() for s in text.split(".") if s.strip()]
print("sentences:", len(sentences))
tokens = []
for sentence in sentences:
words = sentence.lower().split()
tokens.extend(words)
print(f" {len(words)} tokens: {words}")
counts = {}
for token in tokens:
counts[token] = counts.get(token, 0) + 1
top = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:3]
print("tokens:", len(tokens), " vocabulary:", len(counts))
print("type-token ratio:", round(len(counts) / len(tokens), 2))
print("most common:", top)Line by line
- The text has three sentences ending in
.,!and?. Replacing!and?with.lets onesplit(".")find all three. - The list comprehension keeps only non-empty pieces, after
strip()removes the spaces around them. - Each sentence is lowercased and split on whitespace, so "The" and "the" become the same token.
- The dictionary
countsmaps each token to how often it occurs. - Sorting by
(-count, word)puts the most frequent words first and breaks ties alphabetically, so the result never depends on luck. - 16 tokens but only 9 distinct words: "the" alone is 5 of the 16, just as Zipf's law predicts.
Expected result
sentences: 3
6 tokens: ['the', 'cat', 'sat', 'on', 'the', 'mat']
4 tokens: ['the', 'dog', 'sat', 'too']
6 tokens: ['did', 'the', 'cat', 'see', 'the', 'dog']
tokens: 16 vocabulary: 9
type-token ratio: 0.56
most common: [('the', 5), ('cat', 2), ('dog', 2)]
In the real world
Real pipelines face messier text: web pages full of HTML, emojis, several languages in one message, and abbreviations such as "Dr." that end with a full stop but do not end a sentence. Libraries such as spaCy and NLTK provide trained sentence splitters and tokenizers for many languages. Large language models also start with a pipeline: their tokenizer splits text into sub-word pieces before the network ever sees it, which you will build yourself in the byte-pair encoding lesson.
Common mistakes
- Splitting sentences on every full stop, which breaks "Dr. Smith" and "3.5" in two.
- Forgetting to lowercase, so "The" and "the" are counted as different words.
- Leaving punctuation attached, so "dog" and "dog?" become two vocabulary entries.
- Trusting a single statistic: a type-token ratio depends heavily on how long the text is.
Best practices
- Look at your raw text before writing any code; real data always has surprises.
- Keep each pipeline stage a small function you can test on its own.
- Break ties in sorted results explicitly, so the output is the same every run.
- Print a few tokens after each stage to check that it did what you expected.
Summary
NLP makes computers work with human language: classifying, extracting, searching, tagging and generating text. Language is hard because it is ambiguous, varied and depends on context. Most systems follow a pipeline from raw text to sentences, tokens, normalized tokens, numbers and finally a model, and simple counts of tokens and words are the first step.
Quick check
Key takeaways
- NLP turns messy human text into structured data that a program can count, compare and learn from.
- Most systems follow the same pipeline: clean, split, normalize, turn into numbers, then model.
- Simple statistics such as token counts and word frequencies are the first step of almost every NLP project.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.