Natural Language Processing: A Walkthrough with Great Expectations

Concepts, code and practice questions from the improved notebook (NLP_improved.ipynb), applied to Charles Dickens’s novel.

Natural Language Processing (NLP) teaches computers to read and extract meaning from human language. Text is messy (punctuation, capitalization, ambiguity), so the work follows a pipeline: load → split into units → clean → count, tag or score → visualize. We run it on Great Expectations (1861), a long novel with a rich cast, which lets us check whether the statistics recover its characters and tone.

1. Setup and libraries

import os, re
from collections import Counter
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import nltk
from nltk.corpus import stopwords, wordnet
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import sent_tokenize
from nltk.util import ngrams
from nltk.collocations import BigramCollocationFinder, BigramAssocMeasures
from wordcloud import WordCloud
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

TEXT_FILE = "great_expectations.txt"   # change to your copy of the novel

Downloading NLTK resources

for pkg in ["punkt", "punkt_tab", "stopwords", "wordnet", "omw-1.4",
            "averaged_perceptron_tagger", "averaged_perceptron_tagger_eng"]:
    nltk.download(pkg, quiet=True)

NLTK ships its code separately from its data. Punkt models find sentence and word boundaries; stopwords are word lists; WordNet is a lexical database for lemmatization; the perceptron tagger assigns parts of speech (newer NLTK versions need the _eng file).

2. Loading the text

with open(TEXT_FILE, "r", encoding="utf-8") as fh:
    filedata = fh.read()
print(f"{len(filedata):,} characters")
print(filedata[:500])

The file is read into one Python string. Always give encoding="utf-8": Dickens’s text has curly quotes and em dashes that break on some systems otherwise. Printing a slice is a sanity check.

The text has hard line breaks in mid-sentence. They come from file layout, not grammar, so they must be handled before splitting sentences (section 9).

3. Corpus and its structure

from nltk.corpus.reader.plaintext import PlaintextCorpusReader
folder = os.path.dirname(os.path.abspath(TEXT_FILE))
corpus = PlaintextCorpusReader(folder, os.path.basename(TEXT_FILE), encoding="utf-8")
raw_text = corpus.raw()

corpus_stats = pd.Series({"files": len(corpus.fileids()),
    "paragraphs": len(corpus.paras()), "sentences": len(corpus.sents()),
    "words": len(corpus.words())})

paragraphs: 3908   sentences: 9190   words: 226639

The reader builds the hierarchy paragraphs → sentences → words. These descriptive statistics are the first thing to check in any text dataset. The reader uses its own rules, so later counts will not match exactly.

4. Tokenization

spaced_text = raw_text.replace("—", " — ")
tokens_raw = nltk.word_tokenize(spaced_text)

Total tokens: 225604   ['Chapter', 'I', '.', 'My', 'father', '’', 's', 'family', 'name', ...]

Quick check: token vs type?

5. Cleaning: punctuation, case and short words

We apply the normalization steps one at a time and record the token count after each, so each effect is visible.

TITLES = {"mr", "mrs"}
WORD_RE = re.compile(r"[a-z]+(?:['’-][a-z]+)*")

def is_valid(tok):
    """True for alphabetic words of 3+ letters (or a title)."""
    if tok in TITLES:
        return True
    return len(tok) > 2 and not tok.endswith("n't") and WORD_RE.fullmatch(tok) is not None

tokens_no_punct = [t for t in tokens_raw if any(c.isalnum() for c in t)]
tokens_lower    = [t.lower().rstrip(".") for t in tokens_no_punct]
tokens_words    = [t for t in tokens_lower if is_valid(t)]

Quick check: why can lower-casing be a bad idea?

6. Frequency distributions

freq_all = nltk.FreqDist(tokens_words)
print("Top 20 words:", freq_all.most_common(20))

[('the', 8119), ('and', 6956), ('that', 3022), ('was', 2809), ('you', 2214), ('had', 2087), ...]

The top words are grammatical glue and would top the list for almost any English text. That motivates stop-word removal.

7. Stop words

stop_words = set(stopwords.words("english"))
tokens_nostop = [t for t in tokens_words if t not in stop_words]

In the original run this shrank the data to 82,143 tokens, so more than half of the words were stop words. Character names now appear (Joe, Pip, Herbert, Havisham, Wemmick, Estella…), but dialogue words such as said and would still dominate.

Customizing the list

custom_stop_words = {"said", "say", "would", "could", "come"}
stop_words |= custom_stop_words          # set union
tokens_clean = [t for t in tokens_nostop if t not in stop_words]

Stop-word lists are domain-dependent: in a novel, said is noise; in a study of dialogue it might be the main signal. The original removal of these words left 79,353 tokens.

Quick check: why add said and would?

8. Visualizing word frequency

The notebook now plots frequencies three ways.

8.1 Bar chart

top = pd.DataFrame(freq_clean.most_common(25), columns=["word", "count"])
sns.barplot(data=top, y="word", x="count", color="steelblue")

A bar chart is the easiest way to compare exact counts of the top words.

8.2 Cumulative frequency plot

freq_clean.plot(50, cumulative=True)

A cumulative plot shows a running total. A steep early rise that flattens means a few words account for a large share of the text, typical of language (Zipf’s law).

8.3 Word cloud

cloud = WordCloud(stopwords=stop_words, max_words=100, background_color="white",
                  width=900, height=450).generate(" ".join(tokens_clean))
plt.imshow(cloud, interpolation="bilinear"); plt.axis("off")

A word cloud sizes each word by frequency. " ".join(...) turns the token list back into one string, which WordCloud expects. It is good for a quick impression, not for precise comparison.

Quick check: what does a steep-then-flat cumulative curve tell you?

9. Sentences and part-of-speech tagging

9.1 Splitting into sentences

POS tagging and sentiment work best on whole sentences, so we build a table with one row per sentence, chapter by chapter (the novel has 59 chapters).

parts = re.split(r"(?m)^Chapter [IVXLC]+\.\s*$", raw_text)
chapter_texts = parts[1:]

ABBREV_END = re.compile(r"\b(Mr|Mrs|Ms|Dr|St)\.$")

def split_sentences(text):
    text = re.sub(r"\s+", " ", text.replace("—", " — ")).strip()
    merged, carry = [], ""
    for s in sent_tokenize(text):
        s = f"{carry} {s}".strip()
        if ABBREV_END.search(s):
            carry = s; continue
        merged.append(s); carry = ""
    if carry: merged.append(carry)
    return merged

rows = [(ch, s) for ch, txt in enumerate(chapter_texts, start=1)
        for s in split_sentences(txt)]
df = pd.DataFrame(rows, columns=["chapter", "sentence"])

9.2 Part-of-speech (POS) tagging

sample = df.loc[1, "sentence"]
nltk.pos_tag(nltk.word_tokenize(sample))

# tag every sentence in context
tagged_sents = [nltk.pos_tag(nltk.word_tokenize(s)) for s in df["sentence"]]
tag_counts = Counter(tag for sent in tagged_sents for _, tag in sent)

The notebook then charts the 12 most common tags, and lists the most frequent nouns, adjectives and verbs with a helper:

def top_by_pos(prefixes, n=10):
    c = Counter()
    for sent in tagged_sents:
        for tok, tag in sent:
            t = tok.lower().rstrip(".")
            if tag.startswith(prefixes) and is_valid(t) and t not in stop_words:
                c[t] += 1
    return c.most_common(n)

top_by_pos(("NN",)); top_by_pos(("JJ",)); top_by_pos(("VB",))

Quick check: why tag whole sentences rather than the filtered token list?

10. Lemmatization

lemmatizer = WordNetLemmatizer()
lemmatizer.lemmatize("called")             # 'called'  (assumed noun)
lemmatizer.lemmatize("called", pos="v")    # 'call'
lemmatizer.lemmatize("better", pos="a")    # 'good'
lemmatizer.lemmatize("mice")               # 'mouse'

WordNetLemmatizer looks words up in WordNet. It assumes every word is a noun unless told otherwise, so verbs like called, saw and came were left unchanged in the original, and the surname Jaggers became jagger.

def wn_pos(tag):
    if tag.startswith("J"): return wordnet.ADJ
    if tag.startswith("V"): return wordnet.VERB
    if tag.startswith("R"): return wordnet.ADV
    return wordnet.NOUN

def lemmatize_sentence(tagged):
    out = []
    for tok, tag in tagged:
        t = tok.lower().rstrip(".")
        if not is_valid(t) or t in stop_words:
            continue
        lemma = t if tag in ("NNP", "NNPS") else lemmatizer.lemmatize(t, wn_pos(tag))
        if lemma not in stop_words:
            out.append(lemma)
    return out

sent_lemmas = [lemmatize_sentence(s) for s in tagged_sents]   # one list per sentence
lemmas = [w for s in sent_lemmas for w in s]                  # flat list for the book

Quick check: stemming vs lemmatization?

11. N-grams and collocations

def count_ngrams(n, top=10):
    c = Counter(g for sent in sent_lemmas for g in ngrams(sent, n))
    return c.most_common(top)

count_ngrams(2, 15); count_ngrams(3, 15)

In the original run, the top bigrams were miss havisham (299), mr. jagger (214), mr. wopsle (110), mr. pumblechook (89), dear boy (62) and mrs. joe (60). Top trigrams included pale young gentleman (19), mill pond bank (12) and three jolly bargeman (7). Your new counts will differ somewhat because of the improved cleaning.

Collocations with PMI

Raw counts favour pairs that include very common words. Collocations use PMI (pointwise mutual information) to find pairs that occur together more often than chance would predict, which highlights true phrases and names.

finder = BigramCollocationFinder.from_documents(sent_lemmas)
finder.apply_freq_filter(10)            # ignore pairs seen fewer than 10 times
for pair in finder.nbest(BigramAssocMeasures().pmi, 20):
    print(" ".join(pair))

Quick check: how many trigrams are in a text of N tokens?

Quick check: why can high-PMI bigrams be dominated by rare words?

12. Sentiment analysis (VADER)

analyzer = SentimentIntensityAnalyzer()
scores = df["sentence"].apply(analyzer.polarity_scores).apply(pd.Series)
df = pd.concat([df.drop(columns=["neg", "neu", "pos", "compound"], errors="ignore"), scores], axis=1)

Classifying sentences two ways

strict = pd.Series({"positive": (df["compound"] > 0).sum(),
                    "negative": (df["compound"] < 0).sum(),
                    "neutral":  (df["compound"] == 0).sum()})

df["label"] = np.select([df["compound"] >= 0.05, df["compound"] <= -0.05],
                        ["positive", "negative"], default="neutral")

The strict rule matches the original (any positive score is positive). The VADER thresholds treat scores between −0.05 and 0.05 as neutral, which is the recommended convention. The original strict run gave:

The spike at 0 is made of sentences with no words in VADER’s lexicon, which therefore score exactly neutral. Ignoring it, scores spread widely on both sides.

Sentiment over the course of the book

chapter_sent = df.groupby("chapter")["compound"].mean()   # average per chapter
df["compound"].rolling(300).mean()                         # smoothed sentence-level arc

df.nlargest(5, "compound")[["chapter", "compound", "sentence"]]    # most positive
df.nsmallest(5, "compound")[["chapter", "compound", "sentence"]]   # most negative

Averaging per chapter and smoothing across sentences reveals the story’s emotional arc. Listing the most extreme sentences is a useful sanity check on whether the scores make sense.

Quick check: what does a compound score of exactly 0.0 mean, and why so many?

Quick check: why might VADER misjudge this novel?

13. Characters across the book

Because proper nouns were kept intact, we can see where each main character is mentioned. In a dispersion plot, each mark is one occurrence at that position in the text.

characters = ["pip", "joe", "estella", "havisham", "magwitch", "jaggers", "herbert", "wemmick", "biddy"]
nltk.Text(lemmas).dispersion_plot(characters)

Compare when each character first appears and where mentions cluster, then check whether that matches the story as you know it (for example, Magwitch is largely absent from the opening chapters).

14. Pipeline summary

summary = pd.DataFrame({"tokens": step_counts})
summary["removed vs previous"] = -summary["tokens"].diff().fillna(0).astype(int)

Key lessons

Glossary

15. Practice questions

Part A: Conceptual questions

1. List the main steps of the text pipeline in this notebook, in order.

2. Which is larger for a typical book: tokens or types? Why?

3. Why is a set better than a list for stop words?

4. Jaggers became jagger in the original. Why, and how was it fixed?

5. Which technique finds the most distinctive phrases rather than the most frequent?

6. Interpret: neg=0.0, neu=0.8, pos=0.2, compound=0.42.

7. Name two reasons a word cloud alone is a poor basis for conclusions.

Part B: Coding exercises

Try each in a notebook cell first. All use variables created earlier in NLP_improved.ipynb, so run it from the top.

1. Compute the type–token ratio of tokens_clean. What does a higher value suggest?

ttr = len(set(tokens_clean)) / len(tokens_clean)
print(f"Type-token ratio: {ttr:.3f}")

2. Show the 15 most common words of 8 letters or more.

long_words = nltk.FreqDist(t for t in tokens_clean if len(t) >= 8)
print(long_words.most_common(15))

3. Find the 10 most common trigrams that do not contain mr or mrs.

c = Counter(g for sent in sent_lemmas for g in ngrams(sent, 3)
            if not {"mr", "mrs"} & set(g))
for g, n in c.most_common(10):
    print(" ".join(g), n)

4. Create a word cloud using only the adjectives.

adjectives = [t.lower().rstrip(".") for sent in tagged_sents for t, tag in sent
              if tag.startswith("JJ") and is_valid(t.lower().rstrip("."))
              and t.lower() not in stop_words]
cloud = WordCloud(background_color="white", width=900, height=450).generate(" ".join(adjectives))
plt.figure(figsize=(11, 6)); plt.imshow(cloud, interpolation="bilinear"); plt.axis("off"); plt.show()

5. Which chapter has the most negative average sentiment, and which the most positive?

chapter_sent = df.groupby("chapter")["compound"].mean()
print("Most negative:", chapter_sent.idxmin(), round(chapter_sent.min(), 3))
print("Most positive:", chapter_sent.idxmax(), round(chapter_sent.max(), 3))

6. Compare average sentiment of sentences mentioning Estella with those mentioning Joe.

for name in ["Estella", "Joe"]:
    mask = df["sentence"].str.contains(rf"\b{name}\b")
    print(f"{name}: {mask.sum()} sentences, mean compound = {df.loc[mask, 'compound'].mean():.3f}")

7. Challenge: write top_words(chapter_number, n=10) returning the n most frequent lemmas of one chapter.

def top_words(chapter_number, n=10):
    idx = df.index[df["chapter"] == chapter_number]
    c = Counter(w for i in idx for w in sent_lemmas[i])
    return c.most_common(n)

print(top_words(1))
print(top_words(59))

Going further