Project: Text Analyzer

What you'll learn
- How to plan a project as a set of small, testable functions
- How to combine files, generators, regex, and
collections.Counterin one program - How to collect results in a dataclass and save them as JSON and as a text report
- How to test the pure parts of a program with simple test functions
- How to handle real-world text problems: case, punctuation, encoding, and empty files
Why it matters
Word counters, readability checkers, SEO tools, spam filters, and search engines all start with the same basic job: read text, split it into words and sentences, count things, and find patterns. In this project you build your own text analyzer from scratch. It is small enough to finish in one sitting, but it uses almost everything from the last few modules, and it shows how separate skills fit together into a real program.
Goal and features
Your program reads a text file and produces a report like this:
Report for cafe.txt
===================
Characters: 591 (489 without spaces)
Words: 93
Sentences: 10
...
Top words:
cafe 4
tea 3
...
Emails: aziza@silkroadcafe.uz, jobs@silkroadcafe.uzFeatures:
| Feature | Tools you will use |
|---|---|
| Read a text file safely | pathlib, encoding="utf-8" |
| Count characters, words, sentences, paragraphs | regex, generators |
| Most common words, ignoring stop words | Counter, sets |
| Average word and sentence length | functions, guard clauses |
| Longest words | sorted with a key |
| Reading time estimate | simple math |
| Extract emails, URLs, and numbers | re patterns |
| Save a JSON report and a text report | json, dataclasses.asdict |
| Check the logic | test functions with assert |
Design
Before writing code, list the functions and how data flows between them. Each function does one job, and most of them are pure: they take text or words and return a result, without printing or touching files. Pure functions are easy to test.
cafe.txt
read_text(path)text(str)
remove_links(text)
iter_words(text)generatorwords(list)top_words(words)longest_words(words)reading_time(n)
count_sentences(text)
count_paragraphs(text)
extract(text)emails, urls, numbers
analyze(text)TextStats(dataclass)
format_report(stats)report.txt
json.dumps(asdict(stats))report.json
Only read_text, save_reports, and main touch files or the screen. Everything else is pure.
Step 1: Create a sample file and read it
A real user would pass their own file. To keep everything runnable, the program writes a sample text first. Always give encoding="utf-8" when reading and writing text.
from pathlib import Path
SAMPLE_TEXT = """\
The Silk Road Cafe opened in Samarkand in 2019. It started with 12 tables,
a small kitchen, and a big idea: serve green tea the way grandmothers make it.
Today the cafe welcomes about 350 guests every day!
Aziza, the owner, answers every message herself. You can write to her at
aziza@silkroadcafe.uz or visit https://silkroadcafe.uz/menu to see the menu.
Is the tea really the best in Samarkand? Many guests think so.
Next year the cafe team plans a second cafe in Bukhara. The budget is 450000000 som,
and guests can expect the same tea. Questions about jobs go to jobs@silkroadcafe.uz.
"""
Path("cafe.txt").write_text(SAMPLE_TEXT, encoding="utf-8")
def read_text(path):
return Path(path).read_text(encoding="utf-8")
text = read_text("cafe.txt")
print(text[:60])
print("Characters:", len(text))
print("Empty?", not text.strip())The Silk Road Cafe opened in Samarkand in 2019. It started w
Characters: 591
Empty? Falsenot text.strip() is True for a file that is empty or contains only spaces and newlines. You will use this check in main().
Step 2: Split text into words
The obvious approach, text.split(), is not good enough:
text = "Tea, tea and TEA! Is the tea good? It's great."
print(text.split())['Tea,', 'tea', 'and', 'TEA!', 'Is', 'the', 'tea', 'good?', "It's", 'great.']"Tea,", "tea", and "TEA!" would count as three different words. You need to lowercase the text and keep only letters. A regex does both jobs. The pattern [a-z]+(?:'[a-z]+)? means "ASCII letters a-z, optionally followed by an apostrophe and more letters", so it's stays one word. That is enough for the English sample in this project, but it is not a general word pattern: "o‘zbek gʻalaba café" becomes ['o', 'zbek', 'g', 'alaba', 'caf'], because ‘, ʻ and é are not in a-z. A generator yields words one at a time:
import re
WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
def iter_words(text):
for match in WORD_RE.finditer(text.lower()):
yield match.group()
text = "Tea, tea and TEA! Is the tea good? It's great."
print(list(iter_words(text)))['tea', 'tea', 'and', 'tea', 'is', 'the', 'tea', 'good', "it's", 'great']There is one more trap. Emails and URLs contain letters too, so aziza@silkroadcafe.uz would become the "words" aziza, silkroadcafe, uz. Remove them before counting words:
import re
WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)*\.[a-z]{2,}", re.IGNORECASE)
URL_RE = re.compile(r"https?://[^\s,]+[^\s,.]")
def remove_links(text):
text = URL_RE.sub(" ", text)
return EMAIL_RE.sub(" ", text)
def iter_words(text):
for match in WORD_RE.finditer(text.lower()):
yield match.group()
line = "Write to aziza@silkroadcafe.uz or visit https://silkroadcafe.uz/menu today."
print(list(iter_words(line)))
print(list(iter_words(remove_links(line))))['write', 'to', 'aziza', 'silkroadcafe', 'uz', 'or', 'visit', 'https', 'silkroadcafe', 'uz', 'menu', 'today']
['write', 'to', 'or', 'visit', 'today']The URL pattern ends with [^\s,.] so a period right after a link is not included in it.
Why a generator?
iter_words never builds a list itself. In this project you call list() on it because you need the words several times, but the same function could feed Counter directly while reading a huge file line by line.
Step 3: Characters, sentences, and paragraphs
A sentence ends with ., !, or ? (maybe several, like ?!) followed by whitespace or the end of the text. Paragraphs are separated by a blank line.
import re
SENTENCE_END_RE = re.compile(r"[.!?]+(?:\s|$)")
def count_sentences(text):
return len(SENTENCE_END_RE.findall(text.strip()))
def count_paragraphs(text):
return len([p for p in re.split(r"\n\s*\n", text) if p.strip()])
text = """Hello from Bukhara! Is it hot today?! Yes.
Visit silkroadcafe.uz for 3.5 hours of tea.
The end."""
print("Characters:", len(text))
print("Without spaces:", len(re.sub(r"\s", "", text)))
print("Sentences:", count_sentences(text))
print("Paragraphs:", count_paragraphs(text))Characters: 98
Without spaces: 79
Sentences: 5
Paragraphs: 3Requiring whitespace after the punctuation means silkroadcafe.uz and 3.5 do not end a sentence. Abbreviations like Dr. Karimov would still be counted wrongly. Perfect sentence detection is surprisingly hard, and "good enough" is fine for this project.
Step 4: Most common and longest words
Counter counts the words, and a set of stop words removes the boring ones ("the", "and", "in") that would otherwise always win:
from collections import Counter
STOP_WORDS = {"the", "and", "in", "a", "is", "to", "of"}
def top_words(words, n=5, stop_words=STOP_WORDS):
counts = Counter(w for w in words if w not in stop_words)
return counts.most_common(n)
def longest_words(words, n=3):
unique = set(words)
return sorted(unique, key=lambda w: (-len(w), w))[:n]
words = ["the", "tea", "in", "samarkand", "and", "the", "tea", "in", "bukhara",
"is", "the", "best", "tea", "in", "uzbekistan"]
print(top_words(words))
print(top_words(words, n=2, stop_words=set()))
print(longest_words(words))[('tea', 3), ('samarkand', 1), ('bukhara', 1), ('best', 1), ('uzbekistan', 1)]
[('the', 3), ('tea', 3)]
['uzbekistan', 'samarkand', 'bukhara']Two details matter here:
stop_wordsis a parameter with a default, so tests and other users can pass their own set.- The sort key
(-len(w), w)sorts by length (longest first), then alphabetically. Without the second part, words of equal length could come out in a different order each run, because sets have no fixed order. Stable output makes testing possible.
Step 5: Averages and reading time
Averages divide by a count, and a count can be zero. Guard against that, or an empty file will crash the program:
WORDS_PER_MINUTE = 200
def average(total, count):
return round(total / count, 2) if count else 0.0
def reading_time(word_count, wpm=WORDS_PER_MINUTE):
return round(word_count / wpm, 1)
words = ["green", "tea", "in", "samarkand"]
print("Avg word length:", average(sum(map(len, words)), len(words)))
print("Avg sentence length:", average(len(words), 1))
print("Empty text:", average(0, 0))
print("Reading time for 1,500 words:", reading_time(1500), "min")Avg word length: 4.75
Avg sentence length: 4.0
Empty text: 0.0
Reading time for 1,500 words: 7.5 minAn adult reads roughly 200 to 250 words per minute. Keeping the number in a named constant makes it easy to change.
Step 6: Extract emails, URLs, and numbers
The email and URL patterns are the ones from Step 2, and NUMBER_RE matches whole numbers and decimals like 3.5:
import re
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)*\.[a-z]{2,}", re.IGNORECASE)
URL_RE = re.compile(r"https?://[^\s,]+[^\s,.]")
NUMBER_RE = re.compile(r"\b\d+(?:\.\d+)?\b")
def remove_links(text):
text = URL_RE.sub(" ", text)
return EMAIL_RE.sub(" ", text)
def extract(text):
urls = URL_RE.findall(text)
emails = EMAIL_RE.findall(text)
numbers = NUMBER_RE.findall(remove_links(text))
return emails, urls, numbers
text = "Call 2 friends, pay 45000 som or 3.5 USD. Mail info@tour2026.uz, see https://tour2026.uz/prices."
emails, urls, numbers = extract(text)
print(emails)
print(urls)
print(numbers)['info@tour2026.uz']
['https://tour2026.uz/prices']
['2', '45000', '3.5']Numbers are searched in the text without links, otherwise the 2026 inside the domain name would be reported as a number.
Step 7: Collect results in a dataclass
A function that returns 13 values needs a clear container. A dataclass (from Dataclasses) gives each value a name, a nice repr, and a free conversion to a dictionary with asdict():
from dataclasses import dataclass, asdict
@dataclass
class TextStats:
words: int
sentences: int
top_words: list
stats = TextStats(words=93, sentences=10, top_words=[("cafe", 4), ("tea", 3)])
print(stats)
print(stats.words)
print(asdict(stats))TextStats(words=93, sentences=10, top_words=[('cafe', 4), ('tea', 3)])
93
{'words': 93, 'sentences': 10, 'top_words': [('cafe', 4), ('tea', 3)]}The analyze(text) function then calls all the helpers and fills in one TextStats. You will see it in the full program below.
Step 8: Write the reports
The text report is built by a pure function that returns a string. Saving is a separate, tiny function. This way you can test the formatting without creating files:
import json
from dataclasses import dataclass, asdict
from pathlib import Path
@dataclass
class TextStats:
words: int
sentences: int
top_words: list
def format_report(stats, title="Text report"):
lines = [title, "=" * len(title), f"Words: {stats.words}", f"Sentences: {stats.sentences}", "Top words:"]
for word, count in stats.top_words:
lines.append(f" {word:<10} {count}")
return "\n".join(lines)
stats = TextStats(words=93, sentences=10, top_words=[("cafe", 4), ("tea", 3)])
Path("report.txt").write_text(format_report(stats) + "\n", encoding="utf-8")
Path("report.json").write_text(json.dumps(asdict(stats), indent=2), encoding="utf-8")
print(Path("report.txt").read_text(encoding="utf-8"))
print(Path("report.json").read_text(encoding="utf-8"))Text report
===========
Words: 93
Sentences: 10
Top words:
cafe 4
tea 3
{
"words": 93,
"sentences": 10,
"top_words": [
[
"cafe",
4
],
[
"tea",
3
]
]
}JSON has no tuples, so ("cafe", 4) is saved as a list ["cafe", 4]. When you load the file back, you get lists.
The full program
Here is everything together. Copy it into text_analyzer.py and run it. To analyze your own file, change the path in main() and delete the line that writes the sample.
import json
import re
from collections import Counter
from dataclasses import dataclass, asdict
from pathlib import Path
SAMPLE_TEXT = """\
The Silk Road Cafe opened in Samarkand in 2019. It started with 12 tables,
a small kitchen, and a big idea: serve green tea the way grandmothers make it.
Today the cafe welcomes about 350 guests every day!
Aziza, the owner, answers every message herself. You can write to her at
aziza@silkroadcafe.uz or visit https://silkroadcafe.uz/menu to see the menu.
Is the tea really the best in Samarkand? Many guests think so.
Next year the cafe team plans a second cafe in Bukhara. The budget is 450000000 som,
and guests can expect the same tea. Questions about jobs go to jobs@silkroadcafe.uz.
"""
STOP_WORDS = {
"a", "about", "and", "at", "be", "every", "for", "go", "her", "in", "is",
"it", "many", "of", "or", "see", "so", "the", "to", "with", "you", "can",
"think", "way", "make", "it's",
}
WORDS_PER_MINUTE = 200
WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
SENTENCE_END_RE = re.compile(r"[.!?]+(?:\s|$)")
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)*\.[a-z]{2,}", re.IGNORECASE)
URL_RE = re.compile(r"https?://[^\s,]+[^\s,.]")
NUMBER_RE = re.compile(r"\b\d+(?:\.\d+)?\b")
@dataclass
class TextStats:
characters: int
characters_no_spaces: int
words: int
sentences: int
paragraphs: int
average_word_length: float
average_sentence_length: float
reading_time_minutes: float
top_words: list
longest_words: list
emails: list
urls: list
numbers: list
def read_text(path):
return Path(path).read_text(encoding="utf-8")
def remove_links(text):
text = URL_RE.sub(" ", text)
return EMAIL_RE.sub(" ", text)
def iter_words(text):
for match in WORD_RE.finditer(text.lower()):
yield match.group()
def count_sentences(text):
return len(SENTENCE_END_RE.findall(text.strip()))
def count_paragraphs(text):
return len([p for p in re.split(r"\n\s*\n", text) if p.strip()])
def top_words(words, n=5, stop_words=STOP_WORDS):
counts = Counter(w for w in words if w not in stop_words)
return counts.most_common(n)
def longest_words(words, n=3):
unique = set(words)
return sorted(unique, key=lambda w: (-len(w), w))[:n]
def reading_time(word_count, wpm=WORDS_PER_MINUTE):
return round(word_count / wpm, 1)
def extract(text):
urls = URL_RE.findall(text)
emails = EMAIL_RE.findall(text)
numbers = NUMBER_RE.findall(remove_links(text))
return emails, urls, numbers
def analyze(text):
words = list(iter_words(remove_links(text)))
sentences = count_sentences(text)
emails, urls, numbers = extract(text)
word_count = len(words)
return TextStats(
characters=len(text),
characters_no_spaces=len(re.sub(r"\s", "", text)),
words=word_count,
sentences=sentences,
paragraphs=count_paragraphs(text),
average_word_length=round(sum(map(len, words)) / word_count, 2) if words else 0.0,
average_sentence_length=round(word_count / sentences, 2) if sentences else 0.0,
reading_time_minutes=reading_time(word_count),
top_words=top_words(words),
longest_words=longest_words(words),
emails=emails,
urls=urls,
numbers=numbers,
)
def format_report(stats, title="Text report"):
lines = [
title,
"=" * len(title),
f"Characters: {stats.characters} ({stats.characters_no_spaces} without spaces)",
f"Words: {stats.words}",
f"Sentences: {stats.sentences}",
f"Paragraphs: {stats.paragraphs}",
f"Avg word length: {stats.average_word_length}",
f"Avg sentence: {stats.average_sentence_length} words",
f"Reading time: {stats.reading_time_minutes} min",
"",
"Top words:",
]
for word, count in stats.top_words:
lines.append(f" {word:<12} {count}")
lines.append("Longest words: " + ", ".join(stats.longest_words))
lines.append("Emails: " + (", ".join(stats.emails) or "-"))
lines.append("URLs: " + (", ".join(stats.urls) or "-"))
lines.append("Numbers: " + (", ".join(stats.numbers) or "-"))
return "\n".join(lines)
def save_reports(stats, json_path, text_path):
Path(json_path).write_text(json.dumps(asdict(stats), indent=2), encoding="utf-8")
Path(text_path).write_text(format_report(stats) + "\n", encoding="utf-8")
def main():
Path("cafe.txt").write_text(SAMPLE_TEXT, encoding="utf-8")
text = read_text("cafe.txt")
if not text.strip():
print("The file is empty, nothing to analyze.")
return
stats = analyze(text)
print(format_report(stats, "Report for cafe.txt"))
save_reports(stats, "report.json", "report.txt")
loaded = json.loads(Path("report.json").read_text(encoding="utf-8"))
print()
print("Saved report.json with keys:", ", ".join(list(loaded)[:4]), "...")
print("Saved report.txt:", Path("report.txt").stat().st_size, "bytes")
if __name__ == "__main__":
main()Report for cafe.txt
===================
Characters: 591 (489 without spaces)
Words: 93
Sentences: 10
Paragraphs: 3
Avg word length: 4.15
Avg sentence: 9.3 words
Reading time: 0.5 min
Top words:
cafe 4
tea 3
guests 3
samarkand 2
silk 1
Longest words: grandmothers, questions, samarkand
Emails: aziza@silkroadcafe.uz, jobs@silkroadcafe.uz
URLs: https://silkroadcafe.uz/menu
Numbers: 2019, 12, 350, 450000000
Saved report.json with keys: characters, characters_no_spaces, words, sentences ...
Saved report.txt: 485 bytesNotice the structure: constants and compiled patterns at the top, small pure functions in the middle, and all input/output in read_text(), save_reports(), and main(). The if __name__ == "__main__": line (from Modules and Imports) lets you import these functions in a test file without running the program.
Testing the pure functions
Because most functions are pure, they are easy to test. In a real project, put these in test_text_analyzer.py, write from text_analyzer import ... at the top, and run pytest. Here the functions are repeated and the tests are called by hand, so the block runs on its own:
import re
from collections import Counter
WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
SENTENCE_END_RE = re.compile(r"[.!?]+(?:\s|$)")
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)*\.[a-z]{2,}", re.IGNORECASE)
def iter_words(text):
for match in WORD_RE.finditer(text.lower()):
yield match.group()
def count_sentences(text):
return len(SENTENCE_END_RE.findall(text.strip()))
def top_words(words, n=5, stop_words=frozenset()):
counts = Counter(w for w in words if w not in stop_words)
return counts.most_common(n)
def longest_words(words, n=3):
return sorted(set(words), key=lambda w: (-len(w), w))[:n]
def reading_time(word_count, wpm=200):
return round(word_count / wpm, 1)
def test_iter_words_ignores_case_and_punctuation():
assert list(iter_words("Tea, TEA! tea?")) == ["tea", "tea", "tea"]
def test_iter_words_keeps_apostrophes():
assert list(iter_words("It's Aziza's cafe")) == ["it's", "aziza's", "cafe"]
def test_iter_words_empty_text():
assert list(iter_words("")) == []
assert list(iter_words("123 !!! 45")) == []
def test_count_sentences():
assert count_sentences("One. Two! Three?") == 3
assert count_sentences("Wait?! Really...") == 2
assert count_sentences("Pi is 3.14 and the site is cafe.uz") == 0
assert count_sentences("") == 0
def test_top_words_uses_stop_words():
words = ["the", "tea", "the", "tea", "plov"]
assert top_words(words, n=1, stop_words={"the"}) == [("tea", 2)]
def test_longest_words_ties_are_alphabetical():
assert longest_words(["tea", "plov", "nan", "samsa", "lagman"], n=2) == ["lagman", "samsa"]
def test_reading_time():
assert reading_time(0) == 0.0
assert reading_time(1000) == 5.0
assert reading_time(300, wpm=250) == 1.2
def test_email_pattern():
text = "Mail aziza@cafe.uz or JOBS@Cafe.UZ, not @cafe or aziza@"
assert EMAIL_RE.findall(text) == ["aziza@cafe.uz", "JOBS@Cafe.UZ"]
tests = [
test_iter_words_ignores_case_and_punctuation,
test_iter_words_keeps_apostrophes,
test_iter_words_empty_text,
test_count_sentences,
test_top_words_uses_stop_words,
test_longest_words_ties_are_alphabetical,
test_reading_time,
test_email_pattern,
]
for test in tests:
test()
print("passed:", test.__name__)passed: test_iter_words_ignores_case_and_punctuation
passed: test_iter_words_keeps_apostrophes
passed: test_iter_words_empty_text
passed: test_count_sentences
passed: test_top_words_uses_stop_words
passed: test_longest_words_ties_are_alphabetical
passed: test_reading_time
passed: test_email_patternLook at what the tests cover: normal input, edge cases (empty text, only numbers), and tricky cases found while building (apostrophes, 3.14, ties). Each bug you fix is a good reason to add a test, as you learned in Choosing Test Cases.
Try it
A compact version of the analyzer runs right here. Paste in your own text, or change the stop words.
Common mistakes
1. Reading a file without an encoding
Without encoding=, Python uses the computer's default, which can differ between Windows and Linux or macOS, so the same program may read a file on one computer and fail on another. And text files are not always saved as UTF-8. A file saved in an old Windows encoding fails even when you ask for UTF-8:
from pathlib import Path
Path("old.txt").write_bytes("Ташкент, Самарканд".encode("cp1251"))
text = Path("old.txt").read_text(encoding="utf-8")UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 0: invalid continuation byteFix: always pass encoding="utf-8" so your program behaves the same on every computer, and catch UnicodeDecodeError to show a friendly message (or try another encoding).
2. Forgetting to lowercase
from collections import Counter
words = "Tea is good. TEA is cheap. tea is hot.".replace(".", "").split()
print(Counter(words).most_common(2))
print(Counter(w.lower() for w in words).most_common(2))[('is', 3), ('Tea', 1)]
[('tea', 3), ('is', 3)]Fix: normalize case once, in the function that produces words, so every other function gets clean data.
3. Leaving punctuation attached
from collections import Counter
text = "Plov, plov and more plov!"
print(Counter(text.lower().split()))Counter({'plov,': 1, 'plov': 1, 'and': 1, 'more': 1, 'plov!': 1})"plov,", "plov", and "plov!" are counted separately. Fix: extract words with a regex like [a-z]+ instead of split().
4. Crashing on an empty file
text = ""
words = text.split()
average_length = sum(len(w) for w in words) / len(words)ZeroDivisionError: division by zeroFix: check if not text.strip(): early in main(), and guard every division: total / count if count else 0.0. Test with an empty string on purpose.
Challenges
These extend the project. Try each one before opening the hints.
1. Ask for the file name
Let the user choose the file: use input("File to analyze: "), and print a friendly message if the file does not exist instead of a traceback.
Hints
Path(name).exists()tells you whether the file is there. Or wrapread_textintry/except FileNotFoundError.- Keep the check in
main(), not in the pure functions.
2. Words per paragraph
Add a field words_per_paragraph (a list of integers) to TextStats, and show it in the text report like Words per paragraph: 31, 34, 28.
Hints
- Reuse
re.split(r"\n\s*\n", text)fromcount_paragraphs, then count words in each part withiter_words. sum(1 for _ in iter_words(paragraph))counts without building a list.- Remember to add a test for a text with one paragraph and a text with a blank line.
3. Stop words from a file
Load stop words from stop_words.txt (one word per line, lines starting with # are comments) instead of the hard-coded set.
Hints
- Write a generator pipeline like in Iterators and Generators: read lines, strip, skip blanks and comments.
- Build the set with
{line.lower() for line in ...}. - Pass the set to
top_words(words, stop_words=...). That is why it was a parameter.
4. Compare two texts
Analyze two files and print a side-by-side comparison: words, sentences, average sentence length, and the words that appear in the top 10 of both texts.
Hints
- Call
analyze()twice. You get twoTextStatsobjects. - Build sets of words from each
top_wordslist and use&for the intersection. - Format columns with f-strings like
f"{label:<20}{a:>8}{b:>8}".
5. Streaming analyzer for huge files
Rewrite word counting so it works on a file of several gigabytes: never call read_text() or list() on all the words. Count words, most common words, and the longest word while reading line by line.
Hints
- Open the file with
open(path, encoding="utf-8")and loop over lines. Each line goes throughremove_linksanditer_words. Counter.update(iterable)adds counts from a generator.- Keep a running word count and the current longest word in variables instead of a list.
- Sentences can span lines, so counting them exactly in streaming mode is harder. Decide whether an approximation (count per line) is acceptable, and write that decision in a comment.
Recap
- Plan a project as a list of small functions and a data flow before writing code.
- Normalize text early: lowercase it, extract words with a regex, and remove emails and URLs before counting.
Counter.most_common()plus a stop-word set finds meaningful frequent words, and a two-part sort key makes results stable.- Guard every division and check for empty input, so edge cases do not crash the program.
- A dataclass collects many results under clear names, and
asdict()turns it into JSON. - Keep calculations pure and file/screen work at the edges. Pure functions are easy to test with plain
assertorpytest.