Python Iterators and Loops — for, while, Generators, and Beyond

Introduction

Why This Matters

Loops are everywhere in Python, but most developers use them without understanding what powers them under the hood. Python's iteration protocol — the __iter__ and __next__ methods — is what makes for loops work on lists, strings, files, dicts, and any custom object you create. Understanding this protocol unlocks the ability to build memory-efficient generators, lazy pipelines, and infinite sequences.

Generators, introduced in Python 2.2 and significantly enhanced since, are one of Python's most important tools for processing large data. They allow you to iterate over sequences that may be too large to fit in memory, process data streams, and create pipelines of transformations without building intermediate lists.

This knowledge is particularly valuable for data engineering, backend development, and any Python role where performance and scalability matter.

How for Loops Work

Python's for loop calls iter() on the object, then repeatedly calls next() until StopIteration.

numbers = [1, 2, 3]
 
# What Python does internally:
iterator = iter(numbers)       # calls numbers.__iter__()
print(next(iterator))         # 1 — calls iterator.__next__()
print(next(iterator))         # 2
print(next(iterator))         # 3
# next(iterator)              # raises StopIteration
 
# The for loop does all of this automatically:
for n in numbers:
    print(n)

The Iterator Protocol

An object is iterable if it implements __iter__. An iterator implements both __iter__ and __next__.

class CountUp:
    """Iterator that counts from start to stop."""
    def __init__(self, start: int, stop: int):
        self.current = start
        self.stop = stop
 
    def __iter__(self):
        return self
 
    def __next__(self) -> int:
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += 1
        return value
 
for num in CountUp(1, 6):
    print(num)   # 1 2 3 4 5

Generators — The Simple Way

Generator functions use yield to lazily produce values.

def count_up(start: int, stop: int):
    """Generator version of CountUp."""
    current = start
    while current &lt; stop:
        yield current
        current += 1
 
for num in count_up(1, 6):
    print(num)  # 1 2 3 4 5
 
# Generator expressions
squares = (x ** 2 for x in range(10))  # lazy, no list created
total = sum(x ** 2 for x in range(1_000_000))  # memory efficient

Infinite Generators

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b
 
# Use itertools.islice to take first N values
import itertools
 
first_10 = list(itertools.islice(fibonacci(), 10))
print(first_10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

itertools — Essential Loop Utilities

import itertools
 
# chain: iterate multiple iterables as one
for item in itertools.chain([1, 2], [3, 4], [5]):
    print(item)  # 1 2 3 4 5
 
# islice: take first N from any iterable
for n in itertools.islice(fibonacci(), 5):
    print(n)  # 0 1 1 2 3
 
# enumerate with start
for i, val in enumerate(["a", "b", "c"], start=1):
    print(i, val)  # 1 a, 2 b, 3 c
 
# zip_longest
from itertools import zip_longest
for a, b in zip_longest([1, 2, 3], ["x", "y"], fillvalue="-"):
    print(a, b)  # 1 x, 2 y, 3 -
 
# groupby
data = sorted([{"dept": "eng", "name": "Alice"}, {"dept": "eng", "name": "Bob"}, {"dept": "hr", "name": "Carol"}], key=lambda x: x["dept"])
for dept, group in itertools.groupby(data, key=lambda x: x["dept"]):
    print(dept, [p["name"] for p in group])

Generator Pipelines

Generators can be chained into lazy data pipelines:

def read_lines(path: str):
    with open(path) as f:
        yield from f
 
def parse_ints(lines):
    for line in lines:
        stripped = line.strip()
        if stripped.isdigit():
            yield int(stripped)
 
def filter_large(nums, threshold: int):
    for n in nums:
        if n > threshold:
            yield n
 
# Chain into a pipeline — no intermediate lists
pipeline = filter_large(parse_ints(read_lines("numbers.txt")), threshold=100)
for value in pipeline:
    print(value)

yield from

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)  # recursively yield from sub-list
        else:
            yield item
 
print(list(flatten([1, [2, [3, 4]], 5])))  # [1, 2, 3, 4, 5]

Common Mistakes

Modifying a list while iterating over it — causes skipped elements
Consuming a generator twice — it is exhausted after the first iteration
Using a list where a generator expression would suffice
Forgetting that zip() stops at the shortest iterable — use zip_longest() if needed
Not using enumerate() and manually tracking an index variable

Best Practices

Use generator expressions for memory-efficient transformations
Use yield from to delegate to sub-generators cleanly
Prefer itertools over hand-rolled loops for common iteration patterns
Iterate files line by line (for line in f) instead of f.readlines()
Use any() and all() with generator expressions for short-circuit evaluation

Key Takeaways

Python's for loop uses the iterator protocol: __iter__() and __next__()
Any object implementing __iter__ and __next__ is an iterator
Generator functions use yield to lazily produce values one at a time
Generator expressions (expr for x in iterable) are memory-efficient alternatives to list comprehensions
itertools provides powerful tools: chain, islice, groupby, zip_longest, product
Generators are exhausted after one pass — store results in a list if you need to iterate again
yield from delegates iteration to another iterable, simplifying recursive generators

Introduction#

Why This Matters#

How for Loops Work#

The Iterator Protocol#

Generators — The Simple Way#

Infinite Generators#

itertools — Essential Loop Utilities#

Generator Pipelines#

yield from#

Common Mistakes#

Best Practices#

Key Takeaways#