Chain-of-Thought Prompting — Complete Guide with Python Examples (2026)

Introduction

Why Chain-of-Thought Prompting Matters

Chain-of-Thought (CoT) prompting instructs a language model to produce intermediate reasoning steps before delivering a final answer. Introduced in a landmark 2022 paper by Wei et al., CoT prompting improved accuracy on grade-school math benchmarks by over 50% for large models.

The insight is simple: when a model writes out its reasoning, it is less likely to make logical leaps that produce wrong answers. This technique is especially powerful for math, logic puzzles, multi-step analysis, and any task where errors compound across steps.

Zero-Shot Chain-of-Thought

Adding "Let's think step by step" to a prompt elicits reasoning without any examples:

from openai import OpenAI
 
client = OpenAI()
 
def zero_shot_cot(question: str) -> dict:
    """Solve a problem using zero-shot chain-of-thought."""
    # Step 1: Generate reasoning
    reasoning_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": f"{question}\n\nLet's think step by step."
            }
        ],
        temperature=0.2
    )
    reasoning = reasoning_response.choices[0].message.content
 
    # Step 2: Extract final answer
    answer_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": f"{question}\n\nLet's think step by step."},
            {"role": "assistant", "content": reasoning},
            {"role": "user", "content": "Therefore, the final answer is:"}
        ],
        temperature=0
    )
    answer = answer_response.choices[0].message.content
 
    return {"reasoning": reasoning, "answer": answer}
 
result = zero_shot_cot(
    "A store has 48 apples. They sell 3/4 of them and receive a new shipment of 20. How many apples are there now?"
)
print(result["answer"])

Few-Shot Chain-of-Thought

Providing worked examples with explicit reasoning chains produces more reliable and structured output:

FEW_SHOT_COT_PROMPT = """Solve each problem by showing your work step by step.
 
Problem: A train travels at 60 mph for 2.5 hours. How far does it travel?
Reasoning:
- Distance = speed * time
- Distance = 60 mph * 2.5 hours = 150 miles
Answer: 150 miles
 
Problem: A class has 30 students. 40% are girls. How many boys are there?
Reasoning:
- Girls = 30 * 0.40 = 12
- Boys = 30 - 12 = 18
Answer: 18 boys
 
Problem: {question}
Reasoning:"""
 
def few_shot_cot(question: str) -> str:
    prompt = FEW_SHOT_COT_PROMPT.format(question=question)
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1
    )
    return response.choices[0].message.content
 
answer = few_shot_cot(
    "A factory produces 240 units per day. It operates 5 days a week. "
    "How many units does it produce in 4 weeks?"
)
print(answer)

Self-Consistency: Majority Voting Over Multiple Chains

Self-consistency samples multiple reasoning paths and picks the most common answer. It reduces variance and improves accuracy by 5–20% over single-sample CoT:

from collections import Counter
import re
 
def self_consistent_cot(question: str, num_samples: int = 5) -> dict:
    """
    Generate multiple reasoning chains and select the most common answer.
    """
    answers = []
    reasoning_chains = []
 
    for i in range(num_samples):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "user",
                    "content": f"{question}\n\nThink step by step and give a final numeric answer."
                }
            ],
            temperature=0.7  # Variety across samples
        )
        content = response.choices[0].message.content
        reasoning_chains.append(content)
 
        # Extract the last number mentioned as the answer
        numbers = re.findall(r'\b\d+(?:\.\d+)?\b', content)
        if numbers:
            answers.append(numbers[-1])
 
    # Majority vote
    if answers:
        vote_counts = Counter(answers)
        best_answer = vote_counts.most_common(1)[0][0]
    else:
        best_answer = "Unable to determine"
 
    return {
        "answer": best_answer,
        "vote_distribution": dict(Counter(answers)),
        "sample_count": num_samples,
        "chains": reasoning_chains
    }
 
result = self_consistent_cot(
    "If a rectangle has a perimeter of 36 cm and its length is twice its width, what is the area?"
)
print(f"Answer: {result['answer']}")
print(f"Vote distribution: {result['vote_distribution']}")

Tree of Thought: Exploring Multiple Reasoning Branches

Tree of Thought (ToT) extends CoT by generating multiple candidate reasoning paths, evaluating each, and selecting the best:

def tree_of_thought(problem: str, num_branches: int = 3) -> dict:
    """
    Explore multiple reasoning approaches and select the most promising.
    """
    # Step 1: Generate diverse approaches
    branches_prompt = f"""Problem: {problem}
 
Generate {num_branches} different approaches to solve this problem.
For each approach, describe the strategy and first reasoning step.
 
Format:
Approach 1: [strategy name]
First step: [reasoning]
 
Approach 2: [strategy name]
First step: [reasoning]"""
 
    branches_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": branches_prompt}],
        temperature=0.8
    )
    branches = branches_response.choices[0].message.content
 
    # Step 2: Evaluate and select the most promising approach
    eval_prompt = f"""Problem: {problem}
 
These approaches were suggested:
{branches}
 
Which approach is most likely to lead to a correct solution and why?
Then solve the problem using that approach, showing all steps."""
 
    solution_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": eval_prompt}],
        temperature=0.1
    )
 
    return {
        "branches": branches,
        "solution": solution_response.choices[0].message.content
    }
 
result = tree_of_thought(
    "A farmer has chickens and rabbits. There are 20 heads and 56 legs total. "
    "How many of each animal are there?"
)
print(result["solution"])

CoT for Code Debugging

Chain-of-thought reasoning is highly effective for debugging — asking the model to trace execution step by step:

def debug_with_cot(code: str, error: str) -> str:
    prompt = (
        "You are a Python expert. Debug this code by reasoning through each step.\n\n"
        f"Code:\n{code}\n\nError: {error}\n\n"
        "Debug process:\n"
        "1. Identify what each line does\n"
        "2. Trace the execution path that leads to the error\n"
        "3. Pinpoint the root cause\n"
        "4. Propose a fix with explanation\n\n"
        "Begin debugging:"
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1
    )
    return response.choices[0].message.content
 
broken_code = "def calculate_average(numbers):\n    total = sum(numbers)\n    return total / len(numbers)\n\nresult = calculate_average([])"
diagnosis = debug_with_cot(broken_code, "ZeroDivisionError: division by zero")
print(diagnosis)

Measuring CoT Improvement

Track how much CoT improves accuracy on your specific task:

def benchmark_cot(test_cases: list[dict]) -> dict:
    """Compare direct answer vs chain-of-thought accuracy."""
    direct_correct = 0
    cot_correct = 0
 
    for case in test_cases:
        question = case["question"]
        expected = str(case["expected"]).lower().strip()
 
        # Direct answer
        direct_resp = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": question}],
            temperature=0
        )
        direct_answer = direct_resp.choices[0].message.content.strip().lower()
        if expected in direct_answer:
            direct_correct += 1
 
        # CoT answer
        cot_resp = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "user",
                 "content": f"{question}\nThink step by step, then give the final answer."}
            ],
            temperature=0.1
        )
        cot_answer = cot_resp.choices[0].message.content.strip().lower()
        if expected in cot_answer:
            cot_correct += 1
 
    n = len(test_cases)
    return {
        "direct_accuracy": direct_correct / n,
        "cot_accuracy": cot_correct / n,
        "improvement": (cot_correct - direct_correct) / n
    }

Common Mistakes

Skipping the reasoning extraction step — CoT generates reasoning AND answer together; parse them separately for downstream use.
Using temperature=0 for self-consistency — Defeats the purpose; you need temperature >= 0.5 for diverse chains.
Applying CoT to simple tasks — It adds latency and cost without benefit on straightforward retrieval tasks.
Ignoring chain quality — A flawed reasoning chain can still produce a wrong answer; validate logic, not just answers.
Not providing worked examples — Zero-shot CoT works but few-shot CoT with domain examples is significantly more accurate.

Best Practices

Use few-shot CoT with 3–5 worked examples for math, logic, and multi-step analysis tasks
Apply self-consistency (5–10 samples) when accuracy matters more than cost
Use Tree of Thought for open-ended problems with multiple valid solution strategies
Separate the reasoning step from the answer extraction step in production pipelines
Set temperature=0.1–0.3 for single-sample CoT; use 0.5–0.8 for self-consistency sampling
Validate reasoning chains on a test set before deploying in production

Key Takeaways

Chain-of-thought prompting improved LLM accuracy on math benchmarks by over 50% in the original research
Adding "Let's think step by step" (zero-shot CoT) is the simplest and most impactful single improvement to most prompts
Self-consistency samples 5–10 reasoning chains and picks the majority answer, reducing variance by 5–20%
Tree of Thought generates multiple solution branches, evaluates them, and pursues the most promising — ideal for complex open-ended problems
CoT is most valuable for multi-step reasoning tasks: math, logic, debugging, and multi-hop question answering
Few-shot CoT with worked examples outperforms zero-shot CoT on specialized or domain-specific tasks
Chain-of-thought adds latency and cost; evaluate whether your task warrants it over a direct prompt
Parsing reasoning and final answer as separate fields prevents downstream failures in automated pipelines

Introduction#

Why Chain-of-Thought Prompting Matters#

Zero-Shot Chain-of-Thought#

Few-Shot Chain-of-Thought#

Self-Consistency: Majority Voting Over Multiple Chains#

Tree of Thought: Exploring Multiple Reasoning Branches#

CoT for Code Debugging#

Measuring CoT Improvement#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading