CrewAI — Build Collaborative Multi-Agent Systems in Python (2026)

Sanjeev SharmaSanjeev Sharma
8 min read

Advertisement

Introduction

Why CrewAI?

CrewAI is a Python framework for building networks of autonomous AI agents that work together like a real team. Where LangChain focuses on chains of operations and AutoGen focuses on agent conversations, CrewAI focuses on role-based agent collaboration with explicit task assignments.

The framework introduces the metaphor of a "crew" — each agent has a role, a goal, and a backstory that shapes its reasoning. Tasks are assigned to specific agents, and the crew executes them in sequence or in parallel depending on your configuration. CrewAI is particularly effective for content production, research pipelines, and multi-step data processing workflows.

Installation and Setup

pip install crewai crewai-tools
import os
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool, FileReadTool, WebsiteSearchTool
 
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["SERPER_API_KEY"] = "your-serper-key"  # For web search tool

Defining Agents

Each agent has a distinct role, goal, and backstory — these three fields heavily influence how the agent reasons:

from crewai import Agent
from crewai_tools import SerperDevTool
 
search_tool = SerperDevTool()
 
# Research agent: finds information
researcher = Agent(
    role="Senior Research Analyst",
    goal="Find accurate, up-to-date information on the assigned topic and identify key trends",
    backstory="""You are a former academic researcher turned industry analyst.
You have 15 years of experience synthesizing complex information from multiple sources.
You always cite your sources and flag when information is uncertain or contested.
You are skeptical of marketing claims and prioritize peer-reviewed or primary sources.""",
    tools=[search_tool],
    verbose=True,
    allow_delegation=False,    # This agent does not delegate to others
    max_iter=5,                # Maximum tool-use iterations per task
    llm="gpt-4o"
)
 
# Writing agent: produces polished content
writer = Agent(
    role="Technical Content Writer",
    goal="Transform research findings into clear, engaging, and technically accurate content",
    backstory="""You are a technical writer with a computer science background.
You write for developers and engineers who value accuracy over marketing fluff.
You use concrete examples, avoid jargon without explanation, and structure content for scannability.
You never invent facts — if the research does not cover something, you say so.""",
    verbose=True,
    allow_delegation=False,
    llm="gpt-4o"
)
 
# Editor agent: quality control
editor = Agent(
    role="Editorial Director",
    goal="Ensure content is factually accurate, well-structured, and meets publication standards",
    backstory="""You have edited technical content for major developer publications for 10 years.
You check for factual accuracy, logical flow, consistent terminology, and readability.
You provide specific, actionable feedback. You approve content only when it meets high standards.""",
    verbose=True,
    allow_delegation=True,    # Can send work back to writer
    llm="gpt-4o"
)

Defining Tasks

Tasks specify what each agent must do, what inputs they receive, and what output they must produce:

from crewai import Task
 
def create_content_pipeline(topic: str) -> tuple:
    """Create a three-stage content pipeline for a given topic."""
 
    research_task = Task(
        description=f"""Research the following topic thoroughly: {topic}
 
Your deliverable must include:
1. A factual overview (what it is, why it matters)
2. Current state in 2026 (recent developments, adoption)
3. Key technical concepts (3-5 core ideas with brief explanations)
4. Real-world use cases (at least 3 concrete examples)
5. Limitations and challenges
6. Comparison with alternatives if relevant
 
Cite specific sources where possible. Flag any claims you are uncertain about.""",
        expected_output="A structured research report with sections, facts, sources, and flagged uncertainties",
        agent=researcher,
    )
 
    writing_task = Task(
        description=f"""Using the research provided, write a comprehensive technical blog post about: {topic}
 
Requirements:
- Target audience: senior software engineers
- Length: 1000-1200 words
- Structure: Introduction, 4-5 main sections with h2 headings, Conclusion
- Include at least one Python code example that is complete and runnable
- Use concrete numbers and benchmarks from the research
- Avoid fluff sentences — every sentence must add information""",
        expected_output="A complete, publication-ready technical blog post in markdown",
        agent=writer,
        context=[research_task],        # This task uses research_task's output
    )
 
    editing_task = Task(
        description="""Review the drafted blog post for:
1. Factual accuracy against the research report
2. Technical correctness of any code examples
3. Logical flow and structure
4. Readability (flag sentences over 25 words)
5. Missing context or unexplained jargon
 
Provide: (a) specific line-level feedback, and (b) a revised final version if changes are needed.
Approve with 'APPROVED' only when the post meets publication standards.""",
        expected_output="Editorial feedback and a final approved version of the blog post",
        agent=editor,
        context=[research_task, writing_task],
    )
 
    return research_task, writing_task, editing_task

Running the Crew

from crewai import Crew, Process
 
def run_content_crew(topic: str) -> str:
    """Run the full content pipeline for a given topic."""
    research_task, writing_task, editing_task = create_content_pipeline(topic)
 
    crew = Crew(
        agents=[researcher, writer, editor],
        tasks=[research_task, writing_task, editing_task],
        process=Process.sequential,    # Tasks run in order
        verbose=True,
        memory=True,                   # Agents share short-term memory
        max_rpm=20,                    # Rate limit API calls
    )
 
    result = crew.kickoff()
    return str(result)
 
output = run_content_crew("vector databases and their role in RAG systems")
print(output)

Hierarchical Process: Manager-Worker Pattern

For complex tasks, use a hierarchical process where a manager agent delegates and coordinates:

from crewai import Agent, Crew, Process
 
manager = Agent(
    role="Engineering Manager",
    goal="Coordinate the team to deliver high-quality software solutions efficiently",
    backstory="You are a senior engineering manager who excels at breaking down problems and delegating effectively.",
    allow_delegation=True,
    llm="gpt-4o"
)
 
backend_dev = Agent(
    role="Backend Engineer",
    goal="Write efficient, secure Python backend code",
    backstory="Expert in FastAPI, PostgreSQL, and distributed systems.",
    allow_delegation=False,
    llm="gpt-4o"
)
 
qa_engineer = Agent(
    role="QA Engineer",
    goal="Write comprehensive test suites that catch bugs and regressions",
    backstory="Expert in pytest, test coverage analysis, and property-based testing.",
    allow_delegation=False,
    llm="gpt-4o"
)
 
build_task = Task(
    description="Build a rate-limiting middleware for a FastAPI application. Limit: 100 requests per minute per IP.",
    expected_output="Production-ready Python code with tests",
    agent=manager,  # Manager receives the task and delegates
)
 
crew = Crew(
    agents=[manager, backend_dev, qa_engineer],
    tasks=[build_task],
    process=Process.hierarchical,  # Manager delegates to workers
    manager_llm="gpt-4o",
    verbose=True,
)
 
result = crew.kickoff()

Adding Custom Tools

from crewai.tools import BaseTool
from pydantic import BaseModel, Field
import httpx
 
class GitHubSearchInput(BaseModel):
    query: str = Field(description="GitHub search query")
    language: str = Field(default="python", description="Programming language filter")
 
class GitHubSearchTool(BaseTool):
    name: str = "GitHub Code Search"
    description: str = "Search GitHub for code examples and repositories"
    args_schema: type[BaseModel] = GitHubSearchInput
 
    def _run(self, query: str, language: str = "python") -> str:
        """Search GitHub and return formatted results."""
        # Replace with real GitHub API call using your token
        url = f"https://api.github.com/search/repositories?q={query}+language:{language}&sort=stars"
        headers = {"Authorization": "token YOUR_GITHUB_TOKEN"}
        try:
            response = httpx.get(url, headers=headers, timeout=10)
            data = response.json()
            items = data.get("items", [])[:5]
            results = [
                f"- {item['full_name']}: {item['description']} ({item['stargazers_count']} stars)"
                for item in items
            ]
            return "\n".join(results) if results else "No results found"
        except Exception as e:
            return f"Search failed: {e}"
 
# Attach to an agent
researcher_with_github = Agent(
    role="Open Source Researcher",
    goal="Find the best open-source solutions on GitHub",
    backstory="Expert at discovering and evaluating open-source projects.",
    tools=[GitHubSearchTool()],
    llm="gpt-4o"
)

Common Mistakes

  1. Overlapping agent roles — If two agents have similar roles, the LLM cannot determine who should act on a given task.
  2. Tasks without expected output — Vague expected_output fields produce inconsistent results across runs.
  3. Missing context links — If a task needs another task's output, pass it via the context parameter.
  4. Allowing delegation on all agents — Most agents should have allow_delegation=False; only manager agents should delegate.
  5. No rate limiting — Multi-agent crews can exhaust OpenAI rate limits quickly; always set max_rpm.
  6. Ignoring verbose output during development — Set verbose=True during development to understand agent reasoning.

Best Practices

  • Write detailed agent backstories — they significantly influence reasoning quality and role adherence
  • Always specify expected_output in precise, measurable terms (not "a good summary" but "a 300-word summary with 3 key findings")
  • Use Process.sequential for pipelines where task order matters; use Process.hierarchical when coordination is needed
  • Enable memory=True so agents share context without repeating information in every message
  • Test each agent individually before running the full crew to isolate problems
  • Monitor token consumption per crew run — large crews with many rounds can be expensive

Key Takeaways

  • CrewAI models multi-agent collaboration as a "crew" where each agent has a role, goal, and backstory that shapes reasoning
  • Agent backstories are not cosmetic — detailed, specific backstories produce measurably better role adherence
  • The context parameter in Task links tasks together so later agents receive prior agents' outputs automatically
  • Sequential process runs tasks in order; hierarchical process uses a manager agent to delegate and coordinate
  • Custom tools are created by subclassing BaseTool and implementing _run — type annotations drive the LLM schema
  • Setting allow_delegation=True only on manager agents prevents routing confusion in large crews
  • Always set max_rpm in production to avoid hitting API rate limits during parallel or rapid task execution
  • CrewAI is best suited for content pipelines, research workflows, and multi-step automation — not real-time interactive applications

Advertisement

Sanjeev Sharma

Written by

Sanjeev Sharma

Full Stack Engineer · E-mopro

Related reading