Pandas for Data Analysis — The Complete Python Guide for 2026

Introduction

Why This Matters

Pandas is the de facto standard for data analysis in Python. Whether you are cleaning messy CSV exports, aggregating sales data, exploring a dataset before training a machine learning model, or building a data pipeline — pandas is the tool. It is used by data scientists, data engineers, backend developers, and analysts worldwide.

In 2026, pandas 2.x (backed by Apache Arrow) delivers significantly better performance and memory efficiency than its predecessors. The integration with tools like Polars, DuckDB, and Jupyter notebooks makes it even more versatile. Understanding pandas deeply makes you a more effective contributor to any data-driven team.

This guide covers the essential operations you will use daily: loading data, filtering, groupby, merging, handling nulls, and exporting results.

Installation and Import

pip install pandas openpyxl

import pandas as pd
import numpy as np
 
# Suppress scientific notation for display
pd.set_option("display.float_format", "{:.2f}".format)

Creating DataFrames and Series

# From a dictionary
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "age": [30, 25, 35, 28],
    "department": ["Engineering", "Marketing", "Engineering", "HR"],
    "salary": [95000, 60000, 110000, 72000],
})
 
# Series
s = pd.Series([10, 20, 30, 40], name="scores", index=["a", "b", "c", "d"])
 
# From CSV / Excel
df = pd.read_csv("data.csv")
df = pd.read_excel("data.xlsx", sheet_name="Sheet1")

Exploring a DataFrame

print(df.shape)        # (4, 4) — rows, columns
print(df.dtypes)       # column data types
print(df.info())       # concise summary
print(df.describe())   # statistical summary (numeric columns)
print(df.head(3))      # first 3 rows
print(df.tail(2))      # last 2 rows
print(df.columns.tolist())  # ['name', 'age', 'department', 'salary']
print(df.isnull().sum())    # count nulls per column

Selecting Data

# Column selection
df["name"]                    # Series
df[["name", "salary"]]        # DataFrame
 
# Row selection
df.iloc[0]                    # first row by position
df.iloc[0:2]                  # first two rows
df.loc[0]                     # row by label/index
df.loc[0:2, "name":"age"]     # rows 0-2, columns name to age

Filtering

# Simple condition
engineers = df[df["department"] == "Engineering"]
 
# Multiple conditions (use & and | with parentheses)
senior_engineers = df[(df["department"] == "Engineering") & (df["salary"] &gt; 100000)]
 
# isin
tech_depts = df[df["department"].isin(["Engineering", "Marketing"])]
 
# query() — readable string syntax
high_earners = df.query("salary > 80000 and age &lt; 35")
 
# String filtering
df[df["name"].str.startswith("A")]
df[df["name"].str.contains("li", case=False)]

Adding and Modifying Columns

# New column
df["annual_bonus"] = df["salary"] * 0.10
 
# Conditional column
df["level"] = df["salary"].apply(lambda s: "senior" if s > 90000 else "junior")
 
# Using np.where
df["is_senior"] = np.where(df["salary"] > 90000, True, False)
 
# Rename columns
df = df.rename(columns={"name": "full_name", "age": "employee_age"})
 
# Drop columns
df = df.drop(columns=["annual_bonus"])

Handling Missing Data

# Detect
print(df.isnull().sum())
print(df.notnull().any())
 
# Drop rows with any null
df_clean = df.dropna()
 
# Drop rows where specific column is null
df_clean = df.dropna(subset=["salary"])
 
# Fill nulls
df["salary"] = df["salary"].fillna(df["salary"].median())
df["department"] = df["department"].fillna("Unknown")

GroupBy and Aggregation

# Average salary by department
df.groupby("department")["salary"].mean()
 
# Multiple aggregations
summary = df.groupby("department").agg(
    avg_salary=("salary", "mean"),
    max_salary=("salary", "max"),
    count=("name", "count"),
).reset_index()
 
print(summary)

Sorting and Ranking

# Sort by single column
df_sorted = df.sort_values("salary", ascending=False)
 
# Sort by multiple columns
df_sorted = df.sort_values(["department", "salary"], ascending=[True, False])
 
# Rank within group
df["salary_rank"] = df.groupby("department")["salary"].rank(ascending=False)

Merging and Joining

departments = pd.DataFrame({
    "department": ["Engineering", "Marketing", "HR"],
    "budget": [500000, 200000, 150000],
})
 
# Inner join (only matching rows)
merged = df.merge(departments, on="department", how="inner")
 
# Left join (all from left, matching from right)
merged = df.merge(departments, on="department", how="left")

Exporting Results

# CSV
df.to_csv("output.csv", index=False)
 
# Excel
df.to_excel("output.xlsx", sheet_name="Employees", index=False)
 
# JSON
df.to_json("output.json", orient="records", indent=2)
 
# Display with tabulate
from tabulate import tabulate
print(tabulate(df, headers="keys", tablefmt="pipe"))

Common Mistakes

Modifying a slice without using .copy() — triggers SettingWithCopyWarning
Using df["col"] = value inside a function on a slice — doesn't modify original
Comparing floats with == — use .round() or np.isclose()
Forgetting reset_index() after groupby or filtering
Using iterrows() for row-by-row operations — always prefer vectorized operations

Best Practices

Use vectorized operations (not loops or iterrows()) for performance
Always call .copy() when creating a subset you intend to modify
Use pd.read_csv(..., dtype={"id": str}) to control column types on load
Profile memory with df.info(memory_usage="deep") before processing large files
Use pd.NA instead of np.nan for nullable integer and string columns

Key Takeaways

pd.DataFrame is the core data structure — a 2D table with labeled rows and columns
Use .iloc[] for positional indexing and .loc[] for label-based indexing
Filter with boolean conditions: df[df["col"] > value]; use & and | for multiple
groupby().agg() is the standard way to compute aggregates by group
merge() performs SQL-style joins between DataFrames
Avoid iterrows() — vectorized pandas operations are 10-100x faster
Pandas 2.x uses Apache Arrow under the hood for significantly better memory and speed

Introduction#

Why This Matters#

Installation and Import#

Creating DataFrames and Series#

Exploring a DataFrame#

Selecting Data#

Filtering#

Adding and Modifying Columns#

Handling Missing Data#

GroupBy and Aggregation#

Sorting and Ranking#

Merging and Joining#

Exporting Results#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading