Bot Traffic Killing Your APIs — When 80% of Your Traffic Isn't Human

Introduction

Bot traffic is not background noise — for most public APIs, it is the majority of traffic. Scrapers, credential stuffers, inventory manipulators, and price monitors collectively generate more requests than legitimate users on most e-commerce and SaaS products. Every one of those requests pays your infrastructure bill and degrades latency for real users. The defense is layered and must work at multiple points in the stack simultaneously.

Recognizing Bot Traffic Patterns

Before implementing defenses, you need to know what you are fighting:

Velocity attacks — simple but costly. Thousands of requests per minute from one IP with identical payloads and no pause between requests. Easy to detect, high infrastructure impact.

Credential stuffing — more subtle. Many different usernames against the login endpoint, distributed across IPs but with consistent timing patterns and high failure rates. Bots are testing stolen credential lists.

Content scraping — sequential page traversal with no CSS or image requests, no session cookies, no pause between pages. The bot is walking your catalog programmatically.

Inventory manipulation — add-to-cart events with no checkout follow-through, triggered immediately after price-change events. No browse behavior precedes the cart action.

Account creation abuse — signups using disposable email domains, identical IP, slightly varied input data, no email verification click-through.

Fix 1: Multi-Dimensional Rate Limiting

Rate limiting on IP alone is insufficient — bots rotate IPs. Limit on IP, user identity, and endpoint simultaneously:

import { RateLimiterRedis } from 'rate-limiter-flexible';
import { createClient } from 'redis';
 
const redis = createClient({ url: process.env.REDIS_URL });
 
// Burst limiter: catches immediate spikes
const burstLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: 'rl_burst',
  points: 30,        // 30 requests
  duration: 10,      // per 10 seconds
  blockDuration: 60, // block for 60s if exceeded
});
 
// Sustained limiter: catches persistent bots that stay under burst limits
const sustainedLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: 'rl_sustained',
  points: 500,
  duration: 3600,
  blockDuration: 3600,
});
 
async function rateLimitMiddleware(req, res, next) {
  const ip = req.ip;
  const userId = req.user?.id;
 
  try {
    await Promise.all([
      burstLimiter.consume(ip),
      sustainedLimiter.consume(ip),
      userId ? burstLimiter.consume(`user_${userId}`) : Promise.resolve(),
    ]);
    next();
  } catch (err) {
    const retryAfter = Math.ceil((err.msBeforeNext ?? 60000) / 1000);
    res.set({ 'Retry-After': retryAfter });
    return res.status(429).json({ error: 'Too many requests', retryAfter });
  }
}

The two-layer approach catches both burst attacks and sustained low-and-slow bots that stay under per-second limits but hammer over hours.

Fix 2: Bot Fingerprinting

Score every request based on header patterns and request timing. Humans using browsers send consistent, predictable headers; bots often skip or mangle them:

function calculateBotScore(req) {
  let score = 0;
 
  // Missing browser headers that real clients always send
  if (!req.headers['accept-language']) score += 20;
  if (!req.headers['accept-encoding']) score += 20;
 
  // Known bot user-agent strings
  const ua = req.headers['user-agent'] ?? '';
  const botPatterns = [/bot/i, /crawler/i, /spider/i, /curl/i, /wget/i, /python-requests/i];
  if (botPatterns.some((p) => p.test(ua))) score += 60;
 
  // API requests without referer are suspicious (browsers always send it)
  if (req.path.startsWith('/api/') && !req.headers['referer']) score += 10;
 
  // Requests arriving faster than humanly possible
  const lastSeen = recentRequests.get(req.ip);
  if (lastSeen && Date.now() - lastSeen < 50) score += 30;
 
  recentRequests.set(req.ip, Date.now());
  return score;
}
 
// Apply to request pipeline
app.use((req, res, next) => {
  req.botScore = calculateBotScore(req);
 
  if (req.botScore >= 80) {
    return res.status(403).json({ error: 'Request blocked' });
  }
 
  next();
});

Scores between 40 and 80 should trigger stricter rate limits or CAPTCHAs rather than hard blocks, to avoid false positives on legitimate clients like mobile apps.

Fix 3: Credential Stuffing Detection

Track login failure patterns per IP to catch bots testing stolen passwords across many accounts:

async function detectCredentialStuffing(redis, ip, email, success) {
  const key = `login_attempts:${ip}`;
  await redis.lpush(key, JSON.stringify({ email, success, ts: Date.now() }));
  await redis.expire(key, 3600);
 
  const raw = await redis.lrange(key, 0, 99);
  const attempts = raw.map((a) => JSON.parse(a));
 
  const uniqueEmails = new Set(attempts.map((a) => a.email)).size;
  const failureRate = attempts.filter((a) => !a.success).length / attempts.length;
 
  // Credential stuffing signature: many accounts, mostly failures
  if (uniqueEmails > 10 && failureRate > 0.8) {
    await redis.set(`blocked_ip:${ip}`, '1', { EX: 86400 });
    await alerting.critical(`Credential stuffing from ${ip}: ${uniqueEmails} accounts, ${Math.round(failureRate * 100)}% failures`);
  }
 
  // Single-account brute force
  const accountAttempts = attempts.filter((a) => a.email === email);
  if (accountAttempts.length > 10) {
    await redis.set(`account_locked:${email}`, '1', { EX: 900 });
  }
}

Fix 4: Protecting High-Value Scraped Endpoints

For product catalogs, pricing pages, and content that bots love to scrape:

// Force small page sizes — bots need many more requests to complete a scrape
app.get('/api/products', async (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const limit = Math.min(parseInt(req.query.limit) || 20, 20); // cap at 20
 
  // Add artificial delay for high-bot-score anonymous traffic
  if (!req.user && req.botScore > 30) {
    await new Promise((r) => setTimeout(r, 500 + Math.random() * 500));
  }
 
  const products = await db.query(
    'SELECT id, name, price FROM products LIMIT $1 OFFSET $2',
    [limit, (page - 1) * limit]
  );
 
  // Suppress pagination cursor for suspected bots — raises scraping cost
  const nextPage = req.botScore < 30 ? page + 1 : undefined;
  res.json({ products: products.rows, nextPage });
});
 
// Honeypot: a URL that only bots will visit
app.get('/api/internal/all-products', (req, res) => {
  redis.set(`blocked_ip:${req.ip}`, '1', { EX: 86400 });
  logger.warn({ ip: req.ip, ua: req.headers['user-agent'] }, 'Honeypot triggered');
  res.status(404).json({ error: 'Not found' });
});

The honeypot approach is highly effective: any IP that requests a URL that is only known to scrapers (never linked from your UI) can be blocked with zero false positives.

Fix 5: Cloudflare at the Edge

Cloudflare WAF rules block the most obvious bots before they reach your servers:

# WAF rules (Cloudflare dashboard or Terraform)
rules:
  - name: Block known bot user agents
    expression: >
      (http.user_agent contains "python-requests") or
      (http.user_agent contains "Go-http-client") or
      (http.user_agent eq "")
    action: block
 
  - name: Login endpoint rate limit
    expression: 'http.request.uri.path eq "/api/auth/login"'
    action: rate_limit
    ratelimit:
      requests_per_period: 5
      period: 60
      mitigation_timeout: 300
 
  - name: Challenge anonymous VPN/proxy traffic
    expression: 'cf.client.bot or ip.geoip.is_in_european_union = false'
    action: managed_challenge

Cloudflare's bot score (cf.client.bot) is a trained classifier based on behavioral signals. Enable it before writing any custom fingerprinting code.

Defense Stack Summary

Layer	Protects Against	False Positive Risk
Cloudflare WAF	Volumetric, known bots	Low
Burst rate limiting	DDoS, automated spikes	Low
Sustained rate limiting	Low-and-slow bots	Medium
Bot fingerprinting	Headless browsers, scrapers	Medium
Credential stuffing detection	Account takeover	Low
Honeypot endpoints	Aggressive catalog scrapers	None

Key Takeaways

Bot traffic routinely exceeds legitimate user traffic on public APIs; measure it separately in your dashboards before tuning defenses
Two-layer rate limiting (burst + sustained) catches both immediate spikes and low-and-slow bots that evade per-second limits
Bot fingerprinting scores requests on missing browser headers, known bot user-agent strings, and sub-50ms request timing
Credential stuffing detection blocks IPs with more than 10 unique accounts and above 80% login failure rate over a rolling hour
Honeypot endpoints that are never linked from the UI catch aggressive scrapers with zero false positives
Cloudflare WAF with managed bot rules handles volumetric attacks before they consume origin capacity
Artificial response delays for high-bot-score traffic raise the cost of scraping without blocking legitimate users
Every bot defense layer should emit metrics so you can measure its effectiveness and tune thresholds based on real traffic data

Introduction#

Recognizing Bot Traffic Patterns#

Fix 1: Multi-Dimensional Rate Limiting#

Fix 2: Bot Fingerprinting#

Fix 3: Credential Stuffing Detection#

Fix 4: Protecting High-Value Scraped Endpoints#

Fix 5: Cloudflare at the Edge#

Defense Stack Summary#

Key Takeaways#

Related reading