API Response Optimization — Compression, Streaming, and Payload Minimization

Introduction

API payload size directly impacts latency, especially for mobile users on slow networks. A 10MB response takes 30 seconds on 3G, while a 1MB response takes 3 seconds. Compression, streaming, and intelligent field selection turn bloated APIs into snappy experiences. Production systems use multiple strategies layered together.

Brotli vs Gzip: Compression Ratios and CPU Trade-offs
Streaming JSON Responses
Pagination and Cursor Encoding
Field Selection (Sparse Fieldsets)
HTTP/2 Server Push
Response Compression Configuration
Content-Encoding Negotiation
Checklist
Conclusion

Brotli vs Gzip: Compression Ratios and CPU Trade-offs

Brotli compresses better than gzip but costs more CPU. The choice depends on your constraints:

import compression from 'compression';
import express from 'express';
import fs from 'fs';

const app = express();

// Gzip middleware (built into compression library)
app.use(compression({
  level: 6, // 0-9, higher = better compression but more CPU
  threshold: 1024, // Only compress > 1KB
}));

// Brotli middleware (requires brotli support)
import { createBrotliCompress } from 'zlib';

const brotliMiddleware = (req: express.Request, res: express.Response, next: express.NextFunction) => {
  const acceptEncoding = req.headers['accept-encoding'] || '';

  if (!acceptEncoding.includes('br')) {
    next();
    return;
  }

  // Check Content-Type (only compress compressible types)
  const contentType = res.getHeader('content-type')?.toString() || '';
  const isCompressible = [
    'application/json',
    'application/javascript',
    'text/html',
    'text/css',
    'text/plain',
    'image/svg+xml',
  ].some(type => contentType.includes(type));

  if (!isCompressible) {
    next();
    return;
  }

  // Apply brotli compression
  const brotli = createBrotliCompress({
    params: {
      // BROTLI_PARAM_QUALITY: 0-11, higher = better but slower
      [11]: 6, // Quality: balance speed and ratio
    },
  });

  res.setHeader('Content-Encoding', 'br');
  res.setHeader('Vary', 'Accept-Encoding');

  const originalSend = res.send;
  res.send = function(data: any) {
    if (!data) return originalSend.call(this, data);

    if (typeof data === 'object') {
      data = JSON.stringify(data);
    }

    brotli.write(data);
    brotli.end();

    brotli.pipe(res);

    return res;
  };

  next();
};

// Compression decision matrix
interface CompressionStrategy {
  contentType: string;
  minSize: number; // Don't compress below this
  recommended: 'gzip' | 'brotli' | 'none';
  reason: string;
}

const strategies: CompressionStrategy[] = [
  {
    contentType: 'application/json',
    minSize: 500, // JSON compresses well
    recommended: 'brotli',
    reason: 'Highly compressible, worth brotli overhead',
  },
  {
    contentType: 'application/javascript',
    minSize: 1000,
    recommended: 'brotli',
    reason: 'Large files benefit from better compression',
  },
  {
    contentType: 'text/html',
    minSize: 500,
    recommended: 'brotli',
    reason: 'HTML has high redundancy',
  },
  {
    contentType: 'image/jpeg',
    minSize: Infinity, // Don't compress (already compressed)
    recommended: 'none',
    reason: 'Already compressed format',
  },
  {
    contentType: 'image/png',
    minSize: Infinity,
    recommended: 'none',
    reason: 'Already compressed format',
  },
];

// Compression ratio example
const compressionRatios = {
  gzip: {
    json: 0.15, // 15% of original (85% reduction)
    javascript: 0.25,
    html: 0.20,
  },
  brotli: {
    json: 0.10, // 10% of original (90% reduction)
    javascript: 0.18,
    html: 0.15,
  },
};

// When payload size differences matter
const examples = [
  { size: 1024, gzip: 154, brotli: 102, benefit: 'Minimal' },
  { size: 10 * 1024, gzip: 1536, brotli: 1024, benefit: 'Good' },
  { size: 100 * 1024, gzip: 15360, brotli: 10240, benefit: 'Excellent' },
  { size: 1024 * 1024, gzip: 153600, brotli: 102400, benefit: 'Critical' },
];

Streaming JSON Responses

For large datasets, stream responses instead of buffering:

import { Readable } from 'stream';
import express from 'express';

// Problem: Loading entire dataset into memory
app.get('/api/users', async (req, res) => {
  const users = await db.user.findMany(); // All in memory!
  res.json(users);
});

// Solution: Stream responses
app.get('/api/users/stream', async (req, res) => {
  res.setHeader('Content-Type', 'application/x-ndjson'); // Newline-delimited JSON
  res.setHeader('Cache-Control', 'no-cache');

  const stream = db.user.findMany().stream(); // Database cursor

  stream.on('data', user => {
    res.write(JSON.stringify(user) + '\n');
  });

  stream.on('end', () => {
    res.end();
  });

  stream.on('error', (error) => {
    console.error('Stream error:', error);
    res.status(500).end();
  });
});

// Client-side parsing of ndjson
async function parseNDJSON<T>(response: Response): Promise<T[]> {
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  const results: T[] = [];

  try {
    while (true) {
      const { done, value } = await reader.read();

      if (done) break;

      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split('\n');

      // Process complete lines
      for (let i = 0; i < lines.length - 1; i++) {
        if (lines[i].trim()) {
          results.push(JSON.parse(lines[i]));
        }
      }

      // Keep incomplete last line in buffer
      buffer = lines[lines.length - 1];
    }

    // Process final line
    if (buffer.trim()) {
      results.push(JSON.parse(buffer));
    }
  } finally {
    reader.releaseLock();
  }

  return results;
}

// Streaming with backpressure (don't overwhelm client)
app.get('/api/events/stream', async (req, res) => {
  res.setHeader('Content-Type', 'application/x-ndjson');

  const stream = db.event.findMany().stream();

  stream.on('data', event => {
    const shouldContinue = res.write(JSON.stringify(event) + '\n');

    if (!shouldContinue) {
      // Client can't keep up, pause source
      stream.pause();
    }
  });

  res.on('drain', () => {
    // Client ready for more
    stream.resume();
  });

  stream.on('end', () => res.end());
  stream.on('error', err => {
    console.error('Stream error:', err);
    res.status(500).end();
  });
});

// Cursor-based pagination (more efficient than offset)
app.get('/api/items-paginated', async (req, res) => {
  const limit = Math.min(parseInt(req.query.limit as string) || 20, 100);
  const cursor = req.query.cursor as string | undefined;

  const items = await db.item.findMany({
    take: limit + 1, // Fetch one extra to determine hasMore
    ...(cursor && {
      skip: 1, // Skip the cursor item itself
      cursor: { id: cursor },
    }),
  });

  const hasMore = items.length > limit;
  const data = hasMore ? items.slice(0, limit) : items;
  const nextCursor = hasMore ? data[data.length - 1]?.id : null;

  res.json({
    data,
    hasMore,
    nextCursor,
  });
});

Pagination and Cursor Encoding

Cursor-based pagination is more robust than offset-based:

import { Buffer } from 'buffer';

// Opaque cursor encoding (hides implementation)
function encodeCursor(value: any): string {
  return Buffer.from(JSON.stringify(value)).toString('base64');
}

function decodeCursor(cursor: string): any {
  return JSON.parse(Buffer.from(cursor, 'base64').toString());
}

app.get('/api/posts', async (req, res) => {
  const pageSize = 20;
  const cursor = req.query.cursor ? decodeCursor(req.query.cursor as string) : null;

  const posts = await db.post.findMany({
    take: pageSize + 1,
    ...(cursor && {
      skip: 1,
      cursor: { id: cursor.id },
    }),
    orderBy: { createdAt: 'desc' },
  });

  const hasMore = posts.length > pageSize;
  const data = posts.slice(0, pageSize);

  res.json({
    data,
    pagination: {
      pageSize,
      hasMore,
      nextCursor: hasMore ? encodeCursor({ id: data[data.length - 1].id }) : null,
    },
  });
});

// Keyset pagination (most efficient)
// Uses indexed columns (createdAt + id) for stable ordering
app.get('/api/events', async (req, res) => {
  const pageSize = 50;
  const afterCreatedAt = req.query.afterCreatedAt ? new Date(req.query.afterCreatedAt as string) : null;
  const afterId = req.query.afterId as string | undefined;

  const events = await db.event.findMany({
    take: pageSize + 1,
    where: afterCreatedAt
      ? {
          OR: [
            { createdAt: { gt: afterCreatedAt } },
            {
              createdAt: { eq: afterCreatedAt },
              id: { gt: afterId || '' },
            },
          ],
        }
      : undefined,
    orderBy: [{ createdAt: 'asc' }, { id: 'asc' }],
  });

  const hasMore = events.length > pageSize;
  const data = events.slice(0, pageSize);
  const lastEvent = data[data.length - 1];

  res.json({
    data,
    pagination: {
      hasMore,
      nextParams: hasMore
        ? {
            afterCreatedAt: lastEvent.createdAt.toISOString(),
            afterId: lastEvent.id,
          }
        : null,
    },
  });
});

Field Selection (Sparse Fieldsets)

Let clients request only fields they need:

import express from 'express';

// Client: GET /api/users?fields=id,name,email
// Returns: { id, name, email } (excludes phone, address, etc)

interface FieldSelection {
  include?: string[];
  exclude?: string[];
}

function parseFieldSelection(query: any): FieldSelection {
  const fields = query.fields ? (query.fields as string).split(',') : null;

  return {
    include: fields || undefined,
  };
}

app.get('/api/users', async (req, res) => {
  const fieldSelection = parseFieldSelection(req.query);

  const users = await db.user.findMany({
    select: fieldSelection.include
      ? Object.fromEntries(fieldSelection.include.map(f => [f, true]))
      : true, // Default: all fields
  });

  res.json(users);
});

// GraphQL-like field selection for nested objects
app.get('/api/posts', async (req, res) => {
  // ?fields=id,title,author{id,name},comments{id,text}
  const fields = parseGraphQLFields(req.query.fields as string);

  const posts = await db.post.findMany({
    select: {
      id: fields.has('id'),
      title: fields.has('title'),
      body: fields.has('body'),
      author: fields.has('author')
        ? {
            select: {
              id: fields.author?.has('id'),
              name: fields.author?.has('name'),
              email: fields.author?.has('email'),
            },
          }
        : false,
      comments: fields.has('comments')
        ? {
            select: {
              id: fields.comments?.has('id'),
              text: fields.comments?.has('text'),
            },
          }
        : false,
    },
  });

  res.json(posts);
});

function parseGraphQLFields(fieldString: string): Map<string, Set<string>> {
  // Simple parser: "id,title,author{id,name}"
  const result = new Map<string, Set<string>>();
  // Implementation of GraphQL field parsing
  return result;
}

HTTP/2 Server Push

Proactively send related resources:

import spdy from 'spdy';
import fs from 'fs';

// HTTP/2 server with push
const options = {
  key: fs.readFileSync('./server.key'),
  cert: fs.readFileSync('./server.cert'),
};

spdy.createServer(options, async (req, res) => {
  if (req.url === '/') {
    // HTML document requires styles and script
    const pushResources = [
      { path: '/styles.css', type: 'style' },
      { path: '/app.js', type: 'script' },
      { path: '/logo.png', type: 'image' },
    ];

    // Push related resources
    for (const resource of pushResources) {
      const stream = res.push(resource.path, {
        request: { accept: '*/*' },
        response: { 'content-type': contentTypeFor(resource.type) },
      });

      const file = fs.createReadStream('.' + resource.path);
      file.pipe(stream);
    }

    // Send HTML
    res.setHeader('Content-Type', 'text/html');
    res.end('<html><!-- HTML references pushed resources --></html>');
  }
}).listen(443);

function contentTypeFor(type: string): string {
  const types: { [key: string]: string } = {
    style: 'text/css',
    script: 'application/javascript',
    image: 'image/png',
  };
  return types[type] || 'application/octet-stream';
}

// Note: HTTP/2 push is complex and often less effective than careful bundling
// Use only for critical resources known to improve real-world performance

Response Compression Configuration

Complete production setup:

import compression from 'compression';
import express from 'express';

const app = express();

// Selective compression middleware
app.use(compression({
  // Only compress if beneficial
  level: (req, res) => {
    const userAgent = req.headers['user-agent'] || '';

    // Lower compression for slow clients
    if (userAgent.includes('mobile')) {
      return 1; // Lower CPU usage
    }

    return 6; // Default
  },

  // Only compress these types
  type: [
    'application/json',
    'application/javascript',
    'text/html',
    'text/css',
    'text/plain',
    'text/xml',
    'text/x-component',
    'text/x-cross-domain-policy',
    'image/svg+xml',
  ],

  // Minimum size for compression
  threshold: 1024,

  // Filter responses by status, headers, etc
  filter: (req, res) => {
    if (res.getHeader('x-no-compression')) {
      return false;
    }

    return compression.filter(req, res);
  },
}));

// Example endpoints
app.get('/api/large', (req, res) => {
  const data = Array.from({ length: 10000 }, (_, i) => ({
    id: i,
    name: `Item ${i}`,
    description: `This is item number ${i}`.repeat(5),
  }));

  res.setHeader('Cache-Control', 'public, max-age=3600');
  res.json(data);
});

app.get('/csv-export', (req, res) => {
  res.setHeader('Content-Type', 'text/csv; charset=utf-8');
  res.setHeader('Content-Disposition', 'attachment; filename=export.csv');

  // Stream CSV (already handles compression via middleware)
  const stream = generateCSVStream();
  stream.pipe(res);
});

Content-Encoding Negotiation

Respect client capabilities:

app.use((req, res, next) => {
  const acceptEncoding = req.headers['accept-encoding'] || '';

  // Determine supported encodings
  const supportsBrotli = acceptEncoding.includes('br');
  const supportsGzip = acceptEncoding.includes('gzip');
  const prefersCompression = supportsBrotli || supportsGzip;

  // Store on request for use by middleware
  (req as any).compression = {
    supportsBrotli,
    supportsGzip,
    prefersCompression,
  };

  next();
});

Checklist

Conclusion

API response optimization combines multiple strategies: compression reduces wire size, streaming reduces memory usage, pagination enables incremental loading, and field selection eliminates unnecessary data. Together, they transform bloated APIs into nimble services that perform well on slow networks and at scale.