Monitoring and Observability Guide 2026 — Prometheus, Grafana, and OpenTelemetry

Introduction

Why This Matters

You should never learn about a production problem from a user complaint. Proper observability means you know something is wrong before most users are affected. The three pillars — logs, metrics, and traces — together answer what happened, how often, and why. This guide builds a complete stack using open-source tools used by production engineering teams worldwide.

The Three Pillars of Observability

Pillar	What it answers	Tools
Logs	What happened (events with context)	Pino, Loki, CloudWatch
Metrics	How many / how fast (numbers over time)	Prometheus, Grafana
Traces	Why it is slow (journey through services)	OpenTelemetry, Tempo

Structured Logging with Pino

// lib/logger.ts
import pino from 'pino'
 
export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  transport: process.env.NODE_ENV === 'development'
    ? { target: 'pino-pretty', options: { colorize: true } }
    : undefined,
  base: {
    service: 'api',
    version: process.env.npm_package_version,
    env: process.env.NODE_ENV,
  },
  redact: {
    paths: ['req.headers.authorization', 'body.password', 'body.creditCard'],
    censor: '[REDACTED]',
  },
  serializers: {
    err: pino.stdSerializers.err,
    req: (req) => ({
      method: req.method,
      url: req.url,
      remoteAddress: req.socket?.remoteAddress,
    }),
  },
})
 
// Request middleware
export function requestLogger() {
  return (req: any, res: any, next: () => void) => {
    const start = Date.now()
    const requestId = crypto.randomUUID()
    req.log = logger.child({ requestId })
 
    res.on('finish', () => {
      const duration = Date.now() - start
      req.log.info({
        method: req.method,
        url: req.url,
        statusCode: res.statusCode,
        duration,
      }, 'Request completed')
 
      if (duration > 1000) {
        req.log.warn({ duration }, 'Slow request detected')
      }
    })
    next()
  }
}

Prometheus Metrics

import { Registry, Counter, Histogram, Gauge } from 'prom-client'
 
const register = new Registry()
 
const httpRequestTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register],
})
 
const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
  registers: [register],
})
 
const activeUsers = new Gauge({
  name: 'active_users_total',
  help: 'Active users in last 5 minutes',
  registers: [register],
})
 
export function metricsMiddleware() {
  return (req: any, res: any, next: () => void) => {
    const start = process.hrtime.bigint()
    res.on('finish', () => {
      const duration = Number(process.hrtime.bigint() - start) / 1e9
      const labels = {
        method: req.method,
        route: req.route?.path || 'unknown',
        status: res.statusCode.toString(),
      }
      httpRequestTotal.inc(labels)
      httpRequestDuration.observe(labels, duration)
    })
    next()
  }
}
 
// Expose /metrics endpoint for Prometheus scraping
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType)
  res.end(await register.metrics())
})

Prometheus and Grafana Stack

# docker-compose.yml — monitoring stack
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
 
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
 
volumes:
  prometheus_data:
  grafana_data:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
scrape_configs:
  - job_name: nodejs-app
    static_configs:
      - targets: ['app:3000']
    metrics_path: /metrics
 
  - job_name: postgres
    static_configs:
      - targets: ['postgres-exporter:9187']

Distributed Tracing with OpenTelemetry

// tracing.ts — Initialize before app startup
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { Resource } from '@opentelemetry/resources'
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
 
const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'api',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://tempo:4318/v1/traces',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
      '@opentelemetry/instrumentation-redis': { enabled: true },
    }),
  ],
})
 
sdk.start()
process.on('SIGTERM', () => sdk.shutdown())

Alerting Rules

# prometheus/alerts.yml
groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5%"
 
      - alert: SlowResponseTime
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency above 1s"
 
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage above 90%"

Common Mistakes

Logging without structure — plaintext logs cannot be filtered or queried efficiently in production; always use JSON
No request ID in logs — without a correlation ID, tracing a single request across multiple log lines is nearly impossible
Alerting on every metric — alert only on symptoms (user-facing errors, high latency) not causes (CPU, memory) to reduce noise
No sampling on traces — tracing 100% of requests at high traffic is expensive; use 10% sampling in production
No SLOs defined — without Service Level Objectives, you cannot tell if your alerts are well-calibrated

Best Practices

Use child loggers with logger.child({ requestId, userId }) to add context to every log within a request scope
Define SLOs before writing alert rules — e.g., 99.9% of requests under 500ms — then alert on SLO burn rate
Store Grafana dashboards as JSON in version control and provision them via the provisioning directory
Use the RED method for services: Request rate, Error rate, Duration — three metrics capture most service health
Instrument business metrics (orders created, users registered) alongside technical metrics for full visibility

Key Takeaways

Structured JSON logging (with Pino) makes logs searchable by any field and parsable by log aggregation tools
Prometheus uses a pull model — it scrapes your /metrics endpoint every 15 seconds; no push infrastructure required
Histograms in Prometheus allow you to calculate P50, P95, P99 latency across any time window using histogram_quantile()
OpenTelemetry provides vendor-neutral auto-instrumentation for HTTP, Express, PostgreSQL, and Redis with zero code changes
Alert on the symptoms users experience (5xx errors, slow P95 latency) not on infrastructure metrics like CPU
The RED method (Rate, Errors, Duration) is the minimal viable metrics set for any microservice
Grafana dashboards provisioned as JSON files in version control can be reproduced exactly across environments
Distributed tracing is essential for debugging latency in microservices — a single slow database query can hide anywhere

Introduction#

Why This Matters#

The Three Pillars of Observability#

Structured Logging with Pino#

Prometheus Metrics#

Prometheus and Grafana Stack#

Distributed Tracing with OpenTelemetry#

Alerting Rules#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading