OpenTelemetry — Unified Observability for Modern Applications

Introduction

Why This Matters

OpenTelemetry (OTel) is the CNCF standard for application observability — a single set of APIs, SDKs, and tools for collecting traces, metrics, and logs from any language and sending them to any backend. Before OTel, switching from Datadog to Jaeger required re-instrumenting every service. With OTel, you instrument once and route telemetry to any vendor. It is now the second-most-active CNCF project after Kubernetes.

Core Concepts

Traces: A tree of spans representing a single request across multiple services
Spans: Individual units of work (one HTTP call, one DB query) with start time, duration, and attributes
Metrics: Numeric measurements aggregated over time (same as Prometheus metrics)
Logs: Structured log records that can be correlated with traces via trace context
Collector: A standalone process that receives, processes, and exports telemetry

Node.js Instrumentation

// tracing.js — load this FIRST before any other imports
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions');
 
const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME || 'my-service',
    [SEMRESATTRS_SERVICE_VERSION]: process.env.SERVICE_VERSION || '1.0.0',
    environment: process.env.NODE_ENV || 'production',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://otel-collector:4318/v1/metrics',
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
      '@opentelemetry/instrumentation-redis': { enabled: true },
    }),
  ],
});
 
sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

// server.js — application code
require('./tracing'); // Must be first import
 
const express = require('express');
const { trace, metrics, context } = require('@opentelemetry/api');
 
const app = express();
const tracer = trace.getTracer('my-service');
const meter = metrics.getMeter('my-service');
 
// Custom metrics
const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total number of HTTP requests',
});
const requestDuration = meter.createHistogram('http_request_duration_ms', {
  description: 'HTTP request duration in milliseconds',
  unit: 'ms',
});
 
app.get('/users/:id', async (req, res) => {
  const span = tracer.startSpan('get-user', {
    attributes: {
      'user.id': req.params.id,
      'http.method': 'GET',
    },
  });
 
  const start = Date.now();
 
  try {
    // Wrap async work in span context
    const user = await context.with(trace.setSpan(context.active(), span), async () => {
      return await getUserFromDatabase(req.params.id);
    });
 
    span.setStatus({ code: 1 }); // OK
    requestCounter.add(1, { method: 'GET', route: '/users/:id', status: '200' });
    res.json(user);
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: 2, message: err.message }); // ERROR
    requestCounter.add(1, { method: 'GET', route: '/users/:id', status: '500' });
    res.status(500).json({ error: err.message });
  } finally {
    requestDuration.record(Date.now() - start, { route: '/users/:id' });
    span.end();
  }
});

Python Instrumentation

# tracing.py
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
 
resource = Resource.create({"service.name": "my-python-service", "service.version": "1.0.0"})
 
# Configure tracing
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces"))
)
trace.set_tracer_provider(tracer_provider)
 
# Configure metrics
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="http://otel-collector:4318/v1/metrics"),
    export_interval_millis=30000
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
 
# Auto-instrument FastAPI and SQLAlchemy
FastAPIInstrumentor.instrument()
SQLAlchemyInstrumentor().instrument()

OpenTelemetry Collector

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          static_configs:
            - targets: ['localhost:8888']
 
processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 500
    spike_limit_mib: 100
  resource:
    attributes:
      - action: insert
        key: environment
        value: production
 
exporters:
  # Send traces to Jaeger
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  # Send metrics to Prometheus
  prometheus:
    endpoint: "0.0.0.0:8889"
  # Send to Datadog
  datadog:
    api:
      key: ${DD_API_KEY}
      site: datadoghq.com
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/jaeger, datadog]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus, datadog]

Context Propagation

// Context propagation passes trace context across HTTP calls
// Auto-instrumentation handles this automatically for most HTTP clients
 
// For manual HTTP calls, propagate W3C Trace Context headers
const { propagation, context } = require('@opentelemetry/api');
 
async function callDownstreamService(userId) {
  const headers = {};
  propagation.inject(context.active(), headers);
  // headers now contains: traceparent: 00-4bf92f3577b34da6...-00f067aa0ba902b7-01
 
  const response = await fetch(`http://user-service/users/${userId}`, { headers });
  return response.json();
}

Common Mistakes

Importing application code before require('./tracing') — auto-instrumentation patches modules at import time, so tracing must load first
Creating new tracer/meter instances in every function — create them once at module level and reuse
Recording spans for every function call — only instrument meaningful units of work (external calls, significant processing)
Not setting service.name resource attribute — makes traces impossible to filter by service in Jaeger or Tempo
Ignoring span status — always set span.setStatus() and call span.recordException() in error handlers

Best Practices

Use auto-instrumentation for HTTP, database, and messaging libraries — it covers 80% of spans with zero code changes
Deploy the OTel Collector as a sidecar or DaemonSet — decouple applications from backend-specific endpoints
Use semantic conventions for attribute names (http.method, db.system, rpc.service) for interoperability
Sample traces at the Collector level for high-volume services — head-based sampling at 10-20% reduces storage
Correlate logs with traces by injecting trace_id and span_id into log records
Export to multiple backends simultaneously via the Collector — useful during vendor evaluation

Key Takeaways

OpenTelemetry is the CNCF standard for instrumentation — vendor-neutral APIs that separate instrumentation from export
Auto-instrumentation patches Node.js/Python modules automatically — no code changes required for HTTP, DB, and cache libraries
The OTel Collector receives, processes, and routes telemetry — it decouples applications from specific observability backends
Traces consist of spans representing units of work; spans carry attributes, events, and parent-child relationships
W3C Trace Context (traceparent header) propagates trace context across HTTP service boundaries
Semantic Conventions define standard attribute names — use them for consistent querying across services and tools
Sampling at 10-20% of traces is common for high-throughput services to control storage and cost
OTel supports traces, metrics, and logs — a single SDK and Collector pipeline can replace multiple separate agents

Introduction#

Why This Matters#

Core Concepts#

Node.js Instrumentation#

Python Instrumentation#

OpenTelemetry Collector#

Context Propagation#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading