Grafana Loki Log Aggregation 2026 — The Prometheus-Native Logging Stack

Introduction

Why This Matters

Elasticsearch is expensive and operationally complex to maintain. Loki stores only log metadata labels — not full-text indexes — making it 10x cheaper to operate at scale. Combined with Grafana for dashboards and Promtail for log shipping, you get the PLG stack (Promtail + Loki + Grafana): a unified interface for logs, metrics, and traces without managing multiple backends. For teams already using Prometheus, Loki is the natural complement.

The PLG Stack Architecture

Application logs (stdout/file)
         ↓
   Promtail (agent, runs on every node)
         ↓
   Loki (aggregation, indexes labels only)
         ↓
   Grafana (visualization, alerting, LogQL)

Docker Compose: Full PLG Stack

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
 
  promtail:
    image: grafana/promtail:2.9.4
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
 
  grafana:
    image: grafana/grafana:10.3.3
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_FEATURE_TOGGLES_ENABLE: traceqlEditor
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
 
volumes:
  loki-data:
  grafana-data:

Loki Configuration

# loki-config.yml
auth_enabled: false
 
server:
  http_listen_port: 3100
 
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
 
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
 
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_length: 721h
  max_entries_limit_per_query: 50000
 
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100
 
ruler:
  alertmanager_url: http://alertmanager:9093

Promtail: Shipping Logs to Loki

# promtail-config.yml
server:
  http_listen_port: 9080
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://loki:3100/loki/api/v1/push
 
scrape_configs:
  # Docker container logs
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
    pipeline_stages:
      - json:
          expressions:
            level: level
            msg: msg
      - labels:
          level:
 
  # Application log files
  - job_name: application
    static_configs:
      - targets: [localhost]
        labels:
          job: myapp
          environment: production
          __path__: /var/log/app/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            status: status
            method: method
            duration: duration
      - labels:
          level:
          method:
          status:

Structured Logging: Make Logs Queryable

// logger.ts — JSON logging with Pino
import pino from 'pino'
 
export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  timestamp: pino.stdTimeFunctions.isoTime,
  base: {
    service: 'myapp-api',
    environment: process.env.NODE_ENV,
    version: process.env.APP_VERSION,
  },
  redact: {
    paths: ['req.headers.authorization', '*.password', '*.token'],
    censor: '[REDACTED]',
  },
})
 
// Request logging middleware
export function requestLogger(req: any, res: any, next: () => void) {
  const start = Date.now()
 
  res.on('finish', () => {
    logger.info({
      msg: 'HTTP request',
      method: req.method,
      path: req.path,
      status: res.statusCode,
      duration: Date.now() - start,
      userId: req.user?.id,
      traceId: req.headers['x-trace-id'],
    })
  })
 
  next()
}

LogQL: Querying Logs Like Metrics

# Basic stream selection
{service="myapp-api", environment="production"}
 
# Filter by content
{service="myapp-api"} |= "ERROR"
{service="myapp-api"} != "healthcheck"
 
# Parse JSON and filter by field value
{service="myapp-api"} | json | level="error"
{service="myapp-api"} | json | status >= 500
 
# Path pattern matching
{service="myapp-api"} | json | path =~ "/api/.*"
 
# Count errors per minute — metric query
sum(rate({service="myapp-api"} | json | level="error" [1m])) by (path)
 
# P95 response time
quantile_over_time(0.95,
  {service="myapp-api"} | json | unwrap duration [5m]
) by (path)
 
# Error rate percentage
(
  sum(rate({service="myapp-api"} | json | status >= 500 [5m]))
  /
  sum(rate({service="myapp-api"} | json [5m]))
) * 100

Kubernetes Log Collection

# kubernetes/promtail-daemonset.yml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      serviceAccountName: promtail
      containers:
        - name: promtail
          image: grafana/promtail:2.9.4
          args:
            - -config.file=/etc/promtail/config.yml
          env:
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers

Loki Alert Rules

# loki-rules.yml
groups:
  - name: application-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate({service="myapp-api"} | json | level="error" [5m]))
            /
            sum(rate({service="myapp-api"} | json [5m]))
          ) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on myapp-api"
 
      - alert: SlowRequests
        expr: |
          quantile_over_time(0.95,
            {service="myapp-api"} | json | unwrap duration [5m]
          ) > 2000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 response time above 2000ms"
 
      - alert: AppDown
        expr: |
          sum(rate({service="myapp-api"} [1m])) == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "No logs received from myapp-api — application may be down"

Loki vs Alternatives

Feature	Loki	Elasticsearch	Datadog Logs
Self-hosted cost	Very low	High	N/A
Managed cost	Low	High	High
Full-text search	No	Yes	Yes
Label-based search	Yes	Yes	Yes
Grafana native	Yes	Plugin	Plugin
Metrics from logs	Yes (LogQL)	Via pipeline	Yes
Setup complexity	Low	High	None

Common Mistakes

Logging in plaintext instead of JSON — Promtail cannot extract fields from unstructured text for label-based filtering
Too many high-cardinality labels — avoid labels like user_id or request_id on every log line; they explode the label index
No positions.yaml persistence — without it, Promtail re-ships logs from the start on every restart
Querying large time ranges without limits — use limit in LogQL queries; scanning 30 days of logs is expensive
Not connecting Loki to Grafana Tempo — trace ID extraction in Grafana lets you jump from a log line directly to its distributed trace

Best Practices

Use structured JSON logs with consistent field names (level, msg, status, duration) so LogQL parses them reliably
Keep label cardinality low — labels are the Loki index; high-cardinality labels (user ID, trace ID) should stay in log content
Use pipeline_stages in Promtail to extract labels from JSON logs rather than querying raw log lines at query time
Store Loki chunks in S3 for production — local filesystem storage does not scale and is not durable
Provision Grafana dashboards and Loki data sources as code so they are reproducible across environments

Key Takeaways

Loki indexes only labels — not log content — making it 10x cheaper to store and operate than Elasticsearch at the same volume
Promtail is the log shipping agent; it tails files, Docker container logs, and Kubernetes pod logs using service discovery
LogQL combines label filtering with content filtering and supports metric extraction from logs using rate() and quantile_over_time()
A DaemonSet runs Promtail on every Kubernetes node so all pod logs are automatically collected without per-pod configuration
Loki alert rules fire on LogQL metric queries just like Prometheus — you can alert on error rate derived from log data
High-cardinality labels like request IDs or user IDs will cause Loki performance issues — keep them in log content, not labels
Grafana Explore lets you switch between Loki (logs), Prometheus (metrics), and Tempo (traces) in a single UI using correlation
quantile_over_time() in LogQL extracts P50/P95/P99 latency from log fields without a separate metrics pipeline

Introduction#

Why This Matters#

The PLG Stack Architecture#

Docker Compose: Full PLG Stack#

Loki Configuration#

Promtail: Shipping Logs to Loki#

Structured Logging: Make Logs Queryable#

LogQL: Querying Logs Like Metrics#

Kubernetes Log Collection#

Loki Alert Rules#

Loki vs Alternatives#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading