9 articles
Build a production monitoring stack with Prometheus and Grafana: metrics collection, PromQL queries, alerting with Alertmanager, Kubernetes monitoring, and dashboard design for DevOps teams.
Implement OpenTelemetry for vendor-neutral distributed tracing, metrics, and logs: SDK setup for Node.js and Python, collector configuration, Jaeger integration, and context propagation across microservices.
Master agent evaluation: trajectory analysis, tool accuracy, task completion rates, efficiency scoring, and LLM-as-judge evaluation frameworks.
Build a comprehensive analytics backend for AI features. Track queries, user satisfaction, funnel conversion, and detect anomalies in AI system behavior.
Build automated evaluation pipelines with LLM-as-judge, DeepEval metrics, and RAGAS to catch quality regressions before users see them.
Comprehensive guide to evaluating LLM performance in production using offline metrics, online evaluation, human sampling, pairwise comparisons, and continuous monitoring pipelines.
Deploy OpenTelemetry with auto-instrumentation, custom spans, metrics, and the Collector pipeline. Export to Jaeger, Tempo, or Datadog.
Master the RAGAS framework and build evaluation pipelines that measure faithfulness, context relevance, and answer quality without expensive human annotation.
Build comprehensive monitoring for RAG systems tracking retrieval quality, generation speed, user feedback, and cost metrics to detect quality drift in production.