Published on

AI Analytics Backend — Tracking User Behavior, Query Patterns, and Business Metrics

Authors
  • Name
    Twitter

Introduction

You can''t improve what you don''t measure. An AI analytics backend reveals how users interact with your AI features, which queries are confusing the model, where users drop off, and when your system is behaving abnormally. This post covers building a comprehensive event pipeline, analyzing query patterns, tracking user satisfaction, measuring business impact, and detecting anomalies.

AI Interaction Event Schema

Design an event schema that captures everything needed for analysis without overwhelming storage.

interface AIInteractionEvent {
  eventId: string;
  timestamp: Date;
  tenantId: string;
  userId: string;
  sessionId: string;
  query: string;
  queryEmbedding?: number[]; // Optional: for semantic analysis
  responseId: string;
  response: string;
  model: string;
  temperature: number;
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
  cost: number; // in cents
  userSource: "web" | "mobile" | "api" | "slack";
  queryCategory?: string; // Optional: user or model-assigned category
  responseLengthCharacters: number;
  stopReason?: string; // Why did LLM stop? max_tokens, stop_sequence, etc.
  errorOccurred: boolean;
  errorType?: string;
  metadata?: Record<string, unknown>;
}

class AIEventCollector {
  private eventQueue: AIInteractionEvent[] = [];
  private batchSize = 100;
  private flushIntervalMs = 5000;

  async recordInteraction(event: Omit<AIInteractionEvent, "eventId">): Promise<void> {
    const fullEvent: AIInteractionEvent = {
      ...event,
      eventId: `event-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`,
    };

    this.eventQueue.push(fullEvent);

    // Flush if batch size reached
    if (this.eventQueue.length >= this.batchSize) {
      await this.flush();
    }
  }

  async flush(): Promise<void> {
    if (this.eventQueue.length === 0) return;

    const eventsToFlush = [...this.eventQueue];
    this.eventQueue = [];

    // Write to event store (data warehouse, time series DB, etc.)
    await this.persistEvents(eventsToFlush);
  }

  private async persistEvents(events: AIInteractionEvent[]): Promise<void> {
    console.log(`Persisting ${events.length} events`);
    // Implementation: write to BigQuery, Snowflake, ClickHouse, etc.
  }

  startPeriodicFlush(): void {
    setInterval(() => {
      this.flush();
    }, this.flushIntervalMs);
  }
}

// Event schema with helpful defaults
function createAIEvent(
  tenantId: string,
  userId: string,
  query: string,
  response: string,
  model: string,
  latencyMs: number,
  inputTokens: number,
  outputTokens: number
): Omit<AIInteractionEvent, "eventId"> {
  return {
    timestamp: new Date(),
    tenantId,
    userId,
    sessionId: `session-${userId}-${Date.now()}`,
    query,
    response,
    model,
    temperature: 0.7,
    inputTokens,
    outputTokens,
    latencyMs,
    cost: (inputTokens * 0.003 + outputTokens * 0.015) / 100, // Example pricing
    userSource: "web",
    responseLengthCharacters: response.length,
    errorOccurred: false,
  };
}

Query Analytics

Analyze popular queries, zero-result rates, and semantic query clustering.

interface QueryAnalytics {
  query: string;
  count: number;
  uniqueUsers: number;
  averageLatency: number;
  avgSatisfactionScore: number;
  resultedInError: number;
}

interface QueryCluster {
  clusterId: string;
  queries: string[];
  semanticSimilarity: number;
  commonTopic: string;
}

class QueryAnalyzer {
  async analyzePopularQueries(
    tenantId: string,
    limit: number = 100
  ): Promise<QueryAnalytics[]> {
    // In production, query from data warehouse
    const events = await this.fetchEvents(tenantId);

    // Group by query
    const queryGroups = new Map<string, AIInteractionEvent[]>();

    events.forEach((event) => {
      if (!queryGroups.has(event.query)) {
        queryGroups.set(event.query, []);
      }
      queryGroups.get(event.query)!.push(event);
    });

    // Calculate analytics per query
    const analytics: QueryAnalytics[] = Array.from(queryGroups.values()).map(
      (group) => ({
        query: group[0].query,
        count: group.length,
        uniqueUsers: new Set(group.map((e) => e.userId)).size,
        averageLatency:
          group.reduce((sum, e) => sum + e.latencyMs, 0) / group.length,
        avgSatisfactionScore: 0.5, // From separate satisfaction tracking
        resultedInError: group.filter((e) => e.errorOccurred).length,
      })
    );

    return analytics
      .sort((a, b) => b.count - a.count)
      .slice(0, limit);
  }

  async analyzeZeroResultRate(
    tenantId: string
  ): Promise<{
    zeroResultRate: number;
    affectedQueries: string[];
  }> {
    const events = await this.fetchEvents(tenantId);

    const zeroResultEvents = events.filter(
      (e) => e.response.toLowerCase().includes("no result") ||
        e.response.toLowerCase().includes("unable to find") ||
        e.response.toLowerCase().includes("i don''t know")
    );

    return {
      zeroResultRate: (zeroResultEvents.length / events.length) * 100,
      affectedQueries: [...new Set(zeroResultEvents.map((e) => e.query))].slice(
        0,
        10
      ),
    };
  }

  async clusterSimilarQueries(
    tenantId: string
  ): Promise<QueryCluster[]> {
    const events = await this.fetchEvents(tenantId);
    const uniqueQueries = [...new Set(events.map((e) => e.query))];

    // Group queries that address similar topics
    // In production, use embedding-based clustering
    const clusters: QueryCluster[] = [];

    const topicKeywords = {
      pricing: ["price", "cost", "plans", "subscription"],
      troubleshooting: ["problem", "error", "broken", "not working"],
      features: ["feature", "capability", "can", "support"],
    };

    for (const [topic, keywords] of Object.entries(topicKeywords)) {
      const clusterQueries = uniqueQueries.filter((q) =>
        keywords.some((kw) => q.toLowerCase().includes(kw))
      );

      if (clusterQueries.length > 0) {
        clusters.push({
          clusterId: topic,
          queries: clusterQueries,
          semanticSimilarity: 0.75,
          commonTopic: topic,
        });
      }
    }

    return clusters;
  }

  private async fetchEvents(tenantId: string): Promise<AIInteractionEvent[]> {
    // In production, query data warehouse
    return [];
  }
}

User Satisfaction Signals

Track explicit ratings and implicit engagement signals that indicate satisfaction.

interface SatisfactionEvent {
  eventId: string;
  timestamp: Date;
  userId: string;
  responseId: string;
  type: "thumbs-up" | "thumbs-down" | "follow-up" | "share" | "save";
  rating?: number; // 1-5 scale
}

interface SatisfactionMetrics {
  avgRating: number;
  thumbsUpRate: number;
  followUpRate: number;
  shareRate: number;
  saveRate: number;
  netSatisfactionScore: number; // Thumbs up % - thumbs down %
}

class SatisfactionTracker {
  private satisfactionEvents: SatisfactionEvent[] = [];

  recordSatisfaction(
    userId: string,
    responseId: string,
    type: SatisfactionEvent["type"]
  ): void {
    this.satisfactionEvents.push({
      eventId: `satisfaction-${Date.now()}`,
      timestamp: new Date(),
      userId,
      responseId,
      type,
    });
  }

  getSatisfactionMetrics(
    responseIds?: string[]
  ): SatisfactionMetrics {
    let events = this.satisfactionEvents;

    if (responseIds) {
      events = events.filter((e) => responseIds.includes(e.responseId));
    }

    const thumbsUp = events.filter((e) => e.type === "thumbs-up").length;
    const thumbsDown = events.filter((e) => e.type === "thumbs-down").length;
    const followUps = events.filter((e) => e.type === "follow-up").length;
    const shares = events.filter((e) => e.type === "share").length;
    const saves = events.filter((e) => e.type === "save").length;

    const total = events.length;

    return {
      avgRating: 3.5, // From explicit ratings if collected
      thumbsUpRate: (thumbsUp / total) * 100,
      followUpRate: (followUps / total) * 100,
      shareRate: (shares / total) * 100,
      saveRate: (saves / total) * 100,
      netSatisfactionScore: ((thumbsUp - thumbsDown) / total) * 100,
    };
  }

  getResponseSatisfaction(responseId: string): {
    positive: number;
    negative: number;
    neutral: number;
  } {
    const events = this.satisfactionEvents.filter(
      (e) => e.responseId === responseId
    );

    return {
      positive: events.filter((e) => e.type === "thumbs-up").length,
      negative: events.filter((e) => e.type === "thumbs-down").length,
      neutral: events.filter((e) => !["thumbs-up", "thumbs-down"].includes(e.type)).length,
    };
  }
}

Funnel Analysis

Track user progression through key conversion stages driven by AI features.

type FunnelStage = "ai_query" | "response_received" | "satisfied" | "action_taken" | "converted";

interface FunnelEvent {
  userId: string;
  sessionId: string;
  stage: FunnelStage;
  timestamp: Date;
  details?: Record<string, unknown>;
}

interface FunnelMetrics {
  stage: FunnelStage;
  uniqueUsers: number;
  conversionRate: number; // % moving to next stage
}

class FunnelAnalyzer {
  private funnelEvents: FunnelEvent[] = [];

  recordFunnelEvent(
    userId: string,
    sessionId: string,
    stage: FunnelStage
  ): void {
    this.funnelEvents.push({
      userId,
      sessionId,
      stage,
      timestamp: new Date(),
    });
  }

  analyzeFunnelMetrics(): FunnelMetrics[] {
    const stages: FunnelStage[] = [
      "ai_query",
      "response_received",
      "satisfied",
      "action_taken",
      "converted",
    ];

    const stageCounts = new Map<FunnelStage, Set<string>>();

    stages.forEach((stage) => {
      const users = new Set(
        this.funnelEvents
          .filter((e) => e.stage === stage)
          .map((e) => e.userId)
      );
      stageCounts.set(stage, users);
    });

    const metrics: FunnelMetrics[] = [];

    for (let i = 0; i < stages.length; i++) {
      const stage = stages[i];
      const nextStage = stages[i + 1];
      const currentCount = stageCounts.get(stage)?.size || 0;
      const nextCount = nextStage ? stageCounts.get(nextStage)?.size || 0 : currentCount;

      metrics.push({
        stage,
        uniqueUsers: currentCount,
        conversionRate:
          currentCount > 0 ? (nextCount / currentCount) * 100 : 0,
      });
    }

    return metrics;
  }

  getBottlenecks(): Array<{
    from: FunnelStage;
    to: FunnelStage;
    dropOffPercent: number;
  }> {
    const metrics = this.analyzeFunnelMetrics();
    const bottlenecks = [];

    for (let i = 0; i < metrics.length - 1; i++) {
      const dropOff = 100 - metrics[i].conversionRate;

      if (dropOff > 20) {
        // Significant drop-off
        bottlenecks.push({
          from: metrics[i].stage,
          to: metrics[i + 1].stage,
          dropOffPercent: dropOff,
        });
      }
    }

    return bottlenecks;
  }
}

Cohort Analysis by AI Feature

Compare metrics across user groups using different AI features or versions.

interface UserCohort {
  cohortId: string;
  name: string;
  description: string;
  members: Set<string>;
  createdAt: Date;
  criteria: {
    aiFeatureVersion?: string;
    userSegment?: string;
    geography?: string;
  };
}

interface CohortMetrics {
  cohortId: string;
  avgSatisfaction: number;
  totalQueries: number;
  uniqueUsers: number;
  conversionRate: number;
  avgLatency: number;
  costPerUser: number;
}

class CohortAnalyzer {
  private cohorts = new Map<string, UserCohort>();
  private events: AIInteractionEvent[] = [];

  createCohort(
    name: string,
    description: string,
    criteria: UserCohort["criteria"]
  ): UserCohort {
    const cohort: UserCohort = {
      cohortId: `cohort-${Date.now()}`,
      name,
      description,
      members: new Set(),
      createdAt: new Date(),
      criteria,
    };

    this.cohorts.set(cohort.cohortId, cohort);
    return cohort;
  }

  addMemberToCohort(cohortId: string, userId: string): void {
    const cohort = this.cohorts.get(cohortId);

    if (cohort) {
      cohort.members.add(userId);
    }
  }

  getCohortMetrics(cohortId: string): CohortMetrics | null {
    const cohort = this.cohorts.get(cohortId);

    if (!cohort) return null;

    const cohortEvents = this.events.filter((e) =>
      cohort.members.has(e.userId)
    );

    if (cohortEvents.length === 0) {
      return {
        cohortId,
        avgSatisfaction: 0,
        totalQueries: 0,
        uniqueUsers: 0,
        conversionRate: 0,
        avgLatency: 0,
        costPerUser: 0,
      };
    }

    const totalCost = cohortEvents.reduce((sum, e) => sum + e.cost, 0);
    const uniqueUsers = new Set(cohortEvents.map((e) => e.userId)).size;

    return {
      cohortId,
      avgSatisfaction: 0.5, // Would come from satisfaction tracker
      totalQueries: cohortEvents.length,
      uniqueUsers,
      conversionRate: 0.45, // Would come from funnel analyzer
      avgLatency:
        cohortEvents.reduce((sum, e) => sum + e.latencyMs, 0) /
        cohortEvents.length,
      costPerUser: totalCost / uniqueUsers,
    };
  }

  compareCohorts(
    cohortIds: string[]
  ): Record<string, CohortMetrics | null> {
    const comparison: Record<string, CohortMetrics | null> = {};

    cohortIds.forEach((id) => {
      comparison[id] = this.getCohortMetrics(id);
    });

    return comparison;
  }
}

Cost Per User Segment

Track and optimize costs across different user segments.

interface CostAnalysis {
  segmentId: string;
  segmentName: string;
  totalCost: number;
  userCount: number;
  costPerUser: number;
  costPerQuery: number;
  topCostDrivers: Array<{ type: string; cost: number }>;
}

class CostSegmentAnalyzer {
  private events: AIInteractionEvent[] = [];

  analyzeCostBySegment(
    segmentName: string,
    userIds: string[]
  ): CostAnalysis {
    const userSet = new Set(userIds);
    const segmentEvents = this.events.filter((e) =>
      userSet.has(e.userId)
    );

    const totalCost = segmentEvents.reduce((sum, e) => sum + e.cost, 0);
    const queryCount = segmentEvents.length;

    // Identify top cost drivers
    const costByModel = new Map<string, number>();

    segmentEvents.forEach((e) => {
      costByModel.set(e.model, (costByModel.get(e.model) || 0) + e.cost);
    });

    const topCostDrivers = Array.from(costByModel.entries())
      .sort(([, a], [, b]) => b - a)
      .slice(0, 3)
      .map(([type, cost]) => ({ type, cost }));

    return {
      segmentId: `segment-${Date.now()}`,
      segmentName,
      totalCost,
      userCount: userIds.length,
      costPerUser: totalCost / userIds.length,
      costPerQuery: totalCost / queryCount,
      topCostDrivers,
    };
  }

  optimizeCostForSegment(
    segmentName: string,
    userIds: string[]
  ): Array<{
    recommendation: string;
    potentialSavings: number;
  }> {
    const analysis = this.analyzeCostBySegment(segmentName, userIds);

    const recommendations = [];

    // Check for expensive models
    if (analysis.topCostDrivers[0]?.cost > analysis.totalCost * 0.6) {
      recommendations.push({
        recommendation: `Switch ${analysis.topCostDrivers[0]?.type} to cheaper model`,
        potentialSavings: analysis.topCostDrivers[0]?.cost * 0.3 || 0,
      });
    }

    // Check for high token usage
    const avgTokens = this.events
      .filter((e) => new Set(userIds).has(e.userId))
      .reduce((sum, e) => sum + e.inputTokens + e.outputTokens, 0) /
      this.events.filter((e) => new Set(userIds).has(e.userId)).length;

    if (avgTokens > 1000) {
      recommendations.push({
        recommendation: "Implement prompt compression or context truncation",
        potentialSavings: analysis.totalCost * 0.2,
      });
    }

    return recommendations;
  }
}

Anomaly Detection in Usage Patterns

Detect unusual patterns that might indicate problems or abuse.

interface AnomalyAlert {
  alertId: string;
  timestamp: Date;
  type: "spike" | "drop" | "error_rate" | "latency" | "unusual_pattern";
  severity: "low" | "medium" | "high";
  metric: string;
  expectedValue: number;
  actualValue: number;
  description: string;
}

class AnomalyDetector {
  private alerts: AnomalyAlert[] = [];
  private baseline = new Map<string, { mean: number; stdDev: number }>();

  async detectAnomalies(
    events: AIInteractionEvent[]
  ): Promise<AnomalyAlert[]> {
    const newAlerts: AnomalyAlert[] = [];

    // Detect request rate spike
    const requestsPerMinute = this.calculateRequestsPerMinute(events);
    const rateBaseline = this.baseline.get("requests_per_minute");

    if (rateBaseline && requestsPerMinute > rateBaseline.mean + 3 * rateBaseline.stdDev) {
      newAlerts.push({
        alertId: `alert-${Date.now()}`,
        timestamp: new Date(),
        type: "spike",
        severity: "high",
        metric: "requests_per_minute",
        expectedValue: rateBaseline.mean,
        actualValue: requestsPerMinute,
        description: `Request rate spiked to ${requestsPerMinute} (expected ~${rateBaseline.mean})`,
      });
    }

    // Detect error rate increase
    const errorRate = this.calculateErrorRate(events);

    if (errorRate > 0.05) {
      newAlerts.push({
        alertId: `alert-${Date.now()}`,
        timestamp: new Date(),
        type: "error_rate",
        severity: "high",
        metric: "error_rate",
        expectedValue: 0.01,
        actualValue: errorRate,
        description: `Error rate elevated to ${(errorRate * 100).toFixed(2)}%`,
      });
    }

    // Detect latency increase
    const avgLatency = events.reduce((sum, e) => sum + e.latencyMs, 0) / events.length;
    const latencyBaseline = this.baseline.get("latency_ms");

    if (latencyBaseline && avgLatency > latencyBaseline.mean * 2) {
      newAlerts.push({
        alertId: `alert-${Date.now()}`,
        timestamp: new Date(),
        type: "latency",
        severity: "medium",
        metric: "latency_ms",
        expectedValue: latencyBaseline.mean,
        actualValue: avgLatency,
        description: `Average latency doubled to ${avgLatency.toFixed(0)}ms`,
      });
    }

    this.alerts.push(...newAlerts);
    return newAlerts;
  }

  private calculateRequestsPerMinute(events: AIInteractionEvent[]): number {
    const now = Date.now();
    const oneMinuteAgo = now - 60000;

    const recentEvents = events.filter(
      (e) => e.timestamp.getTime() > oneMinuteAgo
    );

    return recentEvents.length;
  }

  private calculateErrorRate(events: AIInteractionEvent[]): number {
    if (events.length === 0) return 0;

    const errorCount = events.filter((e) => e.errorOccurred).length;

    return errorCount / events.length;
  }

  updateBaseline(metric: string, values: number[]): void {
    const mean = values.reduce((a, b) => a + b, 0) / values.length;
    const variance =
      values.reduce((sum, val) => sum + Math.pow(val - mean, 2), 0) /
      values.length;
    const stdDev = Math.sqrt(variance);

    this.baseline.set(metric, { mean, stdDev });
  }
}

Checklist

  • Design event schema capturing query, response, latency, cost, and errors
  • Batch events in memory and flush periodically for efficiency
  • Track popular queries and identify zero-result rates
  • Collect explicit satisfaction ratings and implicit engagement signals
  • Implement funnel analysis to identify conversion bottlenecks
  • Compare metrics across cohorts (features, user segments, versions)
  • Analyze costs by segment and identify optimization opportunities
  • Monitor for spikes, drops, and elevated error rates in real-time

Conclusion

An analytics backend transforms raw usage data into actionable insights. Start by designing a comprehensive event schema that captures what matters: queries, responses, latency, costs, errors. Collect events efficiently with batching and async persistence. Analyze popular queries and identify semantic clusters. Track user satisfaction through explicit ratings and implicit signals. Use funnels to find where users drop off. Compare cohorts to understand which features or user segments perform best. Monitor costs by segment to optimize spending. Finally, detect anomalies in real-time to catch problems before they impact users. By building these analytics capabilities, you''ll continuously improve your AI system based on real production data.