The ControlTheory Glossary

The software development lifecycle when most code is written by AI assistants (Claude Code, Cursor, Codex, Copilot) rather than typed by a human. The bottleneck shifts away from authoring and toward reviewing, debugging, and understanding what the code does at runtime.

Notifying a human (or another system) when a metric or log condition crosses a threshold. Alerts are usually defined as code (for example, PromQL rules in Prometheus) and routed through a separate component like Alertmanager, PagerDuty, or a Slack webhook.

The component that handles alerts fired by Prometheus. It deduplicates them, groups related ones together, silences noise during maintenance, and routes each alert to the right destination (PagerDuty, email, a Slack channel). Prometheus decides when a rule is breached; Alertmanager decides who hears about it and how. On the AI side, Möbius adds the Slack-facing alerting layer on top of correlated telemetry.

Identifying data points that deviate from the established pattern in a log stream, metric, or trace. Useful for catching the failures you didn’t think to write a static threshold for.

The practice of tracking the health and speed of an application from the user’s perspective (request latency, error rates, throughput, and slow transactions), usually with auto-instrumentation that traces each request end to end. APM overlaps heavily with observability; the difference is mostly emphasis, with APM centered on application-level performance and observability spanning logs, metrics, and traces across the whole system. Where classic APM lives in a dashboard you switch to, Dstl8 folds the same application-level signal back into the editor as runtime feedback.

Black-box monitoring tests a system from the outside the way a user would (hit an endpoint, check the response) without knowing anything about its internals. White-box monitoring reads the system’s own exposed internals: the metrics, logs, and traces it emits about itself. Black-box catches “is it down”; white-box explains “why,” which is why most teams run both.

A Prometheus exporter that probes endpoints from the outside over HTTP, HTTPS, TCP, ICMP, or DNS, and reports whether they responded, how fast, and with what status. It’s how you get black-box “is this URL up and healthy” checks into a Prometheus-based stack, alongside the white-box metrics your services expose about themselves.

A written review after an incident that focuses on the system and process failures that allowed it, not on which person made the mistake. Removing blame is what makes engineers honest about what actually happened, which is the only way the same failure stops recurring. A core SRE practice, and the timeline it’s built on is exactly what Möbius assembles through its root-cause analysis and impact assessment.

The number of unique values a label or attribute can take. User IDs and request IDs are high cardinality; a region label is low. High cardinality is good for debugging and expensive to index in metric systems, which is why traditional TSDBs push back on it.

AWS’s built-in logs and metrics service. Collects logs from Lambda, ECS, EC2, API Gateway, and most other AWS services. Queried with CloudWatch Logs Insights. ControlTheory ingests CloudWatch as a source so AWS workloads show up next to the rest of your deployment chain.

How a trace stays connected as a request crosses service boundaries. Each service reads the incoming trace context, attaches it to its own spans, and passes it along. The W3C Trace Context standard defines the traceparent and tracestate HTTP headers that carry it, so tracing works even when each hop runs different software. Without it, a distributed trace fragments into disconnected pieces.

The practice of constantly sampling where a running program spends its CPU, memory, and other resources in production, not just in a one-off local profiler run. Often called the “fourth pillar” alongside logs, metrics, and traces, it answers “which function is burning the CPU right now” with low enough overhead to leave on permanently. Tools include Parca, Pyroscope, and eBPF-based profilers.

ControlTheory’s approach of constantly summarizing raw telemetry into smaller, higher-signal artifacts (patterns, regressions, anomalies) so you can scan the result instead of reading every log line. The “Dstl8” name comes from this.

Linking events from different sources by a shared identifier (trace ID, request ID, deploy SHA, timestamp window). The core problem when one user request crosses Vercel, Supabase, and Railway in three different log schemas.

The record of where data came from and everything that happened to it on the way: its source, every transformation, and every table or model it flowed through to reach a dashboard or report. It’s what lets you trace a wrong number back to the join that broke it, and assess the blast radius of a schema change before you ship it. A core building block of data observability.

The ability to understand the health of data flowing through pipelines and warehouses (freshness, volume, schema changes, distribution drift, and lineage) rather than the health of the services running the code. Applies the same outputs-first idea as system observability to datasets, catching a broken dbt model or a silently empty table before a dashboard lies to someone.

The set of platforms a request passes through in production. For example, Vercel edge → Supabase Postgres → Railway worker. Each link has its own logs, its own schema, and its own failure modes, which is why correlating across them is the hard part of debugging.

Following a single request as it moves through multiple services, recording each hop as a span. Tracing tools (OpenTelemetry, Jaeger, Tempo) stitch the spans together into a tree you can read like a stack trace across machines.

ControlTheory’s runtime feedback product. Connects to your deployment chain (Vercel, Supabase, AWS, Kubernetes, OTLP, GitHub), correlates events across sources, detects patterns and regressions, and feeds findings into Claude Code, Cursor, or Codex through an MCP server.

eBPF (extended Berkeley Packet Filter) lets you run small sandboxed programs inside the Linux kernel without changing kernel source or loading modules. In observability it’s used to capture network, syscall, and performance data straight from the kernel, generating metrics, traces, and profiles with no code changes to the application being watched. It powers a wave of auto-instrumentation and profiling tools that work even on services you can’t recompile.

A serverless function that runs at the CDN edge close to the user (Vercel Edge Functions, Cloudflare Workers, Deno Deploy). Faster cold starts than traditional Lambda, but a more constrained runtime: no full Node API, tighter memory limits, and different debugging surfaces.

System behavior that isn’t visible from any single component but only appears when components interact at runtime. Common when AI-generated code wires together libraries or services that the person prompting didn’t know would meet.

A small process that scrapes data from a third-party system and exposes it in Prometheus format. Examples: node_exporter for host metrics, kube-state-metrics for Kubernetes objects, blackbox_exporter for HTTP probes.

A lightweight, fast log and metrics processor and forwarder, written in C, widely used as the collection agent in cloud-native pipelines (a graduated CNCF project, part of the Fluentd family). It tails logs from files or container runtimes, parses and filters them, and ships them onward, making it one of the common first stops in an observability pipeline before data reaches a backend.

The four measurements Google’s SRE practice recommends watching on any user-facing system: latency (how long requests take), traffic (how many there are), errors (how many fail), and saturation (how full the system is). A deliberately small starting set. Instrument these four first, then add detail where the signals point. Dstl8 watches for regressions across signals like these across your whole deployment chain rather than one service at a time.

ControlTheory’s open-source terminal UI for log analysis. Reads from stdin, files, or a live stream; surfaces patterns and anomalies in real time; runs with no account and no config. brew install gonzo.

Telemetry with many unique label combinations, for example one row per user, per request, or per trace. Essential for debugging individual problems, expensive in traditional metric stores, and central to the modern observability cost problem.

The process of detecting, diagnosing, and resolving a production issue. Usually involves paging the on-call, opening an incident channel, triaging across services, fixing the immediate problem, and writing a postmortem afterward.

Surfacing runtime data inside the tool the developer is already in (Claude Code, Cursor, Codex) instead of forcing a switch to a dashboard. The goal is to close the loop between writing code and seeing how it behaves without a context switch.

Adding code (or auto-instrumentation libraries) that emits telemetry from your application. The OpenTelemetry SDKs are the current standard across most languages and replace the older mix of StatsD clients, Prometheus client libraries, and vendor-specific agents.

Filtering, sampling, and routing telemetry at the collector layer so you only store and pay for the data you actually use. The most common path to lowering a Datadog or Splunk bill without losing the signal you need to debug.

An open-source distributed tracing backend (originally built at Uber, now a graduated CNCF project) that stores trace data and stitches spans into a searchable tree you can read like a stack trace across machines. Commonly paired with OpenTelemetry: instrument with OTel, export to Jaeger to store and visualize. Alternatives in the same role include Grafana Tempo and Zipkin.

A service that listens to the Kubernetes API and emits metrics about the state of cluster objects: how many pods are running, deployment replica counts, job status, node conditions. Distinct from the metrics-server (which reports live CPU/memory usage): kube-state-metrics tells you what the cluster thinks should be true, which is what most Kubernetes alerts are built on.

An open-source container orchestrator. Most cloud-native applications run on it. Produces its own telemetry stream (pod logs, container events, kubelet metrics, audit logs) which is usually one of the louder sources in any observability pipeline.

A key-value pair attached to a metric or log line for filtering and grouping. Prometheus and Loki are built around label-based indexing: queries operate on label sets first and on raw text second (or not at all).

Collecting log lines from many sources into one queryable store. Loki, Elasticsearch, and CloudWatch Logs are common backends; the collection layer is usually Fluent Bit, Vector, or the OpenTelemetry Collector.

Severity tags on a log line: typically TRACE, DEBUG, INFO, WARN, ERROR, FATAL. Most production systems filter at INFO or WARN by default and only enable DEBUG during an active investigation.

The end-to-end handling of log data: collecting it from every source, parsing and enriching it, storing it somewhere queryable, and aging it out on a retention schedule. The umbrella over log aggregation, structured logging, and log retention, and usually where the observability bill grows fastest as a system scales, the point where intelligent telemetry control earns its keep by filtering and routing before storage.

How long you keep log data before deleting or archiving it. Short retention is cheap but loses history you might need to debug a slow-burning regression or satisfy an audit; long retention is the opposite. The usual compromise is tiered: keep recent logs hot and searchable, then roll older ones into cheap cold storage.

Grafana Loki’s query language. Filters by label selectors first, then by line content. Similar in spirit to PromQL but designed for log streams instead of time-series metrics.

Loki and Prometheus are sister projects from Grafana that store different things. Prometheus stores metrics: numeric time series queried with PromQL. Loki stores logs: text lines, indexed only by labels and queried with LogQL. They share the same label model on purpose, so a label set that selects a metric in Prometheus selects the matching logs in Loki, letting you pivot between the two during an investigation.

An open protocol that lets AI coding tools (Claude Code, Cursor, Codex) call external tools and read external context. ControlTheory ships an MCP server that exposes runtime data so the AI can answer questions against production reality, not just the code in the editor.

A numeric measurement sampled over time: request rate, error count, queue depth, p99 latency, CPU usage. Stored in a time-series database and read by dashboards and alerting rules.

ControlTheory’s set of AI agents inside Dstl8. Möbius handles root-cause analysis, impact assessment, fix recommendations, and Slack-side alerting on top of the correlated telemetry stream.

The average time between when a problem starts and when anyone or anything notices it. The front half of an incident’s clock: MTTD covers detection, MTTR covers everything from detection to resolution. Cutting MTTD is largely a monitoring-and-anomaly-detection problem. You can’t fix what you haven’t seen, and it’s where pattern-based runtime tools aim to shave the most time.

The average time from the moment an incident is detected to the moment it’s resolved. The headline reliability metric most teams track, and the one runtime feedback tools are aimed at reducing.

The ability to understand what a running system is doing from its outputs alone: logs, metrics, traces, events. Distinct from monitoring, which only checks conditions you already knew to watch.

The path telemetry takes from where it’s emitted to where it’s stored (collection, processing, filtering, sampling, routing), usually built around an agent like the OpenTelemetry Collector, Fluent Bit, or Vector. The layer where you control cost and shape data before it lands in a backend, and the natural place to apply intelligent telemetry control.

Monitoring checks conditions you already knew to watch: a dashboard of predefined metrics and alerts. Observability is the broader ability to ask new questions of a running system from its raw outputs, including ones you didn’t anticipate when you built it. Monitoring tells you a known thing broke; observability helps you investigate an unknown one, which is what pattern-based tools like Gonzo and Dstl8 are built to surface.

A vendor-neutral standard, set of SDKs, and collector for emitting telemetry. Covers traces, metrics, and logs in one project. Now the default instrumentation choice across most languages and frameworks, and a CNCF-graduated project.

A standalone binary that receives, processes, and exports telemetry. The usual place to do sampling, filtering, redaction, attribute enrichment, and routing before data hits any backend.

OpenTelemetry and Prometheus solve overlapping but different problems. Prometheus is a complete metrics system: it scrapes, stores, and queries time-series data with PromQL. OpenTelemetry is a vendor-neutral way to generate and ship telemetry (traces, metrics, and logs) to whatever backend you choose, including Prometheus. In practice many teams instrument with OpenTelemetry and still store metrics in Prometheus; Dstl8 accepts OTLP directly so either path lands in the same correlated view.

The wire protocol used to ship telemetry between OpenTelemetry SDKs, collectors, and backends. gRPC by default, HTTP/JSON optional. Most modern observability tools accept OTLP directly.

The response time that 99% of requests come in under. The slowest 1% are worse. A “tail latency” measure, and it matters because averages hide pain: a healthy average can still mean the slowest 1% of requests are timing out for real users. Teams track p95, p99, and p99.9 to see the experience at the edges, not the middle. That’s the kind of pattern Gonzo and Dstl8 surface from a live stream instead of an after-the-fact query.

Grouping similar log lines (or other events) into clusters so you can see the shape of activity at a glance instead of reading every line. The first thing Gonzo and Dstl8 do on incoming streams.

The stdout and stderr output of the containers running inside a Kubernetes pod, retrievable with kubectl logs. They’re ephemeral: when a pod is rescheduled or crashes, its logs can vanish with it, so production clusters run a node-level agent (Fluent Bit, Vector, or the OpenTelemetry Collector) to ship them to durable storage before that happens. Dstl8 ingests Kubernetes as a source so pod logs land next to the rest of the deployment chain.

An open-source time-series database and metric monitoring system. The CNCF’s second graduated project after Kubernetes. Most cloud-native metric pipelines start with a Prometheus server or a Prometheus-compatible backend.

A Prometheus metric type that only ever goes up (or resets to zero on restart): total requests served, errors seen, bytes sent. You almost never read a counter’s raw value; you wrap it in rate() to get a per-second rate over a window, which is what turns “5,402,118 requests total” into the useful “120 requests/sec right now.”

A Prometheus metric type that can go up and down, representing a value at a moment in time: current memory in use, queue depth, active connections, temperature. Unlike a counter, you read a gauge directly: its latest value is the answer. Use a counter for things that accumulate, a gauge for things that fluctuate.

A Prometheus metric type that samples observations into preconfigured buckets. For example, counting how many requests fell into each latency band. It exposes _bucket, _sum, and _count series, and you compute quantiles like p99 at query time with histogram_quantile(). The standard way to track latency distributions in Prometheus; summaries are the alternative when you need quantiles computed client-side instead.

Time-series data points in the Prometheus format: a metric name, a set of labels, and a numeric value sampled over time. The four core types are counters (only go up), gauges (go up and down), histograms (bucketed distributions), and summaries (quantiles). They’re scraped from an HTTP endpoint your app or an exporter exposes, and queried with PromQL. At scale the question becomes which of them you actually need to keep, the job of intelligent telemetry control at the collector layer.

Prometheus’s query language. Lets you slice, aggregate, and transform time-series data by labels. The de facto standard for metric queries; most TSDBs implement at least a subset.

A platform-as-a-service for deploying containers, databases, and cron jobs from a Git repo. Common in the Vercel + Supabase + Railway deployment-chain pattern and a frequent source of “why is this worker silently dead” tickets.

Measures the experience of actual users as it happens: page load time, Core Web Vitals, JavaScript errors, and interaction delays captured from real browsers and devices in the field. The opposite of synthetic monitoring, which runs scripted checks against your endpoints: RUM tells you what users genuinely experienced, including the slow networks and old phones you’d never reproduce in a test.

A lightweight way to monitor a request-driven service by watching three things: Rate (requests per second), Errors (how many fail), and Duration (how long they take). A service-level companion to the resource-level USE method, and a fast way to get a new service usefully instrumented without deciding on a hundred metrics first.

A bug introduced by a change that previously worked. In AI-assisted development, regressions often appear runs after the merge that caused them, when production traffic finally hits a new code path.

Working backward from a symptom (a failed request, a fired alert, a customer complaint) to the underlying change or condition that produced it. The slowest part of incident response in most teams.

The distance between what an AI coding tool knows about the code and what’s actually happening to it in production. Closes as runtime telemetry is fed back into the editor; widens as the AI ships more code faster than anyone can read.

The cycle of writing code, deploying it, observing how it behaves, and using that information in the next change. Short loops produce better software; AI-generated code makes the loop both more important and harder to close by hand.

Keeping a subset of telemetry instead of all of it. Head sampling decides at the start of a trace; tail sampling waits and keeps the interesting traces (errors, slow requests, specific users). Both trade fidelity for cost.

A runtime where you don’t manage the server. The platform spins up an instance per request (AWS Lambda, Vercel Functions, Cloudflare Workers, Supabase Edge Functions). Trades operational simplicity for cold-start latency and a more constrained debugging surface.

Service Level Indicator (a thing you measure), Service Level Objective (an internal target for that thing), Service Level Agreement (a contractual promise to a customer). Reliability work is usually organized around these three.

One unit of work in a distributed trace: a function call, an HTTP request, a database query. Spans carry a start time, end time, parent ID, and a set of attributes. A trace is a tree of spans.

The key-value pairs attached to a span that describe what that unit of work was doing: the HTTP route, the database statement, a user ID, a status code. They turn a trace from a bare timing tree into something searchable, letting you filter to “spans where http.status_code = 500” instead of reading every trace by hand. OpenTelemetry defines semantic conventions so common attributes are named consistently across tools, and consistent attributes are what let Dstl8 correlate the same request across Vercel, Supabase, and Railway.

The practice, originating at Google, of applying software engineering to operations. Defines reliability with SLOs, error budgets, and blameless postmortems instead of ad-hoc heroics.

Emitting logs as key-value pairs (usually JSON) instead of free-form strings. Makes logs queryable by field and dramatically cheaper to search and aggregate at scale. The first change most teams make when their log volume gets serious.

An open-source Firebase alternative built on Postgres. Provides auth, database, storage, and edge functions. Its logs (Postgres, auth, edge runtime, realtime) are commonly stitched into a deployment-chain view because no single one tells the whole story.

Runs scripted, simulated transactions against your application on a schedule (load a page, walk through a checkout, hit an API) from known locations, whether or not real users are active. It’s black-box monitoring: catches outages and regressions before customers do, and gives you a clean, consistent baseline because the test is identical every run. Pair it with real user monitoring to cover both “does it work” and “what are users actually getting.”

The data a system emits about its own behavior: the logs, metrics, and traces it produces as it runs. The raw material of observability: you can’t understand a system from its outputs if it isn’t emitting any. Modern telemetry is increasingly standardized on OpenTelemetry, which replaced an older patchwork of vendor agents and per-language clients. It’s also the volume that drives the bill, which is why intelligent telemetry control exists to filter and route it before storage.

Logs (discrete events with detail), metrics (numeric measurements over time), and traces (the path of a single request across services). The framing is a useful starting map (each pillar answers a different question), though it’s increasingly treated as a floor rather than a ceiling, with continuous profiling sometimes called a fourth pillar. The harder problem in practice isn’t the pillars themselves but correlating across them, which is what Dstl8 does on the live stream.

A database optimized for numeric data indexed by timestamp. Prometheus, InfluxDB, VictoriaMetrics, and Mimir are common examples. Stores metrics; not designed for logs or traces, though some have been stretched in that direction.

The full record of one request’s path through a distributed system, made up of spans. A trace ID is the identifier that ties those spans together, and the most useful key for correlating logs and metrics back to a specific request.

The unique identifier shared by every span in a single distributed trace: the key that ties together each hop a request makes across services. The most useful value to log on every line and attach to every metric, because it’s what lets you jump from “this log line” to “the entire request it belonged to.” Joining those scattered signals by trace ID across an otherwise disconnected deployment chain is precisely what Dstl8 automates.

Failures you didn’t think to monitor for because you didn’t know they could happen. The class of problem traditional dashboards miss by design and pattern-based runtime tools are built to surface.

A way to check the health of a resource (a CPU, disk, network link, or memory pool) by looking at three things: Utilization (how busy it is), Saturation (how much work is queued waiting for it), and Errors. The resource-level counterpart to the request-level RED method, and a fast checklist for finding a bottleneck in infrastructure.

A high-performance observability data pipeline tool (written in Rust, now part of Datadog) that collects, transforms, and routes logs, metrics, and traces. Sits in the same slot as Fluent Bit and the OpenTelemetry Collector: the processing layer where you filter, reshape, and route telemetry before it reaches a backend, and where you control how much of it you end up paying to store.

A platform for deploying frontends and edge functions, primarily for Next.js. Common starting point in modern deployment chains and a regular source of “edge function timed out” and “function size limit exceeded” mysteries.