Live Tailing Grafana Loki Logs with Gonzo

Observability teams often use Grafana Loki for centralized log storage and querying. But when you need to watch logs as they happen (i.e. Live Tailing Loki) —for debugging, troubleshooting, or monitoring—live tailing becomes essential. That’s where Gonzo comes in: an open-source, OTLP-native, terminal-based log viewer (think k9s for logs) with support for Loki’s native formats.

We recently extended Gonzo with custom formats to support Loki’s JSON output, making it easy to plug into either the logcli utility or Loki’s Live Tail API. This unlocks a powerful workflow: use Loki as your log store, and Gonzo as your real-time lens.

Why Live Tailing with Loki?

While Loki is well known for its powerful query language (LogQL) and long-term retention, live tailing has often been more difficult. Developers and SREs still need a way to:

• Debug in real time: Watch logs as code changes roll out.
• Validate pipelines: Ensure that OpenTelemetry, sidecars, or agents are shipping logs correctly into Loki.
• Shorten feedback loops: Catch issues instantly without switching between dashboards and terminals.
• Stay in-flow: Operate inside a terminal, right next to kubectl, stern, and k9s.

By combining Loki’s live tailing with Gonzo, you can stay connected to the stream while still benefiting from Gonzo’s OTLP-native filters, search, and (optional) AI-powered assist features.

Gonzo now supports extensible log formats 🙌, and as part of that functionality, we just included two Loki JSON format examples – one that can be used with the popular logcli tool, and one that can be used with the Loki live tail API endpoint.

Option 1: Using Loki logcli

Loki ships with a CLI utility called logcli. With Gonzo’s loki-stream format, you can follow a query in real time:

logcli query --addr=http://localhost:3100 --follow '{service_name=~".+"}' -o jsonl 2>/dev/null | gonzo --format=loki-stream

logcli query --addr=http://localhost:3100 --follow '{service_name=~".+"}' -o jsonl 2>/dev/null | gonzo --format=loki-stream

This connects logcli directly to Gonzo, formatting logs for easy viewing.

Option 2: Using Loki’s Live Tail API (WebSocket)

You can also connect Gonzo directly to Loki’s Live Tail API. Using wscat:

wscat -c 'ws://localhost:3100/loki/api/v1/tail?query={service_name=~".%2B"}&limit=50' | gonzo --format=loki-batch

wscat -c 'ws://localhost:3100/loki/api/v1/tail?query={service_name=~".%2B"}&limit=50' | gonzo --format=loki-batch

Here, Gonzo listens on its loki-batch format to handle streaming log batches from Loki.

Try It Yourself: Quick Test with Docker Compose

Want to try this locally? Here’s a quick setup using Loki, Grafana, and the OpenTelemetry Collector.

docker-compose.yaml

version: "3.8"
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - loki

  otel-collector:
    image: otel/opentelemetry-collector:0.88.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    depends_on:
      - loki
    ports:
      - "4317:4317"
      - "4318:4318"

version: "3.8"
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - loki

  otel-collector:
    image: otel/opentelemetry-collector:0.88.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    depends_on:
      - loki
    ports:
      - "4317:4317"
      - "4318:4318"

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  otlphttp/loki:
    endpoint: <http://loki:3100/otlp>
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  otlphttp/loki:
    endpoint: <http://loki:3100/otlp>
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]

Generate Some Logs with otelgen

Once Loki and the collector are running, generate some OpenTelemetry logs using a utility like otelgen or telemetrygen, and stream them into Loki – targeting a Mac container endpoint here:

docker run --rm ghcr.io/krzko/otelgen:latest \
  --otel-exporter-otlp-endpoint host.docker.internal:4317 \
  --insecure logs

docker run --rm ghcr.io/krzko/otelgen:latest \
  --otel-exporter-otlp-endpoint host.docker.internal:4317 \
  --insecure logs

Now tail them in real time with Gonzo:

logcli query --addr=http://localhost:3100 --follow '{service_name=~".+"}' -o jsonl 2>/dev/null | gonzo --format=loki-stream

logcli query --addr=http://localhost:3100 --follow '{service_name=~".+"}' -o jsonl 2>/dev/null | gonzo --format=loki-stream

Wrapping Up

With Gonzo’s new Loki JSON support, you can plug into Loki either via logcli or directly with the Live Tail API. This makes it easier to debug, validate, and monitor logs in real time—right from your terminal.

Try Gonzo today: https://github.com/control-theory/gonzo

Got other formats you’d like to see supported?, drop us an issue!

Back

Share