Integrating Docker and Caddy Logs with OpenTelemetry Collector: A Comprehensive Guide

The integration of Docker and Caddy logs into an OpenTelemetry Collector requires a systematic approach that leverages containerization best practices, structured logging configurations, and OpenTelemetry’s flexible data collection capabilities. This guide provides a detailed methodology for extracting, processing, and exporting logs from Docker containers and Caddy servers to an OpenTelemetry Collector, ensuring observability across distributed systems.

Key Findings Summary

To achieve seamless log integration:

Docker Logs: Configure Docker’s JSON logging driver to write logs to /var/lib/docker/containers, then use OpenTelemetry’s filelog receiver to collect and parse these logs6 10.
Caddy Logs: Enable Caddy’s HTTP request logging in the Caddyfile, format logs as JSON for machine readability, and route logs to a shared Docker volume accessible by the Collector7 8.
OpenTelemetry Collector: Deploy the Collector as a Docker service with volume mounts for log directories, using pipelines to process Docker/Caddy logs and export them via OTLP/gRPC4 10.
Trace Correlation: Implement Caddy’s tracing directive to inject OpenTelemetry-compatible trace IDs into logs, enabling cross-service correlation in observability platforms8 14.

Docker Log Management for OpenTelemetry

Docker Log Configuration

Docker containers emit logs to stdout and stderr by default, stored as JSON files in /var/lib/docker/containers/<container-id>/<container-id>-json.log. To optimize for OpenTelemetry:

JSON Log Driver: Ensure Docker uses the json-file driver with metadata enrichment:

text

# docker-compose.yml services: caddy: logging: driver: json-file options: tag: "{{.Name}}|{{.ImageName}}|{{.ID}}"

This adds container name, image, and ID to each log entry, facilitating later parsing6 10.
Log Retention: Limit log file size to prevent disk exhaustion:

text

options: max-size: "10m" max-file: "3"

OpenTelemetry Collector Configuration

Deploy the Collector as a Docker service with access to Docker logs:

Volume Mount: Share the host’s Docker logs directory with the Collector:

text

services: otel-collector: volumes: - /var/lib/docker/containers:/var/lib/docker/containers:ro
Filelog Receiver: Configure the Collector to tail Docker logs:

text

receivers: filelog: include: [/var/lib/docker/containers/*/*-json.log] operators: - type: json_parser timestamp: parse_from: attributes.time layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes.log to: body

This extracts the log message body and parses timestamps10 6.

Caddy Server Log Integration

Enabling Structured Logging

Caddy’s default logs omit HTTP requests. Enable access logging in the Caddyfile:

text

example.com { log { output file /var/log/caddy/access.log format json } }

This writes JSON-formatted access logs to /var/log/caddy/access.log, including fields like request duration and response status7.

Enriching Logs with Tracing Context

Activate Caddy’s OpenTelemetry integration to inject trace IDs:

text

tracing { span http_request } log { format json { time_format "iso8601" trace_id {http.vars.trace_id} span_id {http.vars.span_id} } }

This appends OpenTelemetry trace_id and span_id to logs, enabling correlation with distributed traces8 14.

Mount a persistent volume for Caddy logs:

text

services: caddy: volumes: - caddy-logs:/var/log/caddy otel-collector: volumes: - caddy-logs:/var/log/caddy:ro volumes: caddy-logs:

This allows the Collector to read Caddy logs without direct host access1 10.

OpenTelemetry Collector Deployment

Pipeline Configuration

Define a log processing pipeline in otel-collector-config.yaml:

text

receivers: filelog: include: - /var/lib/docker/containers/*/*-json.log - /var/log/caddy/access.log operators: # Docker log parsing - type: json_parser id: docker_parser timestamp: parse_from: attributes.time layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: remove field: attributes.time # Caddy log parsing - type: json_parser id: caddy_parser parse_from: body processors: batch: timeout: 5s exporters: otlp: endpoint: "otel-backend:4317" tls: insecure: true service: pipelines: logs: receivers: [filelog] processors: [batch] exporters: [otlp]

Docker Compose Deployment

Orchestrate services with Docker Compose:

text

version: '3' services: caddy: image: caddy:latest volumes: - ./Caddyfile:/etc/caddy/Caddyfile - caddy-logs:/var/log/caddy ports: - "80:80" otel-collector: image: otel/opentelemetry-collector-contrib:latest command: ["--config=/etc/otel/config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel/config.yaml - /var/lib/docker/containers:/var/lib/docker/containers:ro - caddy-logs:/var/log/caddy:ro volumes: caddy-logs:

Advanced Considerations

Log Sampling and Filtering

Reduce noise by sampling logs in the Collector:

text

processors: probabilistic_sampler: sampling_percentage: 20

Apply to non-critical logs to decrease volume4.

Multi-Line Log Parsing

For non-JSON logs, use multiline operators:

text

operators: - type: multiline line_start_pattern: '^\d{4}-\d{2}-\d{2}'

This aggregates stack traces into single log entries2 5.

Security Hardening

Role-Based Access: Restrict Collector service permissions in Docker:

text

otel-collector: user: "1000:1000" read_only: true
TLS Encryption: Secure OTLP exports with certificates:

text

exporters: otlp: tls: cert_file: /etc/otel/certs/client.crt key_file: /etc/otel/certs/client.key

Conclusion

Integrating Docker and Caddy logs with OpenTelemetry involves:

Configuring Docker and Caddy to emit structured logs with metadata.
Deploying the OpenTelemetry Collector with tailored filelog receivers.
Correlating logs with traces via Caddy’s OpenTelemetry integration.

For production systems, augment this base with log sampling, alerting pipelines, and secured OTLP exports. The provided configurations offer a foundation for full-stack observability, adaptable to Prometheus, Jaeger, or commercial backends.

Integrating Docker and Caddy Logs with OpenTelemetry Collector: A Comprehensive Guide

Key Findings Summary

Docker Log Management for OpenTelemetry

Docker Log Configuration

OpenTelemetry Collector Configuration

Caddy Server Log Integration

Enabling Structured Logging

Enriching Logs with Tracing Context

Docker Volume for Log Sharing

OpenTelemetry Collector Deployment

Pipeline Configuration

Docker Compose Deployment

Advanced Considerations

Log Sampling and Filtering

Multi-Line Log Parsing

Security Hardening

Conclusion

Citations: