Traditional observability architectures often force teams into a fragmented paradigm: time-series databases for metrics, document stores for logs, and specialized APM platforms for distributed traces. This decoupled approach introduces operational complexity, duplicate infrastructure overhead, and context switching during active incident responses.
By unifying telemetry around OpenTelemetry (OTel), leveraging ClickHouse Cloud as a single columnar analytical store, and serving visualizations via Grafana, organizations can establish a high-throughput, cost-effective observability control plane.
Architectural Layout & Component Responsibilities
The architecture is divided into three distinct operational layers: Edge Ingestion, Centralized Processing, and Analytical Storage & Visualization.
[ Local Host / App ] ──OTLP/gRPC──► [ Central OTel Collector ] ──HTTPS/8443──► [ Cloud ClickHouse ] ◄──SQL── [ Local Grafana ]
1. The Edge Layer: Host Nodes & Application Runtimes
At the edge, services run either auto-instrumented runtime agents or native OpenTelemetry SDKs.
Dual-Protocol Ingestion: The edge node standardizes on the OpenTelemetry Protocol (OTLP). Applications emit binary-encoded data via gRPC (Port 4317) for low-latency batch streaming, or HTTP/JSON (Port 4318) for environments behind strict proxy boundaries.
Edge Agent Processing: Lightweight collector instances operating in "agent mode" reside on the host. They collect local system metrics, tail local log files, and buffer trace spans in memory before forwarding them over local loopbacks (or mTLS tunnels) to the central processing plane.
2. The Centralized Processing Control Plane
The central OpenTelemetry Collector operates as a stateless data-processing engine. It decouples application runtimes from the downstream database storage layer.
Receiver Phase: Ingestion pipelines expose OTLP listeners that handle incoming trace, metric, and log payloads simultaneously.
Processor Pipeline:
Memory Limiter: Protects the collector from out-of-memory panics by enforcing backpressure when heap usage spikes.
Resource Detection & Enrichment: Injects global metadata (e.g., environment tags, cloud region, host IDs) into telemetry records before write operations.
Transformation & Redaction: Cleans raw payloads, redacts sensitive attributes (like PII or API tokens), and normalizes log severities.
Batching: Aggregates granular payloads into larger memory blocks to minimize raw network request overhead to the cloud storage layer.
Exporter Phase: Formats processed batches and prepares them for HTTPS transmission.
3. Encrypted Transport & Cloud Storage
The central collector streams batched telemetry to Cloud ClickHouse over encrypted HTTPS (Port 8443 / SSL-TLS).
Columnar Storage Optimization: Unlike traditional row-oriented or document-based search engines, ClickHouse stores data column by column. Because log messages, trace IDs, and metric labels share highly repetitive patterns, columnar compression algorithms (such as LZ4 or ZSTD) achieve compression ratios between 7x and 11x.
Table Schema Partitioning:
otel.otel_logs: Stores structured logs alongside string-indexed TraceId and SpanId attributes for cross-signal correlation.
otel.otel_traces: Stores span data, parent-child execution trees, and duration latencies.
otel.otel_metrics_*: Stores gauges, sums, and histogram samples in optimized sub-tables.
4. Local Visualization & Cross-Signal Correlation
A locally deployed Grafana instance interfaces directly with Cloud ClickHouse using native SQL plugins.
Unified Context Switching: Instead of relying on proprietary backend indexes, Grafana queries ClickHouse directly. Because logs, metrics, and traces reside in the same database engine, users can click on a latency spike in a metric panel, extract the corresponding TraceId, and immediately surface all associated application logs without changing tools or context.
Key Theoretical Advantages
Vendor Neutrality: Standardizing on OpenTelemetry ensures that data collection instrumentation remains entirely decoupled from the underlying storage technology.
Storage Efficiency: Utilizing a single OLAP database for all three observability pillars eliminates data duplication and reduces disk footprint through columnar compression.
High-Throughput Ingestion: ClickHouse is designed for massive parallel batch writes, allowing the architecture to ingest millions of telemetry events per second with predictable resource utilization.