Consolidating all telemetry data in a central location streamlines maintenance. You can also adjust the pipeline configuration centrally and consistently.
The data from your instrumented application can be sent to an OpenTelemetry collector. The collector is a component of OpenTelemetry that receives data (spans, metrics, logs etc) process (pre-processes data), and exports the data (sends it off to a backend that you want to talk to). The structure of any Collector configuration file consists of four classes of pipeline components that access telemetry data:
Receivers collect telemetry from one or more sources. They can be pull or push based, and may support one or more data sources. Like OTLP, Zipkin, Prometheus.
receivers:
otlp:
protocols:
grpc: # Receive data over gRPC
http: # Receive data over HTTP
Once the data is received , the collector can process the data. Processors transform or enrich the telemetry data after it has been received but before it is exported. Common uses include batching, filtering, transforming, enriching, batching, dropping unwanted data or adding metadata.
processors:
batch: # Groups telemetry data into batches for efficient exporting
timeout: 5s
send_batch_size: 1024
attributes: # Modify or add attributes to telemetry data
actions:
- key: environment
value: production
action: insert
An exporter is a component of OpenTelemetry and is how data gets sent to different systems/back-ends. Include Jaeger, Prometheus, Elasticsearch, etc.
exporters:
jaeger:
endpoint: "http://jaeger:14250"
tls:
insecure: true
prometheus:
endpoint: "0.0.0.0:8889" # Prometheus scraper will scrape metrics from this endpoint
Connectors are specialized components that combine receiving, processing, and exporting telemetry. They are often used to integrate with external telemetry pipelines seamlessly.
connectors:
signalfx:
ingest_url: "https://ingest.signalfx.com"
api_token: "your-api-token"
The configurations are in YAML format. At the top level, each components (receivers, processors and exporters) has an ID that serves unique reference.
A collector can run as a
As part of Standalone, you can horizontally scalable collector service deal with low and high-tide telemetry more efficiently and robustly.
Install Collector (Ex: Create container) and a collector config file (Ex: otel-collector-config.yml)
docker-compose.yml
services:
otelcol:
image: otel/opentelemetry-collector-contrib:0.97.0
restart: unless-stopped
command: ["--config=/etc/otel-collector-config.yml", "${OTELCOL_ARGS}"]
volumes:
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
ports:
- 4317:4317
- 4318:4318
In the application server, import OTLPLogExporter / OTLPMetricExporter / OTLPTraceExporter
Ex: For log exporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
logger_provider.add_log_record_processor(SimpleLogRecordProcessor(exporter=OTLPLogExporter(insecure=True))
Ex: For Metric exporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
def create_otlp_reader(export_interval: int) -> MetricReader:
exporter = OTLPMetricExporter(insecure=True)
Ex: For Trace exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
def create_tracing_pipeline() -> BatchSpanProcessor:
exporter = OTLPSpanExporter(insecure=True)
Config receivers
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
Find all receivers (Ex: Splunk etchttps://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) in the GitHub repository of the OpenTelemetry Collector
Data processing happens according to rules or settings defined for each processor, which might include filtering, dropping, renaming, or recalculating telemetry, among other operations.
There are some pre configured processors available ready to use without building the pipeline Processors
Ex: configuration for the log deduplication processor use logdedupprocessor
receivers:
filelog:
include: [./example/*.log]
processors:
logdedup:
interval: 60s
log_count_attribute: dedup_count
timezone: 'America/Los_Angeles'
exporters:
googlecloud:
service:
pipelines:
logs:
receivers: [filelog]
processors: [logdedup]
exporters: [googlecloud]
exporters:
otlphttp/prometheus:
endpoint: "http://prometheus:9090/api/v1/otlp"
tls:
insecure: true
otlp/jaeger:
endpoint: "jaeger:4317"
tls:
insecure: true
These pipelines configure how metrics and traces are received, processed, and exported.
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, memory_limiter]
exporters: [otlp, logging]
metrics:
receivers: [otlp, prometheus]
processors: [batch, resourcedetection]
exporters: [prometheusremotewrite, logging]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp, logging]
✅ Main Page: Click Here