MyObservability

Recommended architecture

1. Setup OpenTelemetry Collector in the edge of your organization network
2. Configure OpenTelemetry Collector to receive, process and export
3. Instrument application and export telemetry to OpenTelemetry Collector

OpenTelemetry Collector Setup

Depends on your requirement use following collector

1. OpenTelemetry Collector (also known as “otelcol”): Includes only the core components (basic receivers, processors, exporters).
2. OpenTelemetry Collector Contrib (also known as “otelcol-contrib”): Includes contrib receivers/exporters (e.g., Prometheus, Kafka, Splunk HEC, Loki, AWS X-Ray, filelog, etc.).
3. OpenTelemetry Collector for Kubernetes (also known as “otelcol-k8s”): A Kubernetes-tailored distribution of the Collector.
4. OpenTelemetry Collector OTLP (also known as “otelcol-otlp”): Lightweight and high-performance — useful for gateways that only need to forward data.

Example Setup

Ex:

Step:1 Install OpenTelemetry Collector on a Central Server

1. Download & extract Opentelemetry Collector: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions
2. update Collector Config: otel-collector-config.yaml

   • Setup Grafana Tempo end point and Splunk HEC endpoint to collect as below

   • Traces -> Grafana Tempo
   • Profiling -> Grafana Tempo
   • Metrics -> Splunk Core (Hec/OTLP)
   • Logs -> Splunk Core (HEC)

receivers:
  otlp:
    protocols:
      grpc:
      http

exporters:
  otlphttp/tempo:
    endpoint: http://<tempo-host>:4318
    tls:
      insecure: true

  # Splunk HEC for logs
  splunk_hec/logs:
    token: "<SPLUNK_HEC_TOKEN>"
    endpoint: https://<splunk-hec-host>:8088
    source: otel-logs
    sourcetype: otel
    index: main
    tls:
      insecure_skip_verify: true

  # Splunk Metrics exporter
  splunk_hec/metrics:
    token: "<SPLUNK_HEC_TOKEN>"
    endpoint: https://<splunk-hec-host>:8088
    source: otel-metrics
    index: main
    tls:
      insecure_skip_verify: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/tempo]

    logs:
      receivers: [otlp]
      exporters: [splunk_hec/logs]

    metrics:
      receivers: [otlp]
      exporters: [splunk_hec/metrics]

Step 2: Configure Java App to Send Telemetry data to the Collector

In this example we are using Auto instrumentation with Java agent

1. Download OTel Java agent: wget https:‌//github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.8.0/opentelemetry-javaagent.jar
2. Attch to Java process: java -javaagent:./opentelemetry-javaagent.jar -jar my_app.jar
3. Setup environment variable

java \
-javaagent:opentelemetry-javaagent.jar \
-Dotel.service.name=my-java-app \
-Dotel.exporter.otlp.endpoint=http://<otel-collector-ip>:4317 \
-Dotel.logs.exporter=otlp \
-Dotel.metrics.exporter=otlp \
-Dotel.traces.exporter=otlp \
-Dotel.profiling.enabled=true \
-Dotel.profiling.exporter=otlp \
-jar my-app.jar

Step 3: Validate Grafana Tempo & Splunk Setup

• In Grafana, use the Tempo data source to view traces and flame graphs (profiling).
• In Splunk Cloud, configure HEC endpoint, token, and ensure logs/metrics are indexed correctly.

---

---

Next Chapter: Instrumentation

Main Page: Click Here

---