Tracing in the Apollo Router

Collect tracing information

The Apollo Router supports OpenTelemetry, with exporters for:

Jaeger
Zipkin
Datadog
OpenTelemetry Protocol (OTLP) over HTTP or gRPC.

The Apollo Router generates spans that include the various phases of serving a request and associated dependencies. This is useful for showing how response time is affected by:

Sub-request response times
Query shape (sub-request dependencies)
Apollo Router post-processing

Span data is sent to a collector such as Jaeger, which can assemble spans into a gantt chart for analysis.

To get the most out of distributed tracing, all components in your system should be instrumented.

The trace_config section contains common configuration that will be used by all exporters. It is optional and will fall back on env variables as specified by the OpenTelemetry spec if service_name is not set.

router.yaml

1
telemetry:
2
  tracing:
3
    trace_config:
4
      service_name: "router"
5
      service_namespace: "apollo"
6
      # Optional. Either a float between 0 and 1 or 'always_on' or 'always_off'
7
      sampler: 0.1
8
      
9
      # Optional. Use a parent based sampler. This enables remote spans help make a decision on if a span is sampeld or not.  
10
      # https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#parentbased
11
      parent_based_sampler: false
12
      
13
      # Optional limits 
14
      max_attributes_per_event: 10
15
      max_attributes_per_link: 10
16
      max_attributes_per_span: 10
17
      max_events_per_span: 10
18
      max_links_per_span: 10
19
      
20
      # Attributes particular to an exporter that have not
21
      # been explicitly handled in Router configuration.
22
      attributes: 
23
        some.config.attribute: "config value"

If service_name is set then environment variables are not used. However, it is possible to embed environment variables into your router config using Unix ${key:default} syntax.

If no environment variable is set and service_name is not present then router will be used as default service name.

Propagation

The propagation section allows you to configure which propagators are active in addition to ones automatically activated by virtue of using an exporter.

router.yaml

1
telemetry:
2
  tracing:
3
    propagation:
4
      # https://www.w3.org/TR/baggage/
5
      baggage: false
6
      
7
      # https://www.datadoghq.com/
8
      datadog: false
9
      
10
      # https://www.jaegertracing.io/ (compliant with opentracing)
11
      jaeger: false 
12
      
13
      # https://www.w3.org/TR/trace-context/
14
      trace_context: false
15
      
16
      # https://zipkin.io/ (compliant with opentracing)
17
      zipkin: false

Specifying explicit propagation is generally only required if you are using an exporter that supports multiple trace ID formats. For example OpenTelemetry Collector, Jaeger or OpenTracing compatible exporters.

Using Datadog

The Apollo Router can be configured to connect to either the default agent address or a URL.

router.yaml

1
telemetry:
2
  tracing:
3
    datadog:
4
      # Either 'default' or a URL
5
      endpoint: default

Using Jaeger

The Apollo Router can be configured to export tracing data to Jaeger either via an agent or http collector.

Agent config

router.yaml

1
telemetry:
2
  tracing:
3
    jaeger:
4
      agent:
5
        # Either 'default' or an url 
6
        endpoint: docker_jaeger:14268

Collector config

If you are using Kubernetes then you can inject your secrets into configuration via env variable.

router.yaml

1
telemetry:
2
  tracing:
3
    jaeger:
4
      collector: 
5
        endpoint: "http://my-jaeger-collector"
6
        username: "${JAEGER_USERNAME}"
7
        password: "${JAEGER_PASSWORD}"

OpenTelemetry Collector via OTLP

OpenTelemetry Collector is a horizontally scalable collector that can be used to receive, process and export your telemetry data in a pluggable way.

If you find that the built-in telemetry features of the Apollo Router are missing some desired functionality e.g. exporting to Kafka then it's worth considering this option.

router.yaml

1
telemetry:
2
  tracing:
3
    otlp:
4
      # Either 'default' or a URL
5
      endpoint: default
6
      
7
      # Optional protocol (Defaults to grpc)
8
      protocol: grpc
9
      
10
      # Optional Grpc configuration
11
      grpc:
12
        domain_name: "my.domain"
13
        key:
14
          file: ""
15
          # env: ""
16
        ca:
17
          file: ""
18
          # env: ""
19
        cert:
20
          file: ""
21
          # env: ""
22
        metadata:
23
          foo: bar
24
      
25
      # Optional Http configuration
26
      http:
27
        headers:
28
          foo: bar   
29
      
30
      # Optional timeout in humatime form       
31
      timeout: 2s

Using Zipkin

The Apollo Router can be configured to export tracing data to Zipkin either via an agent or http collector.

Agent config

router.yaml

1
telemetry:
2
  tracing:
3
    zipkin:
4
      agent:
5
        # Either 'default' or an url 
6
        endpoint: http://my_zipkin_agent.dev

Collector config

router.yaml

1
telemetry:
2
  tracing:
3
    zipkin:
4
      collector: 
5
        endpoint: "http://my-zipkin-collector"

Edit on GitHub

Collecting metrics

Overview

Tracing in the Apollo Router

Collect tracing information

Common configuration

Trace config

Propagation

Using Datadog

Using Jaeger

Agent config

Collector config

OpenTelemetry Collector via OTLP

Using Zipkin

Agent config

Collector config