OpenTelemetry Driver

Introduction

Note: This driver is only available with Kipchak Enterprise.

The OpenTelemetry driver sends your API's traces, metrics and logs to an OpenTelemetry Collector over OTLP, from where they reach Grafana Tempo, Jaeger, Honeycomb, Datadog, Elastic or any other backend that accepts OTLP.

• Instrumented drivers: every request made through the HTTP driver (and so the OPA, Vault, ClickHouse and other drivers built on it), every Doctrine query, every Valkey command and every Kafka message gets a span, with no code changes.
• Traces across services: outgoing HTTP requests and Kafka messages carry the W3C trace context, so the service called, or the consumer of the message, continues the same trace.
• Logs with their trace: every line written through the Logger driver carries its trace_id and span_id, and logs can be exported to the collector too.
• Built for worker mode: telemetry is exported after the response has been sent, without waiting for the collector, and an unreachable collector never slows requests down.
• Your own spans and metrics: OpenTelemetry::span() traces any piece of work, and OpenTelemetry::meter() records any measurement.

The OpenTelemetry middleware adds a server span for every request, so the spans above belong to the request that caused them, and request metrics.

Installation

To install this driver, you need access to the Enterprise Composer repository at https://php.pkgs.1x.ax/.

composer require kipchak/driver-opentelemetry kipchak/middleware-opentelemetry

Initialise the driver in drivers/drivers.php, after the Logger driver, and the middleware last in middlewares/middlewares.php, so that it runs first:

// drivers/drivers.php
use Kipchak\Driver\OpenTelemetry\OpenTelemetry;

Config::initialise($container);
Logger::initialise($container);
OpenTelemetry::initialise($container);
Http::initialise($container);

// middlewares/middlewares.php
use Kipchak\Middleware\OpenTelemetry\OpenTelemetry;

Error::initialise($app);
// ... other middleware
OpenTelemetry::initialise($app);

Telemetry is exported after each response has been sent, in the worker loop of html/index.php. The starter project calls $app->terminate() there from version 3.19 (kipchak/core 3.15). In an existing API, add it after frankenphp_handle_request():

$keepRunning = \frankenphp_handle_request($handler);

// Work registered with $app->onTerminate(), such as exporting telemetry, runs here, after the response has
// been sent
$app->terminate();

The instrumentation of each driver needs a recent version of it: kipchak/driver-http 2.15, kipchak/driver-doctrine 1.51, kipchak/driver-valkey 1.34 and kipchak/driver-kafka 1.16. An older version is simply not instrumented, and the driver logs a notice naming the update needed.

Configuration

Copy the sample config to config/kipchak.opentelemetry.php:

<?php

use function Kipchak\Core\env;

return [
    'enabled' => true,

    'service' => [
        'name' => env('OTEL_SERVICE_NAME', 'orders-api'),  // defaults to the name in kipchak.api
        'version' => env('APP_VERSION', ''),
        'environment' => env('APP_ENV', ''),             // deployment.environment.name
    ],
    'resource' => [],                                     // more resource attributes, such as ['team' => 'payments']

    'exporter' => [
        'endpoint' => env('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://otel-collector:4318'),
        'protocol' => 'http/protobuf',                    // or 'http/json'
        'headers' => [],                                  // such as ['Authorization' => 'Bearer ...']
        'compression' => 'gzip',                          // or 'none'
        'timeout' => 5,
    ],

    'traces' => ['enabled' => true, 'sample_ratio' => 1.0],
    'metrics' => ['enabled' => true, 'interval' => 60],
    'logs' => ['correlate' => true, 'export' => false, 'level' => 'info'],

    'export' => [
        'batch_size' => 512,
        'max_queue_size' => 2048,
        'delay' => 1,
        'retry_after' => 30,
    ],

    'instrument' => [
        'http' => true,                                   // or ['query' => true] to record query strings
        'doctrine' => true,                               // or ['statements' => false] to leave out SQL text
        'valkey' => true,                                 // or ['cache_dir' => '/path'] for the generated class
        'kafka' => true,
    ],

    'server' => [],                                       // the middleware's settings
];

The config is checked when the worker starts, and a mistake stops it with a message naming the setting.

What is traced

Doctrine records SQL text but never bound parameters: queries built by Doctrine use placeholders. If your API writes values into SQL text, set 'doctrine' => ['statements' => false]. Valkey spans record the command, never keys or values. A failed call, an exception or a 5xx response marks its span as an error.

The metrics follow the OpenTelemetry semantic conventions: http.server.request.duration (by route), http.client.request.duration and db.client.operation.duration.

Your own spans and metrics

use Kipchak\Driver\OpenTelemetry\OpenTelemetry;

$total = OpenTelemetry::span('price basket', fn () => $pricing->total($basket), ['basket.items' => count($basket)]);

OpenTelemetry::meter()->createCounter('orders.placed')->add(1, ['channel' => 'web']);

$traceId = OpenTelemetry::traceId();   // for an error response or a support ticket

A span started in span() is the current span while its callback runs, so the driver calls made inside it belong to it. An exception marks it as an error and is rethrown. OpenTelemetry::tracer() gives the full OpenTelemetry API for anything else.

A Kafka consumer continues the producer's trace for the work it does with a message:

$message = $consumer->consume();

OpenTelemetry::span('fulfil order', fn () => $orders->fulfil($message), parent: OpenTelemetry::extract($message->headers));

OpenTelemetry::inject($headers) adds the trace context to headers for calls the instrumentation does not cover. In a CLI command that ends quickly, OpenTelemetry::flush() exports everything before it exits; anything left is exported when the process ends in any case.

Exporting in worker mode

Between requests a FrankenPHP worker waits for the next request with nothing running, so the driver exports after a response has been sent, in $app->terminate(), and writes each export to the collector's socket without waiting for its answer. Exports never delay a response, and a slow collector never holds up a worker.

• A worker that was idle before a request exports everything after it, so nothing waits in a worker that may now stay idle.
• A busy worker exports when it holds batch_size records or the oldest has waited delay, so under load each worker sends a few large exports rather than one per request.
• If traffic stops abruptly, each worker may hold up to delay seconds of records until its next request, or until it exits; liveness probes keep this short. Workers export what they hold when they exit.
• If the collector cannot be reached, exports pause for retry_after, and records produced in the meantime are dropped rather than held, so an outage costs each worker one failed export per period and no memory. Each pause is logged once per worker, with the number of records dropped.

In testing on the Kipchak Enterprise runtime image, with the collector stopped under load, every request succeeded, throughput fell by 10 to 15%, and exports resumed when the collector returned.

Performance

A span costs about 15 microseconds to create and export on the Kipchak runtime image. In testing, a route that wrote and read a row through Doctrine, set and read a Valkey key, called a second API over HTTP, ran a span of its own and wrote a log line took 957 microseconds without telemetry, 1,234 with every span recorded and logs exported, and 1,085 with 10% of traces sampled. A route that spends its time on real database and network work sees a much smaller share.

Over 20,500 requests to that route in worker mode, across two APIs, every span, log line and metric reached the collector, each request was one trace from the first API into the second, and worker memory stayed constant.

Valkey

phpredis has no hooks, so the driver has the Valkey driver connect with a subclass of \Redis or \RedisCluster that traces each command. The subclass is generated from the installed phpredis extension's own signatures, so it matches whatever version is installed, and written once to instrument.valkey.cache_dir (the system temporary directory by default) so that OPcache keeps it. Where nothing can be written, it is compiled in memory instead.

Commands queued in a pipeline or a transaction are traced as one PIPELINE or EXEC span. Pools that set options.class themselves, RedisArray pools, Predis and Relay are not instrumented.

Git Repository

The source code for this driver is available on 1x.ax at https://1x.ax/mamluk/enterprise/kipchak/drivers/opentelemetry.