OpenTelemetry + Laravel: Production Observability Beyond Telescope

Laravel Telescope is great for development. Laravel Pulse is good for high-level metrics. But for production debugging at scale — tracing a request across queues, databases, external APIs, and microservices — you need OpenTelemetry.

OpenTelemetry (OTel) is the industry standard for observability. It collects traces, metrics, and logs and exports them to any backend (Jaeger, Grafana Tempo, Datadog, New Relic).

Telescope vs Pulse vs OpenTelemetry

Telescope = local dev debugging. Pulse = production dashboard for Laravel-specific metrics. OpenTelemetry = full observability for production distributed systems.

Three Pillars of Observability

Traces  → What happened step by step (request → DB → cache → API → response)
Metrics → What are the numbers (request count, latency p99, error rate)
Logs    → What was said (error messages, context, stack traces)

OpenTelemetry unifies all three with correlation — a trace ID links logs and metrics to specific request flows.

Setting Up OpenTelemetry in Laravel

Install the PHP SDK

composer require open-telemetry/sdk
composer require open-telemetry/exporter-otlp
composer require open-telemetry/transport-grpc

# Auto-instrumentation for common libraries
composer require open-telemetry/contrib-auto-laravel
composer require open-telemetry/contrib-auto-pdo
composer require open-telemetry/contrib-auto-http-client

Environment Configuration

# .env
OTEL_SERVICE_NAME=my-blog
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_PHP_AUTOLOAD_ENABLED=true

Docker Compose for Local Development

# docker-compose.yml
services:
  app:
    build: .
    environment:
      OTEL_SERVICE_NAME: my-blog
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686" # Jaeger UI
      - "14268:14268"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

OTel Collector Config

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889

  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, logging]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Manual Tracing in Laravel

Creating Spans

// app/Services/PostService.php

use OpenTelemetry\API\Globals;
use OpenTelemetry\API\Trace\StatusCode;

class PostService
{
    public function getPost(string $slug): Post
    {
        $tracer = Globals::tracerProvider()->getTracer('blog');

        $span = $tracer->spanBuilder('PostService.getPost')
            ->setAttribute('post.slug', $slug)
            ->startSpan();

        $scope = $span->activate();

        try {
            $post = Cache::remember("post_{$slug}", 3600, function () use ($slug, $tracer) {
                $dbSpan = $tracer->spanBuilder('database.query')
                    ->setAttribute('db.system', 'mysql')
                    ->setAttribute('db.statement', 'SELECT * FROM posts WHERE slug = ?')
                    ->startSpan();

                $post = Post::where('slug', $slug)->firstOrFail();

                $dbSpan->setAttribute('db.rows_affected', 1);
                $dbSpan->end();

                return $post;
            });

            $span->setAttribute('post.id', $post->id);
            $span->setAttribute('post.title', $post->title);
            $span->setStatus(StatusCode::STATUS_OK);

            return $post;
        } catch (\Exception $e) {
            $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());
            $span->recordException($e);
            throw $e;
        } finally {
            $span->end();
            $scope->detach();
        }
    }
}

Middleware for Request Tracing

// app/Http/Middleware/TraceRequests.php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use OpenTelemetry\API\Globals;
use OpenTelemetry\API\Trace\StatusCode;

class TraceRequests
{
    public function handle(Request $request, Closure $next): mixed
    {
        $tracer = Globals::tracerProvider()->getTracer('http');

        $span = $tracer->spanBuilder("HTTP {$request->method()} {$request->path()}")
            ->setAttribute('http.method', $request->method())
            ->setAttribute('http.url', $request->fullUrl())
            ->setAttribute('http.route', $request->route()?->uri() ?? 'unknown')
            ->setAttribute('http.user_agent', $request->userAgent() ?? '')
            ->setAttribute('http.client_ip', $request->ip())
            ->startSpan();

        $scope = $span->activate();

        try {
            $response = $next($request);

            $span->setAttribute('http.status_code', $response->getStatusCode());

            if ($response->getStatusCode() >= 400) {
                $span->setStatus(StatusCode::STATUS_ERROR);
            } else {
                $span->setStatus(StatusCode::STATUS_OK);
            }

            return $response;
        } catch (\Exception $e) {
            $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());
            $span->recordException($e);
            throw $e;
        } finally {
            $span->end();
            $scope->detach();
        }
    }
}

Custom Metrics

// app/Providers/AppServiceProvider.php

use OpenTelemetry\API\Globals;

public function boot(): void
{
    $meter = Globals::meterProvider()->getMeter('blog');

    // Counter: total page views
    $pageViews = $meter->createCounter(
        'blog.page_views',
        'views',
        'Total page views'
    );

    // Histogram: response time
    $responseTime = $meter->createHistogram(
        'blog.response_time',
        'ms',
        'Response time in milliseconds'
    );

    // Store in container for use elsewhere
    $this->app->singleton('metrics.page_views', fn () => $pageViews);
    $this->app->singleton('metrics.response_time', fn () => $responseTime);
}

// Usage in controller
class BlogController extends Controller
{
    public function show(string $slug)
    {
        $start = microtime(true);

        $post = $this->postService->getPost($slug);

        // Record metrics
        app('metrics.page_views')->add(1, ['slug' => $slug]);
        app('metrics.response_time')->record(
            (microtime(true) - $start) * 1000,
            ['route' => 'blog.show']
        );

        return view('blog.show', compact('post'));
    }
}

Structured Logging with Trace Context

Link logs to traces:

// app/Logging/OtelFormatter.php

namespace App\Logging;

use Monolog\Formatter\JsonFormatter;
use Monolog\LogRecord;
use OpenTelemetry\API\Trace\Span;

class OtelFormatter extends JsonFormatter
{
    public function format(LogRecord $record): string
    {
        $span = Span::getCurrent();
        $context = $span->getContext();

        $record->extra['trace_id'] = $context->getTraceId();
        $record->extra['span_id'] = $context->getSpanId();
        $record->extra['service'] = config('app.name');

        return parent::format($record);
    }
}

Now every log line includes trace_id — you can jump from a log entry directly to the full trace in Jaeger.

Tracing Queue Jobs

// app/Jobs/ProcessPost.php

use OpenTelemetry\API\Globals;
use OpenTelemetry\API\Trace\SpanKind;

class ProcessPost implements ShouldQueue
{
    public function handle(): void
    {
        $tracer = Globals::tracerProvider()->getTracer('queue');

        $span = $tracer->spanBuilder('ProcessPost')
            ->setSpanKind(SpanKind::KIND_CONSUMER)
            ->setAttribute('job.name', 'ProcessPost')
            ->setAttribute('post.id', $this->post->id)
            ->startSpan();

        $scope = $span->activate();

        try {
            // Process post...
            $this->post->generateSearchIndex();
            $this->post->clearCache();

            $span->setStatus(\OpenTelemetry\API\Trace\StatusCode::STATUS_OK);
        } catch (\Exception $e) {
            $span->recordException($e);
            $span->setStatus(\OpenTelemetry\API\Trace\StatusCode::STATUS_ERROR);
            throw $e;
        } finally {
            $span->end();
            $scope->detach();
        }
    }
}

Sending to Cloud Providers

Datadog

OTEL_EXPORTER_OTLP_ENDPOINT=https://trace.agent.datadoghq.com
OTEL_EXPORTER_OTLP_HEADERS=DD-API-KEY=your_api_key

Grafana Cloud

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64_encoded_credentials>

Dashboards & Alerts

With traces and metrics flowing to Grafana:

Dashboard 1: Request Overview
  - p50, p95, p99 latency
  - Request rate (rpm)
  - Error rate (%)
  - Top slow endpoints

Dashboard 2: Database Performance
  - Query count per request
  - Slow query detection
  - Connection pool usage

Dashboard 3: External Dependencies
  - API call latency
  - Cache hit/miss ratio
  - Queue depth & processing time

Alert: response_time_p99 > 2000ms for 5 minutes
Alert: error_rate > 5% for 2 minutes
Alert: queue_depth > 1000

Performance Impact

OpenTelemetry adds overhead. Strategies to minimize:

Sampling — don't trace every request in production:

// Trace 10% of requests (probabilistic)
putenv('OTEL_TRACES_SAMPLER=parentbased_traceidratio');
putenv('OTEL_TRACES_SAMPLER_ARG=0.1');

In-Practice guidance:

• Development: 100% sampling (trace everything)
• Staging: 50-100% sampling
• Production: 1-10% sampling (depends on traffic volume)
• Always trace error responses at 100%

Batching — export in batches, not per-span:

OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
OTEL_BSP_SCHEDULE_DELAY=5000

Conclusion

OpenTelemetry is more setup than Telescope or Pulse, but it gives you production-grade observability:

1. Traces show you exactly what happened in each request
2. Metrics show you trends and patterns
3. Logs with trace context let you drill into specifics
4. Vendor-agnostic — switch from Jaeger to Datadog without code changes

Start with auto-instrumentation (install the contrib packages), then add custom spans for your business logic. You'll wonder how you ever debugged without it.