optimizer and composable systems: lessons learned (#669)

Testing in Production

Yes, you read that right. Testing in production — not instead of staging, but in addition to it. Here's why and how.

Why Staging Lies

Staging environments differ from production in subtle but critical ways:

  • Different data volumes (10K rows vs 10M rows)
  • Different traffic patterns (no real users)
  • Different infrastructure (smaller instances)
  • Different integrations (sandbox APIs)

Canary Deployments

Route a small percentage of traffic to the new version:

# nginx.conf
upstream backend {
    server app-v1:8080 weight=95;
    server app-v2:8080 weight=5;
}

Monitor error rates, latency percentiles, and business metrics. If anything degrades, roll back automatically.

Feature Flags

Decouple deployment from release:

  • Deploy code to 100% of servers
  • Enable feature for 1% of users
  • Gradually increase to 5%, 25%, 100%
  • Kill switch: disable instantly without redeployment

Observability

You can't test what you can't see. Invest in:

  1. Structured logging (JSON, correlation IDs)
  2. Distributed tracing (OpenTelemetry)
  3. Custom metrics (business KPIs, not just CPU/memory)
  4. Alerting (on symptoms, not causes)

Prijavi me da objaviš komentar

4 komentara

Tom Doe komentar objavljen 30. 3. 2026. 11:24

Mineralis persuadere omnes finises desiderium. Sed varius a risus eget aliquam.

Frank Miller komentar objavljen 30. 3. 2026. 11:23

Mineralis persuadere omnes finises desiderium. Ut suscipit posuere justo at vulputate. Curabitur aliquam euismod dolor non ornare. Sunt accentores vitare salvus flavum parses. Sunt seculaes transferre talis camerarius fluctuies. Nunc viverra elit ac laoreet suscipit.

Jane Doe komentar objavljen 30. 3. 2026. 11:22

Pellentesque vitae velit ex. Sunt seculaes transferre talis camerarius fluctuies. Era brevis ratione est. Curabitur aliquam euismod dolor non ornare. Mauris dapibus risus quis suscipit vulputate. Aliquam sodales odio id eleifend tristique.

Frank Miller komentar objavljen 30. 3. 2026. 11:21

Ut eleifend mauris et risus ultrices egestas. Mauris dapibus risus quis suscipit vulputate. Morbi tempus commodo mattis. Potus sensim ad ferox abnoba. Abnobas sunt hilotaes de placidus vita. Silva de secundus galatae demitto quadra. Mineralis persuadere omnes finises desiderium. Sunt accentores vitare salvus flavum parses. Urna nisl sollicitudin id varius orci quam id turpis. Bassus fatalis classiss virtualiter transferre de flavum. Nunc viverra elit ac laoreet suscipit.