Observability

Metrics and the Four Golden Signals

Use metrics and the four golden signals to quantify service demand, failures, responsiveness, and capacity pressure without confusing aggregates with user experience.

Intermediate14 min read
Observability lessonDelivery and reliability foundationsLearn

Use metrics and the four golden signals to quantify service demand, failures, responsiveness, and capacity pressure without confusing aggregates with user experience.

What you will be able to do

  • Distinguish traffic rate from error ratio in a realistic metrics and the four golden signals case.
  • Interpret the delivery evidence and boundary associated with latency distribution.
  • Choose an appropriate action involving saturation level without exceeding the named operational scope.
  • Verify golden-signal view through an observable service result and reproducible handoff.

01

Frame Metrics and the Four Golden Signals

Use metrics and the four golden signals to quantify service demand, failures, responsiveness, and capacity pressure without confusing aggregates with user experience.

A search service slows during a product launch. Its CPU average looks normal, so the team must examine traffic, errors, latency distribution, and saturation together.

Keep the delivery target, declared intent, execution evidence, reliability boundary, and recovery choice separate. Start with observable state and preserve enough context for another operator to reproduce the decision.

02

Traffic rate

Traffic measures the demand placed on a service, commonly as requests or transactions per interval. Within metrics and the four golden signals, this role answers a separate delivery or reliability question and keeps its own evidence.

Compare the launch request rate with the established baseline. Apply that action to the named service case before widening the rollout, infrastructure scope, or incident response.

Respect this boundary: do not interpret raw cumulative counts without a time window. The observable result is specific: the team identifies the size and timing of the demand increase.

03

Error ratio

Errors measure requests that fail explicitly, implicitly, or by policy relative to total demand. Within metrics and the four golden signals, this role answers a separate delivery or reliability question and keeps its own evidence.

Calculate the failed-search ratio for the same route and interval. Apply that action to the named service case before widening the rollout, infrastructure scope, or incident response.

Respect this boundary: do not mix client cancellations with server faults without labeling them. The observable result is specific: the metric shows which share of searches failed.

04

Latency distribution

Latency describes how long service work takes and is best interpreted as a distribution or percentile. Within metrics and the four golden signals, this role answers a separate delivery or reliability question and keeps its own evidence.

Inspect high-percentile search latency beside the median. Apply that action to the named service case before widening the rollout, infrastructure scope, or incident response.

Respect this boundary: do not let an average hide a slow tail affecting real users. The observable result is specific: the distribution exposes the delayed fraction of requests.

05

Saturation level

Saturation measures how constrained a service resource is, often through queueing or utilization near a limit. Within metrics and the four golden signals, this role answers a separate delivery or reliability question and keeps its own evidence.

Measure worker queue depth and connection-pool occupancy during the spike. Apply that action to the named service case before widening the rollout, infrastructure scope, or incident response.

Respect this boundary: do not use cpu alone when another resource limits throughput. The observable result is specific: queue pressure identifies the constrained search resource.

06

Golden-signal view

The four golden signals form a connected view of traffic, errors, latency, and saturation for a service. Within metrics and the four golden signals, this role answers a separate delivery or reliability question and keeps its own evidence.

Align all four signals for the same service boundary and time window. Apply that action to the named service case before widening the rollout, infrastructure scope, or incident response.

Respect this boundary: do not compare differently filtered panels and call them one incident view. The observable result is specific: the dashboard connects user slowdown with demand and capacity evidence.

07

Apply Metrics and the Four Golden Signals to One Service Change

Use one bounded delivery decision: A search service slows during a product launch. Its CPU average looks normal, so the team must examine traffic, errors, latency distribution, and saturation together.

First, compare the launch request rate with the established baseline. Then, calculate the failed-search ratio for the same route and interval. Keep both observations attached to the exact revision, environment, or service window.

Next, inspect high-percentile search latency beside the median. After that, measure worker queue depth and connection-pool occupancy during the spike. Close the work only after you align all four signals for the same service boundary and time window.

08

Recap Before Practice and Prove

Traffic rate: Traffic measures the demand placed on a service, commonly as requests or transactions per interval. In this service case, compare the launch request rate with the established baseline. Preserve the boundary: do not interpret raw cumulative counts without a time window.

Error ratio: Errors measure requests that fail explicitly, implicitly, or by policy relative to total demand. In this service case, calculate the failed-search ratio for the same route and interval. Preserve the boundary: do not mix client cancellations with server faults without labeling them.

Latency distribution: Latency describes how long service work takes and is best interpreted as a distribution or percentile. In this service case, inspect high-percentile search latency beside the median. Preserve the boundary: do not let an average hide a slow tail affecting real users.

Saturation level: Saturation measures how constrained a service resource is, often through queueing or utilization near a limit. In this service case, measure worker queue depth and connection-pool occupancy during the spike. Preserve the boundary: do not use cpu alone when another resource limits throughput.

Golden-signal view: The four golden signals form a connected view of traffic, errors, latency, and saturation for a service. In this service case, align all four signals for the same service boundary and time window. Preserve the boundary: do not compare differently filtered panels and call them one incident view.

NEXT STEP

Turn reading into recall

Practice the concepts without a timer, with coaching and retry available after every answer.

Open guided practice