AWS X-Ray

A distributed tracing service that follows a single request as it moves through every service it touches — API Gateway, Lambda, ECS, EC2, Elastic Beanstalk, downstream AWS services, and even on-premises systems — and stitches the result into one connected timeline. Where CloudWatch Logs tells you what your application printed and CloudTrail tells you which AWS API was called, X-Ray tells you where a specific request's time actually went and which hop broke it.

Observability — distributed tracing Sampled by default Works across Lambda / ECS / EC2 / Elastic Beanstalk
2000
Daemon UDP listener port
64 KB
Segment document size limit
1/sec
Default sampling reservoir
5%
Default sampling fixed rate

What X-Ray Actually Is

The Problem It Solves

Core Components

⚠️ The Recurring Exam Theme

Almost every X-Ray question tests one of three things: (1) can you route "which service is slow / which service is failing" to X-Ray's service map and trace timeline rather than to logs, (2) do you know that annotations are indexed/searchable and metadata is not — the single most commonly tested X-Ray fact, and (3) do you know how tracing actually gets turned on per compute platform (Lambda's Active Tracing toggle vs. the daemon on EC2/ECS/on-prem vs. built-in support in Elastic Beanstalk).

How a Trace Is Built — Request Walkthrough

1 — Client Request

A client calls POST /orders. No X-Ray trace header exists yet.

2 — Edge / Entry Point

Amazon API Gateway (with X-Ray tracing enabled on the stage) generates a new Root trace ID, opens its own segment, and returns/propagates the X-Amzn-Trace-Id header forward on the downstream call.

3 — Compute Layer

An AWS Lambda function (Active Tracing enabled) receives the propagated trace header and opens its own segment attributed to the same trace ID.

4 — Instrumentation (SDK)

Inside the function, the X-Ray SDK wraps the AWS SDK DynamoDB client. Every GetItem/PutItem call becomes a subsegment nested under the Lambda segment, recording its own start/end time and any fault.

5 — Downstream Response

DynamoDB responds. The subsegment closes and records its duration plus any throttle/fault status.

6 — Assembly

The X-Ray backend receives segments from both API Gateway and Lambda — all tagged with the same trace ID — and stitches them into a single trace.

7 — Visualization

The service map renders API Gateway → Lambda → DynamoDB as connected nodes; the trace timeline (waterfall view) shows exactly which hop consumed the most latency or produced the fault.

Service map — nodes/edges + health color Trace timeline — per-segment/subsegment waterfall Filter expressions — search by annotation

Final Summary

Must Memorize
  • Annotations = indexed/searchable; Metadata = never searchable
  • Trace = segments (+ subsegments) sharing one trace ID
  • Default sampling = 1 req/sec reservoir + 5% fixed rate
  • Lambda needs no daemon — Active Tracing is built in
  • EC2/ECS/on-prem need the X-Ray daemon (UDP port 2000)
Must Understand
  • Sampling rule priority/reservoir/fixed-rate mechanics
  • Trace header propagation across HTTP vs. async (SNS/SQS) boundaries
  • Fargate needs the daemon as a task sidecar container — no host to install on
  • X-Ray vs. CloudWatch Logs vs. CloudTrail — three different questions
Can De-prioritize
  • Exact SDK method/function names per language
  • Console navigation specifics
  • Historical/legacy public-preview details

Exam appearance probability: HIGH

Core Components — Deep Dive

The settings and mechanics inside this tab are where most X-Ray exam questions actually live — especially sampling rule behavior and the annotations/metadata split.

2.1 Trace Foundation
WhatThe full record of one request end-to-end
Identified byA single trace ID shared by every segment involved
2.2 Segment High exam relevance
WhatOne service/resource's contribution to a trace
ContainsName, ID, start/end time, HTTP data, error/fault/throttle flags, annotations, metadata
2.3 Subsegment High exam relevance
WhatA more granular timing block nested inside a segment
Typical useDownstream calls — SQL query, outbound HTTP call, AWS SDK call (DynamoDB, S3, SNS)
2.4 Sampling Rules — Mechanics Frequently tested
Default rule1 request/sec reservoir + 5% fixed rate of the remainder
ReservoirMinimum requests/sec always sampled, unconditionally
Fixed rate% of requests beyond the reservoir that get sampled
2.5 Annotations vs. Metadata THE classic exam trap

This single distinction shows up, in some form, on nearly every X-Ray exam encounter. Memorize it cold.

PropertyAnnotationsMetadata
Indexed / searchableYes — usable in console filter expressions, GetTraceSummaries, and X-Ray groupsNo — never indexed, never searchable
Value typesSimple scalars only: string, number, booleanAny JSON-serializable value — objects, arrays, nested data
Typical useA field you'll filter/group traces by — order ID, customer tier, feature flag, regionVerbose debug context — full request/response payloads, stack traces
Where it's visibleFilter expressions, trace summaries, groupsOnly inside the raw trace detail view
⚠️ The trap, stated plainly

"Search/filter traces by a custom business field" → annotation, always. If the requirement is "attach extra context for a human debugging this trace later" and searching isn't needed → metadata. Putting a searchable field into metadata by mistake means it will silently never show up in a filter expression.

2.6 Groups & Filter Expressions Medium
WhatA saved filter expression that scopes traces for dashboards, metrics, and Insights
Exampleservice("payments-api") { fault = true }
2.7 X-Ray Insights Advanced
WhatAutomatic anomaly detection on fault/error/latency for a group
OutputAn "insight" — timeline, probable root cause, contributing services — with no manual threshold configuration
2.8 Service Map — Reading It

AWS Exam Thinking

Requirement → Keywords → Expected Answer → why every distractor fails.

Identify which downstream service is causing latency in a microservices app
latency across servicesmicroservicesbottleneck
Expected Answer

AWS X-Ray service map + trace timeline

DistractorWhy it's wrong
CloudWatch Logs InsightsSearches free text; no built-in cross-service, per-request timeline unless you manually correlate request IDs
CloudTrailRecords management/API calls, not application request performance
VPC Flow LogsNetwork-layer IP traffic metadata, not application-level timing
Search traces by a custom business field (e.g. order ID, tenant, customer tier)
search/filter tracescustom field
Expected Answer

Record it as an X-Ray annotation

DistractorWhy it's wrong
MetadataNot indexed — cannot appear in a filter expression, ever
CloudWatch Logs lineNot part of the X-Ray trace; requires separate manual correlation
Custom subsegment nameRenames a timeline label, doesn't create a structured/searchable field
Trace a Lambda function with the least operational overhead
serverlessno daemon to manage
Expected Answer

Enable Active Tracing in the Lambda function configuration

DistractorWhy it's wrong
Manually deploy the X-Ray daemonLambda has built-in auto-instrumentation; no daemon needed or possible
Install the daemon on the underlying hostLambda has no accessible host
Use VPC Flow LogsNetwork metadata only, no relationship to application tracing
Reduce X-Ray cost on a high-traffic service without losing rare-error visibility
cost controlstill catch rare errors
Expected Answer

Custom sampling rule — appropriately sized reservoir + a low fixed rate

DistractorWhy it's wrong
Disable X-Ray entirelyLoses all tracing — doesn't satisfy "still catch rare errors"
Increase Lambda memoryNo relationship to X-Ray sampling
Rely on X-Ray Insights aloneInsights analyzes already-sampled data; it doesn't control the sampling rate
Enable tracing on ECS with the Fargate launch type
Fargateno accessible host
Expected Answer

Add the X-Ray daemon as a sidecar container in the task definition

DistractorWhy it's wrong
"X-Ray isn't supported on Fargate"False — the sidecar pattern is the standard, documented approach
Install the daemon on the hostFargate provides no accessible EC2 host
Assume it's automaticThere is no default/automatic daemon on Fargate — it must be explicitly added
A trace unexpectedly "restarts" across an SNS/SQS boundary
async messagingbroken trace continuity
Expected Answer

Trace header wasn't propagated by the producer — must be explicitly forwarded (e.g. as a message attribute) by an instrumented SDK

DistractorWhy it's wrong
Missing execution roleWould cause invocation/permission failures, not a silently new trace
Active Tracing is account-limited to one functionFabricated — no such limit exists
Daemon required for propagationPropagation is a header-passing concern, not a daemon function
Attribute an incident: which service failed, who changed the config, what error was logged
multi-tool incident investigation
Expected Answer

X-Ray for latency/fault location, CloudTrail for the config-change API call, CloudWatch Logs for the error text

DistractorWhy it's wrong
X-Ray for all threeOverclaims scope — no visibility into IAM API calls or free-text log content
CloudTrail for all threeNo concept of request latency or log text
Any single-service answerEach question maps to a fundamentally different service by design
Guarantee zero sampling loss — trace 100% of production requests
complete capturecompliance/audit-grade
Expected Answer

Custom sampling rule — reservoir ≥ peak volume, fixed rate 100%

DistractorWhy it's wrong
"X-Ray can't disable sampling"Sampling is fully configurable; it's a deliberate cost/completeness tradeoff, not a hard limit
X-Ray Insights captures 100% automaticallyInsights analyzes existing sampled data; it doesn't change the sampling rate
Switching SDK for daemon changes samplingThe daemon is a transport mechanism, not a sampling override

Integrations & Architecture Example

Amazon API Gateway Entry point
WhatEnable X-Ray tracing per stage
WhyGenerates the edge segment and originates/propagates the trace header downstream

REST APIs and HTTP APIs both support X-Ray tracing, but the configuration surface differs slightly — see 06-APIGateway-Study-Guide.html for the REST vs. HTTP API distinction in depth.

AWS Lambda Compute — auto-instrumented
WhatActive Tracing toggle in function configuration
WhyZero-daemon tracing — the Lambda service itself generates the segment
IAMExecution role needs xray:PutTraceSegments / xray:PutTelemetryRecords (commonly via AWSXRayDaemonWriteAccess)
Amazon ECS (EC2 and Fargate launch types) Daemon required
EC2 launch typeRun the X-Ray daemon as a container (or host process) alongside the app
Fargate launch typeAdd the daemon as a sidecar container in the same task definition — shares the task's network namespace so the app container reaches it on localhost:2000
Amazon EC2 Daemon required
WhatInstall and run the X-Ray daemon on the instance
IAMInstance profile role needs X-Ray write permissions
NetworkOnly local UDP 2000 traffic needed — no inbound security group rule required from outside the instance
AWS Elastic Beanstalk Built-in support
WhatEnable the "X-Ray daemon" option in environment configuration
WhyBeanstalk runs and manages the daemon on environment instances for you — no manual install

See 04-ElasticBeanstalk-Study-Guide.html for how this fits into a Beanstalk environment's overall configuration model.

Amazon CloudWatch (ServiceLens)
WhatCorrelates X-Ray traces with CloudWatch metrics, logs, and alarms in one view
WhyPivot from a service-map node or a metric spike directly into the relevant traces/logs for that resource
Amazon SNS / SQS — async trace propagation Trap
WhatTrace context propagation across a messaging boundary is not automatic the way it is for a direct HTTP call
Why it mattersAn instrumented SDK path must carry the trace header (e.g., as a message attribute); otherwise the downstream consumer starts a disconnected trace

End-to-End Architecture Example

Serverless order-processing API with full tracing

Client → API Gateway (HTTP API, X-Ray tracing enabled on the stage, Cognito User Pool authorizer validates the JWT) → Lambda (Active Tracing enabled, X-Ray SDK wraps the AWS SDK clients) → DynamoDB (order write, captured as a subsegment) and SNS (publishes an order-confirmed event, trace context forwarded via message attribute by the instrumented SDK) → a second Lambda (also Active Tracing enabled) subscribed to the topic, sending the confirmation email.

Best Practices & Common Exam Traps

When to Use X-Ray — and When Not To

Use X-Ray when...
  • A request crosses multiple services and you need to know which hop is slow or failing
  • You need a visual, aggregated service map of a distributed architecture
  • You need per-request, per-hop timing (a waterfall view), not just aggregate metrics
  • You need to search/filter traces by a business-meaningful field (via annotations)
Don't reach for X-Ray when...
  • You need free-text log search/alerting → CloudWatch Logs (Logs Insights) — X-Ray isn't a log store
  • You need an audit trail of who called which AWS API → AWS CloudTrail — X-Ray traces app-level requests, not IAM/management activity
  • You need infrastructure metrics (CPU, memory) → CloudWatch Metrics / Container Insights
  • You need guaranteed, unsampled capture of every request for legal retention → a dedicated logging/archival solution — X-Ray is sampled by default and not designed as a system of record

Best Practices

Must Know
  • Enable Active Tracing on every Lambda function in a traced path
  • Use annotations for anything you'll ever filter/group by
  • Keep metadata for verbose, non-searchable debug context only
  • Size the sampling reservoir to catch rare errors at low traffic
Good Practice
  • Instrument every outbound AWS SDK/HTTP client so subsegments are complete
  • Deploy the daemon as an ECS/Fargate sidecar, not a bolt-on afterthought
  • Use groups + filter expressions to scope dashboards and X-Ray Insights
  • Explicitly propagate the trace header across SNS/SQS boundaries
Advanced Practice
  • Enable X-Ray Insights on high-value groups for automatic anomaly detection
  • Correlate via CloudWatch ServiceLens for one unified metrics+traces+logs view
  • Trace on-premises components into the same distributed trace via the SDK + local daemon relay
  • When unnecessary: a single-service, no-fan-out application gets little value from X-Ray — CloudWatch Logs/metrics alone may be sufficient

Common Exam Traps

MisconceptionReality
"Annotations and metadata are basically interchangeable"Only annotations are indexed/searchable via filter expressions; metadata is never searchable, by design
"X-Ray traces 100% of requests automatically"Default sampling only guarantees 1 req/sec + 5% of the rest; full capture requires a deliberate custom rule
"Lambda tracing needs zero configuration"Active Tracing must be explicitly enabled, and the execution role needs X-Ray write permissions
"The X-Ray daemon runs automatically on ECS/Fargate"On EC2 launch type you must run it yourself; on Fargate it must be added as a sidecar container — there's no default agent
"X-Ray is a log aggregation/storage service"It's a distributed tracing/APM service; use CloudWatch Logs for free-text log search
"Trace headers propagate automatically across every AWS boundary"Automatic for instrumented direct HTTP calls; async boundaries like SNS/SQS require the producer to forward the trace context
"Sampling out a request costs nothing extra, so just sample everything"A higher fixed sampling rate directly increases trace-recording cost — sampling exists to trade completeness for cost control

X-Ray vs. CloudWatch Logs vs. CloudTrail

Since X-Ray doesn't have close functional lookalikes the way GuardDuty/Inspector/Macie do, the comparison that actually matters on this exam is which of these three observability services answers which question.

DimensionX-RayCloudWatch LogsCloudTrail
What it tracksOne request's path & timing across services (an app-level trace)Free-text log output your app/AWS service writesAPI calls made against AWS services (who/what/when at the control plane)
GranularitySegment/subsegment timeline for a single requestArbitrary log lines/eventsOne event per API call
Sampled?Yes, by default (sampling rules control %)No — every line you log is storedNo — every management-plane API call is recorded; data events are selectively enabled
Primary use caseLatency/error root cause across a distributed architectureDebugging via free-text/query search, alarms on log patternsSecurity/compliance audit — "who deleted this bucket"
Typical exam trigger phrase"identify which service in the chain is slow/failing""search/query application log output," "alarm on an error count in logs""who made this API call," "audit trail for compliance"

Memory Anchors

Hands-On Lab

Three self-contained exercises that turn the conceptual annotations/metadata and Active Tracing material above into something you actually click and run. All console steps assume the eu-west-1 (Ireland) region and an existing Lambda function — swap my-orders-function for your own function name throughout. Each CLI block is AWS CLI v2 syntax.

Exercise 1 — Enable Active Tracing and Read Your First Trace

⚠️ Why this matters for the exam

"Trace a Lambda function with the least operational overhead" is a recurring scenario stem, and the expected answer is always the Active Tracing toggle — never a manually installed daemon. This exercise makes the muscle memory permanent: one console toggle (or one CLI flag) plus one managed IAM policy, nothing else.

  1. In the Lambda console (region eu-west-1), open my-orders-functionConfiguration tab → Monitoring and operations tools → click Edit → toggle Active tracing to On → Save. The function's configuration now shows Active Tracing = Active; every new invocation generates an X-Ray segment automatically.
  2. Reproduce the same change from the CLI instead (useful to confirm what the console toggle actually does under the hood):
    aws lambda update-function-configuration \
      --region eu-west-1 \
      --function-name my-orders-function \
      --tracing-config Mode=Active
    "TracingConfig": { "Mode": "Active" } is echoed back in the CLI response.
  3. Attach the AWSXRayDaemonWriteAccess managed policy to the function's execution role so it's allowed to call xray:PutTraceSegments / xray:PutTelemetryRecords:
    aws iam attach-role-policy \
      --role-name my-orders-function-role \
      --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

    Find the exact role name first if you don't have it memorized: aws lambda get-function-configuration --region eu-west-1 --function-name my-orders-function --query Role --output text returns the role ARN; the role name is the last path segment.

    No error is returned; aws iam list-attached-role-policies --role-name my-orders-function-role now lists AWSXRayDaemonWriteAccess.
  4. Invoke the function a handful of times to generate sampled traces:
    for i in 1 2 3 4 5; do
      aws lambda invoke \
        --region eu-west-1 \
        --function-name my-orders-function \
        --payload '{"orderId":"ord-1001","customerTier":"standard"}' \
        --cli-binary-format raw-in-base64-out \
        out-$i.json
    done
    Five out-N.json files are written locally, each with "StatusCode": 200 in the invoke response.
  5. Open the X-Ray console → Traces (region eu-west-1) → the five invocations should appear within about a minute, each with a trace ID and a duration. Click one, then open Service map. The trace timeline shows a single segment named my-orders-function; the service map shows one green node for the function with no downstream edges yet, since no instrumented AWS SDK calls have been added.

Exercise 2 — Custom Subsegment + Annotation, Then Filter By It

⚠️ Why this matters for the exam

This is the annotations-vs-metadata distinction from the Components tab, done hands-on instead of memorized. If you record customer_tier as metadata by mistake, the filter expression in step 5 will silently return zero results — living proof of why the exam always routes "search/filter by a business field" to an annotation.

  1. Add the X-Ray SDK to your deployment package. Python example (requirements.txt):
    aws-xray-sdk>=2.14.0
    Node.js example (package.json):
    npm install aws-xray-sdk-core --save
    The SDK is bundled into the next deployment package/layer alongside your handler code.
  2. Instrument the handler — open a custom subsegment, add one annotation (searchable) and one metadata field (not searchable) so you can see the difference land in the console later.

    Python:

    from aws_xray_sdk.core import xray_recorder
    
    def handler(event, context):
        order_id = event.get("orderId", "unknown")
        customer_tier = event.get("customerTier", "standard")
    
        subsegment = xray_recorder.begin_subsegment("validate-order")
        subsegment.put_annotation("customer_tier", customer_tier)
        subsegment.put_metadata("raw_event", event)
        try:
            # ... order validation logic ...
            result = {"orderId": order_id, "status": "validated"}
        finally:
            xray_recorder.end_subsegment()
    
        return result

    Node.js equivalent:

    const AWSXRay = require('aws-xray-sdk-core');
    
    exports.handler = async (event) => {
      const orderId = event.orderId || 'unknown';
      const customerTier = event.customerTier || 'standard';
    
      const segment = AWSXRay.getSegment();
      const subsegment = segment.addNewSubsegment('validate-order');
      subsegment.addAnnotation('customer_tier', customerTier);
      subsegment.addMetadata('raw_event', event);
    
      try {
        // ... order validation logic ...
        return { orderId, status: 'validated' };
      } finally {
        subsegment.close();
      }
    };
    Handler code now opens/closes a named subsegment on every invocation and tags it with one annotation and one metadata field.
  3. Redeploy the updated code:
    aws lambda update-function-code \
      --region eu-west-1 \
      --function-name my-orders-function \
      --zip-file fileb://function.zip
    "LastUpdateStatus": "Successful" appears in the response once the update finishes propagating.
  4. Invoke again with a couple of different customerTier values so you have something to filter between:
    aws lambda invoke --region eu-west-1 --function-name my-orders-function \
      --payload '{"orderId":"ord-2001","customerTier":"enterprise"}' \
      --cli-binary-format raw-in-base64-out out-enterprise.json
    
    aws lambda invoke --region eu-west-1 --function-name my-orders-function \
      --payload '{"orderId":"ord-2002","customerTier":"standard"}' \
      --cli-binary-format raw-in-base64-out out-standard.json
    Both invocations return "StatusCode": 200, and each produces a new trace containing a validate-order subsegment.
  5. In the X-Ray console → Traces, use the filter expression:
    annotation.customer_tier = "enterprise"
    Only the trace from the enterprise invocation appears. Now try metadata.raw_event = "..." in the filter bar — the console rejects it or returns nothing, because metadata is never indexed. Opening the trace detail view directly (not the filter bar) is the only place raw_event is visible.

Exercise 3 — Read a Service Map for Root-Cause Analysis

⚠️ Why this matters for the exam

Several quiz-style questions above hinge on reading service map color, not just knowing the sampling/annotation trivia. This exercise deliberately breaks a downstream call so you see, first-hand, how a fault and a latency spike each render differently on the map — the visual pattern-matching the exam expects you to already have when a scenario describes symptoms instead of showing you a screenshot.

  1. Add an instrumented downstream call plus a deliberate artificial delay and a conditional failure. Python example, extending Exercise 2's handler:
    import time
    import random
    import boto3
    from aws_xray_sdk.core import patch_all
    
    patch_all()  # auto-wraps boto3 clients so calls become subsegments
    dynamodb = boto3.client("dynamodb", region_name="eu-west-1")
    
    def handler(event, context):
        order_id = event.get("orderId", "unknown")
    
        # Artificial delay to simulate a slow downstream dependency
        time.sleep(2.5)
    
        # Artificial fault ~50% of the time to populate the service map with red
        if random.random() < 0.5:
            raise Exception("Simulated downstream failure for order " + order_id)
    
        dynamodb.put_item(
            TableName="Orders",
            Item={"orderId": {"S": order_id}, "status": {"S": "validated"}}
        )
        return {"orderId": order_id, "status": "validated"}
    The function now sometimes succeeds slowly (2.5s+) and sometimes throws, in addition to whichever behavior it had before.
  2. Redeploy and invoke it 8-10 times in a loop so both outcomes occur:
    aws lambda update-function-code --region eu-west-1 \
      --function-name my-orders-function --zip-file fileb://function.zip
    
    for i in $(seq 1 10); do
      aws lambda invoke --region eu-west-1 --function-name my-orders-function \
        --payload "{\"orderId\":\"ord-30$i\"}" \
        --cli-binary-format raw-in-base64-out out-$i.json
      sleep 1
    done
    Roughly half the invocations return a normal 200 response with elevated duration; the rest return an unhandled function error.
  3. Open X-Ray console → Service map (region eu-west-1). The my-orders-function node is no longer solid green — it shows a red slice/ring proportional to the fault rate, and hovering it surfaces average latency alongside the fault percentage.
  4. Click the node → View traces, then open one of the failed traces and one of the slow-but-successful traces side by side. The failed trace's segment is flagged with a red "Fault" indicator and the exception message from the raised exception appears in the segment detail. The slow-but-successful trace shows a normal (non-fault) segment whose duration bar is visibly wider than a typical invocation, making the 2.5s artificial delay obvious in the waterfall view without reading any code.
  5. Clean up before moving on, so this exercise's artificial fault/delay doesn't linger in a real deployment — redeploy the Exercise 2 version of the handler (without the time.sleep and random.random block). Subsequent invocations return to fast, consistently successful responses, and the service map node gradually returns to solid green as new healthy traces accumulate.

Flashcards — 22 Cards

Click card to flip. Mark right or wrong to track score.

Click to reveal answer
1 / 22
Mark:   Score: 0/0

Practice Quiz — 13 Questions

DVA-C02 scenario style, Easy → Specialty. Select an answer to reveal the explanation.

out of 13 correct