A distributed tracing service that follows a single request as it moves through every service it touches — API Gateway, Lambda, ECS, EC2, Elastic Beanstalk, downstream AWS services, and even on-premises systems — and stitches the result into one connected timeline. Where CloudWatch Logs tells you what your application printed and CloudTrail tells you which AWS API was called, X-Ray tells you where a specific request's time actually went and which hop broke it.
Almost every X-Ray question tests one of three things: (1) can you route "which service is slow / which service is failing" to X-Ray's service map and trace timeline rather than to logs, (2) do you know that annotations are indexed/searchable and metadata is not — the single most commonly tested X-Ray fact, and (3) do you know how tracing actually gets turned on per compute platform (Lambda's Active Tracing toggle vs. the daemon on EC2/ECS/on-prem vs. built-in support in Elastic Beanstalk).
A client calls POST /orders. No X-Ray trace header exists yet.
Amazon API Gateway (with X-Ray tracing enabled on the stage) generates a new Root trace ID, opens its own segment, and returns/propagates the X-Amzn-Trace-Id header forward on the downstream call.
An AWS Lambda function (Active Tracing enabled) receives the propagated trace header and opens its own segment attributed to the same trace ID.
Inside the function, the X-Ray SDK wraps the AWS SDK DynamoDB client. Every GetItem/PutItem call becomes a subsegment nested under the Lambda segment, recording its own start/end time and any fault.
DynamoDB responds. The subsegment closes and records its duration plus any throttle/fault status.
The X-Ray backend receives segments from both API Gateway and Lambda — all tagged with the same trace ID — and stitches them into a single trace.
The service map renders API Gateway → Lambda → DynamoDB as connected nodes; the trace timeline (waterfall view) shows exactly which hop consumed the most latency or produced the fault.
Exam appearance probability: HIGH
The settings and mechanics inside this tab are where most X-Ray exam questions actually live — especially sampling rule behavior and the annotations/metadata split.
X-Amzn-Trace-Id HTTP header, format: Root=1-5e1b4151-5ac6c58dc39a1b9b4f7c9b7c;Parent=53995c3f42cd8ad8;Sampled=1 — Root (trace ID), Parent (the calling segment's ID, if any), Sampled (1 = record, 0 = don't).This single distinction shows up, in some form, on nearly every X-Ray exam encounter. Memorize it cold.
| Property | Annotations | Metadata |
|---|---|---|
| Indexed / searchable | Yes — usable in console filter expressions, GetTraceSummaries, and X-Ray groups | No — never indexed, never searchable |
| Value types | Simple scalars only: string, number, boolean | Any JSON-serializable value — objects, arrays, nested data |
| Typical use | A field you'll filter/group traces by — order ID, customer tier, feature flag, region | Verbose debug context — full request/response payloads, stack traces |
| Where it's visible | Filter expressions, trace summaries, groups | Only inside the raw trace detail view |
"Search/filter traces by a custom business field" → annotation, always. If the requirement is "attach extra context for a human debugging this trace later" and searching isn't needed → metadata. Putting a searchable field into metadata by mistake means it will silently never show up in a filter expression.
service("payments-api") { fault = true }Requirement → Keywords → Expected Answer → why every distractor fails.
AWS X-Ray service map + trace timeline
| Distractor | Why it's wrong |
|---|---|
CloudWatch Logs Insights | Searches free text; no built-in cross-service, per-request timeline unless you manually correlate request IDs |
CloudTrail | Records management/API calls, not application request performance |
VPC Flow Logs | Network-layer IP traffic metadata, not application-level timing |
Record it as an X-Ray annotation
| Distractor | Why it's wrong |
|---|---|
| Metadata | Not indexed — cannot appear in a filter expression, ever |
| CloudWatch Logs line | Not part of the X-Ray trace; requires separate manual correlation |
| Custom subsegment name | Renames a timeline label, doesn't create a structured/searchable field |
Enable Active Tracing in the Lambda function configuration
| Distractor | Why it's wrong |
|---|---|
| Manually deploy the X-Ray daemon | Lambda has built-in auto-instrumentation; no daemon needed or possible |
| Install the daemon on the underlying host | Lambda has no accessible host |
| Use VPC Flow Logs | Network metadata only, no relationship to application tracing |
Custom sampling rule — appropriately sized reservoir + a low fixed rate
| Distractor | Why it's wrong |
|---|---|
| Disable X-Ray entirely | Loses all tracing — doesn't satisfy "still catch rare errors" |
| Increase Lambda memory | No relationship to X-Ray sampling |
| Rely on X-Ray Insights alone | Insights analyzes already-sampled data; it doesn't control the sampling rate |
Add the X-Ray daemon as a sidecar container in the task definition
| Distractor | Why it's wrong |
|---|---|
| "X-Ray isn't supported on Fargate" | False — the sidecar pattern is the standard, documented approach |
| Install the daemon on the host | Fargate provides no accessible EC2 host |
| Assume it's automatic | There is no default/automatic daemon on Fargate — it must be explicitly added |
Trace header wasn't propagated by the producer — must be explicitly forwarded (e.g. as a message attribute) by an instrumented SDK
| Distractor | Why it's wrong |
|---|---|
| Missing execution role | Would cause invocation/permission failures, not a silently new trace |
| Active Tracing is account-limited to one function | Fabricated — no such limit exists |
| Daemon required for propagation | Propagation is a header-passing concern, not a daemon function |
X-Ray for latency/fault location, CloudTrail for the config-change API call, CloudWatch Logs for the error text
| Distractor | Why it's wrong |
|---|---|
| X-Ray for all three | Overclaims scope — no visibility into IAM API calls or free-text log content |
| CloudTrail for all three | No concept of request latency or log text |
| Any single-service answer | Each question maps to a fundamentally different service by design |
Custom sampling rule — reservoir ≥ peak volume, fixed rate 100%
| Distractor | Why it's wrong |
|---|---|
| "X-Ray can't disable sampling" | Sampling is fully configurable; it's a deliberate cost/completeness tradeoff, not a hard limit |
| X-Ray Insights captures 100% automatically | Insights analyzes existing sampled data; it doesn't change the sampling rate |
| Switching SDK for daemon changes sampling | The daemon is a transport mechanism, not a sampling override |
REST APIs and HTTP APIs both support X-Ray tracing, but the configuration surface differs slightly — see 06-APIGateway-Study-Guide.html for the REST vs. HTTP API distinction in depth.
xray:PutTraceSegments / xray:PutTelemetryRecords (commonly via AWSXRayDaemonWriteAccess)See 04-ElasticBeanstalk-Study-Guide.html for how this fits into a Beanstalk environment's overall configuration model.
Client → API Gateway (HTTP API, X-Ray tracing enabled on the stage, Cognito User Pool authorizer validates the JWT) → Lambda (Active Tracing enabled, X-Ray SDK wraps the AWS SDK clients) → DynamoDB (order write, captured as a subsegment) and SNS (publishes an order-confirmed event, trace context forwarded via message attribute by the instrumented SDK) → a second Lambda (also Active Tracing enabled) subscribed to the topic, sending the confirmation email.
order_id, customer_tier) let the team filter directly to a specific customer's trace during a support investigation.| Misconception | Reality |
|---|---|
| "Annotations and metadata are basically interchangeable" | Only annotations are indexed/searchable via filter expressions; metadata is never searchable, by design |
| "X-Ray traces 100% of requests automatically" | Default sampling only guarantees 1 req/sec + 5% of the rest; full capture requires a deliberate custom rule |
| "Lambda tracing needs zero configuration" | Active Tracing must be explicitly enabled, and the execution role needs X-Ray write permissions |
| "The X-Ray daemon runs automatically on ECS/Fargate" | On EC2 launch type you must run it yourself; on Fargate it must be added as a sidecar container — there's no default agent |
| "X-Ray is a log aggregation/storage service" | It's a distributed tracing/APM service; use CloudWatch Logs for free-text log search |
| "Trace headers propagate automatically across every AWS boundary" | Automatic for instrumented direct HTTP calls; async boundaries like SNS/SQS require the producer to forward the trace context |
| "Sampling out a request costs nothing extra, so just sample everything" | A higher fixed sampling rate directly increases trace-recording cost — sampling exists to trade completeness for cost control |
Since X-Ray doesn't have close functional lookalikes the way GuardDuty/Inspector/Macie do, the comparison that actually matters on this exam is which of these three observability services answers which question.
| Dimension | X-Ray | CloudWatch Logs | CloudTrail |
|---|---|---|---|
| What it tracks | One request's path & timing across services (an app-level trace) | Free-text log output your app/AWS service writes | API calls made against AWS services (who/what/when at the control plane) |
| Granularity | Segment/subsegment timeline for a single request | Arbitrary log lines/events | One event per API call |
| Sampled? | Yes, by default (sampling rules control %) | No — every line you log is stored | No — every management-plane API call is recorded; data events are selectively enabled |
| Primary use case | Latency/error root cause across a distributed architecture | Debugging via free-text/query search, alarms on log patterns | Security/compliance audit — "who deleted this bucket" |
| Typical exam trigger phrase | "identify which service in the chain is slow/failing" | "search/query application log output," "alarm on an error count in logs" | "who made this API call," "audit trail for compliance" |
Three self-contained exercises that turn the conceptual annotations/metadata and Active Tracing material above into something you actually click and run. All console steps assume the eu-west-1 (Ireland) region and an existing Lambda function — swap my-orders-function for your own function name throughout. Each CLI block is AWS CLI v2 syntax.
"Trace a Lambda function with the least operational overhead" is a recurring scenario stem, and the expected answer is always the Active Tracing toggle — never a manually installed daemon. This exercise makes the muscle memory permanent: one console toggle (or one CLI flag) plus one managed IAM policy, nothing else.
my-orders-function → Configuration tab → Monitoring and operations tools → click Edit → toggle Active tracing to On → Save.
The function's configuration now shows Active Tracing = Active; every new invocation generates an X-Ray segment automatically.
aws lambda update-function-configuration \ --region eu-west-1 \ --function-name my-orders-function \ --tracing-config Mode=Active
"TracingConfig": { "Mode": "Active" } is echoed back in the CLI response.
AWSXRayDaemonWriteAccess managed policy to the function's execution role so it's allowed to call xray:PutTraceSegments / xray:PutTelemetryRecords:
aws iam attach-role-policy \ --role-name my-orders-function-role \ --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
Find the exact role name first if you don't have it memorized: aws lambda get-function-configuration --region eu-west-1 --function-name my-orders-function --query Role --output text returns the role ARN; the role name is the last path segment.
aws iam list-attached-role-policies --role-name my-orders-function-role now lists AWSXRayDaemonWriteAccess.
for i in 1 2 3 4 5; do
aws lambda invoke \
--region eu-west-1 \
--function-name my-orders-function \
--payload '{"orderId":"ord-1001","customerTier":"standard"}' \
--cli-binary-format raw-in-base64-out \
out-$i.json
done
Five out-N.json files are written locally, each with "StatusCode": 200 in the invoke response.
my-orders-function; the service map shows one green node for the function with no downstream edges yet, since no instrumented AWS SDK calls have been added.
This is the annotations-vs-metadata distinction from the Components tab, done hands-on instead of memorized. If you record customer_tier as metadata by mistake, the filter expression in step 5 will silently return zero results — living proof of why the exam always routes "search/filter by a business field" to an annotation.
requirements.txt):
aws-xray-sdk>=2.14.0Node.js example (
package.json):
npm install aws-xray-sdk-core --saveThe SDK is bundled into the next deployment package/layer alongside your handler code.
Python:
from aws_xray_sdk.core import xray_recorder
def handler(event, context):
order_id = event.get("orderId", "unknown")
customer_tier = event.get("customerTier", "standard")
subsegment = xray_recorder.begin_subsegment("validate-order")
subsegment.put_annotation("customer_tier", customer_tier)
subsegment.put_metadata("raw_event", event)
try:
# ... order validation logic ...
result = {"orderId": order_id, "status": "validated"}
finally:
xray_recorder.end_subsegment()
return result
Node.js equivalent:
const AWSXRay = require('aws-xray-sdk-core');
exports.handler = async (event) => {
const orderId = event.orderId || 'unknown';
const customerTier = event.customerTier || 'standard';
const segment = AWSXRay.getSegment();
const subsegment = segment.addNewSubsegment('validate-order');
subsegment.addAnnotation('customer_tier', customerTier);
subsegment.addMetadata('raw_event', event);
try {
// ... order validation logic ...
return { orderId, status: 'validated' };
} finally {
subsegment.close();
}
};
Handler code now opens/closes a named subsegment on every invocation and tags it with one annotation and one metadata field.
aws lambda update-function-code \ --region eu-west-1 \ --function-name my-orders-function \ --zip-file fileb://function.zip
"LastUpdateStatus": "Successful" appears in the response once the update finishes propagating.
customerTier values so you have something to filter between:
aws lambda invoke --region eu-west-1 --function-name my-orders-function \
--payload '{"orderId":"ord-2001","customerTier":"enterprise"}' \
--cli-binary-format raw-in-base64-out out-enterprise.json
aws lambda invoke --region eu-west-1 --function-name my-orders-function \
--payload '{"orderId":"ord-2002","customerTier":"standard"}' \
--cli-binary-format raw-in-base64-out out-standard.json
Both invocations return "StatusCode": 200, and each produces a new trace containing a validate-order subsegment.
annotation.customer_tier = "enterprise"Only the trace from the
enterprise invocation appears. Now try metadata.raw_event = "..." in the filter bar — the console rejects it or returns nothing, because metadata is never indexed. Opening the trace detail view directly (not the filter bar) is the only place raw_event is visible.
Several quiz-style questions above hinge on reading service map color, not just knowing the sampling/annotation trivia. This exercise deliberately breaks a downstream call so you see, first-hand, how a fault and a latency spike each render differently on the map — the visual pattern-matching the exam expects you to already have when a scenario describes symptoms instead of showing you a screenshot.
import time
import random
import boto3
from aws_xray_sdk.core import patch_all
patch_all() # auto-wraps boto3 clients so calls become subsegments
dynamodb = boto3.client("dynamodb", region_name="eu-west-1")
def handler(event, context):
order_id = event.get("orderId", "unknown")
# Artificial delay to simulate a slow downstream dependency
time.sleep(2.5)
# Artificial fault ~50% of the time to populate the service map with red
if random.random() < 0.5:
raise Exception("Simulated downstream failure for order " + order_id)
dynamodb.put_item(
TableName="Orders",
Item={"orderId": {"S": order_id}, "status": {"S": "validated"}}
)
return {"orderId": order_id, "status": "validated"}
The function now sometimes succeeds slowly (2.5s+) and sometimes throws, in addition to whichever behavior it had before.
aws lambda update-function-code --region eu-west-1 \
--function-name my-orders-function --zip-file fileb://function.zip
for i in $(seq 1 10); do
aws lambda invoke --region eu-west-1 --function-name my-orders-function \
--payload "{\"orderId\":\"ord-30$i\"}" \
--cli-binary-format raw-in-base64-out out-$i.json
sleep 1
done
Roughly half the invocations return a normal 200 response with elevated duration; the rest return an unhandled function error.
my-orders-function node is no longer solid green — it shows a red slice/ring proportional to the fault rate, and hovering it surfaces average latency alongside the fault percentage.
time.sleep and random.random block).
Subsequent invocations return to fast, consistently successful responses, and the service map node gradually returns to solid green as new healthy traces accumulate.
Click card to flip. Mark right or wrong to track score.
DVA-C02 scenario style, Easy → Specialty. Select an answer to reveal the explanation.