Play free

AWS · Domain

Data engineering

Lakes, pipelines, streams, data movement, governance, warehouses, data quality and troubleshooting. 6 worlds, 36 stages, about 14.5 hours to complete.

FeedsSAA-C03DEA-C01DVA-C02SAP-C02SCS-C03MLA-C02

Data World 1 · Associate

Lake Foundations

Shape, cut, catalog, register, tier and version a multi-cloud billing lake.

6 stages~2.5 h1 free

  1. 1Shape the FilesFreeSame charges, a hundredth of the bytes: pick the files that let Athena read only what a query needs.
  2. 2Cut by DatePartition by what the queries filter on, as fine as the commonest query and no finer.
  3. 3Find the SchemaA crawler describes what is there, one table per compatible set; it never joins what is different.
  4. 4Register the DayFiles in S3 are not rows until a partition points at them: register each day, or let Athena project it.
  5. 5Keep, Tier, ExpireEvery byte has a temperature and an expiry date. In a versioned bucket, so does every old copy.
  6. 6Open TablesRow-level changes, time travel and compaction need a table format with snapshots. Keep them long enough, and not too long.

Data World 2 · Associate

Pipeline Plumbing

Ingest, trigger, fence, bookmark, orchestrate and capture changes for the billing lake.

6 stages~2.5 h

  1. 1Right PipeEvery source has its pipe: exports deliver, DataSync copies, partners push, AppFlow reads apps, DMS captures changes, and disks beat a thin link.
  2. 2On ArrivalA trigger runs for every object it matches, once per event, one configuration per bucket: match the data, skip the rest, and batch what arrives every few minutes.
  3. 3Inside the FenceA Glue job in a VPC is network interfaces with private addresses: let them talk to each other, give them a path to S3, and a NAT gateway for what lives on the internet.
  4. 4Only the NewA bookmark remembers which files a job read; it cannot see a file rewritten in place or a row changed in a database: write so that reading twice is harmless.
  5. 5Conduct the RunConduct, do not just start: wait for the job, retry what is transient, and tell someone about what is not.
  6. 6Change CaptureChange data capture reads the database's log: keep the log on, in rows, and long enough; capture every operation; and convert the schema before you move the data.

Data World 3 · Associate

Streams in Motion

Size, share, recover and land a payment stream: shards, readers, poison records, buffers and queues.

6 stages~2.5 h

  1. 1Pick the StreamFive flows, five shapes: a log many read, a pipe into S3, Kafka as it is, a window, a queue.
  2. 2Hot ShardOne merchant, one key, one shard: spread the load and keep each card in order.
  3. 3Many ReadersFive readers share a shard's five reads a second; the sixth needs its own pipe.
  4. 4Poison PillOne bad payment, one stuck shard: split it out, give up on it, keep it, keep going.
  5. 5Land It in S3Buffers decide the files: Parquet the analysts can read, hours Athena can find, countries compliance can open.
  6. 6Queue ItRefunds wait in a queue: let the worker read it, set the bad one aside, pay each once, keep up.

Data World 4 · Associate

Lake Guardians

A governed FinOps lake: who sees which cells, and why.

6 stages~2.5 h

  1. 1Who Holds the KeysA grant that changes nothing: find who really decides what the analysts see.
  2. 2Columns and RowsSame table, three views: each principal gets its rows and columns, and not one cell more.
  3. 3Tag the LakeGrant once to a tag, and next week's table is already covered.
  4. 4Share AcrossThree ways out of the account: a shared table, an encrypted extract, a business catalog.
  5. 5Lock the DataOne new key, three broken jobs: find who actually calls KMS.
  6. 6Find the PIIFive tickets about personal data: read the findings, the trail and the timeline before you change anything.

Data World 5 · Associate

Warehouse and Scale

Load, lay out, query, share and price a retailer's warehouse, and pick the right store and engine for each job.

6 stages~2.5 h

  1. 1Pick the StoreFour access patterns, four stores: the warehouse, the key-value table, the vector index in Postgres, the durable cache.
  2. 2Load It RightSixteen slices want sixteen files: split, compress, list them in a manifest, and leave the storefront its reads.
  3. 3Keys to the JoinPut the rows where the join needs them and the dates where the scan can skip them.
  4. 4Query in PlaceQuery the lake, the operational database, the analysts' view and the stream without loading, copying or waiting.
  5. 5Share the WarehouseShare live data without copying it, to a workgroup, to another account, and to a role beside you: what each one may see is decided at a different gate.
  6. 6Right Engine, Right PriceFour workloads, one price list: the cheapest engine that finishes in time, with the least to run.

Data World 6 · Associate

Pipeline Detective

Gate, watch and investigate the billing lake: quality rules, alarms, and five desks of evidence.

6 stages~2.5 h

  1. 1Quality GateWrite the rule for the line that is wrong, quarantine the line, hold the night: a gate per grain.
  2. 2Watch the PipesA rule for the state, a timeout for the hang, an alarm for the lag, and logs you can read while it runs.
  3. 3The Bill DoubledRead the bill by service, then the queries, the objects and the trail: every extra dollar has a setting behind it.
  4. 4Rows Went MissingCount the rows per day, then read the crawler log, the job runs, the quality results and the trail: each gap has one cause.
  5. 5Stream Is LaggingFour tickets on one stream: read the metrics and the logs, then the shard, the sink, the record and the buffer.
  6. 6Who Touched the DataWho revoked, who deleted, who re-encrypted: the trail and the timeline say; and where audit's own questions should live.

Game types in this domain

How each one plays