Doron Segal Founder · CTO · YC W21 Book a call
TALK

The Hard Parts of White-Label Ordering on AWS

Technical lessons from Per Diem, where one platform runs the ordering apps of thousands of independent coffee shops and restaurants under their own brands. What breaks at that shape isn’t any single AWS service — it’s the tenant key, duplicate webhook deliveries, a read that lands on a replica a beat too early, a table nobody gave a lifecycle, and a mobile build from March calling an API from September.

15 slides 4 rules Aurora · SQS · Redshift AWS Conference Download the deck (PDF)
Use ← → to move · G for the grid
  1. Title slide: the hard parts of white-label ordering on AWS — technical lessons from Per Diem, presented by Doron Segal, CTO and co-founder, at an AWS conference. 1 The hard parts of white-label ordering on AWS
  2. Introduction: Doron Segal, CTO and co-founder of Per Diem, leading the engineering behind its white-label ordering platform — and a father of three, so concurrency, retries and unpredictable inputs are part of life at work and at home. 2 Hi, I’m Doron
  3. The restaurant technology spectrum, from more restaurant ownership to more marketplace aggregation: enterprise-owned apps built in-house by McDonald’s, Starbucks and Chick-fil-A; Per Diem in the middle, where independent and regional brands subscribe to branded mobile and web ordering and keep their direct customer relationship; and consumer marketplaces — DoorDash, Uber Eats, Wolt — that aggregate demand and delivery. Uber Direct supports delivery for orders placed through a restaurant’s own app or website. 3 Where white-label sits
  4. One platform behaves like many products. Consumer mobile apps, web ordering, the Per Diem website, the merchant dashboard and the merchant mobile app all share one API and one operational data store — the operational path. Analytics takes a separate decision path into an S3 data lake and Redshift for back-office aggregated decisions. 4 One platform, many products
  5. The Per Diem technical stack as three paths. Interactive traffic: mobile, web and dashboard hit the API on ECS Fargate, which reads and writes Aurora PostgreSQL and ElastiCache / Valkey. External events: Square, DoorDash, Uber and Stripe arrive through API Gateway behind a WAF, into Amazon SQS, then a webhook worker on ECS. Analytics: Firehose to S3 as Parquet to Amazon Redshift. 5 The stack, in three paths
  6. Tenant isolation is also the scaling strategy: the tenant key carries authorization, index locality, cache ownership and a future shard route. The unsafe shape selects from orders ordered by created_at descending with a limit of 50 — cross-tenant exposure and full-table pressure. The Per Diem shape adds where store_id equals the tenant and created_at is less than a keyset cursor, served by a composite index on store_id and created_at descending. 6 The tenant key is the scaling strategy
  7. At-least-once delivery moves correctness into the handler. A queue absorbs bursts, but it also makes duplicate delivery an application concern: event 8341 arrives from Amazon SQS, the handler verifies the signature, claims an idempotency key and applies the state change; a second delivery of event 8341 becomes a no-op. Design rule — acknowledge only after the durable state transition succeeds. 7 At-least-once means the handler owns correctness
  8. Read scaling creates a consistency decision. Aurora reader endpoints increase read capacity, but an immediate read can observe the previous state: creating an order writes through the writer endpoint, and the consistency-sensitive fetch that follows is routed to the writer too, while the default read path uses the reader endpoint. Per Diem routes read-after-write flows through the writer path; most reads still use replicas. 8 Read scaling is a consistency decision
  9. Transactional and analytical data need different paths. The operational database answers one merchant’s current request; Redshift answers historical questions across time. A canonical event goes through Firehose and a JQ partitioner into the S3 lake as typed Parquet, then a bulk COPY into Redshift for aggregation — partition keys reduce the scan surface and bulk COPY respects the columnar engine. Rule: dashboards never turn Aurora into a warehouse. 9 Two questions, two data paths
  10. Large tables need a lifecycle before they become large. Per Diem partitions append-only tables by created_at and treats archival as part of the design: create the table partitioned, keep recent monthly or yearly partitions, archive old partitions to S3, and run historical analytics in Redshift. Partition pruning keeps recent queries bounded, new partitions avoid the expensive conversion of a live table, and a dry-run archive path protects stateful data. 10 Give a big table a lifecycle first
  11. The API serves clients that cannot deploy together — a mobile release can stay active for months after the backend changes. A version-aware API returns a stable response to an older app build and additive behavior to the current one. Preference order: additive contract, then a server-side version gate, then a new endpoint version, and a forced upgrade only as a last resort. 11 Clients you cannot deploy with
  12. Infrastructure as code still needs a blast-radius gate. CDK makes infrastructure repeatable, but stateful replacement remains stateful replacement: a TypeScript CDK change goes through cdk synth and diff, then a review of replacement and data risk, then sandbox, then production. Cross-stack values travel through explicit CloudFormation exports, secrets stay in Secrets Manager, and production images pin a commit-specific ECR tag. 12 IaC still needs a blast-radius gate
  13. The remaining coupling is visible and testable — the most useful architecture review asks where two definitions can drift. One, event schema: the Glue payload struct and the analytics service’s canonical event type must change together. Two, stack contracts: service stacks depend on named CloudFormation exports, so renaming an export is an API change. Three, runtime policy: local Node versions and CodeBuild runtime versions drift unless CI validates them as one policy. 13 Where two definitions can drift
  14. What held up under real product constraints, as four rules: put the tenant key in every boundary; make retries safe before adding throughput; choose consistency per user flow; and treat schemas, clients and stacks as versioned contracts. 14 What held up
  15. Closing slide: questions or suggestions welcome — doron@tryperdiem.com, tryperdiem.com, Per Diem. 15 Thank you

The short version

  • The tenant key is the scaling strategy, not just the security check. store_id does four jobs at once in this platform — it authorizes the request, it gives the index its locality, it owns the cache entry, and it is the shard route you will need later. A query that sorts orders by created_at without it is both a cross-tenant leak and full-table pressure; the same query with WHERE store_id = ? and a keyset cursor rides a composite index on (store_id, created_at DESC).
  • A queue absorbs bursts and hands you a correctness problem. Square, DoorDash, Uber and Stripe events land in SQS, and at-least-once means the same event will arrive twice. Verify the signature, claim an idempotency key, then apply the state change — and acknowledge only after the durable transition succeeds, so the second delivery of event 8341 is a no-op rather than a second order.
  • Consistency is a per-flow decision, not a global setting. Aurora reader endpoints buy read capacity, and a fetch that immediately follows a write can still observe the previous state. Read-after-write flows go through the writer; everything else stays on the replicas.
  • Two questions need two data paths. The operational database answers one merchant’s current request; Redshift answers historical questions across time. Canonical events go through Firehose and a JQ partitioner into S3 as typed Parquet, then a bulk COPY into Redshift. Dashboards never turn Aurora into a warehouse.
  • Give a table its lifecycle before it is large. Append-only tables are partitioned by created_at from the start, old partitions archive to S3, and history lives in Redshift — because partitioning a live table later is the expensive version of this decision.

The clients are the other half of it. A mobile build stays in the wild for months after the backend moves, so the API has to serve versions that cannot deploy together: additive contract first, then a server-side version gate, then a new endpoint version, and a forced upgrade only when nothing else works. The same logic applies to infrastructure — CDK makes a change repeatable, but cdk synth and a diff exist so someone reviews replacement and data risk before production, and cross-stack values travel through explicit CloudFormation exports rather than by hand.

Which leaves the coupling that’s left. The most useful architecture review asks where two definitions can drift apart: the Glue payload struct and the analytics service’s canonical event type, a renamed CloudFormation export that is quietly an API change, and local Node versions against CodeBuild’s. Four rules held up under real product constraints — put the tenant key in every boundary, make retries safe before adding throughput, choose consistency per user flow, and treat schemas, clients and stacks as versioned contracts.

Want this delivered to your team, or a second pair of eyes on a multi-tenant platform of your own? Book a call.