Newsletter #122
Enjoy Newsletter #122.
The worst take in tech: new grad hiring
Scott Kennedy pushes back hard against the spreading narrative that companies should stop hiring new grads because AI can automate entry-level work. According to that argument, junior engineers historically learned the craft by doing simple tasks; now that AI handles those tasks, juniors won’t develop engineering judgment while budgets get spent on tokens instead. Kennedy calls this dead wrong: over the past year, he has seen the strongest new-grad talent pool in more than a decade of recruiting, and Replit doubled its new-grad hiring target to capitalize on it. Driven, curious, high-slope graduates can outperform many experienced hires within a couple of years when given strong mentorship and hard problems; writing them off because of AI is short-sighted.
He also points out a distinct advantage of “AI-native” graduates: having grown up alongside modern AI tools, they have no legacy habits to unlearn and use AI coding workflows with natural fluency—something he sees repeatedly in interviews. Even so, the formula for succeeding as a junior engineer hasn’t changed: curiosity to dig beneath the surface and debug deeply, resilience to learn from failed attempts, humility to ask for help when stuck, and seeking out high-challenge environments. AI changes how the work gets done, but it doesn’t diminish the value of learning speed, grit, and judgment.
Parsing Gigabytes of JSON per Second
This paper introduces simdjson, the first standard-compliant JSON parser capable of parsing gigabytes of JSON per second on a single core of a commodity CPU. The motivation is straightforward: JSON is ubiquitous across the web and distributed systems, and when applications ingest massive volumes of JSON documents, parsing easily becomes a primary CPU bottleneck despite decades of parser research.
What sets simdjson apart from prior validating parsers is its pervasive use of SIMD vector instructions, allowing the processor to inspect and classify dozens of bytes in parallel within a single instruction. As a result, simdjson executes roughly one-quarter or fewer CPU instructions than RapidJSON—the fastest traditional parser benchmarked—while still performing full UTF-8 and structural validation. The authors released the library under a permissive open-source license, making the results easy to reproduce and enabling widespread adoption across the industry.
Six SQL patterns I use to catch transaction fraud
This post presents six practical SQL patterns as a first line of defense for catching anomalous transaction behavior across credit cards, e-commerce, and healthcare billing. The velocity pattern groups transactions into rolling time windows to spot rapid bursts of activity from a single cardholder; combining multiple window sizes catches both automated card-testing scripts firing in seconds and slower fraud sprees spread over hours. The impossible-travel pattern uses LAG window functions paired with geographic distance formulas to flag a card used in two distant cities within an unrealistic timeframe. The suspicious-amount pattern hunts for small round test charges or transactions clustered just below reporting thresholds, while the merchant-anomaly pattern compares each merchant’s activity against its own historical baseline to spot compromised terminals.
The off-hours pattern compares transaction timestamps against a cardholder’s normal spending hours, making it reliable only once an account has built up sufficient history. Finally, a feature-engineering pattern pre-computes derived window columns—such as seconds since previous transaction or trailing 24-hour spend—so combining multiple fraud signals becomes a simple WHERE filter. No single rule is accurate enough in isolation, whereas a transaction tripping three or four patterns at once is almost certainly fraudulent. In production, queries still need careful NULL handling, human review for false positives, and date-range partitioning before running window functions to keep warehouse costs in check.
We replaced Redis with MySQL for inventory reservations—and it scaled (2026)
Shopify migrated its flash-sale inventory reservation system from Redis to MySQL so that temporary checkout reservations and the canonical inventory ledger live inside the exact same ACID transaction. Updating two separate stores without atomicity risks overselling or leaked reservations when failures happen mid-flight. Instead of storing a single quantity counter row per SKU—which would serialize every concurrent buyer behind a single row lock—the new schema represents each individual unit of inventory as its own row, capped at a pool of 1,000 rows per SKU-location pair so tables stay compact while queries remain fast even for massive stock counts. MySQL 8’s SELECT ... FOR UPDATE SKIP LOCKED lets concurrent checkouts skip rows currently locked by other buyers instead of waiting in line, drastically cutting lock contention during flash sales. A composite primary key minimizes per-row lock overhead, READ COMMITTED isolation prevents gap locks from blocking concurrent inventory restocks, and deterministic lock ordering prevents deadlocks.
During rollout, however, the real bottleneck turned out to be connection pool exhaustion: database CPU sat low while requests queued up waiting for connections. Tagging queries with business workflow metadata and measuring at the proxy layer revealed unrelated checkout operations holding database connections open far too long. After cleaning up connection lifetimes and tuning configs, primary database reads dropped by 50%, total transactions fell by 33%, and writer CPU stayed below 50% even during peak flash sales. To migrate safely, Shopify dual-wrote to both Redis and MySQL while Redis remained primary, shadowed and compared results against live traffic, and shifted primary reads incrementally by cohort with instant rollback ready at every step.
Cell-Based Architecture for Resilient Payment Systems
American Express structures its core payment processing platform into self-contained cells, where each cell bundles its own services, databases, and supporting infrastructure. A cell is defined as a fault-isolation boundary rather than a specific deployment technology: an entire cell can fail, undergo maintenance, or be drained of traffic without impacting neighboring cells. Slowly changing reference data like currency exchange rates is replicated into every cell ahead of time to avoid synchronous cross-cell lookups, while stateful transactions are routed deterministically to the cell holding the authoritative state. A Global Transaction Router governs all ingress traffic and strictly forbids services inside one cell from calling directly into another; while this sometimes requires duplicating services across cells, it preserves hard fault isolation and keeps latency low.
When a cell fails mid-transaction, in-flight requests aren’t resumed halfway through on another cell—they are replayed from the beginning on a healthy cell using the original request payload. That retry is safe because a globally unique transaction ID enforces end-to-end idempotency, allowing downstream systems to detect and deduplicate repeated work. Traffic shifts across cells gradually by percentage rather than flipping all at once. The platform also minimizes non-essential dependencies on the critical path by making audit logging and config refreshes asynchronous, gracefully degrading auxiliary features so payment authorization never stalls. True resilience comes from strict blast-radius boundaries and architectural discipline, not just monitoring and retries.
When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes
During a simulated availability-zone failure game day, Datadog discovered that their high-availability PostgreSQL clusters on Kubernetes could not fail over safely. When network latency spiked during the drill, the primary kept accepting writes and replicating asynchronously while standby replicas fell further and further behind. When the primary was terminated, Patroni rightly refused to promote any standby because their replication lag exceeded maximum_lag_on_failover—preventing silent data loss, but leaving the cluster wedged and unable to accept writes until human operators intervened. The first lesson: nominal high availability means nothing if no replica is actually safe to promote when disaster strikes.
Datadog moved to a hybrid replication topology: standby instances eligible for failover use synchronous replication, while read-only replicas stay asynchronous to avoid unnecessary latency penalties. In Patroni, they enabled synchronous_mode, synchronous_mode_strict, and synchronous_commit = remote_apply, guaranteeing a transaction is acknowledged only after its WAL records have been replayed and applied on the synchronous standby. Synthetic benchmarks at this highest durability setting showed average write latency increasing by 53% and throughput dropping by 34%, but rolling the change out gradually across datacenters with soak periods confirmed that real-world application impact remained well within acceptable bounds. If all synchronous standbys become unavailable, the primary halts writes rather than silently risking data loss, turning an invisible durability hazard into an explicit error that callers can retry or queue. Balancing durability and performance requires actively monitoring SyncRep wait events alongside standby health.
From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective
On Christmas Day 2024—the biggest traffic day of the year for digital photo frames—Aura Frames suffered a three-hour PostgreSQL outage when unbounded WAL growth filled their dedicated WAL volume. The culprit was an RDS PostgreSQL 14.1 default that attached replication slots to read replicas: a replication slot forces the primary to retain WAL segments until every replica has consumed them, so when a replica fell behind under holiday load, WAL piled up until the primary ran out of disk space and crashed. The team immediately capped slot retention using max_slot_wal_keep_size (noting that wal_keep_size sets a minimum floor, not a ceiling), choosing to invalidate a lagging replica rather than let the primary go down.
To prepare for Christmas 2025, instead of vertically scaling a single massive primary again, Aura partitioned at the table level (functional sharding): the ten highest-write tables were split out onto seven new primaries, creating eight databases in total. This proved a pragmatic middle ground before needing horizontal row sharding on their largest table. The Ruby on Rails monolith stayed intact, routing queries cleanly via Active Record Multiple Databases, while physical replication moved data over with just 5–10 minutes of cutover downtime. Total provisioned CPU and memory increased roughly 4.7x, and on Christmas Day 2025 the system peaked at 226,000 transactions per second—sustaining over 100,000 TPS for ten straight hours at an average query latency of 25 microseconds. Continuous testing, canary rollouts, and load testing de-risked the data-access changes; temporarily over-provisioning for a predictable annual peak made complete sense, and AWS Blue/Green deployments let them scale the fleet back down afterward to restore cost efficiency.
.gitignore Isn’t the Only Way To Ignore Files in Git
Git supports three distinct scopes for ignoring files, each suited to a different job. .gitignore is committed into the repository, making it the right place for project-wide rules shared by every contributor, like build outputs and dependency folders. .git/info/exclude lives inside your local .git directory and is never tracked or pushed, applying only to your local clone—ideal for personal scratch files, local scripts, or project-specific tool configs that don’t belong in the team’s shared .gitignore. Finally, ~/.config/git/ignore acts as your user-wide global ignore file across every repository on your machine, perfect for OS- or editor-generated noise like macOS .DS_Store files or editor swap files.
You can customize the location of the global ignore file via git config --global core.excludesFile ~/.gitignore_global and reset it to the default with git config --global --unset core.excludesFile. And whenever you aren’t sure which rule across those three layers is causing a file to be ignored, git check-ignore -v <path> is the fastest way to debug it: the command prints the exact ignore file, line number, and pattern that matched—or outputs nothing if the file isn’t ignored.
The only scalable delete in Postgres is DROP TABLE
In PostgreSQL, running a large DELETE doesn’t immediately free disk space—it creates a mountain of follow-up work for the database. Under MVCC, deleted rows remain on disk as dead tuples so concurrent transactions can still see a consistent snapshot; autovacuum eventually marks that space reusable internally, but rarely shrinks the underlying data files on disk (except when completely empty pages sit at the very end of a relation), and indexes and sequential scans still pay the cost of skipping over dead tuples in the meantime. Mass deletes also generate heavy WAL volume that must be replicated downstream, inflating write latency, vacuum debt, and replica lag—while cascading foreign-key deletes can unexpectedly amplify a small delete into gigabytes of churn. VACUUM FULL reclaims OS disk space, but only by taking an expensive exclusive table lock and rewriting the entire table.
By contrast, DROP TABLE and TRUNCATE acquire an AccessExclusiveLock but execute in nearly constant time regardless of row count: they unlink the underlying physical files directly without generating per-row dead tuples or vacuum debt. Consequently, schemas that need regular bulk purges should be designed so large deletes become table drops. For time-series or retention-bound data, time-based table partitioning turns recurring deletes into instantly dropping the oldest partition. For a one-off cleanup where most rows are being deleted, copying the surviving rows into a temporary table, running TRUNCATE on the original, and inserting them back avoids per-row delete WAL (though structural and re-insert WAL still applies); if you cannot afford a long lock, you can dual-write and backfill into a fresh table before swapping names in a brief transaction, taking care to handle foreign keys, sequences, and concurrent writes. When the rows to delete are a small fraction of the table, batched deletes in small transactions remain the practical choice so autovacuum can keep up between batches.
This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.
Comments