Newsletter #11
Enjoy Newsletter #11.
Enabling near real-time data analytics on the data lake
Grab shares how they brought data lake ingestion latency down to just a few minutes instead of waiting on hourly or daily batch jobs. Their legacy Hive-and-Parquet pipeline worked well for analytical scans but made record-level updates and upserts prohibitively expensive, since modifying a single record required rewriting entire Parquet files. Grab migrated to Apache Hudi, tailoring table types to each source’s workload: for high-throughput Kafka event streams, they use Merge On Read (MOR) tables where Apache Flink appends fast Avro row logs and Spark runs periodic background compaction into columnar Parquet files; for lower-throughput sources, they use simpler Copy On Write (COW) tables, accepting a 10–15 minute ingestion latency.
The article dives into practical production lessons: partitioning high-volume Kafka event tables by event hour to speed up writes and bound compaction scope, using Flink CDC to stream binlog changes directly from relational databases, and choosing Hudi’s Bucket Index for deterministic hashing without external index lookups (despite the trade-off that bucket counts are hard to resize later). As a result, business analysts get much fresher operational dashboards, and the fraud detection team can query near-real-time transaction data on the lake without putting extra load on production OLTP databases.
The Ultimate Guide To Software Architecture Documentation
This comprehensive guide covers how to write, structure, visualize, and maintain software architecture documentation. The author pushes back against the “the code is the documentation” myth: source code tells you what the system does at the lowest level, but cannot tell you why architectural trade-offs were made, what non-functional quality attributes drove the design, or how the system fits into the broader business context. Good architecture documentation aligns stakeholders around a shared mental model, speeds up onboarding for new engineers, guides teams when building new features, and streamlines communication with external partners and auditors.
For structure, the author recommends the open-source, process-agnostic arc42 template, advising teams not to fill out every section up front on day one, but to grow the document incrementally as decisions are made. For visualization, the C4 model provides a clean hierarchy across four zoom levels: Context, Containers, Components, and Code. Finally, the post champions “Documentation as Code” and “Diagrams as Code” using tools like AsciiDoc, PlantUML, and Structurizr so architecture docs live in Git alongside the codebase and evolve in lockstep with pull requests.
Beyond the Basics: Designing for a Million Users
This post summarizes the opening chapter of Alex Xu’s System Design Interview, walking through the foundational building blocks for scaling a web application from a handful of users to millions. It starts with a single-server deployment: as traffic grows, latency spikes, a single hardware failure takes the entire site offline, vertical scaling hits cost and physical ceilings, and geographically distant users suffer slow page loads.
The first step is scaling horizontally rather than vertically, pairing a stateless web tier (moving user session data out into a shared persistent store or cache) with a load balancer to distribute traffic across multiple web servers. Next comes choosing the right database (relational SQL vs. NoSQL), adding read replicas and database sharding, placing a cache tier in front of hot database reads, offloading static assets to a CDN, introducing message queues for asynchronous background processing, using geo-routing for global users, and rounding out the stack with centralized logging, metrics, and automation. It serves as a solid primer for anyone preparing for system design interviews.
Decision-Making Pitfalls for Technical Leaders
Chelsea Troy points out that the software industry routinely promotes strong individual contributors into technical leadership roles with zero training in how decision-making changes at scale, highlighting three habits that are harmless as an IC but destructive once you hold authority. The first is assuming context: applying a “best practice” without examining what problem it was designed to solve, leading to mismatched architectures that the leader can’t justify to their team or executives. The second is treating every criterion as an optimization target, which causes decisions to drag on forever or get reopened repeatedly, whipsawing downstream teams. The third is manufacturing false urgency, subjecting the team to constant fire drills over foreseeable problems that should have been planned for.
Her remedies are straightforward: understand the concrete trade-offs of every option within your specific project context, explicitly separate criteria you want to maximize from constraints that only need to meet a “good enough” threshold (satisficing), and actively absorb pressure rather than amplifying it—because people rarely do their best thinking in a panic. Most “emergencies” in software engineering, she argues, are actually leadership failures in disguise.
How I know I’m working with a strong engineer
Sean Goedecke shares a simple gut-check for telling whether you’re working with a strong engineer: count how often two specific thoughts cross your mind while collaborating with them. The first thought is, “Oh, good catch—I hadn’t thought of that!"—triggered when they spot a subtle bug, improve the user experience, or simplify a design, rather than when they pedantically nitpick edge cases that don’t matter. If you find yourself thinking that constantly around someone, you should probably defer to their judgment on big technical calls.
The second thought is, “Nice, that’ll work,” when they propose the exact sensible approach you would have come up with yourself. That reaction signals reliability and steady execution—which the author notes is actually high praise. Engineers can be great in many different ways, of course, but this two-question heuristic is a surprisingly reliable intuitive litmus test.
Ugly Code and Dumb Things
Armin Ronacher, creator of Flask, Jinja, and Werkzeug, reflects on two competing passions in software: crafting clean, reusable library abstractions versus hacking together fast, pragmatic solutions for real users. He points to Flamework—an open-source PHP framework capturing Flickr’s early engineering ethos of “doing the simplest thing that could possibly work.” Its code looks messy to modern eyes, concatenating raw SQL strings and using global state, yet it nailed the problems that actually mattered for scaling Flickr: user-based sharding, database replication, and production observability.
Clean, aesthetically pleasing code doesn’t guarantee product success if it doesn’t solve a real problem for real people, while “ugly” code that ships often embodies the exact right compromises needed to iterate quickly. Both mindsets are valid, though they rarely coexist peacefully in the same codebase. The real art, Ronacher concludes, is knowing when to transition from duct-tape pragmatism to solid architectural foundations—because successful projects eventually grow up and need to be re-engineered.
Development Philosophy
This page from Sentry’s engineering handbook distills over a decade of lessons into their core development philosophy. Sentry emphasizes that code is not art—it is a tool for solving customer problems—so engineers must constantly weigh building clean abstractions against shipping pragmatic “duct tape,” and ask experienced teammates when unsure where on that spectrum a task falls. Even “correctness” can be a trap: for instance, forcing rigid static typing gymnastics onto Python can slow down iteration without delivering commensurate value. Duct tape is great for validating ideas quickly, but you can’t build a lasting business on it alone—Sentry is equally willing to invest in multi-year foundational bets, such as replacing shell-outs to llvm-symbolizer with their own custom Rust-based symbolicator service.
Other guiding principles include: avoiding overly “clever” code that future readers (including yourself six months later) will struggle to decipher; encouraging anyone to open a pull request against any repository in the company; standardizing primarily on Python, Rust, and TypeScript; avoiding proprietary database lock-in so Sentry remains easy to self-host; weighing the long-term maintenance cost of every third-party dependency; and keeping main green and deployable at all times.
AI is Stifling Tech Adoption
The author argues that AI coding assistants are quietly stifling the adoption of newer web technologies for two reasons. The first is the knowledge cutoff trap: because foundation models are trained on historical snapshots (at the time of writing, Anthropic’s latest models had a cutoff around April 2024), they know little or nothing about newly released frameworks or major version upgrades. That creates a vicious cycle: poor AI support discourages developers from adopting a new tool, fewer users means fewer blog posts and forum answers, and less public writing leaves future model training runs starved of examples. The second culprit is system prompt bias: many AI coding tools explicitly instruct the model behind the scenes to default to React and Tailwind CSS—to the point where Claude once rewrote the author’s vanilla HTML/CSS/JS snippet into React despite explicit instructions not to.
To test this, the author asked four major AI chatbots to “build a web app” with no stack specified: Claude and ChatGPT defaulted straight to React + Tailwind every time, Gemini used vanilla HTML/CSS/JS but immediately suggested migrating to React, and DeepSeek was more flexible but took several prompts to produce working code. The net effect is that technologies that were dominant right before ChatGPT launched are having their dominance locked in, newcomers are being steered toward a monoculture without realizing it, and AI vendors ought to be far more transparent about the stack biases baked into their system prompts.
February - The Rest of the Story - JVM Weekly vol. 119
This February 27, 2025 issue of JVM Weekly rounds up notable stories from February that didn’t fit into earlier editions. The headline news is that the next LTS release of Scala 3, slated for Q4 2025, will officially drop support for JDK 8; the Scala team is currently debating between JDK 11 and JDK 17 as the new minimum baseline, driven largely by the upcoming removal of memory-access methods in sun.misc.Unsafe, which forces a redesign of how lazy val is compiled under the hood. The issue also highlights a set of J.P. Morgan Java Lead interview questions, a series on SOLID principles in Kotlin, a draft JEP under Project Leyden that streamlines creating AOT caches into a single command-line step, and a bytecode-level auto-parallelization prototype that achieved up to a 9x speedup in the author’s benchmarks.
Rounding out the issue is part two of a series on building agentic AI systems with Quarkus—where application control flow is handed over to an LLM—alongside fresh ecosystem releases including KotlinPoet 2.0, Dokka 2.0, the Ktor CLI, Pulumi Java 1.0, and CheerpJ 3.1, a WebAssembly-powered JVM that runs full Java applications directly in the browser.
Manager Antipatterns
Ted Neward catalogs twelve common engineering management antipatterns—dysfunctional archetypes that companies repeatedly create through poor hiring, promotion, or org design. The roster includes the Absentee Manager (never around), The Radical (wants to burn everything down and rewrite from scratch), The Nepotist (given the team because they’re friends with an executive), The Mouthpiece (merely parrots upper management), The One-Trick Pony (applies the same hammer to every problem), The Chicken Little (panics constantly), The Pushover (avoids all conflict), The Transplant (dropped in from an unrelated domain), The Perfectionist (micromanages every detail), The Promoted Superstar (the best coder forced into management), The Riddler (communicates in cryptic hints instead of clear expectations), and The Player-Coach (stretched thin trying to manage and code simultaneously).
For each archetype, Neward sketches the behavior and offers practical survival strategies from three vantage points: when you report to that manager, when that manager reports to you, and when you realize you have become that manager yourself. It’s a handy field guide for diagnosing leadership dysfunction wherever you sit on the org chart.
Pause – Decision-Making Superpower
The author argues that even if you master every mental model and decision-making framework in the book, none of them will save you if you react on impulse. Recalling an early-career mistake—giving a snap feature estimate live on a client call and then pulling all-nighters to hit the impossible deadline—the author adopted a personal rule: never give an immediate commitment in the room; always ask to follow up after the meeting. Taking a deliberate pause creates space to step back from emotional hijacking, check for blind spots, and gather missing context—and people rarely push back when you simply tell them it’s a personal rule of yours. Nobel laureate Daniel Kahneman famously used the same tactic to stop himself from saying yes to requests on the spot that he would later regret.
Pausing is especially powerful when you receive an infuriating Slack message or email—wait a few hours or sleep on it before replying—or when fielding high-stakes questions during a crisis, such as right after a layoff. And when you genuinely don’t have all the answers yet, state clearly what is within your control and what isn’t, and share the concrete steps currently underway so your team knows you are taking their concerns seriously.
APOSD vs Clean Code
This GitHub repository records a written debate between John Ousterhout, author of A Philosophy of Software Design (APOSD), and Robert “Uncle Bob” Martin, author of Clean Code, hashing out their sharpest disagreements on software design. While both agree that the ultimate goal is making software easier to understand and modify, they clash across three main areas. On method length, Clean Code famously advocates tiny functions of just a few lines, whereas Ousterhout argues that decomposing functions too aggressively creates “shallow” interfaces and tangles of tiny coupled methods that force readers to jump back and forth to reconstruct the control flow.
On code comments, Clean Code treats most comments as a failure to express intent in code, while APOSD views comments as indispensable for capturing high-level interface contracts and design intent that code alone cannot express. On Test-Driven Development (TDD), Martin champions strict red-green-refactor micro-cycles before writing implementation code, whereas Ousterhout worries that strict TDD encourages tactical, line-by-line hacking at the expense of upfront design thinking. Reading the two authors challenge each other over concrete code examples is a masterclass in software design trade-offs and well worth reading to form your own view.
The Software Engineer Spectrum: Speed vs. Accuracy
Drawing on 15 years of experience ranging from IC engineer to CTO, Ben Howdle observes that every software engineer sits somewhere along a spectrum between Speed and Accuracy. Speed-leaning engineers live by “ship now, refine later”—excelling at rapid iteration and getting MVPs in front of users, at the risk of accumulating technical debt and missing edge cases. Accuracy-leaning engineers live by “make sure it’s right before it goes live”—excelling at rock-solid reliability, edge-case coverage, and scalable architecture, at the risk of moving slowly and over-engineering. Neither end of the spectrum is inherently superior; what matters is alignment with the company’s current stage.
Different business stages demand different balances: an early-stage seed startup needs Speed to validate product-market fit before the runway runs out; a scaling Series B company needs a balanced mix as testing and architecture start to matter; and a large enterprise or regulated fintech needs Accuracy where downtime or compliance failures are existential. The post offers self-reflection prompts—such as whether you get more frustrated by slow decisions or by shipping imperfect code—and advises engineering leaders to build balanced teams where both archetypes respect each other and can shift along the spectrum as the product matures.
LLM Ecosystem Predictions
Brian Kihoon Lee, an engineer at Databricks, lays out his predictions for the LLM ecosystem over the next 5–10 years. The cost of LLM inference at a fixed quality level is dropping roughly three times faster than Moore’s Law—halving every six months—driven by hardware advances, systems optimizations, and algorithmic breakthroughs; because all three favor capital-intensive hyperscalers, only a handful of frontier model providers will survive, competing at razor-thin margins. Just as high-level languages beat C and Java on developer iteration speed, simple agent architectures will beat complex, brittle orchestration frameworks: simply swapping in the latest frontier model every 6–12 months, backed by a solid evaluation suite, will beat months of custom prompt-chain tuning on both quality and cost.
He also predicts the market will bifurcate between managed enterprise AI platforms and DIY tooling, with an “Excel for LLMs” emerging to let non-programmers build their own AI workflows. Cost per query will span orders of magnitude—from 100-million-parameter small models to 1-trillion-parameter frontier models—while the economic value unlocked will vary even more wildly: LLMs are poised to replace expensive human coordination overhead inside organizations, personalized tutoring, and customer support with 24/7 availability and consistent quality. By 2030, he argues, the world will look as different from today as the mobile and Internet era looked compared to the 1990s.
The Right Kind of Stubborn
Paul Graham distinguishes between two very different traits that often get lumped together: persistence and obstinacy. Both make people hard to stop, but in opposite ways: the persistent person is like a boat whose engine can’t be throttled back, while the obstinate person is like a boat whose rudder can’t be turned. Persistent people are fiercely attached to the goal and happily change tactics when something isn’t working—in fact, they listen intently to criticism because they want to know if their current approach is flawed; obstinate people are attached to their initial plan—which is almost always the least-informed version of the idea—and reflexively dismiss feedback. Obstinacy only works on simple problems where the first idea happens to be right by luck.
True persistence, Graham argues, has a much richer internal structure built on five distinct qualities: energy to keep trying, imagination to invent new approaches when blocked, resilience to keep morale intact after setbacks, good judgment to weigh expected value, and a specific focus on a concrete problem. For less experienced founders and makers, his advice is to start with a small, well-defined goal and expand your ambition as you gain traction. Because it requires all five ingredients at once, the right kind of stubbornness is much rarer than mere obstinacy—and vastly more powerful.
Bonus: Interesting images from ByteByteGo

This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.
Comments