Newsletter #132
Enjoy Newsletter #132.
Perfection is not over-engineering
The phrase “we don’t want to aim for perfection” comes up in engineering meetings more times than the author can count. He understands the caution—over-engineering has burned out plenty of teams—but argues the industry is conflating two completely different things. Over-engineering is simply solving the wrong problem, usually with good intentions and a pile of accidental complexity attached. A “perfect” solution, by contrast, is very real once requirements are clear and all constraints are on the table: tighten the constraints enough and only one design fits, making it perfect precisely because nothing else satisfies the actual problem. Picking Python on serverless to skip a build step might be ideal for one team and completely wrong for a team that doesn’t know Python or needs predictable low latency.
In the author’s view, every system is a product: internal libraries, APIs, and CLI tools all have real users with real needs. The clearest tell of over-engineering is asking “why did we build it this way?” and getting an answer that falls apart under scrutiny. A classic example is a three-person team maintaining five microservices that share the same underlying data: foreign keys once enforced by the database turn into loosely coupled string IDs, and one service deletes records without the others knowing—all to buy “independent deployability,” a problem the three-person team never actually had. The takeaway: that isn’t an excess of perfection, it’s a failure of requirements gathering. The enemy was never perfection; it was vague requirements.
Claude Is Not a Compiler
Josh Bleecher Snyder argues that Claude is not a compiler—because it is better than one. In the idealized textbook view, software is built in neat horizontal layers: vision becomes strategy, product specs become engineering plans, source code becomes binaries, with each layer owned by a different role making thousands of micro-decisions. In reality, layers constantly leak and rub against one another, and working across layers is where the biggest wins happen—just as the Empire State Building was designed and erected in under a year because the owners, architects, structural engineers, and contractors sat in the same room making cross-cutting trade-offs together. Claude can work vertically across all those layers at once—discussing strategy, architecture, source code, and assembly without needing to schedule a meeting.
Snyder illustrates this with exe.dev’s distributed DNS server. High-level strategic and architectural decisions were settled human-to-human; for the rest, he spun up multiple parallel agent loops to build full implementations with tests, then had another agent compare the resulting designs. What shocked him was how many consequential design decisions the agents quietly made on their own without asking—and how differently each agent resolved them, such as how to handle a database state rollback. Repeating this multi-agent exploration three times let him distill a concise design document that guided the final agent across every critical decision point. A week later, having barely read the raw code himself, he could answer every question from his teammates, and a month into production the DNS server had zero incidents. He calls this “vibe-engineering”—and predicts that soon we’ll just call it software engineering.
In defense of not understanding your codebase
How thoroughly must an engineer understand their codebase? Sean Goedecke argues the answer depends entirely on the environment. Developers working on small, self-contained codebases like Redis will insist you must understand every line; engineers working on massive systems like Google or GitHub know that is literally impossible. Because the first group dominates online discourse, Goedecke steps up to defend the second: in a sufficiently large system, having only a partial understanding isn’t a moral failing—it is the best anyone can ever achieve. At scale, everyone operates with an incomplete, slightly wrong mental model, so effective engineers must get comfortable making educated guesses.
He pushes back against Peter Naur’s famous essay “Programming as Theory Building.” While agreeing that a programmer’s most valuable asset is the mental “theory” of how the program works, Goedecke rejects Naur’s conclusion that once the original team leaves, a codebase is dead and ought to be rewritten from scratch. First, large legacy systems encode thousands of hard-won edge cases that a greenfield rewrite won’t rediscover until production breaks, and successful rewrites almost always happen incrementally by carving up the existing system. Second, “abandoned” codebases are successfully revived in industry all the time: you start by tracing one end-to-end flow and expand your mental map from there. Finally, preserving a pristine mental theory is just one business value among many, routinely traded off whenever teams collaborate or patch urgent security vulnerabilities. Being a professional engineer means aligning with your company’s trade-offs—even when that means working effectively in code you don’t fully understand.
Beyond Happy Path Engineering: the Network
This opening installment of Gabor Koos’s “Beyond Happy Path Engineering” series examines the assumptions that local development environments indulge but production mercilessly punishes, starting with the network. A network call looks like a simple function call, but it crosses a boundary into a system you do not control. Using checkout payment authorization as a running example, Koos notes that “clean” failures like a refused connection are easy to handle compared to a timeout after the request was sent: the operation may have succeeded, failed, or still be sitting in a queue. The foundational rule of distributed systems is that not receiving a response does not mean the operation didn’t happen. Latency is also a failure mode in disguise: a slow downstream service ties up threads and connection pools, averages hide catastrophic tail latency, and slowness cascades into full outages faster than hard errors.
Koos outlines the essential defenses: treat timeouts as explicit design decisions derived from per-request latency budgets; retry only when safe, backing mutations with idempotency keys passed on every attempt alongside exponential backoff and jitter to prevent thundering herds; and propagate deadlines and cancellation contexts downstream so abandoned work stops immediately. Circuit breakers and load shedding protect overloaded services, while graceful degradation separates critical path steps (like taking payment) from deferrable side effects (like sending confirmation emails) using explicit order states like "pending". GitHub’s October 2018 outage—where a network partition lasting under a minute caused over 24 hours of degraded service—vividly illustrates the cost of happy-path network assumptions.
Beyond Happy Path Engineering: Time
Part two of the series turns to time—seemingly the simplest dependency of all: grab the current timestamp, compare two values, and make a decision. The running example is a checkout flow that holds inventory for fifteen minutes using if now() > reservation.expiresAt. If the cleanup worker’s clock drifts two minutes ahead of the checkout server’s clock, items get released back to inventory while the customer is mid-payment. Wall clocks can jump forward or backward mid-execution via NTP adjustments, turning elapsed durations negative, and timestamps across different machines cannot reliably establish event ordering. Worst of all, time bugs fail silently: services stay up, dashboards stay green, and the system quietly makes wrong business decisions.
The core lesson is that “time” actually answers four distinct questions: wall-clock time records when an event happened in the real world, monotonic time measures elapsed durations and timeouts on a single machine, logical time (versions or sequence numbers) orders competing updates, and business time governs billing periods, cutoff windows, and user time zones. Expiration should be modeled as an explicit state transition rather than a passive timestamp comparison: use timestamps only to find candidate rows, then execute a conditional write that transitions the row to expired only if its status is still reserved, making late or duplicate cleanup runs harmless. When local clocks aren’t enough to arbitrate, reconcile against the authoritative external source, such as the payment provider. Cloudflare’s New Year’s 2017 leap-second DNS outage—when subtracting two time.Now() readings yielded a negative number and triggered a panic—serves as a classic cautionary tale.
Beyond Happy Path Engineering: Databases
Part three tackles databases, the layer developers most often treat as an infallible black box. Using an e-commerce checkout flow that works flawlessly on a developer’s laptop because there is zero concurrent traffic, Koos shows how production changes everything: checkouts, admin dashboards, background jobs, analytics queries, and schema migrations all compete for the same connections and locks. A single slow query can exhaust the connection pool, causing incoming requests to queue up, users to hit refresh, and “checkout to go down” even though the database hasn’t crashed. Two concurrent requests can both read stock = 1 and both sell the last item; wrapping code in a transaction doesn’t magically prevent race conditions unless business invariants are enforced explicitly. On top of that come lagging read replicas, commit timeouts that leave the caller unsure whether the write succeeded, deadlocks, and schema migrations running alongside multiple live application versions.
Koos recommends starting from the invariant that must never be violated—such as “never sell more than available inventory”—and enforcing it directly in the atomic write statement (UPDATE ... WHERE stock >= quantity) backed by database constraints rather than application-level checks alone. Keep transactions short, never make network calls inside a transaction, and retry only transient errors. Make writes idempotent, define explicit data-freshness requirements per endpoint before routing reads to replicas, run schema migrations in backward-compatible expand-and-contract phases, and rehearse backup restores regularly. Atlassian’s 2022 outage and GitHub’s 2018 incident both prove that recovery speed matters just as much as data durability.
Beyond Happy Path Engineering: Storage
Part four examines object storage, which looks simpler than a relational database but hides a fundamental trap: your database and your object store are two separate systems with no shared transaction coordinator. Consider a document upload flow: write the file to S3, insert a metadata row in the database, and return the URL. If the blob upload succeeds but the database insert fails, you get an invisible orphaned object; flip the order and a failed upload leaves a database row pointing to a missing blob that serves 404s. Abandoned multipart uploads quietly rack up storage bills, presigned URLs remain valid even after the database record is deleted, concurrent uploads to the same key silently overwrite each other, and CDNs keep serving stale cached files. High durability also doesn’t equal recoverability, because an accidental DELETE is replicated just as durably as a valid write.
Koos advises ordering upload steps so failures fail safe, tracking lifecycle state explicitly in the database (pending, confirmed, failed), and serving only confirmed records. Object keys should use immutable UUIDs or content hashes rather than user-supplied filenames; large files should upload directly from the client via short-lived presigned URLs; and deletions should soft-delete the database row first before a background job purges the blob, backed by a periodic reconciliation sweeper. The 2014 demise of Code Spaces—which shut down permanently after an attacker gained access to their AWS control panel and wiped both production storage and backups—underscores why backups must live outside the blast radius of production credentials.
Everyone Should Know SIMD
SIMD has a reputation for being arcane, and Mitchell Hashimoto regularly meets strong engineers who write it off as too hard to learn or useful only in niche high-performance computing. He argues that reputation is undeserved: the vast majority of practical “process N items at a time” SIMD code follows a single five-step template, and once you internalize that template, writing SIMD is almost as straightforward as writing a standard for loop. Because SIMD lets the CPU process multiple data elements in a single instruction, any loop scanning bytes, characters, or array elements is a candidate once inputs reach thousands or millions of bytes.
The five-step template is: broadcast constants into vectors and initialize accumulators if needed; loop over the input in chunks equal to the vector width; perform parallel comparisons or arithmetic across all lanes; reduce or store the vector result; and handle the trailing remainder elements using your original scalar loop. Hashimoto walks through a real example from the Ghostty terminal emulator written in Zig: finding the end of a run of printable ASCII characters by scanning until a byte is <= 0xF. The scalar loop is a one-liner; the SIMD version adds about 12 lines of code and delivers a theoretical 4x speedup on ARM NEON, 8x on AVX2, and 16x on AVX-512, measuring roughly 5x faster in real benchmarks on an Intel AVX2 machine. While compilers can sometimes auto-vectorize simple loops, whenever a hot path cares about a 5x speedup, Hashimoto prefers explicit, portable SIMD that won’t silently regress after a compiler upgrade or minor code tweak.
htop explained
Pēteris Ņikiforovs admits that for years he used htop without truly understanding half the numbers on screen—for instance, assuming a load average of 1.0 on a dual-core machine meant the CPUs were 50% busy. So he dug into the Linux kernel sources and wrote a comprehensive visual guide breaking down every single bar and column in htop. What makes the guide stand out is that he traces every metric back to its exact kernel source: uptime comes from /proc/uptime, load average comes from /proc/loadavg, and you can verify how htop gathers them yourself by running it under strace to see which files it opens.
Load average isn’t CPU percentage at all—it is the exponentially damped moving average of runnable threads plus threads stuck in uninterruptible sleep waiting on disk or network I/O; the 1-minute figure reflects roughly 63% of the past minute’s load and 37% of the prior average. The guide then demystifies process states (R running/runnable, S interruptible sleep, D uninterruptible sleep, Z zombie terminated but not yet reaped by its parent), NI nice values from -20 to 19 and their relation to kernel priority (PR = 20 + NI), and the memory columns: VIRT includes unbacked virtual reservations and memory-mapped files so it’s rarely actionable, RES is actual resident RAM (though shared libraries and copy-on-write pages get double-counted across processes), and MEM% is simply RES divided by total RAM. He wraps up by auditing every default background process on a fresh Ubuntu Server install to see which ones can be safely stripped out.
The Productivity-Experience Paradox
In the second article from her master’s research on how AI coding tools affect software engineers, Annie Vella shares findings from a longitudinal study tracking the exact same cohort of engineers across two waves six months apart (October 2024 to April 2025). On the productivity front, the news is consistently positive: 84% of respondents reported that AI helped them get more done in both waves. Yet developer experience—measured across the DevEx framework dimensions of feedback loops, cognitive load, and flow state—moved in the opposite direction: the share of engineers reporting a net-negative experience nearly doubled from 14% to 27%, and not a single engineer who fell into the negative group in Wave 1 recovered to an all-positive experience by Wave 2.
The central finding is that productivity and developer experience—long assumed to rise and fall together—are decoupling: tracking the same individuals over time, experience degraded while output held steady, with a near-zero correlation ($r = 0.02$) between changes in flow state and changes in perceived productivity. That decoupling is dangerous because the business case for investing in DevEx has historically been “happy developers are productive developers”; when output stays high, rising cognitive strain stays invisible until it turns into burnout. Flow suffered the sharpest drop (negative reports rose from 7% to 20%) even as feedback loops sped up, and Vella suspects the two are linked: every fast AI response triggers a micro-context switch from creator to reviewer. Borrowing the term “junk flow” to describe a slot-machine-like loop of rapid prompting that mimics flow without deep satisfaction, she concludes that while AI boosts extrinsic output, it risks eroding the intrinsic craft and joy of software engineering.
Prefactoring: Clear the Way for Your New Feature
This short post from Google’s “Tech on the Toilet” series opens with Kent Beck’s famous advice: “First make the change easy (warning: this may be hard), then make the easy change.” When you try to bolt a new feature directly onto code that wasn’t structured to accommodate it, things get messy fast: tweaking one file forces changes in three others, and soon you’re knee-deep in unrelated modules. Prefactoring—short for preparatory refactoring—is the discipline of restructuring existing code to fit the upcoming requirement before writing the new feature, rather than promising yourself you’ll clean it up afterward.
The payoffs are immediate: the new feature slots in naturally, code reviews go much faster because structural refactoring and behavioral changes live in separate commits, fewer bugs slip in because cleanup is isolated from new logic, and rollbacks are safer because each change is small and self-contained. In the post’s example, the first CL simply extracts a get_display_name(user) helper to deduplicate name-formatting logic across the profile page and email templates; the second CL adds middle-name support with a one-line edit inside that helper. Even if you only spot the right abstraction mid-implementation, you can still split the cleanup into a preparatory CL—or spin off reviewer-suggested cleanups in parallel while your main CL is under review. Not every cleanup belongs up front, though: if a refactoring doesn’t directly unblock your feature, save it for later, or fold it in if it’s truly trivial.
This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.
Comments