Newsletter #135
Enjoy Newsletter #135.
Git at any scale
Vicent Martí explains why hosting centralized Git repositories at massive scale is notoriously difficult: Git packfiles are large binary blobs that expect to live on a local POSIX filesystem with fast mmap random access. Early attempts to ditch packfiles and store loose Git objects in distributed key-value stores made git clone unusably slow, because walking the commit and tree graph incurred a network round trip at every hop. Shared network filesystems like NFS, GFS, or DRBD proved slow, brittle, and operationally painful. GitHub’s Spokes replicates repositories at the application layer using a three-phase commit for reference transactions, but push throughput is bottlenecked by the slowest replica, and fixing every repo at three copies over-provisions cold repos while under-provisioning hot CI monorepos—all while requiring a massive central routing database.
Cursor’s Continuity engine flips the architecture by making a write-ahead log (WAL) on S3-compatible object storage the single source of truth. Every git push writes its pack data and ref update to the WAL before acknowledging the client, becoming visible only after the reference transaction succeeds and updates the WAL index—making all pushes strictly linearizable. Standard on-disk Git repositories on local NVMe drives act purely as stateless, disposable caches reconstructed on demand from the WAL, eliminating consensus protocols entirely. Nodes are selected via rendezvous hashing, while write safety relies on S3’s conditional compare-and-swap (If-Match / If-None-Match). Read replicas receive invalidation hints over UDP gossip and verify freshness via conditional GET requests with ETags in under 10 ms. Only the primary node runs packfile compaction; replicas simply download compacted packs from S3. Benchmarks on Cursor’s own monorepo show linear read scaling up to 100 replicas, ~120 pushes/sec on S3 Standard, and over 300 pushes/sec on S3 Express One Zone—laying the foundation for Origin, Cursor’s Git hosting platform built for high-concurrency agent workloads.
Regressive JPEGs
Maurycy explores a delightfully cursed trick with progressive JPEGs, a format designed to transmit low-frequency image components first so a partially downloaded photo appears blurry rather than half-blank. Inside a progressive JPEG, compressed image data is divided into multiple “scans,” each with its own header specifying which color channels (Y, Cb, Cr), which range of DCT frequency bins, and which bit-precision slice it contains. The first scan normally carries the DC (average color) bin for all three channels, while subsequent scans progressively fill in high-frequency AC detail. Because each scan declares its own spectral range, the author realized nothing stops a later scan from overwriting frequency bins that were already rendered—meaning you can concatenate multiple same-resolution images into a single JPEG file (stripping their intermediate SOI, SOF, and EOI markers) so the image morphs from one picture into another as it loads over a slow connection.
The catch is that browsers cap the maximum number of scans they will decode before giving up (likely as a defense against decompression bombs), limiting full-detail progressive overwrites to about nine frames. To cram a real animation into a single JPEG, you have to minimize the number of scans per frame. Because the JPEG spec forbids mixing DC and AC coefficients in the same progressive scan, the smallest possible frame is a single DC-only scan—producing a blocky 8×8-pixel-per-block low-res image. Using one DC scan per frame, Chrome renders roughly 90 frames before bailing out, Firefox keeps going even longer, and ghosting artifacts vanish because there are no leftover AC coefficients from earlier frames bleeding through. The result is an entire multi-second video inside a valid .jpg file—utterly useless in production since playback speed is dictated entirely by network bandwidth, but prime material for a rickroll.
A quick look at zero-knowledge proofs
Max Bernstein and Chris Gregory demystify zero-knowledge proofs (ZKPs) through a classic example that has nothing to do with cryptocurrency: proving a graph is 3-colorable, following Goldreich, Micali, and Wigderson’s 1986 paper. In a ZKP, a prover claims to know a secret solution to a hard problem, and a verifier wants to be convinced the prover truly knows it without learning a single bit about the solution itself. Graph 3-coloring—assigning one of three colors to every vertex so no two connected vertices share a color—is NP-complete to solve but trivial to check locally along any single edge. Each round of the interactive protocol runs in four steps: the prover randomly permutes the three color names, “locks” each vertex’s permuted color inside a sealed box, and sends all the locked boxes to the verifier; the verifier picks one random edge (u, v) and challenges the prover; the prover sends back the keys to only boxes u and v; and the verifier opens those two boxes to verify that their colors are valid and different.
In Python code, a “locked box” is a cryptographic commitment: the hash of the vertex’s color concatenated with a random cryptographic nonce (using hashlib.sha256 and secrets), since hashing the color alone without a random nonce would produce identical hashes for same-colored vertices and leak the coloring structure. Because a cheating prover who doesn’t actually have a valid 3-coloring can still get lucky unless the verifier picks the exact edge that violates the rules, a single round catches a cheater with probability at least $1/m$ (where $m$ is the number of edges). Repeating the protocol with fresh random color permutations and nonces for $m^2$ rounds drives the probability of forgery down exponentially: on a 1,000-edge graph, ~4,600 rounds reduce the soundness error below 1%, and 10,000 rounds drop it to 0.0045%. The authors build a working client-server demo and note that because 3-coloring is NP-complete, any NP problem—from Sudoku to integer factorization—can theoretically be reduced to it, even if the resulting circuit graph balloons quickly in size.
The Pulse: We need to talk about migrations with AI
In this free edition of The Pulse, Gergely Orosz argues that AI coding tools have fundamentally changed the economics of large-scale code migrations that teams previously put off for years as prohibitively expensive. The headline example comes from Asana: according to an OpenAI case study, Asana used Codex to rip out Enzyme and migrate their test suite to React Testing Library in roughly 1.5 engineer-weeks and ~$12,000 in model and compute costs—a migration previously estimated at $6 million and five years of background effort. Similarly, Airbnb migrated 3,500 Enzyme test files in six weeks (versus a 1.5 engineer-year manual estimate), Uber migrated 600,000 JUnit 4 tests to JUnit 5 in four months with two engineers, and Bun ported 530,000 lines of Zig to Rust in two weeks for $165,000 in API spend. By contrast, back in 2021 Sentry spent 1.5 years and roughly ten engineers migrating 95,000 lines of JavaScript to TypeScript.
Across these successes, the same engineering patterns recur: automated retry loops when a migrated file fails to build or pass tests, multi-phase per-file pipelines, engineer-designed verification harnesses, starting with leaf files or frequently touched files first, and refactoring the codebase beforehand so patterns are uniform for the LLM. Orosz notes that Asana’s “$6M saved” headline is likely inflated—manual estimates rarely account for humans getting much faster after the first hundred files—and stresses that every single migration still required experienced engineers to design the pipeline, build the verification gates, and handle edge cases. Even so, the cost-benefit curve for paying down massive technical debt has shifted permanently.
The Accidental Application Runtime
Michael F. Bryan describes how a small Go service managing a 3D-printer farm gradually grew into an tangled mess. It started as a simple dashboard where the HTTP server spawned a single background goroutine to poll printer status. Over time, polling jobs from a cloud backend, persisting state to SQLite, scheduling print jobs, streaming G-code, and reporting telemetry were all bolted onto that same initialization block until NewServer was spawning five long-running background goroutines. When the team needed to add graceful shutdown on SIGTERM, figuring out which goroutines were running, in what order they needed to stop, and who owned closing which Go channels required reading the entire startup file line by line. Bryan calls this an “accidental application runtime”: the service had a real concurrent pub/sub architecture, but nothing in the code’s structure made that topology explicit.
Inspired by the PX4 drone autopilot’s uORB messaging layer, Bryan built a lightweight Go library called backplane. Each subsystem becomes a plain Go function func(ctx, deps...) error whose typed parameters declare its communication topology: a chan<- T parameter publishes to topic T, <-chan T subscribes to a stream of T, and *backplane.Latest[T] polls the latest state snapshot. Topics are keyed by Go types rather than error-prone string names. backplane.New validates the dependency graph before startup, and Run supervises the modules like an errgroup with in-process, backpressured delivery. Rebuilding the printer service as six explicit modules—anchored by a strict commit-then-notify rule where the scheduler commits job state to SQLite before publishing a wake-up message—even let the team auto-generate accurate Mermaid architecture diagrams straight from the function signatures.
Generic Methods in Go 1.27
When Go 1.18 introduced generics in 2022, functions and types could declare type parameters, but methods could not declare additional type parameters of their own. Go 1.27 finally lifts that restriction: methods can now introduce their own type parameters without polluting the receiver struct. Dominik Braun illustrates why this matters using a generic graph node Node[T]. Previously, if you wanted a Map method that transforms a Node[T] into a Node[U], you either had to pollute the struct definition itself as Node[T, U]—forcing every unrelated method on Node to carry U around—or give up on method syntax and write a standalone package-level function Map[T, U any](n *Node[T], ...). Starting in Go 1.27, you can simply write func (n *Node[T]) Map[U any](...) Node[U], scoping U strictly to the Map method.
Why did this take four years to land? The hurdle lay in how Go compiles generics versus how it dispatches interfaces. Go’s compiler primarily uses monomorphization (stenciling specialized machine code for concrete type shapes at compile time), whereas Go interfaces rely on runtime dynamic dispatch via an itable of function pointers. Those two models collide if an interface is allowed to declare a generic method like type Mapper interface { Map[U any]() Node[U] }: the compiler cannot pre-populate a fixed-size runtime vtable for an unbounded set of potential type arguments U, especially when reflection can look up methods dynamically at runtime. Go’s pragmatic compromise in 1.27 is to allow generic methods on concrete types while forbidding them in interface definitions—and a generic method cannot be used to satisfy an interface.
Your executable is a SQLite database
Farid Zakaria, author of sqlelf, takes his previous experiment to its logical conclusion: replacing ELF entirely with SQLite as a native Linux executable format. His prototype, SELF (Structured Executable & Linkable Format), produces real SQLite database files that you can chmod +x and run directly from the shell. His core argument is that ELF is already an ad-hoc, bespoke relational database in denial: .strtab is string interning, .gnu.hash is a hash index, section headers are a catalog table, and symbol versioning is a foreign-key schema. In SELF, an executable’s schema consists of a self_meta table for the ELF header, a segments table storing each program header row with its raw machine code/data as a SQLite BLOB, and a symbols table indexed by standard SQLite B-trees in place of .dynsym and .gnu.hash. Registering the SQLite magic header with Linux’s binfmt_misc lets the kernel hand ./my_app directly to self-exec, a tiny C loader that queries the segments, maps them into memory, performs relocations, and jumps to the entry point.
For dynamic linking, SELF supports two modes: keeping glibc’s standard ld.so and using an rtld-audit hook so a SQL query resolves shared libraries, or using self-ld, an experimental SQL-native dynamic linker. Even cooler, self closure bundles a binary and all its transitive shared-library dependencies into a single self-contained SQLite database file—turning ldd into a simple SQL JOIN. Packing all 723 executables in /usr/bin into a single shared SELF database shrank total size from 644.4 MiB (ELF) down to 611.9 MiB thanks to cross-binary library and symbol deduplication, even though individual standalone SELF files are 2.2x to 3.6x larger than their ELF counterparts. While copying segments out of SQLite blobs prevents cross-process mmap page sharing for .text segments and slows cold start on massive binaries, being able to inspect, strip, or atomically patch executables with UPDATE and ROLLBACK is a fascinating glimpse into an alternate Unix universe.
What happens when a GPU reads memory
Doubleword founder Fergus Finn traces the exact hardware journey of a single global memory load instruction (LDG.E) inside an NVIDIA RTX 4090. Because NVIDIA documents very little of its microarchitectural memory pipeline publicly, Finn reverse-engineered the path using custom cycle-accurate microbenchmarks and hardware performance counters. When a 32-thread warp issues a 4-byte load per thread, the hardware coalescer groups those 32 requests into four 32-byte sectors—one 128-byte cache line. On an L1 miss, the virtual address passes through a 16-entry per-SM L1 TLB (backed by 2 MiB pages) to translate into a physical address, because the L2 cache is physically indexed. From there, the chip-wide crossbar routes the request to one of 36 L2 cache slices selected by an undocumented hash of the physical address bits, which in turn forwards L2 misses to one of 12 memory controllers wired to 12 GDDR6X chips—where the controller must issue an expensive DRAM row ACTIVATE command to open a 1 KiB row before bursting the four 32-byte sectors back.
The latency ladder is stark: an L1 hit takes ~15.4 ns, an L2 hit takes ~127 ns, and a full miss down to GDDR6X DRAM takes ~255 ns—roughly 660 GPU clock cycles during which that warp sits stalled. Activating a new DRAM row costs roughly 15x more time than reading from an already-open row, and hammering a single L2 slice caps throughput at 1,957 million loads/sec versus ~68,000 million loads/sec when spread evenly across all 36 slices (a 34.8x difference). GPUs hide that 660-cycle latency through massive warp-level parallelism—switching to other ready warps every cycle—which is why memory-bound kernels live or die by coalesced access patterns that spread traffic evenly across L2 slices and DRAM banks.
A faster way to calculate the day-of-the-week
Ben Joffe tackles a deceptively simple problem: converting a signed 32-bit Unix epoch day count (1970-01-01 = Thursday) into a day of the week (0..6) as fast as possible across the entire 32-bit integer range. Cassio Neri’s 2024 algorithm serves as the state-of-the-art baseline, but requires an extra correction step because 7 is an awkward divisor in binary. Joffe’s key insight is to exploit the fact that $7 = 2^3 - 1$ is a Mersenne number, which means $N \bmod 7 = \lfloor N \cdot \frac{8}{7} \rfloor \bmod 8$. Multiplying by $\frac{8}{7}$ can be approximated by a single fixed-point multiplication by a magic constant M, and taking $\bmod 8$ comes almost for free by masking or shifting to keep only the top three bits of the fractional part. A bounded 32-bit version reduces to a one-liner—((u32(rd) * M + Z) >> 29)—valid across $\pm 242,000$ years, which sped up weekday calculations in the Rust jiff datetime crate by ~40%.
To cover the full signed 32-bit domain without overflow, Joffe presents several variants: widening to a 64-bit multiply (already adopted by WebKit), adding a bit-shifted correction term that executes in parallel with the multiply, using two multiplies for higher instruction throughput, or collapsing the entire calculation into three x86 instructions via LEA. Switching between Sunday-indexed (0..6) and ISO Monday-indexed (1..7) output only requires swapping the additive constant Z. Better yet, the same Mersenne identity generalizes to any divisor of the form $2^N - 2^K$, including x % 24 and x % 60 for extracting hours, minutes, and seconds. Benchmarks show Joffe’s variants taking just 0.17x to 0.39x the runtime of Neri’s algorithm on an Apple M4 Pro and 0.27x to 0.59x on a Ryzen 9 9950X3D, with patches already merged into WebKit, Elixir, Rust’s jiff, and Zig’s zeit.
Dark mode toggles: two states are enough
Lea Verou argues that website header dark-mode toggles should only expose two visible states to the user, even though the underlying state machine has three (System, Light, and Dark). In her view, exposing a three-way segmented control or dropdown in the navbar solves an imaginary user goal: nobody visits a website thinking “my OS is in light mode and this page looks great in light mode, let me proactively pin it to Light just in case my OS switches to dark mode tonight.” Users click a theme toggle for one reason: the current appearance is uncomfortable to read right now, make it the opposite. Exposing all three states in the header is classic “implementation-model UI” that adds visual clutter, extra clicks, and unnecessary cognitive load. At the same time, a naive two-state toggle is even worse if clicking it once permanently locks localStorage into an override with no way to return to following the OS theme.
Verou’s solution is a single two-state button that cleanly traverses all three underlying states. By default (when following the OS), the button displays the currently active theme and stores nothing in localStorage. Clicking it once switches to the opposite theme and saves that explicit override. Clicking it a second time switches back to the theme that matches the user’s current OS preference—and instead of saving a hard override, it clears the stored setting, returning the site to System mode. Because the override is only re-evaluated when the user clicks the toggle, users whose OS automatically flips between light and dark at sunset can still override a site whenever they want without getting stuck permanently out of sync. Reserve explicit three-way Light / Dark / Auto selectors for dedicated settings pages—not the top navigation bar.
Comments