Newsletter #139
Enjoy Newsletter #139.
What Go Taught Us About Java Garbage Collection
Comparing ParparVM (Codename One’s ahead-of-time Java-to-C virtual machine) against Go’s runtime, the author initially assumed Go’s low memory footprint and snappy pauses came down to escape analysis and stack allocation—only to discover how much came from when Go triggers a GC cycle and how it shares marking work across threads. ParparVM’s first win was lowering its hardcoded 24 MB minimum allocation floor before triggering a GC: throughput and p99 latency barely budged, while resident memory (RSS) dropped from 98 MB down to 38 MB. An attempt to pace GC dynamically against the live heap size failed because internal counters missed objects in partial pages, thread-local mutator pages, and large-object allocations, so the team exposed the allocation floor as a build-time tunable instead. On their GcPause benchmark, scaling parallel mark threads from one to four cut worst-case pause times from 2.2–3.3 seconds down to 0.3–0.9 seconds (compared to ~20 ms in Go), and adding Go-style mutator assist—where an allocating thread that outruns the background collector pitches in to mark a small batch of objects itself—smoothed out allocation spikes further.
The second half tackles image caching and reference types. Previously, ParparVM implemented WeakReference by storing the referent as a normal struct field that the GC traced as a strong reference, while iOS apps relied on a separate strong-reference cache table that simply dumped everything on an OS memory warning. A new PR moves the referent pointer into java.lang.ref.Reference directly, teaches the C translator to register reference discovery hooks instead of tracing referents strongly, and lets the collector decide whether to preserve or clear soft references based on memory pressure. The trickiest concurrency hazard occurs when a mutator thread calls ref.get() mid-mark phase and resurrects a pointer the GC is about to sweep; logging get() reads through a snapshot-at-the-beginning (SATB) read barrier and filtering already-marked objects dropped mark-termination passes from 32 down to 1. Updating a recency timestamp on each get() so frequently used images survive longer boosted cache hit rates under a 160 MB cap from 87.99% to 97.44% while simultaneously shrinking memory footprint from 91 MB to 82.1 MB.
How We Built a Real-Time Stock Pricing System at Scale
Mofid Securities, Iran’s largest stock brokerage, needed to stream real-time stock and fund price ticks to tens of thousands of concurrent traders. Market ticks from the exchange core are ingested in C++ and published to Kafka as Protobuf messages at roughly 1,900 messages per second—spiking to 3x that during volatile market opens. For fan-out distribution, the team chose Redis Pub/Sub over heavier durable brokers: stock prices are ephemeral state where a missed tick is immediately superseded by the next tick a moment later, making at-most-once delivery semantics ideal, and Redpanda Connect pipes messages from Kafka into Redis without writing custom glue services. On the browser side, they picked Server-Sent Events (SSE) over WebSockets because price streaming is strictly unidirectional, runs over standard HTTP, and handles reconnection natively; they used the eventsource package to attach auth headers, batched UI updates on the client to avoid layout thrashing, closed streams when tabs navigated away, and fetched an initial REST snapshot before applying the live delta stream.
Inside the .NET backend, a single hosted background service per Redis channel writes incoming ticks into a bounded System.Threading.Channels ring buffer (capacity 100,000, configured with DropOldest backpressure), while a connection coordinator spins up lightweight per-user channels that filter and forward only the specific ticker symbols currently visible on that user’s screen. Rolled out gradually from 30% to 100% of traffic behind an Unleash feature flag, the production cluster serves tens of thousands of concurrent streams across just six pods, each using ~55 MB of RAM and ~20% CPU at peak market hours with maximum queue depth never exceeding 15 messages. Key operational lessons: disable reverse-proxy response buffering via X-Accel-Buffering: no, dispose per-client channels immediately on disconnect to prevent memory leaks, and send a lightweight heartbeat comment every 20 seconds so intermediate load balancers don’t drop idle connections.
Saving another 100TB of RAM with math (and Rust)
Cloudflare freed up another 100+ TB of RAM across its global edge fleet by shrinking pingora-ketama, the consistent hashing library that Pingora Backend Router uses to route requests to cache servers by URL. Because compliance rules and tiered caching configurations require maintaining a separate consistent hash ring for every unique combination of cache features, some router processes were burning up to 6 GB of RAM just holding Ketama rings in memory. Step one was shrinking each point on the ring in Rust. The original Point struct stored hash: u32 and index: u32 (8 bytes per point). Simply changing index from u32 to u16 saved zero bytes because Rust aligns the struct to a multiple of its largest field (u32, 4-byte alignment), adding 2 bytes of tail padding; packing each entry into a byte array [u8; 6] with helper accessors bypassed the alignment padding and immediately cut ring memory by 25%.
Step two used probability theory to slash the number of virtual nodes (points) placed on the ring per server. Deriving the exact closed-form formula for load variance across $n$ servers with $k$ points each revealed steeply diminishing returns: for 100 servers, NGINX’s default of 160 points per server already drops load standard deviation from ~99% down to ~8%, whereas scaling all the way to 100,000 points per server spends the last 90,000 points buying just a 0.7% reduction in variance—and at extreme point counts, 32-bit hash collisions actually cause load imbalance to worsen again. Dropping the point count by 90% left load distribution virtually unchanged. Because changing hash rings in place would invalidate cached assets and trigger a thundering herd to origin servers, Cloudflare held both the old and new rings in memory temporarily—saving 65% net RAM even while dual-running—and shifted traffic gradually by hashing requests into rollout buckets across data center tiers.
Postgres DELETE Can Actualy Scale
In high-throughput PostgreSQL systems, DELETE has a reputation for being so slow that conventional wisdom says to avoid it altogether in favor of TRUNCATE or dropping time-partitioned tables. The team at DBOS, however, needed row-level DELETE for their Postgres-backed durable workflow engine, which executes tens of thousands of workflows and steps per second while enforcing granular, per-workflow retention policies. What makes DELETE expensive in Postgres is MVCC: deleting a row doesn’t free space immediately—it merely stamps the tuple’s xmax header with the deleting transaction ID, leaving a “dead tuple” behind for autovacuum to clean up later, and because Postgres indexes don’t store visibility metadata, every index scan still has to visit heap pages to check whether matching rows are dead.
DBOS’s initial schema relied on foreign keys with ON DELETE CASCADE from the parent workflow_status table to child tables like operation_outputs. While convenient, cascading deletes scattered random I/O across thousands of heap pages because a long-running workflow’s steps are spread out over time—to the point where deleting expired workflows took longer than executing them! The team dropped the foreign keys and rewrote retention as a two-step batch job: first delete expired rows from workflow_status, then batch-delete orphaned rows from each child table ordered by creation time (maximizing page cache locality) using a NOT EXISTS anti-join against workflow_status so active long-running workflows are never touched. When they hit a second wall—finding the retention cutoff timestamp on the created_at B-tree index occasionally stalled for 20 minutes scanning millions of unvacuumed dead tuples piled up at the front of the index—running an explicit VACUUM right before and after each deletion batch eliminated the stall completely, speeding up retention deletes by 10x.
Where PostgreSQL stores row locks
Unlike many databases that maintain an in-memory lock table for row-level locks, PostgreSQL stores row locks directly inside the tuple header on the data page itself. Specifically, the t_xmax field—which normally records the transaction ID that deleted or updated a tuple—does double duty by storing the XID of the transaction currently locking the row, while bit flags in t_infomask indicate the lock strength and clarify that t_xmax is a lock rather than a deletion. That design means PostgreSQL can lock millions of rows without exhausting shared RAM, at the cost of dirtying the data page (and generating WAL) even on a read-only SELECT ... FOR SHARE. When the locking transaction commits, Postgres doesn’t go back to clear t_xmax; future readers simply check the commit log, see the transaction has finished, and ignore the expired lock marker until VACUUM or a subsequent locker overwrites it. Postgres supports four row-lock modes so foreign-key checks don’t stall updates: inserting a child row takes the weakest FOR KEY SHARE lock on the parent row, a normal UPDATE that doesn’t touch key columns takes FOR NO KEY UPDATE, while DELETE and key-modifying updates take FOR UPDATE (which conflicts with everything—meaning developers who reflexively write SELECT ... FOR UPDATE on parent rows unnecessarily block concurrent child inserts that FOR NO KEY UPDATE would allow).
When a second transaction tries to lock an already-locked row, it waits on the holder’s transactionid lock in pg_locks and acquires a heavyweight tuple lock solely to establish fair queue ordering—which is why pg_locks only shows waiting row lockers, never the session actively holding an uncontended row lock. And when multiple transactions concurrently hold shared locks (FOR KEY SHARE or FOR SHARE) on the same tuple, t_xmax stores a MultiXactId pointing to an on-disk member list in pg_multixact/. The post demonstrates how a single long-running transaction holding FOR NO KEY UPDATE on parent rows while 5,000 autocommit transactions insert child rows creates 5,000 distinct multixacts; at ~5 bytes per member and a hard $2^{32}$ member offset limit (~20 GB), runaway multixact creation can trigger multixact "members" limit exceeded outages or emergency anti-wraparound vacuums even when a table has zero dead tuples—making mxid_age(), relminmxid, and next_multi_offset critical production metrics to monitor.
A 40ms Go GC pause caused by swap
After enabling Linux swap on production nodes to absorb brief memory spikes gracefully, the author noticed something alarming in their Go service metrics: stop-the-world (STW) garbage collection pauses were occasionally spiking to 40 ms—roughly 800x slower than the normal 51 µs median pause. The container cgroup hosted two processes: the latency-sensitive Go service (which read request payloads via io.ReadAll and decoded them with proto.Unmarshal) and a nearly idle helper HTTP server. Initially, the author assumed that because the Linux kernel tracks memory pressure per cgroup, cold pages from the idle helper process would be swapped out first, or that if the Go process itself got swapped out, normal mutator threads would fault those pages back in during everyday execution rather than inside a sub-millisecond STW window. Both assumptions turned out to be wrong.
Attaching a custom eBPF probe between the Go runtime’s STW start and stop tracepoints to count major page faults while all goroutines were frozen revealed the smoking gun: the worst 39,902 µs STW pause suffered 228 major page faults, which alone accounted for 39,013 µs (97.8%) of the pause! Resolving the faulting instruction pointers with addr2line pointed straight to runtime.(*spanSet).reset and runtime.nextMarkBitArenaEpoch inside sweepTerm—the brief STW phase at the start of a new GC cycle. Because the Go runtime stores off-heap GC bookkeeping metadata (like span sets and mark-bit arenas) in memory regions that are reused across cycles but touched only during GC transitions, the Linux kernel’s LRU page reclaimer saw those pages sitting untouched between GC cycles, classified them as cold, and paged them out to disk. The moment the next GC cycle froze every goroutine to reset those metadata structures, the STW thread had to page-fault 228 pages back in from disk sequentially while the entire program sat frozen. Swap also degraded normal heap allocations (io.ReadAll on a 511 KiB payload jumped from 3–5 ms to 105 ms on local NVMe swap and 903 ms on network block storage). The takeaway: swap and garbage-collected runtimes with off-heap metadata are a dangerous mix for latency-sensitive services.
How to optimize when you can’t do anything!
Database veteran Hettie Dombrovskaya shares a real-world PostgreSQL war story where every textbook optimization was off the table. The culprit was an unpartitioned 750 GB table holding 16 billion rows. A critical production query that found all records whose [start_date, end_date] interval covered a given target date alongside three equality filters (a, b, c) had started timing out, taking anywhere from several seconds to several minutes. The textbook fix—a GiST range index on (a, b, c, daterange(start_date, end_date))—was impossible during an active incident because building a new index on 16 billion rows would take well over 24 hours. Meanwhile, the best existing B-tree index was on (a, start_date, end_date), where a had terrible selectivity and filtering start_date <= target_date for a recent target date matched nearly the entire 16-billion-row history.
Every standard SQL rewrite hit a wall: adding an arbitrary lower bound like start_date >= target_date - interval '1 year' wasn’t allowed because the customer genuinely needed long-lived historical records; computing MIN(start_date) grouped by a didn’t help because each a bucket contained ancient rows; and creating a partial index covering only the customer’s specific (a, b, c) combinations brought execution under 100 ms in staging, but only helped that single customer. The breakthrough came from looking closely at a spreadsheet the customer had sent over: it happened to list the exact date each of their ~2,000 (a, b, c) combinations was first created! Loading those ~2,000 rows into a tiny lookup table (and in production, exposing the table from another database via a read-only postgres_fdw foreign table since DDL on the primary database was restricted) allowed the query to join against the lookup table and pass that exact per-combination inception date as a tight lower bound start_date >= lookup.first_seen_date into the existing (a, start_date, end_date) index. Query runtime immediately dropped from minutes to under 100 ms. Her takeaway: when you can’t build an index on a massive table, a tiny auxiliary lookup table encoding domain knowledge can act as your “index.”
Four ways to do super fast packet processing in Go
Andree Toonk notes that parsing network packets in Go is easy—the hard part is moving tens of millions of packets per second between the physical NIC and your Go userspace process without the Linux kernel network stack melting your CPU. Because each high-speed packet I/O mechanism has a completely different C API and memory model, Toonk built packetio, a unified Go library that puts four backends behind a single clean interface. AF_PACKET (the standard raw socket used by tcpdump) works everywhere out of the box, but traverses most of the kernel networking stack. AF_XDP hooks an eBPF program into the NIC driver early to redirect frames straight into a shared userspace/kernel ring buffer (UMEM) while leaving normal kernel networking (like SSH and ARP) intact. DPDK bypasses the kernel entirely via poll-mode drivers in userspace at the cost of taking over the whole NIC and a steep learning curve, while mlx5 Direct Verbs talks directly to the hardware queues of NVIDIA/Mellanox ConnectX NICs via rdma-core without needing the massive DPDK runtime. Switching backends in packetio is practically a one-line change to Open().
Toonk benchmarked all four backends in Go across two AMD EPYC servers wired back-to-back with 100 GbE NVIDIA ConnectX-6 Dx NICs blasting minimum-size 64-byte UDP packets at full line rate (148.8 million packets per second), carefully measuring system-wide CPU utilization including kernel softirq cores. On receive (RX), AF_PACKET topped out at just 2.3 Mpps while burning 52.8 CPU cores; AF_XDP hit 146.5 Mpps using 11.4 cores; and both Direct Verbs and DPDK saturated the full 100 GbE line rate at 148.6 Mpps using just 10 and 8 cores respectively. On transmit (TX), DPDK hit line rate using only 3 cores while AF_XDP needed 20 cores. On full L2 packet forwarding (RX + TX), Direct Verbs took first place at 147.4 Mpps across 12 cores. Notably, measuring only the Go process’s CPU hides the fact that AF_XDP spends over half its CPU budget inside kernel softirq handlers driving the NIC ring. Toonk’s rule of thumb: use Direct Verbs on Mellanox/NVIDIA ConnectX hardware, AF_XDP on general Linux NICs when you still need kernel coexistence, DPDK when you need broad multi-vendor poll-mode driver support, and AF_PACKET only for low-rate packet capture.
How Uber Protects Against Retry Storms
At Uber’s microservice scale, naive retries are a recipe for self-inflicted outages: when a caller service receives an error from a downstream dependency, it typically has no idea whether that downstream service failed on its own or is merely forwarding an error from five hops deeper in the call graph. Retrying transient network blips works fine in isolation, but when a deep leaf service is overloaded, independent retries at every layer of a 7-hop call chain multiply exponentially: if each of the 6 upstream layers retries just once (2x), load on the struggling leaf service amplifies by $2^6 = 64\times$ (or $8\times$ across a 4-hop chain). Even enforcing a standard per-client 10% retry budget still compounds across hops ($1.1^3 \approx 1.33\times$ or higher during partial brownouts) because each caller evaluates its budget locally without realizing upstream callers are also retrying the exact same failed request.
Uber solved this cascading amplification by introducing Error Ownership (x-uber-error-claim). The core insight is simple: any given failure in a call tree has exactly one originating service, and only the immediate parent of that originating service should ever be allowed to retry it. A background control-plane component called Service Dependency Analysis correlates inbound and outbound error rates across Uber’s service mesh to determine whether a service is generating errors internally or passively propagating downstream failures; when a service is the true origin of an error, the shared RPC middleware attaches an x-uber-error-claim header to its error response. Its immediate caller sees the claim, performs its configured retries, and—if all retries still fail—strips the claim header before propagating the error further upstream, signaling every ancestor in the call chain not to retry again. Because the logic lives inside Uber’s standard Go/Java RPC middleware, services got protection automatically without application code changes. During a major production incident on November 18, 2025, when a core leaf service sitting 5+ hops deep began failing, retry budgets alone would have amplified its traffic by 46–135%; Error Ownership blocked 9.5 million redundant retry requests, shrinking the maximum retry-storm blast radius from 25 hops down to 3 (and the average from 20 down to 2).
The Real Cost of a Single Insert in Postgres
How many bytes does PostgreSQL actually write to disk when you insert a single 40-byte row? Tiger Data (the creators of TimescaleDB) measures the exact write amplification factor of single-row INSERT statements in PostgreSQL 16 on a realistic table with three B-tree indexes. First comes storage layout overhead: every heap tuple carries a 23-byte MVCC header (t_xmin, t_xmax, t_cid, t_ctid, infomasks) padded to 24 bytes, plus a 4-byte line pointer (ItemIdData) in the page header—turning a 40-byte payload into 68 bytes on the heap (1.7x amplification before touching an index). Next, PostgreSQL updates every index synchronously inside the same transaction: three indexes mean three B-tree traversals and three modified 8 KiB index leaf pages. While a monotonically increasing timestamp index concentrates inserts into the rightmost leaf page sitting hot in shared buffers, a high-cardinality index (like a random UUID or device ID) scatters writes across the entire B-tree, dirtying cold pages all over the buffer pool. And for durability, inserting into the heap plus three indexes emits four separate WAL records per row, each carrying its own 24-byte WAL record header and block reference metadata.
Running a controlled benchmark of 100,000 single-row INSERT transactions and measuring exact WAL generation via pg_current_wal_lsn() and pg_stat_wal reveals the true multiplier. Immediately after a CHECKPOINT, write amplification hits 10.74x because the first modification to any 8 KiB heap or index page after a checkpoint must write a Full Page Image (FPI) into the WAL to guard against torn pages. Even in a warm steady state with zero checkpoints, write amplification is 8.73x (~345 bytes of WAL per 40-byte row—meaning 50,000 inserts/sec generates ~17 MB/s of raw WAL stream alone, before counting dirty 8 KiB buffer flushes by the checkpointer). Dropping all three indexes drops steady-state WAL amplification down to 2.44x, proving that roughly 70% of write volume comes from index maintenance. To tame write amplification, the author recommends a prioritized playbook: batch inserts via COPY or multi-row INSERT ... VALUES (), () so multiple rows share page locks and WAL overhead; drop unused indexes (check pg_stat_user_indexes.idx_scan); enable wal_compression = lz4 or zstd to shrink full-page writes; and tune max_wal_size and checkpoint_timeout to spread checkpoints further apart.
Comments