Newsletter #109
Enjoy Newsletter #109.
A History of IDEs at Google
Laurent Le Brun, who spent 12 years working on developer tooling at Google, recounts the journey from a fragmented ecosystem of editors to a single IDE used by nearly the entire company. Back in 2011, when someone asked whether Google should standardize on one IDE, the near-unanimous answer was “no” — Jeff Dean argued that forcing a whole team onto a single editor was a recipe for misery. The price of that freedom was that every useful integration — Bazel, code search, formatters — had to be rebuilt for every editor. Around 2013, Cider launched as a web-based editor initially meant for quick doc edits, but thanks to a backend capable of indexing Google’s massive monorepo, it gradually won over Go and then Java developers.
In 2020, the team decided to replace Cider’s custom frontend with the VS Code core while keeping Google’s proprietary backend; Cider V opened to 5,000 engineers in 2021 and accounted for roughly 80% of main-repo development by 2023. Once most engineers shared a single platform, teams had strong incentives to build internal extensions — roughly 100 within two years — and AI features like code completion and automated review-comment resolution could be rolled out company-wide far faster. The takeaway: standardizing tooling creates massive leverage, even if the up-front investment is high and not every company can pull it off.
Finding zombies in our systems: A real-world story of CPU bottlenecks
In early 2025, Pinterest’s Ray ML training jobs kept crashing due to flaky network connections, and it took the Kubernetes platform team over three months to track down the culprit. The first clue was that the EC2 ENA network driver reset itself whenever its transmit queue stalled for more than 5 seconds, a classic symptom of CPU starvation; weirdly, the resets only happened in a single availability zone even though cluster configs looked identical, and rebooting nodes only bought about a week of relief. On 96-vCPU GPU instances, aggregate perf profiles looked completely normal; only per-core, per-second mpstat monitoring revealed a single CPU core pinned at 100% system time for several seconds right when each reset occurred. The team set up automated 2-minute perf record captures and used Netflix’s Flamescope to scrub to the exact second of the spike.
The culprit: right before every reset, kubelet spiked to ~6.5% total CPU (normally under 1%), spending nearly all of that time inside the kernel’s mem_cgroup_nr_lru_pages function. The kernel was tracking nearly 70,000 memory cgroups even though only 240 were actually active — “zombie memcgs” — and walking that list pinned a single core long enough to starve the network driver thread sharing it. The root cause traced back to the AWS Deep Learning AMI (Ubuntu 20.04) base image, which shipped with the Amazon ECS agent enabled by default; unable to join an ECS cluster, the agent crashed in a tight loop, leaking a zombie cgroup on every restart. The fix was simple: disable the ECS agent systemd unit and reboot the fleet. As for why the “healthy” availability zone never broke? Another bug there prevented the ECS agent from ever starting in the first place.
Programming Still Sucks.
Steven Langbroek wrote this essay to answer the familiar question “Will AI take programmers’ jobs?”, and his answer is no: the real culprit is greed, merely dressed up in a new coat called “automation.” Starting from a casual question at a birthday party, he paints software engineering as a burning ship: no map, no crew, sails mounted backward, useless navigation instruments, and a predecessor who abandoned ship after leaving behind half-baked “initiatives.” Layoffs, he argues, are signed off by people who know the consequences full well but bow to spreadsheets and quarterly targets, telling themselves they’ll “fix it later” — a “later” that never comes.
The sharpest pain point is what happens to junior developers. A junior’s value isn’t the code they write today, but the fact that one day they become the senior who understands every dark corner of the system; eliminate apprenticeships to optimize short-term output, and a few years later there is nobody left to take the helm. The emotional heart of the piece is Sara, an older infrastructure specialist who inherited her knowledge from Ben and keeps the company alive via a mysterious cron job nobody else understands. When Sara retires, the company will desperately need a replacement after having dismantled the very pipeline that could have trained one. The disaster, Langbroek concludes, isn’t technological — it’s organizations actively sabotaging themselves.
How Container Filesystem Works: Building a Docker-like Container From Scratch
Ivan Velichko’s tutorial on iximiuz Labs builds a container by hand using nothing but standard Linux tools like unshare, mount, and pivot_root — not to replace Docker, but to give readers a clear mental model of what happens under the hood of docker run. The foundation is the mount namespace: it isolates the mount table — the list of mounted filesystems visible to a process — rather than the underlying physical filesystem itself, something you can verify with findmnt across two namespaces. Velichko emphasizes mount propagation across private, shared, and slave modes: if ignored, mounts created inside a container can leak out to the host, and pivot_root — the safer alternative to chroot — requires that the parent mount not be marked shared.
The hands-on walkthrough mirrors what runc actually does: prepare a rootfs directory from an Alpine image; create mount, pid, cgroup, uts, and net namespaces; mount the /proc, /dev, and /sys pseudo-filesystems with appropriate restrictions; bind-mount container-specific /etc/hosts, /etc/hostname, and /etc/resolv.conf files; swap the root filesystem via pivot_root; and harden the container by remounting sensitive /proc paths read-only and masking others with tmpfs or /dev/null. On the security side, an untrusted rootfs can contain symlinks pointing outside the container, so production runtimes use openat2() with RESOLVE_NO_SYMLINKS and operate via file descriptors to prevent TOCTTOU race conditions. Finally, bind mounts are the exact same mechanism powering Docker’s -v flag when sharing host volumes with a container.
Using AI to write better code more slowly
Amid the “vibe coding” frenzy, Nolan Lawson asks the opposite question: if so many people are using LLMs to crank out sloppy code as fast as possible, can we use the exact same tools to write better code more slowly? His answer is yes. His workflow centers on multi-model code review: having Claude, Codex, and Cursor Bugbot independently review a pull request, rank issues as critical, high, medium, or low, and then manually synthesizing and verifying their findings to filter out false positives. Using distinct models and wiping context between rounds reduces the chance that all of them hallucinate the same nonexistent bug. He then has the agent fix the critical and high issues, repeats the review loop until none remain, weighs whether the minor issues are worth addressing, and throws out the entire pull request if the process exposes a fundamental architectural flaw.
Interestingly, his overall delivery speed hasn’t increased at all: the review process routinely uncovers pre-existing bugs in surrounding code, turning every new feature into a side quest of writing tests and cleaning up legacy debt. For anyone letting agents open hundred-line pull requests they don’t fully grasp, Lawson advises slowing down: ask the agent how the code works and where it could break, have it generate documentation with Mermaid diagrams, or use Matt Pocock’s /grill-me skill until you understand every detail. This approach burns more tokens, but pays off in a deep understanding of failure modes and a much healthier codebase.
How CockroachDB Built Vector Indexing at Scale
ByteByteGo breaks down C-SPANN, the vector indexing algorithm CockroachDB designed in-house because no off-the-shelf approach fit a distributed transactional database. Vectors have no natural total ordering, rendering B-trees useless, so every vector index trades exactness for speed via approximate nearest neighbor (ANN) search. CockroachDB set six hard constraints: no central coordinator node, no reliance on massive in-memory structures, minimal network hops, shardable across nodes, no hot spots, and real-time incremental updates — ruling out both traditional HNSW and IVF. C-SPANN instead uses a wide, shallow hierarchical K-means tree: with a branching factor around 100, a million vectors need just three levels and ten billion vectors need five.
The key architectural decision is storing each partition as ordinary key-value rows inside CockroachDB, allowing the index to automatically inherit range splitting, load balancing, replication, and multi-region placement without a separate infrastructure stack. To cut storage overhead, RaBitQ compresses 1,536-dimensional vectors from ~3 KB down to 200 bytes (a 94% reduction), then uses the full-precision vectors to rerank a small candidate set. For multi-tenancy, prefix columns like (user_id, embedding) create isolated per-tenant trees, combining with REGIONAL BY ROW to satisfy data-residency rules. The trade-offs are stated plainly: C-SPANN trails dedicated vector databases on raw tail latency and currently only supports Euclidean distance in preview, but it is a great fit for workloads where vectors need to live alongside transactional data.
This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.
Comments