Enjoy Newsletter #134.

The startup’s Postgres survival guide

Hatchet CEO Alexander Belanger distills roughly two years of running PostgreSQL in production into a pragmatic field guide that’s much easier to scan during an incident than the official manual. On schema design: use identity columns or UUIDs for primary keys, always use timestamptz, and don’t hesitate to reach for jsonb when strict normalization hurts query performance. On reads: join on primary keys, index columns used in ON clauses just like WHERE clauses, and put ORDER BY columns last in compound indexes with matching sort directions. On writes: keep transactions short, never call external services inside a transaction, always use CREATE INDEX CONCURRENTLY on large tables, keep schema migrations additive, and use NOT VALID when adding check constraints. Because Postgres connections are heavy, always use a connection pooler—preferably PgBouncer or pgxpool. Finally, remember that the query planner is a leaky abstraction driven by statistics in pg_stats: debug slow queries with EXPLAIN (ANALYZE, BUFFERS), visualize plans on explain.dalibo.com, and avoid premature micro-optimization.

The advanced half covers batching writes (for instance, pgx’s SendBatch) for roughly 10x higher throughput, monitoring autovacuum via pg_stat_activity, and guarding against transaction ID wraparound. Fix table bloat with pg_repack rather than locking tables with VACUUM FULL, and clear index bloat with REINDEX INDEX CONCURRENTLY. Hatchet relies heavily on FOR UPDATE SKIP LOCKED to power job queues and distributed lease management across instances. Table partitioning lets autovacuum work on smaller chunks independently and makes dropping old time-series data nearly instantaneous. When migrating massive tables, dual-write new rows via a trigger, backfill historical rows in batches outside a long-running transaction, and lean on unique constraints for idempotency.

What is Orasort? understanding Oracle’s Sorting Aglorithm In Simple Terms

Nadeem Shaikh introduces Orasort, a sorting optimization patented by Oracle (credited to Mark Callaghan). According to the post, the 20-year patent expired in 2024, placing the technique in the public domain where it can speed up database sorting by up to 5x. Traditional string sorting compares values byte by byte, so long strings that share a common prefix—such as email addresses or URLs from the same domain—waste many CPU cycles repeating character comparisons. Orasort instead packs an 8-byte slice of each string into a 64-bit integer register and compares all 8 bytes in a single CPU instruction. If those 8 bytes settle the ordering, the comparison finishes immediately; only when they are identical does it advance to the next 8-byte chunk.

The sorting pipeline pairs this with several other techniques. Rather than moving full table rows around during a sort, the database loads only the sort key and row ID (tuple pointer) into an in-memory sort buffer. When the dataset exceeds available RAM, it spills sorted runs to disk and merges them afterward, writing sorted runs out to disk asynchronously while the CPU continues sorting the next batch in memory. While the author notes that open-source databases like PostgreSQL and MySQL have adopted or experimented with similar abbreviated-key sorting ideas—saving substantial cloud CPU costs—the article doesn’t cite primary sources for the 5x figure or the exact patent details, so treat it as an approachable conceptual overview rather than a rigorous historical reference.

Fornax’s Guide To Ridiculously Fast Ethernet

Fornax, creator of the file-sharing service Pixeldrain, leased a 100 GbE server expecting it to saturate the link out of the box, only to watch the machine crash at around 20 Gbit/s. After more than a year of reading NIC manuals, digging through Linux kernel source code, profiling, and even patching the kernel, Fornax hit the full 100 Gbit/s line rate and wrote up the complete tuning guide. At the sysctl layer, pair the BBR congestion control algorithm with the fq packet scheduler so random packet loss doesn’t tank throughput. Enabling tcp_shrink_window (a Cloudflare patch merged in Linux 6.5) cut TCP buffer memory consumption by roughly 80%, curing peak-hour OOM crashes on a server with 1 TB of RAM. Per-connection socket buffers are raised to a 1 GiB ceiling based on the bandwidth-delay product, and tcp_mem is raised to 40–60% of system RAM instead of the ~5% default.

On the hardware side, Mellanox/NVIDIA ConnectX-5 and ConnectX-6 NICs with stock in-tree kernel drivers proved rock-solid. Via ethtool, match the NIC channel count to CPU cores, max out the RX/TX ring buffers, and disable adaptive interrupt coalescing in favor of a fixed 100 µs interval. In the BIOS, set NUMA to NPS1 and disable SMT on 128+ core CPUs. The single most impactful stability fix was disabling the IOMMU (iommu=pt or off), which stopped the NIC from wedging under heavy load. Fornax also eliminated reverse proxies like nginx and Caddy so packets don’t cross the kernel/userspace boundary multiple times, serving traffic directly from Go’s standard HTTP server with SO_REUSEPORT for zero-downtime restarts. HTTP/2 is preferred, while HTTP/3 (QUIC) was dropped because its throughput barely reached half of TCP’s. Finally, run a minimal Debian install on the latest mainline kernel, as Linux 6.8 and 6.19 brought major TCP throughput gains.

Parity before cutover

Alex O’Callaghan recounts inheriting a critical internal service that existed in two parallel universes: a legacy Python 2 version running in production on on-prem VMs, and a modernized Python 3 rewrite living on a long-lived feature branch waiting to be deployed to cloud Kubernetes. This classic “big-bang cutover” approach bundles every possible risk into a single release and delays real-world feedback until the very end. Engineering effort is doubled because every urgent production bugfix has to be written in Python 2 and forward-ported to the Python 3 branch (or worse, forgotten), and subtle behavioral regressions stay hidden until cutover day when too many variables have changed at once to isolate the culprit.

Instead of marching straight toward the cloud cutover, O’Callaghan proposed “taking a step backward to move forward”: shipping the modernized Python 3 codebase onto the legacy on-prem VMs first. Because the legacy OS only supported up to Python 3.6, the team downgraded the branch from Python 3.7 to 3.6 rather than fighting the OS, and added runtime environment configuration so the exact same codebase could deploy to both on-prem VMs and Kubernetes. From that point on, every commit deployed to both environments simultaneously, surfacing behavioral differences incrementally while the legacy Python 2 service remained an instant rollback away. Once the Python 3 service achieved 100% production parity on-prem, cutting over to Kubernetes became a pure infrastructure switch—changing one variable at a time. The takeaway: if a migration branch has been open for months waiting on another team or platform, it’s probably bundling too many changes at once; look for an intermediate state you can ship to production today, even if it feels like a step sideways.

The new Go JSON API: twice as fast, or 1.5x slower?

Starting in Go 1.27, the encoding/json/v2 package ships in the standard library by default. While its Marshal and Unmarshal functions look almost identical to v1, v2 is not a drop-in replacement due to stricter UTF-8 validation, case-sensitive field matching by default, and other behavioral cleanups. Because the legacy encoding/json (v1) package was also re-engineered to run on top of the new v2 engine under the hood, Daniel Lemire benchmarks three configurations: the original Go 1.26 encoding/json engine (built with GOEXPERIMENT=nojsonv2), the v1 API running on Go 1.27’s new v2 backend, and encoding/json/v2 called directly. Benchmarks run single-threaded on an Apple M4 Max and an Intel Xeon Emerald Rapids using three standard simdjson test files (twitter.json, canada.json, and citm_catalog.json) parsed into generic any values.

Simply upgrading to Go 1.27 without touching a line of code makes v1 Marshal up to 2x faster—for example, twitter.json jumps from 198 MB/s to 374 MB/s. Unmarshal via v1 also speeds up on two of the files, though it regresses slightly on float-heavy canada.json (from 128 MB/s down to 106 MB/s). Calling json/v2 directly makes generic Unmarshal 1.5x to 2.3x faster and Marshal 1.2x to 3x faster than the old engine. However, on a round-trip benchmark of 10,000 small strongly typed Go structs, while json/v2 still unmarshals faster, Marshal is roughly 1.5x slower than the legacy engine—contrary to the Go 1.27 release notes claiming struct marshaling performance is at parity. Lemire’s advice: json/v2 is a clear win for unmarshaling, but benchmark your actual struct workloads before migrating write-heavy serialization paths.

A deep dive into DeepSec

Flavio Copes explores DeepSec, an open-source (Apache 2.0) AI-powered security scanner from Vercel. Unlike tools that only review incoming pull request diffs, DeepSec audits an entire repository—including legacy code that has never had a security review. Its pipeline starts with a fast, free local pattern-matching pass that combines file-path globs with regular expressions to flag candidate files. Next, Codex or Claude Code reads the flagged files, traces data flows across the codebase, checks for existing sanitization or auth guards, and records findings with severity, confidence, and remediation advice. A second agent pass independently verifies each finding to filter out hallucinations, a cheaper triage pass assigns P0–P2 priorities, and all state is persisted in .deepsec/ so interrupted scans can resume cleanly.

Running DeepSec against flaviocopes.com using Codex via a standard ChatGPT subscription scanned 729 files and produced 504 raw findings, including 1 Critical and 16 High. Closer inspection showed that 462 of those findings lived in client-side browser mini-tools on the site. Copes then asked Codex to cluster the findings by root cause and separate real production risks from local helper scripts and code examples, narrowing the actionable list down to payment webhooks, course recovery flows, and public forms. DeepSec cannot prove a codebase is secure, still produces false positives, and sends source code to your chosen model provider, but its workflow highlights an important lesson: raw finding counts are mostly noise—always anchor triage where untrusted input touches authentication, private data, money, or infrastructure.

imthenachoman/How-To-Secure-A-Linux-Server: An evolving how-to guide for securing a Linux server.

This popular open-source guide (CC BY-SA 4.0, over 31,000 GitHub stars) compiles practical steps for hardening a Linux server, born out of the author’s personal notes while building a home Debian server. Because Linux security advice is scattered across hundreds of blog posts—and most developers don’t provision bare-metal Linux servers every day—it is easy to overlook basic hardening steps. The moment a server touches the public internet, automated scanners start probing it, and an unhardened box can quietly have its data exfiltrated or get drafted into a DDoS botnet without the owner noticing. The author recommends reading this guide first to understand the why behind each control, and then layering official CIS Benchmarks on top.

The SSH section covers enforcing public/private key authentication, restricting login access via AllowGroups, locking down sshd_config, removing weak Diffie-Hellman moduli, and enabling 2FA. System basics cover restricting sudo and su, sandboxing applications with Firejail, hiding other users’ processes in /proc (hidepid), enforcing strong password policies with PAM, and enabling unattended-upgrades for security patches. Networking covers configuring UFW as a firewall (including fixing the notorious gotcha where Docker bypasses UFW rules by manipulating iptables directly), using PSAD to detect port scans, and running Fail2Ban and CrowdSec to block brute-force and application-layer attacks. Auditing introduces AIDE, ClamAV, rkhunter, Logwatch, Lynis, and OSSEC, followed by a “Danger Zone” for changes that can lock you out if misconfigured—like kernel sysctl hardening, GRUB passwords, and disabling root login. A community-maintained Ansible playbook is also available to automate the setup.

Postgres 19: How Our Advice Has Changed Since We Wrote It

Christopher Winslett at Crunchy Data revisits the company’s classic blog posts on bulk loading, storage, indexing, and partitioning—many originally written in the Postgres 10 and 11 era—to see how their advice holds up against Postgres 19 (beta). Asynchronous I/O (introduced in Postgres 18) dramatically speeds up sequential scans, bitmap heap scans, and vacuuming; Postgres 19 builds on it with autoscaling I/O workers, EXPLAIN (ANALYZE, IO), parallel autovacuum, and disabling JIT compilation by default. The COPY command keeps getting better: SIMD-accelerated CSV parsing, ON_ERROR set_null to survive dirty rows without aborting an entire load, multi-line header skipping, direct COPY ... TO JSON output, and COPY TO support directly on partitioned tables. LZ4 is now the default TOAST compression algorithm (existing toasted values keep their original compression), and a built-in REPACK command (with CONCURRENTLY) finally brings pg_repack-style online table compaction into core Postgres as a native replacement for VACUUM FULL and CLUSTER.

On indexing, BRIN gains minmax_multi opclasses for imperfectly ordered data and bloom opclasses for UUID equality lookups. B-tree skip scans (added in 18) allow a multi-column index to serve queries that filter only on trailing columns when the leading column has low cardinality, letting teams drop redundant single-column indexes. Native partitioning adds MERGE PARTITIONS and SPLIT PARTITION, though both currently take an ACCESS EXCLUSIVE lock and belong in maintenance windows. The takeaway: the core fundamentals haven’t changed—use COPY for bulk loads, store JSON in jsonb, and partition primarily for data lifecycle management—but Postgres 19 makes executing them much smoother. Keep the same mental model, update your commands, and verify with EXPLAIN (ANALYZE, BUFFERS, IO) after upgrading.

How we saved 100 terabytes of memory by optimizing 1.1.1.1’s DNS cache

Big Pineapple, the Rust platform powering Cloudflare’s 1.1.1.1 public resolver and other DNS services, holds over 250 billion cached DNS entries across its global fleet. At that scale, wasting a single byte per cache entry costs more than 250 GB of RAM globally. Although cached DNS entries are completely immutable once inserted, their Rust structs were still using Vec and String fields, each carrying an unused 8-byte capacity field and potential heap over-allocation. Worse, the enum representing DNS record data was sized to its largest variant, padding tiny A and AAAA records with dead space. The team built a realistic traffic benchmark paired with a custom tracking allocator to measure exact per-entry heap usage alongside insert throughput and lookup latency.

They rolled out five progressive optimizations: replacing Vec<T> and String with boxed slices (Box<[T]>) and Box<str>; flattening three separate record lists (answer, authority, additional) into a single allocation indexed by u16 offsets; omitting the record’s owner name when it matches the query name; boxing oversized enum variants like TXT and NAPTR; and finally storing records as pre-serialized DNS wire-format bytes packed into a single contiguous allocation. With wire-format storage, common records like A, AAAA, and TXT can be memcpy’d straight into outbound DNS response packets without re-serializing. In benchmarks, per-entry memory dropped by 56% (from 953 bytes down to 420 bytes), insert throughput rose 43%, and lookup latency fell 19%. In production, p99 resident memory per server dropped from 9.3 GB to 5.3 GB—saving roughly 100 TB of RAM across the fleet (the equivalent of ~130 servers), which Cloudflare plans to reinvest into larger caches for higher hit rates.

Agentic Code Quality

Addy Osmani argues that traditional human line-by-line code review cannot keep pace with the volume of code generated by AI agents, so quality control must shift into the harness, environment, and automated verification systems surrounding the agent. His central thesis: in an agentic workflow, software quality is bounded by the quality of the constraints you place around the agent. Those constraints include unit tests, property-based tests, acceptance tests, mutation testing (to verify your tests actually catch injected bugs rather than merely exercising lines), and static complexity metrics like cyclomatic complexity. How much human code reading you can safely skip depends on the blast radius: throwaway prototypes need very little, whereas auth, billing, and core infrastructure demand either human review or exceptionally tight automated invariants. The goal is an environment that gives the agent fast, reliable feedback and makes failures cheap and contained.

Constraints should operate at three boundaries: up front (via specs, type systems, and architecture rules), continuously during execution (as compiler and test feedback), and at the production gate. Every layer of the verification stack should own a distinct concern—from type safety and lint rules to performance budgets and security scanning—so compilers, failing tests, and CI policies exert early “backpressure” on the agent long before a human is asked to review a PR. When code generation velocity outstrips a team’s verification capacity, you have only three choices: scale up automated verification, slow down generation, or lower your quality bar (the worst option). Because human attention is now the scarcest resource in engineering, save it for high-leverage decisions about product intent, taste, and system architecture, and pull humans into the loop only after the automated guardrails pass.

Bonus

Videos: How the JVM Really Works

ByteByteGo’s video explains how the Java Virtual Machine works under the hood, from compiling Java source code into bytecode to executing it on the target machine.

Images: MCP vs RAG vs AI Agents 9 Distributed Systems Patterns You Should Know