CTO Corner: this month in data (April 2020 edition)
Here are some of the most interesting articles about databases and distributed systems we’ve found in the past month.
The Timescale team includes a bunch of people who care deeply about databases, distributed systems, and generally everything around data.
Many of us have an academic research background – some of our first engineering hires came out of the computer science department at Princeton, where I’m also a professor – while others are industry veterans with deep expertise.
As such, we stay connected with what’s happening in both the academic and developer communities. So each month, I’ll list and summarize a non-exhaustive assortment of the more interesting papers and articles we’ve read over the last few weeks (and invite you to share your own).
Academic Papers
Millions of Tiny Databases
Three Amazon Web Services engineers wrote this really interesting paper at USENIX’s NSDI '20 conference that describes the Physalia databases, which serves as a configuration master for AWS’s Elastic Block Store (EBS). Because EBS is deployed throughout AWS and requires extremely high availability, the goal wasn’t a single monolithic database (even if a geo-replicated one, see the CAP Theorem), but instead millions of little databases to reduce the “blast radius” of hardware failures.
Firecracker
Another AWS paper from NSDI '20, Firecracker describes the next-generation virtualization/isolation architecture for AWS Lambda. The paper talks about the various limitations that Lambda encountered in its v1 (namely, using Linux containers between functions, and VMs between accounts), and how Firecracker addresses these. And with a memory footprint of only 3MB per microVM (vs. 130MB for QEMU VM). And written in Rust, for you language folks!
Adrian Colyer’s the morning paper
Speaking of the above two papers, Adrian has a wonderful site and mailing list for great, highly readable summaries of recent (and sometimes not-so-recent) papers, with a particular focus on systems and occasionally AI/ML. Adrian reviewed both the Millions of Tiny Databases and Firecracker papers, and his site features dozens more.
Articles from the Industry
History of parallelism in PostgreSQL
This post from Amit Kapila tracks parallelism in PostgreSQL (the foundation of TimescaleDB), starting in PG 9.6. It also dives into what the future may hold for our favorite database, including for PG13 and beyond. This is something that we’ve thought about a lot ourselves, especially as we leverage parallelism and async operations in parallel queries across data nodes in multi-node TimescaleDB.
Dropbox’s new sync engine
I enjoyed this post from Dropbox on the architectural implementation of their new sync engine, particularly around some of the general engineering deliberations behind their decision to completely rewrite code at the core of all of Dropbox’s desktop clients. And another bet on Rust (migrating from Python).
Oldies but Goodies
How complex systems fail
This paper has long been read by aspiring computer scientists and engineers; in fact, my cofounder Ajay and I both read it in the late 1990s as undergrads taking MIT’s 6.033 course with Jerry Salzer. The author, Richard Cook, wrote this from his career thinking about patient safety, which seems particularly relevant in the midst of the COVID-19 global pandemic.
For a longer take, Richard Cook spoke on the subject at O’Reilly Velocity 2012.
Data Domain’s Deduplication File System
Academic conferences often give out “Test of Time” awards 10-15 years after papers are first published. This year’s FAST '20 conference awarded its Test of Time to this paper from FAST 2008, which describes various data structure, layout, and caching optimizations to reduce disk access by 99% for deduplication workloads. (Data Domain was originally founded by my Princeton colleague, Kai Li, and went on to become a highly successful business unit of EMC.)
Closing thoughts
I’d love to hear about what you thought of these papers, as well as get your suggestions for articles, sites, and videos we should read/watch and highlight. You can find me on Twitter (@michaelfreedman) or in the Timescale Community Slack.
Until next month!