Optimize your storage costs with TimescaleDB’s data tiering functionality
Gain significant improvements in Total Cost of Ownership by moving historical or infrequently accessed data to more cost effective storage.
Last month, we officially released TimescaleDB 1.5 with some pretty exciting new features. The theme of the release was helping our users gain significant improvements in Total Cost of Ownership (TCO) which means giving you more control over how and where you store your data.
One way you can do this is by leveraging our new compression capabilities. At a high-level, native compression allows users to store more data in less actual storage. Our early internal benchmarking + beta user testing resulted in an average compression ratio of 20x. This means you could fit 1GB of data on 50MB of storage!
You can read more about native compression in our detailed post and documentation. Now, we’d like to talk a little more about TimescaleDB’s new data tiering functionality called move_chunk.
Reduce storage costs
TimescaleDB’s architecture is comprised of chunks (many individual tables holding the data) and hypertables (an abstraction or a virtual view of many chunks). This is where we get the name move_chunk.
The move_chunk functionality allows users to move individual data chunks between Postgres tablespaces. Essentially, as data ages and you’re not accessing it as frequently, you can change the type of storage you keep this data on by simply adding new tablespaces backed by different classes of storage.
In other words, you can closely manage costs without needing to purge data. You have the option to keep data around, but you don’t need to (and shouldn’t) pay a lot to do so. This flexibility is especially beneficial in environments where historical data analysis is a key part of your business.
The other major advantage associated with move_chunk is query performance. When moving chunks, you spread the read IO load across multiple storage mounts and disk arrays, ensuring that any large hypertables you’re using to perform analytics don’t overrun the disk backing the default tablespace.
NOTE: Move chunks is a TimescaleDB Enterprise feature. If you already have the Enterprise version installed, upgrade to TimescaleDB 1.5 to get started. If you’re interested in trying out the Enterprise version of TimescaleDB, sign up for a 30 day trial. If you have questions about how to use move_chunk with your current deployment (i.e. if you are self-managing in AWS, GCP, Azure or other), we’re happy to help! Contact us here or post in our Slack channel.
How it works
To help you visualize how moving chunks actually works within TimescaleDB, I’ll walk through an example where we are tiering our data based on usage and access patterns.
As you can see in the diagram below, as chunks age, we move them from the left (fast storage) to the right (slow/least expensive storage). This is what you can expect with each position:
- Fast Storage: new recently ingested data, and the data we often query (0-3 months old)
- Slow Storage: historical data that we don’t query very often, but still need access to (4-12 months old)
- Slowest Storage: historical data that we don’t really need, but we choose or need to retain (>1 years old)
Your setup might look different depending on how much data you are accessing and how often you’re analyzing your data. This feature is customizable, so you can adjust it based on your needs.
Follow four simple steps
Now that we know the benefits of moving chunks and how it works, let’s look at how to get started.
At a high-level, you will:
- Create a tablespace in Postgres
- Determine which chunks to move based on age / usage
- Move the predetermined chunks to a different tablespace
- Confirm that the chunks are now on the correct tablespace
...it’s as simple as that! Currently, users perform this function manually, and you can choose to move chunks back to their original tablespace at any time (automated options coming in 2020).
View the full documentation on move chunks for more detailed instructions.
Where to go from here
To learn more about what’s new in TimescaleDB 1.5, including move chunks and native compression, join us Thursday, Nov. 14 for “How to Reduce Your Database Total Cost of Ownership with TimescaleDB.” We’ll send the recording to all registrants, so sign up even if you’re unable to attend live.
To upgrade to TimescaleDB 1.5, follow these instructions.
If you are brand new to TimescaleDB, get started here. (Reminder: to access move chunks, you will need TimescaleDB Enterprise. Request your 30 day trial license here.)
Finally, to stay up to date with all things time-series, sign up for our newsletter below!
