dorkusmalorkus.dev

Mind the Gap

2026-01-07T00:00:00Z

SQL stories seem to be very popular on this blog... Maybe it's because they're so relatable and everyone joins in.

I crack me up.

Thank you, thank you. I'll be here all week. Aaanyway... If you haven't read my previous tiny introduction to the Transactional Outbox Pattern, go and give that a quick read. It's okay. I'll wait. Okay, caught up?

Cool. Let's dive into some of the challenges, quirks and gotchas that my team and I experienced when implementing this pattern in a fairly large service.

Pretty much how we felt at one point in this story. (image credit: travelkillsme.com)

Indexs? Indices? Indexi? Indexes! :)

As a reminder from the previous post, here are the DB tables we're dealing with:

Anatomy of the outbox_table table

However, what I didn't include in the previous post is additional info about indexes on the outbox_table table (and yes, I looked it up, apparently 'indices' is math-related, whereas 'indexes' is appropriate for DBs. The more you know and all that).

Another thing I didn't include is that we're working with MySQL 8.0 with InnoDB engine. Why does that matter? Oooh, you'll see soon enough, dear sweet innocent reader.

We have this migration for our outbox_table table:

1  CREATE TABLE `outbox_table` (
2    `id` bigint unsigned NOT NULL AUTO_INCREMENT,
3    `payload` blob NOT NULL,
4    `headers` json NOT NULL,
5    `payload_type` varchar(255) NOT NULL,
6    `published_at` datetime DEFAULT NULL,
7    `created_at` datetime NOT NULL DEFAULT (UTC_TIMESTAMP()),
8    PRIMARY KEY (`id`),
9    INDEX `idx_outbox_table_created_at_published_at` (`created_at`, `published_at`));

You'll notice that we created a composite index on the created_at, published_at fields. This bit is important, so remember it for later.

Locked in

Our outbox_publisher's job was the following:

Fetch all records in the outbox_table table that were not yet published, with preconfigured batch limit (let's say 25). So, any rows within our limit that had a NULL published_at
Publish all the events for any rows fetched.
Update the published records with the appropriate published_at timestamp.

As most of you will know, developers add indexes to DB tables to speed up certain queries. In this case, our select query looked like this:

1SELECT *
2  FROM outbox_table
3  WHERE published_at IS NULL
4ORDER BY created_at
5LIMIT ?
6FOR UPDATE SKIP LOCKED;

Those DBAs-in-waiting among you will notice a few things right away:

1SELECT … FOR UPDATE

A select for update statement does the following:

It acquires exclusive row locks on the selected rows
As such, it guarantees that no other transaction can modify or delete those rows until the current transaction is committed or rolled back.

Prevents: So this is a protection mechanism against:

two (or more) workers selecting the same unpublished rows simultaneously before either worker updates them.
race conditions between the SELECT and UPDATE statements

1SKIP LOCKED

This tells the DB to not wait up by the phone if this row is ~~out with the bois~~, I mean, locked by another transaction. It prevents workers from blocking each other, and also allows you to scale horizontally by adding more workers since they can read from the same table effectively.

Without SKIP LOCKED, the second worker would either block until the first worker commits or even worse, we'd have a deadlock on our hands.

Shards and Sharts

Why do we need all this complexity? Why not just do a simple select and update? Glad you asked! See, we're not dealing with your average, run-of-the-mill, small-scale, grandmother's application here. We're dealing with a high-throughput system with a sharded database setup.

I said sharD. Stop being gross.

Potential accidental underwear soiling jokes aside, a sharded database means that our data is partitioned across multiple database instances, or shards, based on a certain key (like a user ID). This allows us to distribute the load and improve performance, but it also introduces some challenges when it comes to ensuring data consistency and avoiding conflicts between workers.

I can already hear some of you thinking: "Why not just have one worker per shard?" Well, yeah... that's what we have. However, in this wonderful world of microservices and container orchestration, things can get a bit... unpredictable. As such, there is a non-zero chance that two workers could end up working on the same shard at the same time. Hence, the need for SELECT FOR UPDATE SKIP LOCKED.

Gaps & Ghosts

Alright, so we have our select query with FOR UPDATE SKIP LOCKED, and we're ready to roll. Or so we thought. After shipping this setup to production, we started noticing some weird behavior, mostly with latency spikes on the INSERT operations on the outbox table.

After doing the initial checks, we started to suspect our old friend the SELECT FOR UPDATE query. To confirm our suspicions, we enabled the general query log in MySQL and started analyzing the logs. And sure enough, we're seeing quite a bit of locking happening on some of these rows. But wait, didn't we instruct the SELECT query to skip locked rows? Yes, we did. So what does this have to do with the INSERT operations?

Turns out, there's a little-known behavior in MySQL's InnoDB engine related to gap locks and the way indexes work. A gap lock is a lock on a gap between index records OR a gap before the first or after the last index record. This last one is important.

When we have a composite index on created_at and published_at, the SELECT FOR UPDATE query can acquire a gap lock on the index ranges, not just the rows themselves. It does this to prevent phantom reads.

What's a phantom read? It's a concurrency phenomenon where a transaction reads a set of rows that satisfy a certain condition, but another transaction inserts new rows that also satisfy that condition, leading to 'phantom' rows that weren't there on the initial SELECT.

Typically, range selection phantom issues are handled by setting the REPEATABLE_READS at the MySQL server level. With the InnoDB engine, this configuration uses something called 'next-key locking'. According to the docs, this strategy combines index-row locks with "gap locks" to lock the space between rows, preventing other transactions from inserting new rows within that range.

But are we dealing with range selections? Well, not really.

1SELECT *
2  FROM outbox_table
3  WHERE published_at IS NULL
4ORDER BY created_at
5LIMIT ?
6FOR UPDATE SKIP LOCKED;

So what gives? Not every single INSERT was getting issues, just some of them. Once again turning to the almighty docs, it turns out that the REPEATABLE_READ isolation level and next-key locking can still lead to gap locks being acquired in certain situations, even if we're not doing range selections. InnoDB will lock the space between the records in the index, in order to protect against phantom inserts. But we're using the skip locked clause, so why are we still seeing issues?``

After digging deeper, we found out about something called the 'supremum lock'. I'll let the official documentation explain it best:

Suppose that an index contains the values 10, 11, 13, and 20. The possible next-key locks for this index cover the following intervals, where a round bracket denotes exclusion of the interval endpoint and a square bracket denotes inclusion of the endpoint:

(negative infinity, 10]
(10, 11]
(11, 13]
(13, 20]
(20, positive infinity)

For the last interval, the next-key lock locks the gap above the largest value in the index and the “supremum” pseudo-record having a value higher than any value actually in the index. The supremum is not a real index record, so, in effect, this next-key lock locks only the gap following the largest index value.

This is particularly relevant to us because it can cause locking when attempting to insert records at the end of an index range. So we basically had our insert and select queries doing the following:

C'mon. You know I had to.

MAX to the rescue?

What do we know so far about the issue?

We have INSERT operations that are getting blocked due to next-key locks, since they affect the index on the table.
InnoDB is doing its job in protecting against phantom reads and offering reliable transaction isolation.
This causes issues for our INSERT operations, leading to deadlocks and just generally a bad time.
Thankfully, thanks to the outbox_publisher design, no events are lost since the entire transaction rolls back if the insert fails.

So how did we fix this? (yeah, we did fix this, otherwise I wouldn't be writing this post, now would I?)

Given our problem relates to the ranges covered by the composite index on created_at and published_at, we decided to try and do something about that instead. What if, instead of selecting the entire range covered by the index, we ... you know... don't? We basically figure out the last committed transaction's primary key and select everything before that?

1SELECT MAX(id)
2  FROM outbox_table
3  WHERE published = 0

(the keen readers among you might have noticed we no longer care about the published_at column, in favour of the published boolean column. This was part of the multiple attempts to improve the situation, but this post is already getting too long so let's just run with it, k? k!)

Anyway, with that max ID in hand, we can now do a select like this:

1SELECT *
2  FROM outbox_table
3  WHERE id < <max_unpublished_id>
4  AND published = 0
5ORDER BY id
6LIMIT ?
7FOR UPDATE SKIP LOCKED;

By using our max_id, we are explicitly pruning the search space the DB has to do. So, InnoDB does not need to lock the gap above the highest ID in the table, only the rows below it.

So, by using the following:

LIMIT clause
id < max_unpublished_id
SKIP LOCKED
proper indexing

Another crucial point is that since we're working with id, which is an auto-incremented primary key. Because of this predictable range, we can now prevent InnoDB from locking the gap above our selected rows and no longer stalling our INSERT operations. Yay and such!

So what's next?

Well, ideally, we shouldn't even have had to do any of this. The keen among you may have noticed that the main reason we were running into these issues is because of that darn SELECT FOR UPDATE itself. If we could avoid using it, we could avoid all these issues. So... did we remove it? That, my friends, is a story for another day.

Stay tuned for the SQL! (haha, get it? SQL? Sequel? sigh. Never mind...I'll see myself out.)

Goin' Postal

2025-12-10T00:00:00Z

Intro: Dopamine & Distractions

Welcome to my life

As the year draws to a close, the usual talk of New Year's resolutions starts buzzing around.

I never understood these motivational videos on Youtube or self-help blogs that talk about 'sharing your plans' with your friends. I guess the idea is to create accountability so you'll actually follow through. For me though, this has the opposite effect. I get the dopamine hit from telling everyone, then my brain full-on backstabs my ambitions and witholds the sweet sweet happy hormones. So naturally, I don't do the thing I talked about.

Far-fetched analogy, I know, just bear with me ok? At least it's somewhat better than your average LLM (not really but hey).

Why am I rambling on about this? Well, this pattern is basically what my backend services were stuck in. They would tell everyone about their cool sh*t (aka publishing event messages) and get their little dopamine hit, but then their actual work (DB transaction) would fail.

This was always "fun", since it led to lovely bugs. Much like you and your friends, where you continued living your life assuming your cool friend was off doing all the cool things they were bragging about, the downstream consumers of the event messages were now assuming that the publisher service had successfully completed their transaction. When it then turns out their dopamine addicted *ss didn't actually do the work, we had some neat issues on our hands. Partial failures, data inconsistencies across services, retries causing deduplications, inconsistent reads in downstream systems... you name it!

Transactional Outbox Pattern

Your downstream consumer service be like

Because we got pretty sick of debugging these type of issues, we set about investigating how to avoid some of these issues in the first place. After all, why would you emit an event when your main DB transaction might fail and be rolled back?

Enter the OutBox Publisher Pattern. One of the key guarantees this pattern provides is to use the primary database transaction to record both the domain change AND the intent to publish the event.

It does this by writing both the initial record (e.g. a croissant order) AND the event message metadata and payload to the DB. Next, an asynchronous process can then poll or stream entries in the outbox table, handle the publishing of the event and finally mark the event as dispatched. A secondary process can then periodically clean up the dispatched records so your outbox table doesn't grow too large.

Anatomy of the Outbox Table

Let's dive into a simplified example. Lets say we are running a bakery frontend with an async event system to process the orders in our backend bakery (just roll with it okay, I'm running low on creative analogies here)

In our primary service we have a database with two tables:

Basic table setup

The main constraint for the outbox_publisher table is that each payload is IMMUTABLE. We don't want to mess with event payloads or metadata at all. IF the original record (aka the croissant order) is updated or deleted, we enter a new event in the outbox_publisher table which the downstream consumer services can then handle separately.

Publish & Purge

Now that we have our primary record AND event record successfully written to the database, we can kick off our little postal worker process. This worker can poll for any new entries to the publisher table, handle the publishing of the event, then mark the event as published using the boolean column).

This works beautifully. However, over time and in high-throughput systems, this outbox table can grow quite large. Sure, now we can look at partitioning or other ways to handle that read load, but for simplicity we can just delete any of the dispatched event records.

To do this, we can have a second worker run periodically, check for all outbox records marked as published and delete them. To give ourselves some margin we can only delete records that were published more than 2 weeks ago. This is neat because it allows us more visibility for debugging.

Computer Says No

So when does something like the outbox pattern not fit the bill? When dealing with relational databases that have strong transactional guarantees, this pattern is amazing. For non-transactional systems or in systems where ordering is critical, this might not work very well. But then again, if ordering is critical, what are you doing with async event publishing anyway? :)

In that case, it's better to consider things like CDC pipelines, streaming pipelines or log-based setups. We won't dive into those here.

Enough talk. Show Code

Okay okay, chill. Here it is:

 1class Order(Base):
 2    __tablename__ = "orders"
 3    id = Column(Integer, primary_key=True)
 4    product_name = Column(String, nullable=False)
 5    quantity = Column(Integer, nullable=False)
 6    price = Column(Float, nullable=False)
 7    created_at = Column(DateTime, default=datetime.datetime.utcnow)
 8
 9class Outbox(Base):
10    __tablename__ = "outbox_table"
11    id = Column(Integer, primary_key=True)
12    payload = Column(LargeBinary, nullable=False)
13    headers = Column(JSON, nullable=False)
14    created_at = Column(DateTime, default=datetime.datetime.utcnow)
15    published = Column(Boolean, default=False, index=True)
16    published_at = Column(DateTime)
17
18def create_order(product_name, qty, price):
19    # Create both the order and outbox record and commit to DB
20    s = Session() # fake session from SQLAlchemy for brevity
21    try:
22        order = Order(
23            product_name=product_name,
24            quantity=qty,
25            price=price
26        )
27        s.add(order)
28        s.flush() # order.id now available in SQLAlchemy session object
29
30        payload = {
31            "event": "OrderCreated",
32            "data": {
33                "order_id": order.id,
34                "product_name": order.product_name,
35                "quantity": order.quantity,
36                "price": order.price,
37                "created_at": order.created_at.isoformat()
38            }
39        }
40
41        outbox_entry = Outbox(
42            payload=json.dumps(payload).encode("utf-8"),
43            headers={"content_type": "application/json"}
44        )
45
46        s.add(outbox_entry)
47        s.commit()                   # atomic
48        return order.id
49
50    except Exception:
51        # NOTE: entire transaction gets rolled back, so entity AND event, even if only one of them fails
52        s.rollback()
53        raise
54    finally:
55        s.close()
56

Next, we have our OutboxPublisher worker that runs continiously in a separate deployment.

 1
 2class OutboxPublisher:
 3    def __init__(self, outbox, publisher, batch_size=50, delay_sec=1):
 4        self.outbox = outbox
 5        self.publisher = publisher
 6        self.batch_size = batch_size
 7        self.delay_sec = delay_sec
 8
 9    def run(self, should_stop):
10        while not should_stop():
11            start = time()
12            more = self.process_batch()
13            elapsed = time() - start
14
15            if not more:
16                sleep(max(0, self.delay_sec - elapsed))
17
18    def process_batch(self):
19        with session.begin():
20            messages = self.outbox.get_unpublished(self.batch_size)
21            if not messages:
22                return False
23
24            published = []
25            for m in messages:
26                try:
27                    self.publisher.publish(m.payload, m.headers)
28                    published.append(m)
29                except Exception:
30                    continue
31
32            self.outbox.mark_published(published) # simply updates the outbox_record to `published = 1` and the `published_at with current timestamp`
33
34        return len(messages) == self.batch_size

Lastly, a cleanup worker that can be configured to delete published records that are of a certain age.

 1def delete_published_before(outbox, before_date, limit=500):
 2    total = 0
 3    while True:
 4        with session.begin():
 5            deleted = outbox.delete_published_before(before_date, limit)
 6        if deleted == 0:
 7            break
 8        total += deleted
 9    return total
10

Closing Notes

So, with just a handful of code and a bit of infra / DB work, we're now in a much better place where we can reliably tell the world about the cool shit we're doing, and make sure we've actually done it! :)

Even if our publisher crashes mid-flow, this pattern should allow enough resilience to never lose messages and give our developer team same extra buffer to debug and investigate.

In the next part of this two-part series, we'll dive into some interesting operational considerations and gotcha's that have bitten us in the backside since implementing this pattern in some of our high-throughput services and how we mitigated some of the most common issues. Stay tuned for that, hopefully it won't take another year!

Facing the SnowStorm: Part 2

2024-12-27T00:00:00Z

Recap from Part 1

Welcome back! In part 1 of this two-part series on Snowflake Query Performance, we learned that simply upping the size of our Snowflake virtual warehouse didn’t do much for improving query performance. So, if it's not compute power, what's going on? The issue might actually lie in how the data is structured. In this part, we’re going to dig into clustering and partitioning to see if these could be the missing pieces to the puzzle. Let's see if we can finally get to the bottom of this.

Snowflake Micro-Partitions and Data-Clustering

Micro-partitions

All data in Snowflake tables is automatically divided into micro-partitions, which are contiguous units of storage. Groups of rows in tables are mapped into individual micro-partitions, organized in a columnar fashion. This size and structure allows for extremely granular pruning of very large tables, which can be comprised of millions, or even hundreds of millions, of micro-partitions. Snowflake stores metadata about all rows stored in a micropartition, including:

- The range of values for each of the columns in the micro-partition.

- The number of distinct values.

- Additional properties used for both optimization and efficient query processing.

According to Snowflake, these micropartitions are applied automatically to any data inserted into any table, and is primarily used for the goal of Query Pruning.

The micro-partition metadata maintained by Snowflake enables precise pruning of columns in micro-partitions at query run-time, including columns containing semi-structured data. In other words, a query that specifies a filter predicate on a range of values that accesses 10% of the values in the range should ideally only scan 10% of the micro-partitions.

Rabbit Hole Interlude: Partitions vs Indexes

Snowflake's concept of micro-partitions (let's just call them 'partitions') and the index of most traditional Relational Database Management Systems (RDBMS) serve different but complementary purposes in that they both handle data access patterns.

Partitions in Snowflake use a columnar storage format, and play a huge role in how Snowflake optimizes its underlying data storage and retrieval. Think of them as physical chunks of data allocated together (or closeby) on a disk. Snowflake also keeps metadata about which data is stored in each partition (think of it like a directory of physical stores in a shopping center). This metadata is part of the partition and is stored alongside the actual data.

This helps the Snowflake Query Optimizer with data pruning, aka deciding which parts of a disk are irrelevant to the query it's looking to run based on the metadata (directory) that it knows about each partition. Suppose if you have a WHERE clause that specifies a particular CUSTOMER_ID, all the query optimizer has to do is go to the location on disk that has that particular CUSTOMER_ID as a key, therefore discarding all the other partitions that are irrelevant to its query. This is an oversimplification, which we'll discuss a bit later when we talk about data-clustering.

Similarly in theme (but not in implementation), indexes in MySQL are mainly used to speed up the retrieval or rows from a table. Think of a data structure that exists separately from the actual table data (typically a B-Tree) that stores the values for a particular column (or group of columns). The query optimizer can therefore quickly locate the data without having to scan every single row in a table.

So while both approaches serve the same purpose of speeding up queries by eliminating irrelevant pieces of the data, they go about things quite differently. To remember is that Snowflake's partitions are designed to scan huge amounts of data quickly (across different datasets even), while indexes are much smaller in scope and are usually restricted to a single table.

Data Clustering

In addition to micro-partitions, Snowflake further organizes data in data-clusters

Typically, data stored in tables is sorted/ordered along natural dimensions (e.g. date and/or geographic regions). This “clustering” is a key factor in queries because table data that is not sorted or is only partially sorted may impact query performance, particularly on very large tables.

In Snowflake, as data is inserted/loaded into a table, clustering metadata is collected and recorded for each micro-partition created during the process. Snowflake then leverages this clustering information to avoid unnecessary scanning of micro-partitions during querying, significantly accelerating the performance of queries that reference these columns.

As an analogy, a cluster is a directory of directories (partitions) that allows the Query Optimizer to do an initial pruning of all the micro-partitions it knows about. More information here

Source: Snowflake

Interlude: Clustering Keys

Before we dive in further, I should briefly mention Clustering Keys.

As a table grows in size, the data in some table rows might no longer cluster optimally on desired dimensions.
In order to improve the clustering of your table, you can designate one or more columns as the clustering key for your table, which can help improve the clustering of your underlying table micro-partitions. Since you, the developer, (hopefully) understand the access and query patterns in your data a lot better than the query optimizer, you can use clustering keys to provide Snowflake with hints on how to best organize your data.

This excellent article by Harry Tan does a wonderful job further explaining some of these concepts.

A core design principle of Snowflake is to store metadata alongside its tables for fast lookups. So it is with data clusters. The key metric we're interested in for our problem is Clustering Depth.

We're about to do a deep-dive here, so strap in and hold on to your butts.

What he said

According to Snowflake: The clustering depth for a populated table measures the average depth (1 or greater) of the overlapping micro-partitions for specified columns in a table. The smaller the average depth, the better clustered the table is with regards to the specified columns.

Because multiple micro-partitions can store different types of metadata about the same row or record in a table, there is a potential for overlap between two or more partitions. This means the Query Optimizer needs to check multiple partitions when planning a query, increasing its planning (Snowflake calls this compilation, a remnant from the early Oracle days from its founders) latency. We can reduce this overlap by analysing the clustering depth of a table.

- In the beginning, the range of values in all the micro-partitions overlap.

- As the number of overlapping micro-partitions decreases, the overlap depth decreases.

- When there is no overlap in the range of values across all micro-partitions, the micro-partitions are considered to be in a constant state (i.e. they cannot be improved by clustering).

Source: Snowflake

So, to put in terms our dear friend Grug Developer would be happy with:

Partitions good.
Clusters also good. Clusters contain partition data. This moar good.
Small Clustering Depth Number > Large Clustering Depth Number
Moar black think juice (aka coffee) equals moar thinking power.

We can conclude that our slow query performance must be related to these partitions and clustering. However, Snowflake's SnowSight UI does not offer any dashboards or metrics that can tell us more.

We dig deeper to find that Snowflake does expose us some useful information in the form of SYSTEM$CLUSTERING_INFORMATION call.

Given that most of our data lives in the ORDERS_FINAL table, let's focus our efforts here. We run a query on the metadata Snowflake has on our table and look at the results.

select system$CLUSTERING_INFORMATION('ORDERS_FINAL');

 1{
 2  "cluster_by_keys" : "CUSTOMER_ID"
 3  "notes" : "Clustering key columns contain high cardinality key CUSTOMER_ID which might result in expensive re-clustering. Consider reducing the cardinality of clustering keys. Please refer to https://docs.snowflake.net/manuals/user-guide/tables-clustering-keys.html for more information.",
 4  "total_partition_count" : 60,
 5  "total_constant_partition_count" : 12,
 6  "average_overlaps" : 40.2857,
 7  "average_depth" : 27.1429,
 8  "partition_depth_histogram" : {
 9    "00000" : 0,
10    "00001" : 2,
11    "00002" : 2,
12    "00003" : 0,
13    "00004" : 0,
14    "00005" : 5,
15    "00006" : 0,
16    "00007" : 16,
17    "00008" : 8,
18    "00009" : 0,
19    "00010" : 12,
20    "00011" : 0,
21    "00012" : 0,
22    "00013" : 23,
23    "00014" : 0,
24    "00015" : 0,
25    "00016" : 21
26  },
27  "clustering_errors" : [ ]
28}

Disclaimer: For illustration purposes only. Yes, I made up these numbers to prove a point lol.

Let's explain some of these concepts:

CLUSTER_BY_KEYS
Indicates that the table is clustered by the CUSTOMER_ID field, so Snowflake uses this field to organize the data across micro-partitions.

NOTES
The notes mention that the clustering key (BUSINESS_UUID) has high cardinality, which could lead to expensive re-clustering operations. High cardinality means the column has a large number of unique values, which can result in a broad spread of data across many micro-partitions, potentially reducing the effectiveness of clustering (granted, this is less likely for an INT data type, but you get the idea)

TOTAL_PARTITION_COUNT
The total number of micro-partitions in the table is 60. This indicates how the data is divided at the storage level.

TOTAL_CONSTANT_PARTITION_COUNT
Out of 60 partitions, only 12 are constant. Constant means that these partitions do not overlap with others based on the clustering key. This is typically a bad sign, suggesting that most of the data is poorly-clustered.

AVERAGE_OVERLAPS AND AVERAGE_DEPTH
These metrics show that on average, partitions overlap with about 40 other partitions Additionally, the average depth of these overlaps is around 27. This suggests severe overlapping and depth, indicating that re-clustering could be beneficial especially for those partitions that significantly deviate from this average.

PARTITION_DEPTH_HISTOGRAM
This histogram indicates that most partitions have at least some depth, which is not great. This likely means that the data is too spread out in most partitions.

It looks like our partitioning on the ORDERS_FINAL_TABLE (our fact table) is less than optimal, which could be a possible explanation for our slow queries. We can also run this for every single one of our tables by cobbling together a quick script using the SnowPark API.

 1from itertools import permutations
 2import ast
 3
 4import snowflake.snowpark as snowpark
 5from snowflake.snowpark.functions import col
 6
 7TABLES = ["ORDERS_FACT_FINAL", "CUSTOMER_DIMENSION", "SHIPPING_DIMENSION", # rest of tables... ]
 8
 9def main(session: snowpark.Session):
10    def get_clustering_info(table_name):
11        query = f"SELECT SYSTEM$CLUSTERING_INFORMATION('{table_name}')"
12        print(f"Executing query: {query}")
13        result = session.sql(query).collect()
14        clustering_info = ast.literal_eval(result[0][0])
15
16        print(clustering_info)
17
18        total_partition_count = clustering_info.get("total_partition_count", 0) or 1  # prevent div by zero error
19        total_constant_partition_count = clustering_info.get("total_constant_partition_count", 0) or 1  # prevent div by zero error
20
21        average_depth = clustering_info.get("average_depth", 0)
22        average_overlaps = clustering_info.get("average_overlaps", 0)
23
24        return {
25            "table_name": table_name,
26            "total_partition_count": total_partition_count,
27            "total_constant_partition_count": total_constant_partition_count,
28            "clustering_depth": average_depth,
29            "clustering_depth_percentage": (average_depth / total_partition_count) * 100,
30            "clustering_overlap": average_overlaps,
31            "clustering_overlap_percentage": (average_overlaps / total_partition_count) * 100,
32            "partition_depth_histogram": clustering_info.get("partition_depth_histogram")
33        }
34    clustering_results = [get_clustering_info(table_name) for table_name in TABLES]
35
36    results_df = session.create_dataframe(clustering_results)
37    results_df.sort(col('total_partition_count').asc(), col('clustering_depth_percentage').asc())
38    results_df.show()
39    return results_df
40
41"""
42    TOTAL PARTITION COUNT: 
43    Number of micro-partitions in the table. Higher count means a more fragmented table
44
45    TOTAL_CONSTANT_PARTITION_COUNT:
46    Total number of micro-partitions for which the value of the specified columns 
47    have reached a constant state (i.e. the micro-partitions will not benefit 
48    significantly from reclustering). The number of constant micro-partitions in a 
49    table has an impact on pruning for queries. The higher the number, the more 
50    micro-partitions can be pruned from queries executed on the table, which has a 
51    corresponding impact on performance.
52
53    CLUSTERING DEPTH AVG:
54    Micro-partitions scanned per key range Lower = better clustering
55
56    CLUSTERING DEPTH:
57    (DEPTH / TOTAL PARTITIONS) * 100. A lower percentage == efficient clustering
58
59    CLUSTERING OVERLAP AVG:
60    Overlapping micro-partitions per key range. A lower number == less redundancy
61
62    CLUSTERING OVERLAP:
63    (OVERLAP / TOTAL PARTITIONS) * 100. A lower percentage == efficient storage
64
65    PARTITION DEPTH HISTOGRAM:
66    Distribution of micro-partitions across depth levels, 
67    providing a visual clustering assessment
68"""

It looks like most of our tables are well partitioned, with the exception of our ORDERS_FINAL table.

Given that this is our main FACT table, we're pretty confident at this point that this is our bottleneck.

While we can tell that the partitioning on this table is bad, we still don't know WHY.

After sufficient place-forehead-on-desk-at-high-velocity (aka bang head on desk), we decided to ask the good folks at Snowflake, who were awesome enough to let us sit down with their Field CTO team and have us barrage them with the laundry list of questions we gathered after trying to reverse engineer some of the mechanics of their platform.

Snowflake Micro-Partitions: The Plot Thickens

The team's mental state at this point

With the help of the Snowflake Field CTO team, we investigated the partitioning on our tables. Using some internal tooling, the Snowflake team was quickly able to find that our tables were adding more and more micro-partitions by the minute! Y Tho?

The answer: because micro-partitions are immutable. The immutable nature of micro-partitions allows for the lookup and usage of said partitions to be MUCH more performant at query-time. But the flipside of this performance is that, whenever data gets updated or deleted, Snowflake will add new micro-partitions instead of modifying existing ones.

To offset inefficiencies caused by this immutability (and thus the need to create new partitions), Snowflake also has periodic asynchronous automatic re-clustering tasks running in the background, which reorganizes the data across its different nodes and helps the queries remain performant.

This is further confirmed by this Snowflake Blog article from May 2022:

Snowflake micro-partitions immutable files that are not visible to the Snowflake user; rather, the table acts as a container of micro-partitions and that is what the user will interact with. Some of the interactions are:

- A SQL INSERT operation that will load new records as new micro-partitions....

Looking back at our initial ingestion architecture, which component makes continuous updates to the tables? Right, THE TRIGGERED TASK.

Thomas is done f*cking around

Triggered Task == VILLAIN?

Let's look again at our triggered task definition:

 1-- ...
 2as MERGE INTO ORDERS_FINAL USING (
 3                    SELECT *
 4     FROM ORDERS_STAGING_STREAM
 5     QUALIFY ROW_NUMBER() OVER ( -- this QUALIFY statement is important - see below.
 6      PARTITION BY CUSTOMER_ID, ORDER_ID
 7      ORDER BY CREATED_AT DESC
 8     ) = 1
 9    ) AS STG_STREAM ON ORDERS_FINAL.CUSTOMER_ID = STG_STREAM.CUSTOMER_ID
10    AND ORDERS_FINAL.ORDER_ID = STG_STREAM.ORDER_ID
11    WHEN MATCHED THEN
12    UPDATE -- If relevant records are found in ORDERS_FINAL, update values with the stream's values, as we know that's the latest version of the record
13    SET
14     ORDERS_FINAL.RECORD_UUID = STG_STREAM.RECORD_UUID,
15-- ...

We know that this task runs each time it detects new data in the stream it's "listening" to. If the ORDERS_STAGING_STREAM is constantly receiving data, even duplicates or minor changes, the MERGE operation in this task will repeatedly attempt to update the same records.

Additionally, the QUALIFY ROW_NUMBER() OVER (...) = 1 statement selects the latest record for each CUSTOMER_ID and ORDER_ID. When the underlying data changes, this is fine. But what if it's just a minor, irrelevant change (e.g. timestamp) and all other data remains the same? Still, the MERGE statement sees this as a new record and will update the ORDERS_FINAL entry accordingly. Since Snowflake will create new immutable micro-partitions on a record update, frequent updates (caused by the task) will lead to an exponential increase in the number of micro-partitions created for a table.

Because of this excessive number of micro-partitions, the overlapping data between these partitions will also increase dramatically - which explains the large overlap depth from our previous analysis.

So now, the queries now have to scan a lot more micro-partitions in order to retrieve the data, slowing down the entire query!.

Snowflake's auto-clustering background processes are also having a much harder time keeping up with reconciling the partitions to optimize the data organization, which grinds the whole thing to a near crawl.

This was confirmed by the Snowflake team:

"One potentially challenging pattern that some customers have encountered is with a MERGE or UPDATE against a random subset of the rows in the target table. Even if only an extremely small percentage of the rows in the target table are updated, if those rows are distributed across a high number of micro-partitions, then these micro-partitions (since micro-partitions are immutable) all rows within will need to be completely rewritten to new micro-partitions. At a minimum, this can result in poor DML performance in the data pipeline, and in some cases cause extremely high data latency. This can also adversely affect the clustering of the table, negatively impacting query performance against the target table due to poor pruning."

Mystery solved!

"And I would've gotten away with it if it wasn't for those meddling micro-partitions!"

The solution: Deferred Merge Strategy

Great! Root cause identified and proven. Now, what do we do about it?

The main issue we saw earlier is not necessarily the MERGE statement of the triggered task itself, but rather the frequency with which the task executes said MERGE operations.

The basic idea of a Deferred Merge approach is a slight variation of our initial setup, with the main difference being instead of having a triggered task continuously running MERGE statements (and hence causing the explosion in micro-partitions), we "defer" the merging of data to a scheduled task instead, which runs at a regular pre-defined interval.

But how do you handle keeping your data de-duplicated and up to date?

The proposed solution is to replace our initial setup from this:

Node 1: Staging table (entrypoint)

Node 2: Staging Stream (captures new data being inserted into node 1)

Node 3: Triggered Task (checks for new data entering Node 2 and executes MERGE)

Node 4: Destination (aka Final) table with deduplicated records holding latest version -> targeted by queries.

To this:

Node 1: Staging table (entrypoint)

Node 2: Staging Stream (captures new data being inserted into node 1)

Node 3: Triggered Task (checks for new data entering Node 2 and executes MERGE)

Node 4: DELTA table with deduplicated records holding latest version.

Node 5: Scheduled task running at a set interval which MERGES the data from Node 4 into the final table (aka node 6) AND deletes the processed records from the Delta table.

Node 6: Destination (aka Final) table with deduplicated records holding latest version.

Data Freshness Concerns

What about the queries? How to ensure data freshness if our FINAL table only gets updated on a schedule?

The queries should now target both Node 4 (Delta table) and Node 6 (Final table) using a UNION ALL NOT EXISTS clause. According to Snowflake:

This approach uses a UNION ALL to blend the delta data with the base, with a NOT EXISTS added to ensure that a given key (ID) is only emitted from the base data if it is not emitted by the delta data.

In our case, the `base` data is coming from Node 6, and joined via a UNION ALL with Node 4 to ensure we only return the latest unique records.

So how does this solve our problem?

Because the triggered task no longer continuously updates our FINAL table (Node 6) and instead we use a scheduled task, the MERGE operation happens a LOT less frequently, allowing Snowflake and its auto-clustering to operate efficiently and actually keep the number of micro-partitions low.
We are still using a triggered task to get data into Node 4 (Delta table), but the scheduled task is also responsible for periodically deleting rows from this delta table after they've been processed. This means that this DELTA table will stay small, with at most a few thousand records (depending on the time interval used by Node 5).

Final Query

 1WITH DELTA_DATA AS (
 2  SELECT
 3      RECORD_UUID,
 4      ORDER_ID,
 5      QUANTITY,
 6      SHIPPING_ID
 7  FROM ORDER_DELTA
 8    QUALIFY 1 = ROW_NUMBER() OVER (
 9      PARTITION BY CUSTOMER_ID, ORDER_ID
10      ORDER BY CREATED_AT DESC
11  )
12),
13BASE_DATA AS (
14  SELECT
15      RECORD_UUID,
16      ORDER_ID,
17      QUANTITY,
18      SHIPPING_ID
19  FROM ORDER_FINAL F
20  WHERE NOT EXISTS (
21  SELECT 1
22  FROM DELTA_DATA D
23  WHERE D.RECORD_UUID = F.RECORD_UUID
24  )
25  UNION ALL
26  SELECT
27    RECORD_UUID,
28    ORDER_ID,
29    QUANTITY,
30    SHIPPING_ID
31  FROM DELTA_DATA
32),
33SELECT
34  BASE_DATA.ORDER_ID,
35  BASE_DATA.QUANTITY
36  PRODUCT_DIMENSION.PRODUCT_NAME,
37  CUSTOMER_DIMENSION.CUSTOMER_NAME,
38  STATUS_DIMENSION.STATUS_NAME,
39  SHIPPING_DIMENSION.CREATED_AT AS "Shipped At"
40FROM
41    BASE_DATA
42    INNER JOIN PRODUCT_DIMENSION ON BASE_DATA.PRODUCT_ID = PRODUCT_DIMENSION.PRODUCT_ID
43    INNER JOIN CUSTOMER_DIMENSION ON BASE_DATA.CUSTOMER_ID = CUSTOMER_DIMENSION.CUSTOMER_ID
44    INNER JOIN STATUS_DIMENSION ON BASE_DATA.STATUS_ID = STATUS_DIMENSION.STATUS_ID
45    INNER JOIN SHIPPING_DIMENSION ON BASE_DATA.SHIPPING_ID = SHIPPING_DIMENSION.SHIPPING_ID AND BASE_DATA.CUSTOMER_ID = SHIPPING_DIMENSION.SHIPPING_ID
46WHERE
47    BASE_DATA.SHIPPING_ID IS NOT NULL;

The full document with the proposed Deferred Merge approach as outlined by Snowflake can be found here

Final verdict

We implemented the new proposed deferred merge architecture to run side by side with the initial setup and pointed our queries to the new setup.

The results:

From 21 seconds to 2.1 seconds. That's a whopping 90% performance boost !!!

Conclusion

After a few weeks of blood, sweat, tears and a h*ll of a lot of reverse engineering, you finally did it.

The performance improvements gained by Deferred Merge range from 70% on the slower end of the spectrum to 90% on the faster end.

The queries are now humming along beautifully and the team can continue building some pretty awesome products on top of some pretty awesome data solutions. At the same time, this work is never done, so continuous monitoring and tweaking is always appropriate.

The average durations of these queries are used as an example from our actual implementations, and they do represent real production results.

Remember these are averages so the range of values will vary depending on system load, time of day, etc. AKA your mileage may vary.

Roll Credits

A massive thanks to my team at FreshBooks and the team at Snowflake. You all know who you are!

What's next?

I'll be writing more in the near future about Snowflake's architecture, as well as some other measures my team and I have taken to ensure our data pipeline is as fast and reliable as possible. Stay tuned!

Facing the SnowStorm: Part 1

2024-12-22T00:00:00Z

Introduction

In this post, we'll explore how frequent MERGE operations in Snowflake can drastically degrade query performance, and how implementing a Deferred Merge strategy improved our production query speeds by up to 90%.

Over the past few months, my team and I have had the opportunity to work on implementing a new data pipeline in order to provide large-scale reports to an end-user. The design of this data pipeline was mostly an event-driven architecture, with the final destination being a Snowflake Cloud instance.

This article will discuss some of the pitfalls the team encountered, the largest one being that executing frequent (continuous) MERGE updates on Snowflake tables lead to fragmented data, inefficient partitioning and horrible query compilation/execution latency. If you are a Snowflake user managing large datasets that require very low latency from ingestion to retrieval and have to deal with frequent updates to source tables, read on!

Initial Setup

As stated in the introduction, the goal of the data pipeline was to capture data from different source applications, leveraging events as the primary mechanism, the details of which will be the subject of a future post. The main way we are inserting data into the initial Snowflake table is via the Snowflake Java Ingest SDK, which leverages SnowPipe Streaming under the hood.

For the sake of simplicity (and to not get into hot water with my employer) we'll be using a fictional data model to illustrate, namely that of an e-commerce platform (bonus points for originality /s).

Data Model

We'll be modelling our data using the tried-and-true dimensional model (aka "Star Schema"), as popularized by Ralph Kimball et al in the excellent classic Data Warehouse Toolkit book.

NOTE

The data model we're creating here mainly serves as illustration, since the focus of this post is on how our data moves through the Snowflake Ingestion Pipeline.

Defining the Staging table

Our first table in Snowflake has the following schema:

 1CREATE TABLE orders_staging (
 2    record_uuid BINARY,
 3    order_id NUMBER,
 4    customer_id NUMBER,
 5    product_id NUMBER,
 6    quantity NUMBER,
 7    order_date TIMESTAMP,
 8    status_id  NUMBER, -- e.g., 'Pending', 'Shipped', 'Delivered'
 9    shipping_id NUMBER,
10    created_at TIMESTAMP,
11    PRIMARY KEY (record_uuid, order_id)
12);

A few things to note right away:

This table has a _staging suffix.
This is an APPEND-ONLY table. Why, you ask? Good question! We'll get to that later. For now, just know that this means that every new INSERT will have a globally unique record_uuid, which is generated at the application-level during initial ingestion and is just added as a row to the table without any further checks.

Leveraging Snowflake Streams

To detect changes in the ORDERS_STAGING table, we can leverage Snowflake Streams. According to Snowflake's official documentation: A stream object records data manipulation language (DML) changes made to tables, including inserts, updates, and deletes, as well as metadata about each change, so that actions can be taken using the changed data. This process is referred to as change data capture (CDC).

Let's create an append-only stream:

1create stream ORDERS_STAGING_STREAM on table ORDERS_STAGING append_only = true;

APPEND-ONLY STREAMS in SNOWFLAKE

An append-only stream exclusively tracks row inserts on the underlying table. Update, delete, and truncate operations are not captured by append-only streams. An append-only stream specifically returns the appended rows, making it notably more performant than a standard stream for extract, load, and transform (ELT), and similar scenarios reliant solely on row inserts.

Essentially, ORDERS_STAGING_STREAM acts as a listener for any INSERT operations on the ORDERS_STAGING table. So far, so good. But what do we do with this tracked delta on the data?

Snowflake Triggered Tasks

Recall the APPEND-ONLY nature of the staging table? The reason for this is that we're dealing with an event-driven architecture. Typically, event messages in such systems have the following characteristics:

We cannot control the order in which the events are processed in a distributed system with multiple publishers and consumers.
We cannot guarantee uniqueness of messages, so we need must assume that events might be published and consumed at least once, potentially multiple times.

So, how do we handle potential duplicate messages or out-of-order messages when querying for the data?

That's where two more elements come into play: triggered tasks and a ORDERS_FINAL table. Let's first discuss the _FINAL table.

This table essentially has the same schema as the _STAGING table we've seen before:

 1CREATE TABLE orders_final (
 2    record_uuid BINARY,
 3    order_id NUMBER,
 4    customer_id NUMBER,
 5    product_id NUMBER,
 6    quantity NUMBER,
 7    order_date TIMESTAMP,
 8    status_ID NUMBER,
 9    shipping_id NUMBER,
10    created_at TIMESTAMP,
11    PRIMARY KEY (record_uuid, order_id)
12);

The purpose of the ORDERS_FINAL table is to hold the latest, de-duplicated, and normalized version of the records for our final queries.

So, how do we maintain the most-up-to-date and deduplicated version of each record? This is where triggered tasks come in. According to the Snowflake Documentation on Triggered Tasks:

Tasks use user defined functions to automate and schedule business processes. With a single task you can perform a simple to complex function in your data pipeline. (...) You can also combine tasks with table streams for continuous ELT workflows to process recently changed data. (...) Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or perform other periodic work.

Let's define our triggered task:

 1create task ORDER_STAGING_TO_FINAL_TASK
 2 USER_TASK_MINIMUM_TRIGGER_INTERVAL_IN_SECONDS=30 -- leave at least 30 seconds between each run
 3 when system$stream_has_data('ORDERS_STAGING_STREAM') -- only run if the underlying stream has data captured
 4 as MERGE INTO ORDERS_FINAL USING (
 5                    SELECT *
 6     FROM ORDERS_STAGING_STREAM
 7     QUALIFY ROW_NUMBER() OVER ( -- this QUALIFY statement is important - see below.
 8      PARTITION BY CUSTOMER_ID, ORDER_ID
 9      ORDER BY CREATED_AT DESC
10     ) = 1
11    ) AS STG_STREAM ON ORDERS_FINAL.CUSTOMER_ID = STG_STREAM.CUSTOMER_ID
12    AND ORDERS_FINAL.ORDER_ID = STG_STREAM.ORDER_ID
13    WHEN MATCHED THEN
14    UPDATE -- If relevant records are found in ORDERS_FINAL, update values with the stream's values, as we know that's the latest version of the record
15    SET
16     ORDERS_FINAL.RECORD_UUID = STG_STREAM.RECORD_UUID,
17     ORDERS_FINAL.PRODUCT_ID = STG_STREAM.PRODUCT_ID,
18     ORDERS_FINAL.QUANTITY = STG_STREAM.QUANTITY,
19     ORDERS_FINAL.ORDER_DATE = STG_STREAM.ORDER_DATE,
20     ORDERS_FINAL.STATUS = STG_STREAM.STATUS,
21     ORDERS_FINAL.CREATED_AT= STG_STREAM.CREATED_AT,
22    WHEN NOT MATCHED THEN -- if no match found in ORDERS_FINAL, records are new
23    INSERT
24     (
25      RECORD_UUID,
26      CUSTOMER_ID,
27      PRODUCT_ID,
28      QUANTITY,
29      ORDER_DATE,
30      STATUS,
31      CREATED_AT
32     )
33    VALUES
34     (
35      STG_STREAM.RECORD_UUID,
36      STG_STREAM.CUSTOMER_ID,
37      STG_STREAM.PRODUCT_ID,
38      STG_STREAM.QUANTITY,
39      STG_STREAM.ORDER_DATE,
40      STG_STREAM.STATUS,
41      STG_STREAM.CREATED_AT
42    );

Let's highlight some crucial parts of the task definition once more for the cool kids in the back:

1USER_TASK_MINIMUM_TRIGGER_INTERVAL_IN_SECONDS=30
2/* Sets the minimum interval between consecutive task triggers, 
3which prevents the task from executing too frequently. 
4This can help manage performance and limit unnecessary runs. */

1when system$stream_has_data('ORDERS_STAGING_STREAM')
2/* Task should only run when there is new data available in the 'ORDERS_STAGING_STREAM' 
3which tracks changes to the staging table and ensures that the 
4task only processes new or updated records. */

1MERGE INTO ORDERS_FINAL USING
2/* Synchronizes the 'ORDERS_FINAL' table with the latest data 
3from the `ORDERS_STAGING_STREAM`. Checks for matches based on 
4specified keys and decides whether to update existing 
5records or insert new ones. */

1QUALIFY ROW_NUMBER() OVER (PARTITION BY CUSTOMER_ID, ORDER_ID ORDER BY CREATED_AT DESC) = 1
2/* This clause is critical for ensuring that only the most recent 
3record for each unique combination of 'CUSTOMER_ID' and 'ORDER_ID' 
4is used in the merge, by filtering out older duplicates */

1ON ORDERS_FINAL.CUSTOMER_ID = STG_STREAM.CUSTOMER_ID AND ORDERS_FINAL.ORDER_ID = STG_STREAM.ORDER_ID
2/* Set conditions under which records from the stream and 
3the final table are considered a match. */

1WHEN MATCHED THEN UPDATE
2/* If matching records are found, update the matches in the 
3final table with the data from the staging stream. 
4This is an UPDATE IN PLACE operation. */

1WHEN NOT MATCHED THEN INSERT
2/* If no matching record is found, insert the new record into the final table.*/

With this, we now have a fully functional end-to-end ingestion process, ensuring we always have the most up-to-date and relevant records in our ORDERS_FINAL table, which we can now target with our final end-user queries.

Let's illustrate this process visually:

Additional note on Snowflake Streams

Once the Triggered Task has consumed the data from the underlying append-only stream, that data is deleted from the stream.

Satisfied with this approach, you now repeat the process for all other tables in your dimensional model.

The final step is to define our end-user query. Suppose we want a basic report of shipped orders with relevant customer metadata. We'll write a simple SELECT statement on the ORDERS_FINAL table with JOIN clauses into the relevant dimension tables:

 1SELECT
 2    ORDERS_FINAL.ORDER_ID,
 3    ORDERS_FINAL.QUANTITY,
 4    PRODUCT_DIMENSION.PRODUCT_NAME,
 5    CUSTOMER_DIMENSION.CUSTOMER_NAME,
 6    STATUS_DIMENSION.STATUS_NAME,
 7    SHIPPING_DIMENSION.CREATED_AT AS "Shipped At"
 8FROM
 9    ORDERS_FINAL
10    INNER JOIN PRODUCT_DIMENSION ON ORDERS_FINAL.PRODUCT_ID = PRODUCT_DIMENSION.PRODUCT_ID
11    INNER JOIN CUSTOMER_DIMENSION ON ORDERS_FINAL.CUSTOMER_ID = CUSTOMER_DIMENSION.CUSTOMER_ID
12    INNER JOIN STATUS_DIMENSION ON ORDERS_FINAL.STATUS_ID = STATUS_DIMENSION.STATUS_ID
13    INNER JOIN SHIPPING_DIMENSION ON ORDERS_FINAL.SHIPPING_ID = SHIPPING_DIMENSION.SHIPPING_ID AND ORDERS_FINAL.CUSTOMER_ID = SHIPPING_DIMENSION.SHIPPING_ID
14WHERE
15    ORDERS_FINAL.SHIPPING_ID IS NOT NULL;

Winter is Coming for your Query Performance

Everything seems to be running smoothly. Your pipeline is efficient, reports are accurate, and stakeholders are pleased. Executives are patting you on the back, and your fellow engineers' songs of your heroic deeds echo across the open floorplan office.

Suddenly, your pager alerts you to increasing latency. Observability metric dashboards show the end-user query latency climbing dramatically. What is happening?

That familiar pager sound at 3AM...

Using Snowflake's SnowSight UI's excellent cascading GUI interface for query profiling, you notice something unusual:

Excuse me? The query duration shows 21.0 seconds (!!!!). For a simple SELECT query with a handful of inner joins? That is excessively long for an end-user query, before even considering network latency and application processing time. Well, this is not good.

Examining the query details in the SnowSight Query Profiler reveals:

Our query is spending an awful lot of time on TableScan operations.

This means the query spends a lot of time going through the tables in order to find the relevant data. What gives? Our data model is so clean and our query is not very complex.

While you furrow your brow examining these query statistics, your stakeholders come tapping you on the (virtual) shoulder with that familiar look on their face:

Just kidding. The PMs I work with are just absolute wonderful human beings ^_^

Your breathing intensifies. Beads of sweat start rolling down your face. You may be on to something, but we have real users out there with super slow loading reports! Quickly, you huddle with the team and compile a list of potential solutions. You split your team of engineers and divide and conquer:

Simplify Query Complexity

Even though our original query is pretty simple, we can reduce the number columns in our SELECT statement and further refine our WHERE clauses to better leverage our newfound knowledge about micro-partitions. The more we can help the Query Optimizer do its job, the better.

Enable/Disable Query Acceleration Performance

Topic for a future post! In the meantime, have some tasty docs.

Enable/Disable Search Optimization Service

Again, stay tuned! Here's some moar yummy docs.

Split ingestion and querying activities into separate warehouses

Separate the workloads by dedicating distinct Virtual Warehouse instances to ingestion and querying tasks, respectively. Wait, what's a warehouse? Right! Good question. We'll get to this in a minute.

Use Materialized Views as pre-compute for query

We actually did spend some time on this one. While it was a neat experiment and did provide some speedup on the query (since we essentially just pre-computed the data into a giant wide table), it made ingestion, business logic and maintenance unnecessarily complex. Again, stay tuned for more on this in a future post, where we'll be talking materialized views and task graphs!

Increased USER_TASK_MINIMUM_TRIGGER_INTERVAL_IN_SECONDS

Another theory: maybe the ingestion processes are in resource contention with the query processes? If you'll recall, the USER_TASK_MINIMUM_TRIGGER_INTERVAL sets the minimum interval between consecutive task triggers, which prevents the task from executing too frequently. So, if we force a larger pause between task runs, maybe that will leave some breathing room for the queries and reduce resource contention between compute nodes? Sadly, this did not prove to be all that effective.

Added RELY to primary key constraints

What if we reduced redundant joins in our query? Snowflake does not enforce UNIQUE, PRIMARY KEY, and FOREIGN KEY constraints on standard tables.. So, if we're confident in our data model and its inherent relationships between the different tables, we can use the RELY constraint on UNIQUE, PRIMARY KEY, FOREIGN KEY constraints to help the query optimizer eliminate unnecessary joins.

More details here..

Modify or disable clustering

This also made very little difference. But what is clustering? That's what the other half of our team has been working on, so they'll take over from us in a minute!

First things first, let's talk warehouses.

Snowflake Virtual Warehouses Explored

Remember when I talked about Split ingestion and querying activities into separate warehouses ? Yeah, that didn't work. In order to understand why, we need to take a closer look at what a warehouse actually is.

A Snowflake Virtual Warehouse is a cluster of compute resources in Snowflake that processes queries and transactions. Unlike a traditional RDBMS system with static resources, warehouses in Snowflake can be scaled dynamically based on the workload. They provide a completely isolated query execution environment and other features like pausing and auto-suspending of warehouses. This allows Snowflake to do some pretty cool stuff where they can re-use idle resources / nodes / warehouses to buff up compute intensive tasks on other client's warehouses, basically 'borrowing' compute power without charging you, the client, for it. In return, whenever you need a bit of extra juice for a workload, they can do the same for you. What's that? You want docs? Of course, here ya go.

An important consideration when it comes to warehouses is their size, which ranges from XS all the way to 6X-Large as of time of writing.

According to Snowflake:

The size of a warehouse can impact the amount of time required to execute queries submitted to the warehouse, particularly for larger, more complex queries. In general, query performance scales with warehouse size because larger warehouses have more compute resources available to process queries.

If queries processed by a warehouse are running slowly, you can always resize the warehouse to provision more compute resources.

Check out this post to learn more about Snowflake Compute layer and its Virtual Warehouse architecture.

Previously, we used a single XS warehouse for both ingestion and querying workloads. Instead, we decided to:

Assign an XS warehouse for data ingestion.
Assign an S warehouse for querying.

Data freshness is not impacted by this split because warehouses are part of the compute layer, which are totally separated from the storage layer. Therefore, data doesn't need to be replicated between warehouses.

The storage layer is fully managed by Snowflake and is built on cloud storage solutions like Amazon S3, Google Cloud Storage or Microsoft Azure Blob Storage. This centralized storage is scalable, secure, and independent of compute resources. In the origin days of Snowflake, they decided to simply offload the storage to cloud vendors so they could specify more on the querying, compute layers and caching layers, which turned out to be a very smart move in the end (no need to build your own data centers if you can rent them for a fraction of the cost, amirite?).

So, in theory, we can add as many warehouses as we want - they would all have instant access to the same data. We (as the client) just tell Snowflake which cloud provider we want to use and they take care of the provisioning. The data in the respective buckets is usually still their proprietary data format, although they do also support open-source formats like Parquet, Avro, XML, etc...

Snowflake Platform Overview

So where did the warehouse resizing leave us?

In theory, moar compute power == faster queries, right? After all, that's what the docs tell us. WRONG. Increasing the warehouse size did not improve query latency.

EXPECTATION

REALITY

As it turns out, we missed a critical detail that tells us whether or not it was the available compute resources being the bottleneck for our query speed: If the warehouse does not have enough remaining resources to process a query, the query is queued, pending resources that become available as other running queries complete.

Snowflake provides insight into this via its INFORMATION_SCHEMA.QUERY_HISTORY, which shows query queuing statistics.

A more holistic overview can also be seen in the Warehouses section of the SnowSight Admin UI.

Over the past two weeks, there were no queued queries. So, this means compute resources were sufficient after all?

Sigh...back to the drawing board we go.

So, resizing the warehouse didn’t exactly do what we hoped. The query latency didn’t budge, and it turns out compute resources weren’t the issue after all. With no queries getting stuck in a queue, we know it’s something else. What’s actually slowing things down? It might be time to take a closer look at the way our data is organized. In part 2 of this series, we’ll dive into clustering and partitioning to see if they hold the key to fixing this. See you there!

TIL in UNIX tooling - JQ is awesome!

2024-12-06T00:00:00Z

Disclaimer: this post was originally written in December 2019, but new blog new me amirite?

Ain't no shame in admitting it tho

Once in a while, when working with vendors, I might need to make some comparisons between what different vendors/ partners offer. Because documentation can sometimes be incomplete or not completely up to date, I find myself often turning to the vendor's API for answers, since even a handful of sandbox API endpoints can often give you most, if not all, the answers.

Let's dive into an example.

Context

I really, really, really like flaky pastries. I don't know why, but I'm not satisfied until the entire table is covered in flakes. We could dig further into this strange flex, but it's pretty irrelevant. Anyway, to maximize my chances of indulging in this preference, I want to know what types of pastries are offered by 3 bakeries that are located closest to my house. Instead of walking over and asking like a normal human being, I decide to find the answer through more devious ways: hitting their baker.... API. ....this problem statement is already falling apart.

Anyway, let's say I curl baker_1's GET /pastries endpoint, and get the following result:

 1{
 2    "data": [
 3         // irrelevant stuff
 4        "pastries": [
 5            "name": "French Croissant",
 6            "flake_factor": 10
 7        ], 
 8        "bakery_location": {
 9            "longitude": 6.11499,
10            "latitude": 50.76891
11        }, 
12         // moar irrelevant stuff
13    ]
14}

Ideally, if I am making a comparison of multiple bakeries and their pastries, I might want to share that with my fellow pastry-philes (yes, that's a word). Since spreadsheets are cool and all, I want to do that in a CSV file.

Problem

This leads me to 2 problems:

I need to get this JSON data in a CSV format
I need to be able to repeat this process with minimal effort.

Normally, this could potentially take me quite some time to write a script to parse the json data, filter out the fields I care about and write those to a new file (or output them elsewhere). I would need to do this again for each vendor, assuming the response data will look different each time.

JQ to the rescue

This is where jq comes into play. A "lightweight and flexible command-line JSON processor", it basically lets you slice and dice up JSON data however you want, with the same ease that grep lets you do it on the command line for regular text.

Now that we have our JSON response stored locally somewhere, let's apply jq to it.

Since all I care about for these bakeries is their pastries and their flake-factor, most of the top-level data is irrelevant to me. I need to dig into the pastries array. I can do that with the following command:

jq -r '.data[] | {"type": .pastries[].name, "flake": .pastries[].flake_factor}' bakery.json > baker_filtered.json

The above command will take the entries from the data array (.data[]) and pass it further down the filter. There, I can construct a new JSON object with whatever keys I want (in this case, my standardized pastry ratings), and have the values be the desired output from the original JSON object (in this case, entries from the pastries array). I then pass this into a new baker_filtered json file.

You can do some very powerful things with these commands. I have mostly used it to filter out stuff from JSON responses that I don't care about, but as you can see in this case, I created a brand new JSON object with it.

For more tutorials and the full manual, check out jq's homepage.

Sure, I still need to update the command for each vendor with a different response data structure, but that can be done in a matter of seconds once I've sufficiently grokked the jq syntax.

Lastly, I did say I wanted the final output to be CSV. That can be done with a very quick Python script:

 1import json
 2import csv
 3
 4
 5with open('baker_filtered.json') as json_file:
 6    data = json.load(json_file)
 7
 8with open('flaky_pastry_list.csv', 'w') as write_file:
 9    writer = csv.writer(write_file)
10    for row in data:
11        writer.writerows(data)

And there you have it. I might put together a more real-world example with some actual useful data instead of this contrived example, but I hope this can illustrate some of the cool stuff jq will let you do. And yes, I know I'm late to this party, but hey, we can't know everything, all right?

See you next time.