Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

SQL array flattening: Why doesn't CROSS JOIN UNNEST join every nested value with every row?

This question isn't about solving a particular problem, it's about understanding what's actually happening behind the scenes in a common SQL idiom used to flatten arrays. There's some magic behind the scenes and I want to peek behind the curtain of syntactic sugar and see what's going on.

Let's consider the following table t1 :

t1

Now let's assume we have a function called FLATTEN that takes a column of type array and unpacks each of the arrays in that column so that we're left with one row for each value in each array -- if we run SELECT FLATTEN(numbers_array) AS flattened_numbers FROM t1, we'd expect the following, which we'll call t2

t2

In SQL, the CROSS JOIN combines rows from two tables by combining each row from the first table with each row from the second table. So if we run SELECT id, flattened.flattened_numbers from t1 CROSS JOIN flattened, we get

enter image description here

Now flatten is just an imaginary function, and as you can see it's not very useful to combine it with a CROSS JOIN, because each of the original values of the id column gets mixed with flattened_numbers from each of the original rows. Everything gets mixed up because we don't have a WHERE clause that selects only the rows of the CROSS JOIN that we want.

The pattern people actually use to flatten arrays looks like this: SELECT id, flattened_numbers FROM t1 CROSS JOIN UNNEST(sequences.some_numbers) AS flattened_numbers, which produces

enter image description here

But I don't understand why the CROSS JOIN UNNEST pattern actually works. Because the CROSS JOIN doesn't include a WHERE clause, I would expect it behave just like the FLATTEN function I outlined above, where every unnested value gets combined with every row from t1.

Can someone 'unpack' what's actually going on in the CROSS JOIN UNNEST pattern that ensures that each row is only joined with it's own nested values (and not with the nested values from other rows)?

like image 357
conradlee Avatar asked Dec 06 '17 09:12

conradlee


People also ask

How does cross join Unnest work?

For each row N in the source table, UNNEST flattens the ARRAY from row N into a set of rows containing the ARRAY elements, and then the cross join joins this new set of rows with the single row N from the source table. The following example uses UNNEST to return a row for each element in the array column.

What does Unnest do in SQL?

The UNNEST function returns a result table that includes a row for each element of the specified array. If there are multiple ordinary array arguments specified, the number of rows will match the array with the largest cardinality.

Can you join on an array in SQL?

This is the function to use if you want to concatenate all the values in an array field into one string value. You can specify an optional argument as a separator, and it can be any string. If you do not specify a separator, there will be nothing aded between the values.


1 Answers

The best way to think about this is by looking at what happens on a row-by-row basis. Setting up some input data, we have:

WITH t1 AS (   SELECT 1 AS id, [0, 1] AS numbers_array UNION ALL   SELECT 2, [2, 4, 5] ) ... 

(I'm using a third element for the second row to make things more interesting). If we just select from it, we get output that looks like this:

WITH t1 AS (   SELECT 1 AS id, [0, 1] AS numbers_array UNION ALL   SELECT 2, [2, 4, 5] ) SELECT * FROM t1; +----+---------------+ | id | numbers_array | +----+---------------+ | 1  | [0, 1]        | | 2  | [2, 4, 5]     | +----+---------------+ 

Now let's talk about unnesting. The UNNEST function takes an array and returns a value table of the array's element type. Whereas most BigQuery tables are SQL tables defined as a collection of columns, a value table has rows of some value type. For numbers_array, UNNEST(numbers_array) returns a value table whose value type is INT64, since numbers_array is an array with an element type of INT64. This value table contains all of the elements in numbers_array for the current row from t1.

For the row with an id of 1, the contents of the value table returned by UNNEST(numbers_array) are:

+-----+ | f0_ | +-----+ | 0   | | 1   | +-----+ 

This is the same as what we get with the following query:

SELECT * FROM UNNEST([0, 1]); 

UNNEST([0, 1]) in this case means "create a value table from the INT64 values 0 and 1".

Similarly, for the row with an id of 2, the contents of the value table returned by UNNEST(numbers_array) are:

+-----+ | f0_ | +-----+ | 2   | | 4   | | 5   | +-----+ 

Now let's talk about how CROSS JOIN fits into the picture. In most cases, you use CROSS JOIN between two uncorrelated tables. In other words, the contents of the table on the right of the CROSS JOIN are not defined by the current contents of the table on the left.

In the case of arrays and UNNEST, however, the contents of the value table produced by UNNEST(numbers_array) change depending on the current row of t1. When we join the two tables, we get the cross product of the current row from t1 with all of the rows from UNNEST(numbers_array). For example:

WITH t1 AS (   SELECT 1 AS id, [0, 1] AS numbers_array UNION ALL   SELECT 2, [2, 4, 5] ) SELECT id, number FROM t1 CROSS JOIN UNNEST(numbers_array) AS number; +----+--------+ | id | number | +----+--------+ | 1  | 0      | | 1  | 1      | | 2  | 2      | | 2  | 4      | | 2  | 5      | +----+--------+ 

numbers_array has two elements in the first row and three elements in the second, so we get 2 + 3 = 5 rows in the result of the query.

To answer the question about how this differs from flattening the numbers_array and then performing a CROSS JOIN, let's look at the results of this query:

WITH t1 AS (   SELECT 1 AS id, [0, 1] AS numbers_array UNION ALL   SELECT 2, [2, 4, 5] ), t2 AS (   SELECT number   FROM t1   CROSS JOIN UNNEST(numbers_array) AS number ) SELECT number FROM t2; +--------+ | number | +--------+ | 0      | | 1      | | 2      | | 4      | | 5      | +--------+ 

In this case, t2 is is a SQL table with a column named number with those values. If we perform a CROSS JOIN between t1 and t2, we get a true cross product of all rows:

WITH t1 AS (   SELECT 1 AS id, [0, 1] AS numbers_array UNION ALL   SELECT 2, [2, 4, 5] ), t2 AS (   SELECT number   FROM t1   CROSS JOIN UNNEST(numbers_array) AS number ) SELECT id, numbers_array, number FROM t1 CROSS JOIN t2; +----+---------------+--------+ | id | numbers_array | number | +----+---------------+--------+ | 1  | [0, 1]        | 0      | | 1  | [0, 1]        | 1      | | 1  | [0, 1]        | 2      | | 1  | [0, 1]        | 4      | | 1  | [0, 1]        | 5      | | 2  | [2, 4, 5]     | 0      | | 2  | [2, 4, 5]     | 1      | | 2  | [2, 4, 5]     | 2      | | 2  | [2, 4, 5]     | 4      | | 2  | [2, 4, 5]     | 5      | +----+---------------+--------+ 

So what's the difference between this and the previous query with CROSS JOIN UNNEST(numbers_array)? In this case, the contents of t2 don't change for each row from t1. For the first row in t1, there are five rows in t2. For the second row in t1, there are five rows in t2. As a result, the CROSS JOIN between the two of them returns 5 + 5 = 10 rows in total.

like image 60
Elliott Brossard Avatar answered Oct 09 '22 14:10

Elliott Brossard