Spark SQL from_json documentation

Question

Where can I find more detailed information regarding the schema parameter of the from_json function in Spark SQL? A coworker gave me a schema example that works, but to be honest, I just don't understand and it doesn't look like any of the examples I have found thus far. The documentation found here seems to be lacking.

Answer 1

In the link you shared the from_json function uses this example:

SELECT from_json('{"a":1, "b":0.8}', 'a INT, b DOUBLE');

Spark SQL supports the vast majority of Hive features such as the defining TYPES

The example problem I was facing required me to parse the following JSON object:

{'data': [
    {
       "id":02938, 
       "price": 2938.0, 
       "quantity": 1
    }, 
    {
       "id":123, 
       "price": 123.5, 
       "quantity": 2
    }
]}

The corresponding Spark SQL query would look like this:

SELECT 
    from_json('{"data":[{"id":123, "quantity":2, "price":39.5}]}'), 
    'data array<struct<id:INT, quantity:INT, price:DOUBLE>>').data) AS product_details;

you can couple this with the explode function to extract each element into it's own column.

I recommend this post to learn more about constructing the types for your query.

Refer to this SO post for more examples https://stackoverflow.com/a/55432107/1500443