Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

PostgreSQL: fill NULL values in timeserie query with previous value

Tags:

sql

postgresql

I have a database with time related information. I want a list with values for every minute. Like this:

12:00:00  3
12:01:00  4
12:02:00  5
12:03:00  5
12:04:00  5
12:05:00  3

But when there is no data for some minutes I got a result like this:

12:00:00  3
12:01:00  4
12:02:00  5
12:03:00  NULL
12:04:00  NULL
12:05:00  3

I want to fill the NULL value with the previous NOT NULL value.

This query creates a timeserie for every minute. Then it joins this to the data in my database.

I read something about window functions to fill the NULL value with the previous NOT NULL value, but I can't figure out how to implement this in this query. Can someone push me in the good direction?

I tried this solution, but the NULL values are still there: PostgreSQL use value from previous row if missing

This is my query:

SELECT
    date,
    close
FROM generate_series(
  '2017-11-01 09:00'::timestamp,
  '2017-11-01 23:59'::timestamp,
  '1 minute') AS date
LEFT OUTER JOIN
 (SELECT
    date_trunc('minute', market_summary."timestamp") as day,
    LAST(current, timestamp) AS close
    FROM market_summary
  WHERE created_at >= '2017-11-01 09:00'
    AND created_at < '2017-11-01 23:59'
    GROUP BY day
 ) results
ON (date = results.day)
ORDER BY date
like image 453
Edwin Pauli Avatar asked Nov 02 '17 10:11

Edwin Pauli


People also ask

How do you handle null values in PostgreSQL?

nullif also used with the coalesce function to handle the null values. PostgreSQL nullif function returns a null value if provided expressions are equal. If two expressions provided are equal, then it provides a null value; as a result, otherwise, it will return the first expression as a result.

How do I get previous non-null values in SQL?

With a subquery in the SELECT clause The first method uses a subquery inside the SELECT clause to get the first non-NULL value before the current row. First, we create a subquery that returns the first non-NULL value before the current row. The last non-NULL value carries forward but only after the starting row.

Can we insert null in foreign key column PostgreSQL?

A foreign key containing null values cannot match the values of a parent key, since a parent key by definition can have no null values. However, a null foreign key value is always valid, regardless of the value of any of its non-null parts.

How do I use coalesce in PostgreSQL?

The COALESCE function returns the first of its arguments that is not null. Null is returned only if all arguments are null. It is often used to substitute a default value for null values when data is retrieved for display, for example: SELECT COALESCE(description, short_description, '(none)') ...


1 Answers

I found the following method easier:

Create the given data sample:

WITH example (date,close) AS 
(VALUES 
    ('12:00:00',3),
    ('12:00:01',4),
    ('12:00:02',5),
    ('12:00:03',NULL),
    ('12:00:04',NULL), 
    ('12:00:05',3)
) 
SELECT * INTO temporary table market_summary FROM example;

Query to fill NULL values with the previous filled value

select 
    date, 
    close, 
    first_value(close) over (partition by grp_close) as corrected_close
from (
      select date, close,
             sum(case when close is not null then 1 end) over (order by date) as grp_close
      from   market_summary
) t

Return

date      | close | corrected_close
-----------------------------------
12:00:00  | 3     | 3
12:01:00  | 4     | 4
12:02:00  | 5     | 5
12:03:00  | NULL  | 5
12:04:00  | NULL  | 5
12:05:00  | 3     | 3
  • close: existing value
  • corrected_close: corrected value
like image 131
benjamin berhault Avatar answered Sep 24 '22 12:09

benjamin berhault