Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

BigQuery: convert epoch to TIMESTAMP

I'm trying to range-join two tables, like so

SELECT *
FROM main_table h
INNER JOIN
    test.delay_pairs d
ON
    d.interval_start_time_utc < h.visitStartTime
    AND h.visitStartTime < d.interval_end_time_utc

where h.visitStartTime is an INT64 epoch and d.interval_start_time_utc and d.interval_end_time_utc are proper TIMESTAMPs.

The above fails with

No matching signature for operator < for argument types: TIMESTAMP, INT64. Supported signature: ANY < ANY

Neither wrapping h.visitStartTime in TIMESTAMP() nor CAST(d.interval_start_time_utc AS INT64) work. How do I make the two comparable in BigQuery's Standard SQL dialect?

like image 299
RoyalTS Avatar asked Jun 16 '16 14:06

RoyalTS


People also ask

How do you get the current epoch time in BigQuery?

If you are using Legacy SQL, you can use TIMESTAMP_TO_SEC to convert from timestamp to epoch. So, you can do something like this: SELECT TIMESTAMP_TO_MSEC(timestamp_column) FROM [YOUR_DATASET:YOUR_TABLE]; If you are using Standard SQL, you can use UNIX_SECONDS for the same purpose.

What is the timestamp format in BigQuery?

The format is: YYYY-MM-DD HH:MM:SS (e.g. 2021-05-15 16:45:23). Timestamp type: Date, time, and time zone information are all included in timestamps. If no time zone is given, the format falls back to UTC.

How do I change TimeZone in BigQuery?

To convert any TimeZone DateTime string to UTC, one could use PARSE_TIMESTAMP using the supported TIMESTAMP Formats in BigQuery . Here PARSE_TIMESTAMP parses the IST string to a UTC TIMESTAMP (not string). Adding SAFE as prefix takes care of errors/nulls etc.

What is the difference between datetime and timestamp in BigQuery?

Datetime type: comprises both calendar date and time. It does not store time zone information: YYYY-MM-DD HH:MM:SS (e.g. ). Timestamp type: comprises date, time, and time zone information.


2 Answers

You can use timestamp conversion functions like TIMESTAMP_SECONDS, TIMESTAMP_MILLIS, TIMESTAMP_MICROS

for example, assuming your h.visitStartTime is microseconds since the unix epoch

SELECT *
FROM main_table h
INNER JOIN test.delay_pairs d
ON d.interval_start_time_utc < TIMESTAMP_MICROS(h.visitStartTime)
AND TIMESTAMP_MICROS(h.visitStartTime) < d.interval_end_time_utc  
like image 199
Mikhail Berlyant Avatar answered Oct 17 '22 15:10

Mikhail Berlyant


With standard sql you can use one of those, depending on precision:

  • DATE_FROM_UNIX_DATE - from days epoch to date
  • TIMESTAMP_SECONDS - from seconds epoch to timestamp
  • TIMESTAMP_MILLIS - from milliseconds epoch to timestamp
  • TIMESTAMP_MICROS - from microseconds epoch to timestamp

See documentation here: https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#timestamp_seconds

With legacy sql you can just use TIMESTAMP function and multiply or divide by 1000 to bring it to needed epoch type:

SELECT 
  TIMESTAMP(epoch_in_millis / 1000) AS datetime
FROM 
  my_table
like image 12
Bulat Avatar answered Oct 17 '22 16:10

Bulat