Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to compare two tables and return rows with difference with HIVE

So lets say I have a table with about 180 columns and 100 records. This table is backed up into temporary table and original one is removed. After this migration (change) is run on a pipeline which produces the same table. I want to compare the backed up table to the new one adn rows (records) with any difference to be moved to 3rd table (_result table) so I do:

INSERT OVERWRITE TABLE
  zakj_customers.customers_detail_result
SELECT
  acct_id, IF (a.title != b.title, 1, 0) title, IF (a.fname != b.fname, 1, 0) fname, IF (a.dob != b.dob, 1, 0) dob, IF (a.cr_date != b.cr_date, 1, 0) cr_date
FROM
  zakj_customers.customers_detail a
LEFT OUTER JOIN
  zakj_customers.customers_detail_backup b
ON
  (a.acct_id = b.acct_id)
ORDER BY 
  title DESC,fname DESC,dob DESC,cr_date DESC
HAVING
  title > 0 AND fname > 0 AND dob > 0 AND cr_date > 0
;

So oblivious this query is wrong, I'm not much into SQL, and I'm getting syntax errors, so I can't put it together right and on a ticket it was supplied in this format which is obviously wrong.

Can anyone see the way this could be done?

Cheers

like image 637
Jakub Zak Avatar asked Oct 21 '22 00:10

Jakub Zak


1 Answers

Must use "case when" instead of if:

Case When a.title <> b.title then 1 Else 0 End title

I wouldn't write having but the expression into the where condition:

INSERT Into
  zakj_customers.customers_detail_result
SELECT
  acct_id, a.title, a.fname, dob, a.cr_date
FROM
  zakj_customers.customers_detail a
LEFT OUTER JOIN
  zakj_customers.customers_detail_backup b
ON
  (a.acct_id = b.acct_id)
Where b.acct_id is null or a.title <> b.title or a.fname <> b.fname or a.cr_date <> b.cr_date;

"b.acct_id is null" is required to get the new records beacuse <> would filter out them.

(Ordering is totally unnecessary when inserting records.)

like image 161
Peter Krassoi Avatar answered Oct 23 '22 03:10

Peter Krassoi