PySpark insertInto overwrite

Question

I am trying to insert data from a data frame into a Hive table. I have been able to do so successfully using df.write.insertInto("db1.table1", overwrite = True).

I am just a little confused about the overwrite = True part -- I tried running it multiple times and it seemed to append, not overwrite. There wasn't too much in the docs, but when should I set overwrite to False vs. True?

Answer 1

df.insertInto works only if table already exists in hive.

df.write.insertInto("db.table1",overwrite=False) will append the data to the existing hive table.

df.write.insertInto("db.table1",overwrite=True) will overwrite the data in hive table.

Example:

df.show()
#+----+---+                                                                                                                                                                              
#|name| id|
#+----+---+
#|   a|  1|
#|   b|  2|
#+----+---+

#save the table to hive
df.write.saveAsTable("default.table1")

#from hive
#hive> select * from table1;
#OK
#a       1
#b       2

df.write.insertInto("moch.table1",overwrite=True)

#from hive
#hive> select * from table1;
#OK
#a       1
#b       2

#appending data to hive
df.write.insertInto("moch.table1",overwrite=False)

#from hive
#hive> select * from table1;
#OK
#a       1
#b       2
#a       1
#b       2