I have a java app that is streaming data from possibly large files into a postgres RDS via a JDBC connection.
I am using the following command which works for null integer types.
COPY tableName FROM STDIN with (format csv, delimiter E'\u0001', NULL '', QUOTE E'\u0005')
However this does not work for null date fields (PSQLException: ERROR: invalid input syntax for date: "")
If I modify the command to
COPY tableName FROM STDIN with (format csv, delimiter E'\u0001', NULL '\N', QUOTE E'\u0005')
It works for date fields, but not for integer fields. (PSQLException: ERROR: invalid input syntax for integer: "\N")
I've seen similar questions on here addressing either nulls for integer fields or nulls for date fields, but not both, so I'm wondering if there is a way to specify null that will work with both integer and date fields (or a way to specify multiple different null strings for different data types)
I would really like to use COPY as the performance is much better than parsing the file and inserting each record, so hoping this is possible
An integer column can be null, but '' is an empty string not null. The right syntax for a null integer (or any other sql type) is null .
Example - With INSERT Statement INSERT INTO contacts (first_name, last_name) SELECT first_name, last_name FROM employees WHERE employee_number IS NULL; This PostgreSQL IS NULL example will insert records into the contacts table where the employee_number contains a NULL value.
Empty Strings and NULL Values In CSV files, a NULL value is typically represented by two successive delimiters (e.g. ,, ) to indicate that the field contains no data; however, you can use string values to denote NULL (e.g. null ) or any unique string.
If you COPY data into a table already containing data, the new data will be appended. If you COPY TO a file already containing data, the existing data will be overwritten.
So with Postgres 9.4+ This is achievable using FORCE_NULL
based on the documentation
Match the specified columns' values against the null string, even if it has been quoted, and if a match is found set the value to NULL. In the default case where the null string is empty, this converts a quoted empty string into NULL. This option is allowed only in COPY FROM, and only when using CSV format.
So my modified SQL is as follows:
COPY tableName FROM STDIN with (format csv, delimiter E'\u0001', null '', quote E'\u0005', force_null(dateField1, dateField2,...,dateFieldN));
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With