python inserting and retrieving binary data into mysql

Question

I'm using the MySQLdb package for interacting with MySQL. I'm having trouble getting the proper type conversions.

I am using a 16-byte binary uuid as a primary key for the table and have a mediumblob holding zlib compressed json information.

I'm using the following schema:

CREATE TABLE repositories (
    added_id int auto_increment not null,
    id binary(16) not null,
    data mediumblob not null,
    create_date int not null,
    update_date int not null,
    PRIMARY KEY (added_id),
    UNIQUE(id)
) DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ENGINE=InnoDB;

Then I create a new row in the table using the following code:

data = zlib.compress(json.dumps({'hello':'how are you :D'})
row_id = uuid.uuid(4).hex
added_id = cursor.execute('
    INSERT INTO repositories (id, data, create_date, update_date) 
    VALUES (%s, %s, %s, %s)',
    binascii.a2b_hex(row_id), 
    data, 
    time.time(), 
    time.time()
)

Then to retrieve data I use a similar query:

query = cursor.execute('SELECT added_id, id, data, create_date, update_date ' \
    'FROM repositories WHERE id = %s',
    binascii.a2b_hex(row_id)
)

Then the query returns an empty result.

Any help would be appreciated. Also, as an aside, is it better to store unix epoch dates as integers or TIMESTAMP?

NOTE: I am not having problems inserting the data, just trying to retrieve it from the database. The row exists when I check via mysqlclient.

Thanks Alot!@

Answer 1

One tip: you should be able to call uuid.uuid4().bytes to get the raw bytes. As for timestamps, if you want to perform time/date manipulation in SQL it's often easier to deal with real TIMESTAMP types.

I created a test table to try to reproduce what you're seeing:

CREATE TABLE xyz (
    added_id INT AUTO_INCREMENT NOT NULL,
    id BINARY(16) NOT NULL,
    PRIMARY KEY (added_id),
    UNIQUE (id)
) DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ENGINE=InnoDB;

My script is able to insert and query for the rows using the binary field as a key without problem. Perhaps you are incorrectly fetching / iterating over the results returned by the cursor?

import binascii
import MySQLdb
import uuid

conn = MySQLdb.connect(host='localhost')

key = uuid.uuid4()
print 'inserting', repr(key.bytes)
r = conn.cursor()
r.execute('INSERT INTO xyz (id) VALUES (%s)', key.bytes)
conn.commit()

print 'selecting', repr(key.bytes)
r.execute('SELECT added_id, id FROM xyz WHERE id = %s', key.bytes)
for row in r.fetchall():
    print row[0], binascii.b2a_hex(row[1])

Output:

% python qu.py    
inserting '\x96\xc5\xa4\xc3Z+L\xf0\x86\x1e\x05\xebt\xf7\\\xd5'
selecting '\x96\xc5\xa4\xc3Z+L\xf0\x86\x1e\x05\xebt\xf7\\\xd5'
1 96c5a4c35a2b4cf0861e05eb74f75cd5
% python qu.py
inserting '\xac\xc9,jn\xb2O@\xbb\xa27h\xcd<B\xda'
selecting '\xac\xc9,jn\xb2O@\xbb\xa27h\xcd<B\xda'
2 acc92c6a6eb24f40bba23768cd3c42da

Answer 2

To supplement existing answers, there's also an issue with the following warning when dealing with binary strings in queries:

Warning: (1300, "Invalid utf8 character string: 'ABCDEF'") 

It is reproduced by the following:

cursor.execute('''
    CREATE TABLE `table`(
        bin_field` BINARY(16) NOT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
''')

bin_value = uuid.uuid4().bytes
cursor.execute('INSERT INTO `table`(bin_field) VALUES(%s)', (bin_value,))

Whenever MySQL sees that a string literal in a query isn't valid against current character_set_connection it will emit the warning. There are several solutions to it:

1. Explicitly set _binary charset literal

    INSERT INTO `table`(bin_field) VALUES(_binary %s)
   

2. Manually construct queries with hexadecimal literals

    INSERT INTO `table`(bin_field) VALUES(x'abcdef')
   

3. Change connection charset if you're only working with binary strings

For more details see MySQL Bug 79317.

Update

As @charlax pointed out, there's binary_prefix flag which can be passed to the connection's initialiser to automatically prepend _binary prefix when interpolating arguments. It's supported by recent versions of both, mysql-client and pymysql.