PyMySQL executemany with ON DUPLICATE

Question

I have a list composed of dictionaries called member that I was inserting on a database like so

    # Executes query for each dictionary in member.
    cursor.executemany("INSERT INTO `Citizens` (`Handle`,`Org`,`Role`, "
                       "`Rank`,`Visibility`,`Stars`,`Type`) VALUES "
                       "(%(handle)s,%(sid)s,%(roles)s,%(rank)s,"
                       "%(visibility)s,%(stars)s,%(type)s)", member)
    # Commits changes to the database.
    conn.commit()

It worked perfectly, but as I began having issues with duplicate members I decided to add an ON DUPLICATE clause. The idea is that if a member is duplicate we want to update his Org column by concatenating the new data. So I changed the code to this

    # Executes query for each dictionary in member.
    cursor.executemany("INSERT INTO `Citizens` (`Handle`,`Org`,`Role`, "
                       "`Rank`,`Visibility`,`Stars`,`Type`) VALUES "
                       "(%(handle)s,%(sid)s,%(roles)s,%(rank)s,"
                       "%(visibility)s,%(stars)s,%(type)s) ON DUPLICATE"
                       " KEY UPDATE `Org`=concat(ifnull(`Org`, \"\"), "
                       "\", \", %(sid)s);", member)
    # Commits changes to the database.
    conn.commit()

However in doing so I got the following error:

There was a problem inserting member(s): (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('real_lethality','000',0,'Master','visible',5,'affiliate') ON DUPLICATE KEY UPD' at line 3")
Would you like to see the failed SQL query?[Y/n]
b'INSERT INTO `Citizens` (`Handle`,`Org`,`Role`, `Rank`,`Visibility`,`Stars`,`Type`) VALUES \n(\'fullmetaljim\',\'000\',\'Founder\',\'Master\',\'visible\',5,\'main\') ON DUPLICATE KEY UPDATE `Org`=concat(ifnull(`Org`, ""), ", ", \'000\'),\n(\'real_lethality\',\'000\',0,\'Master\',\'visible\',5,\'affiliate\') ON DUPLICATE KEY UPDATE `Org`=concat(ifnull(`Org`, ""), ", ", \'000\')\n;'

Have I implemented the ON DUPLICATE wrongly? How can I make this work correctly?

Sample member:

[
    {'roles': [],
     'rank': 'No SCB account',
     'type': 'main',
     'stars': 2,
     'visibility': 'visible',
     'sid': 'imperium',
     'handle': 'freakyeagle'
    },
    {'roles': [],
     'rank': 'Fleet Member',
     'type': 'main',
     'stars': 1,
     'visibility': 'visible',
     'sid': 'imperium',
     'handle': 'cadimus'},
    {'roles': [],
     'rank': 'Fleet Member',
     'type': 'main',
     'stars': 1,
     'visibility': 'visible',
     'sid': 'imperium',
     'handle': 'belleal'}
]

EDIT: As it seems this might be a bug in PyMySQL itself, would anyone be able to confirm so?

Answer 1

Looking at your code and query from the error I'd agree that the constructed query by executemany method is incorrect. I'd recommend you build your insert in more manual manner looping through the parameters and just execute it without executemany method.

Also MySQL AFAIK is perfectly ok with using single quotes so if you put query in double quotes use single quotes as internal string qualifiers - this way you don't need to escape all the time.