redis presto connector corrupt key when redis.key-prefix-schema-table=true for json dataFormat

Question

I am trying to setup a working example of presto and redis on my local machine according to the (limited) presto-redis documentation.

Summary of Problem:

When using redis.key-prefix-schema-table=true and prefixing a redis key with dev:simple_table: (as instructed by the presto redis connector page), all key columns are null, and the internal column _key_corrupt is true. All value columns are parsed correctly.

When using redis.key-prefix-schema-table=falseand using a pure JSON key and value strings, both work as expected. (note: I have tried and failed to use the csv dataFormat as well)

What works:

I have an example working with redis server and presto server running on my local machine.

presto's etc/catalog/redis.properties file:

connector.name=redis
redis.table-names=simple_table
redis.default-schema=dev
redis.nodes=127.0.0.1:6379
redis.table-description-dir=/Users/acarson/var/redis/tables
redis.key-delimiter=:
redis.key-prefix-schema-table=false
redis.hide-internal-columns=false

I have a table definition file at /Users/acarson/var/redis/tables/simple_table.json

{
    "tableName": "simple_table",
    "schemaName": "dev",
    "key": {
        "dataFormat": "json",
        "fields": [
            {
                "name": "id",
                "mapping": "id",
                "type": "BIGINT"
            },
            {
                "name": "word",
                "mapping": "word",
                "type": "VARCHAR"
            }
        ]
    },
    "value": {
        "dataFormat": "json",
        "fields": [
            {
                "name": "name",
                "mapping": "name",
                "type": "VARCHAR"
            },
            {
                "name": "number",
                "mapping": "number",
                "type": "BIGINT"
            },
            {
                "name": "boolean",
                "mapping": "boolean",
                "type": "BOOLEAN"
            }
        ]
    }
}

using redis-cli, I insert a value with this command:

 SET '{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'

And I can then query the data with presto cli and the resulting columns show up as expected:

presto:dev> SELECT * FROM simple_table;
 id | word | name | number | boolean |           _key            |                      _value                      | _key_length | _value_length | _key_corrupt | _value_corrupt 
----+------+------+--------+---------+---------------------------+--------------------------------------------------+-------------+---------------+--------------+----------------
 42 | foo  | bar  |      3 | false   | {"id": 42, "word": "foo"} | {"name": "bar", "number": 3, "boolean": "false"} |          25 |            48 | false        | false          

What fails:

I switch the redis.properties value redis.key-prefix-schema-table to true, restart presto, delete all keys in redis-cli with FLUSHALL and then insert a new row using the schema table prefix:

SET 'dev:simple_table:{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'

Querying from presto cli shows _key_corrupt=true and the key values are null, even though the values still work.

presto:dev> SELECT * FROM simple_table;
  id  | word |  name  | number | boolean |                     _key                      |                       _value                        | _key_length | _value_length | _key_corrupt | _value_corrupt 
------+------+--------+--------+---------+-----------------------------------------------+-----------------------------------------------------+-------------+---------------+--------------+----------------
 NULL | NULL | bar    |      3 | false   | dev:simple_table:{"id": 42, "word": "foo"}    | {"name": "bar", "number": 3, "boolean": "false"}    |          42 |            48 | true         | false          

I've increased the log levels of redis and presto for hints, but nothing that is showing me any errors or reason for why the key is corrupt. I have a feeling my syntax for redis keys might be incorrect, but I am following the exact method described both in the presto redis documentation and even when reading through presto-redis source which specifies keys use "schema:table:*" format with this flag.

Here's the presto logs at debug level when the query is run:

2016-10-20T17:09:55.048-0700    INFO    main    com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
2016-10-20T17:10:24.785-0700    DEBUG   query-execution-0   com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is PLANNING
2016-10-20T17:10:24.802-0700    DEBUG   Query-20161021_001024_00000_qx72p-104   com.facebook.presto.redis.RedisTableDescriptionSupplier Considering files: [/Users/acarson/var/redis/tables/simple_table.json]
2016-10-20T17:10:24.849-0700    DEBUG   Query-20161021_001024_00000_qx72p-104   com.facebook.presto.redis.RedisTableDescriptionSupplier Redis table dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}}
2016-10-20T17:10:24.850-0700    DEBUG   Query-20161021_001024_00000_qx72p-104   com.facebook.presto.redis.RedisTableDescriptionSupplier Loaded table definitions: [dev.simple_table]
2016-10-20T17:10:24.850-0700    DEBUG   Query-20161021_001024_00000_qx72p-104   com.facebook.presto.redis.RedisTableDescriptionSupplier Found Table definition for dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}}
2016-10-20T17:10:25.020-0700    DEBUG   query-execution-0   com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is STARTING
2016-10-20T17:10:25.027-0700    DEBUG   query-execution-1   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULING
2016-10-20T17:10:25.064-0700    DEBUG   query-execution-1   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULED
2016-10-20T17:10:25.065-0700    DEBUG   query-execution-2   com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is RUNNING
2016-10-20T17:10:25.119-0700    DEBUG   query-execution-2   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is RUNNING
2016-10-20T17:10:25.165-0700    DEBUG   query-execution-2   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULING
2016-10-20T17:10:25.174-0700    DEBUG   query-execution-2   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULED
2016-10-20T17:10:25.179-0700    DEBUG   query-execution-2   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is RUNNING
2016-10-20T17:10:25.278-0700    INFO    20161021_001024_00000_qx72p.1.0-0-58    com.facebook.presto.redis.RedisJedisManager Creating new JedisPool for 127.0.0.1:6379
2016-10-20T17:10:25.313-0700    DEBUG   20161021_001024_00000_qx72p.1.0-0-58    com.facebook.presto.redis.RedisRecordCursor Scanning new Redis keys from cursor 0 . 0 values read so far
2016-10-20T17:10:25.326-0700    DEBUG   20161021_001024_00000_qx72p.1.0-0-58    com.facebook.presto.redis.RedisRecordCursor Read a total of 1 values with 48 bytes.
2016-10-20T17:10:25.330-0700    DEBUG   20161021_001024_00000_qx72p.1.0-0-58    com.facebook.presto.execution.TaskExecutor  Split 20161021_001024_00000_qx72p.1.0-0 RedisSplit{connectorId=redis, schemaName=dev, tableName=simple_table, keyDataFormat=json, valueDataFormat=json, keyName=null, start=0, end=-1, nodes=[127.0.0.1:6379]} (start = 1477008625258, wall = 72 ms, cpu = 56 ms, calls = 1) is finished
2016-10-20T17:10:25.350-0700    DEBUG   http-worker-77  com.facebook.presto.execution.SqlTask   Aborting task 20161021_001024_00000_qx72p.1.0 output 0
2016-10-20T17:10:25.352-0700    DEBUG   task-notification-1 com.facebook.presto.execution.TaskStateMachine  Task 20161021_001024_00000_qx72p.1.0 is FINISHED
2016-10-20T17:10:25.357-0700    DEBUG   query-execution-0   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is FINISHED
2016-10-20T17:10:25.367-0700    DEBUG   20161021_001024_00000_qx72p.0.0-0-59    com.facebook.presto.execution.TaskExecutor  Split 20161021_001024_00000_qx72p.0.0-0  (start = 1477008625257, wall = 110 ms, cpu = 9 ms, calls = 4) is finished
2016-10-20T17:10:25.369-0700    DEBUG   http-worker-88  com.facebook.presto.execution.SqlTask   Aborting task 20161021_001024_00000_qx72p.0.0 output 0
2016-10-20T17:10:25.372-0700    DEBUG   task-notification-0 com.facebook.presto.execution.TaskStateMachine  Task 20161021_001024_00000_qx72p.0.0 is FINISHED
2016-10-20T17:10:25.379-0700    DEBUG   query-execution-0   com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is FINISHED
2016-10-20T17:10:25.380-0700    DEBUG   query-execution-2   com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHING
2016-10-20T17:10:25.383-0700    DEBUG   query-execution-2   com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHED
2016-10-20T17:10:25.420-0700    INFO    query-execution-2   com.facebook.presto.event.query.QueryMonitor    TIMELINE: Query 20161021_001024_00000_qx72p :: Transaction:[c54dc7fe-8159-434d-b4cc-cb13ad41a5d7] :: elapsed 610ms :: planning 247ms :: scheduling 248ms :: running 0ms :: finishing 363ms :: begin 2016-10-20T17:10:24.773-07:00 :: end 2016-10-20T17:10:25.383-07:00

Answer 1

Maybe this line needs escaping the : Did you try removing it, as : is the default anyways.

redis.key-delimiter=:

Also your prefix has a tailing: which should be left out:

SET 'dev:simple_table{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'

instead of

SET 'dev:simple_table:{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'