Unable to create a composite index, stuck at INSTALLED

Question

I'm unable to create an index. My Gremlin code is as follows:

usernameProperty = mgmt.getPropertyKey('username')
usernameIndex = mgmt.buildIndex('byUsernameUnique', Vertex.class).addKey(usernameProperty).unique().buildCompositeIndex()
mgmt.setConsistency(usernameIndex, ConsistencyModifier.LOCK)
mgmt.commit()

Shortly after I receive two errors:

18:04:57 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@0a00009d2537-ip-10-0-0-1572] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x6549ce71]] 18:04:57 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@0a00009d2537-ip-10-0-0-1572] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x2a2815cc], standardtitantx[0x025dc2c0]]

The status of the index is stuck at INSTALLED:

usernameIndex.getIndexStatus(usernameProperty)
==>INSTALLED

I read that a failed instance could cause the issue, but a check of running instances shows just one:

mgmt.getOpenInstances()
==>0a00009d3011-ip-10-0-0-1572(current)

I've also tried issuing a REGISTER_INDEX action, which also gets evicted from the transaction cache with a similar error message:

mgmt.updateIndex(usernameIndex, SchemaAction.REGISTER_INDEX).get()
mgmt.commit()

I've also tried restarting the server multiple times.

It seems like the registration process is simply timing out, causing an "eviction" from the transaction cache. I've waited 48 hours just to be sure it wasn't a slow process. Normal reads, writes, and associated commits to Titan do seem to be working correctly, I'm just not able to create this index. I'm stuck, is there something else I can try? Is there a way to extend the timeout on that transaction?

I'm running Titan 1.0.0 using a DynamoDB backend (setup with the AWS provided CloudFormation template).

EDIT: Here is the complete command I'm pasting into Gremlin with the addition of the awaitGraphStatus step suggested by @M-T-A:

mgmt = graph.openManagement();
usernameIndex = mgmt.getPropertyKey('usernameIndex');
mgmt.buildIndex('byUsername',Vertex.class).addKey(usernameIndex).unique().buildCompositeIndex();
// I have tried with and without a commit here: mgmt.commit();
mgmt.awaitGraphIndexStatus(graph, 'byUsername').status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();

This results in the following error:

java.lang.NullPointerException at com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:52) at com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:18) at java_util_concurrent_Callable$call.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at groovysh_evaluate.run(groovysh_evaluate:3) at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215) at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:69) at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:185) at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:119) at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:94)

I will also note that the routine to disable and delete an index is also failing.

mgmt = graph.openManagement()
theIndex = mgmt.getGraphIndex('byUsername')
mgmt.updateIndex(theIndex, SchemaAction.DISABLE_INDEX).get()
mgmt.commit()
mgmt.awaitGraphIndexStatus(graph, 'byUsername').status(SchemaStatus.DISABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
m = graph.openManagement()
i = m.getGraphIndex('byUsername')
m.updateIndex(i, SchemaAction.REMOVE_INDEX).get()
m.commit()

19:26:26 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@ac1f3fa810472-ip-172-31-63-1681] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x2314cd97], standardtitantx[0x39f8adc0], standardtitantx[0x09de1b85]]

EDIT 2: Attempting to create a new property key and index within the same transaction DOES WORK! But does this mean that I can't create indexes on existing property keys??

graph.tx().rollback();
mgmt = graph.openManagement();
indexName = 'byUsernameTest2';
propertyKeyName = 'testPropertyName2';
propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType(String.class).cardinality(Cardinality.SINGLE).make();
mgmt.buildIndex(indexName,Vertex.class).addKey(propertyKey).buildCompositeIndex();
mgmt.commit();
graph.tx().commit();
mgmt.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
mgmt.commit();

After a pause, this results in:

This management system instance has been closed

Trying to fetch the new index results in:

mgmt = graph.openManagement();
index = mgmt.getGraphIndex('byUsernameTest2');
propkey = mgmt.getPropertyKey('testPropertyName2');
index.getIndexStatus(propkey);

==>ENABLED

Answer 1

If you're running multiple Titan instances, you should be aware that they need to coordinate before the index will become available.

Furthermore, there are various subtleties around transaction management and under what circumstances transactions will be left open; I believe Titan 1.0.0 doesn't have the latest from TinkerPop in that regard. Have you tried creating the index immediately after boot?

Finally, the index creation process is different depending on whether or not the property keys being indexed have been used before. Are you attempting to index new keys, or existing ones?

Answer 2

You need to wait for the Titan and DynamoDB to get the index registered. You could do so by:

ManagementSystem.awaitGraphIndexStatus(graph, propertyKeyIndexName)
                    .status(SchemaStatus.REGISTERED)
                    .timeout(10, java.time.temporal.ChronoUnit.MINUTES) // set timeout to 10 min
                    .call();

The default timeout is usually not long enough, so you could increase it to 10 minutes, it usually does the trick with Dynamo backened.

Only when the index is in REGISTERED state, you could perform a reindex. ONce the reindex is done, you need to wait until it's ENABLED. by reusing the code sample above and changing the state to ENABLED.

For more info, see the docs.

Edit

Let me share the code that works with me on Berkeley and Dynamo DB backends all the time.

    graph.tx().rollback(); //Never create new indexes while a transaction is active
    TitanManagement mgmt=graph.openManagement();
    PropertyKey propertyKey=getOrCreatePropertyKeyIfNotExist(mgmt, propertyKeyName);
    String indexName = makePropertyKeyIndexName(propertyKey);

    if (mgmt.getGraphIndex(indexName)==null) { 
        mgmt.buildIndex(indexName, Vertex.class).addKey(propertyKey).buildCompositeIndex();
        mgmt.commit(); // you MUST commit mgmt
        graph.tx().commit(); // and commit the transaction too
        ManagementSystem.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED).call();
    }else { // already defined.
        mgmt.rollback();
        graph.tx().rollback();
    }

private static PropertyKey getOrCreatePropertyKeyIfNotExist(TitanManagement mgmt, String s) {
    PropertyKey key = mgmt.getPropertyKey(s);
    if (key != null)
        return key;
    else
        return mgmt.makePropertyKey(s).dataType(String.class).make();
}

private static String makePropertyKeyIndexName(PropertyKey pk) {
    return pk.name() + Tokens.INDEX_SUFFIX;
}

From the error that I saw, it seems like Titan couldn't get the index which means you're waiting for the index that is not even defined. Have a look at the line that causes the error here.

Make sure you're passing the right index name to the awaitGraphIndexStatus.