Apache Ignite storing JSON Map in distributed cache

Question

I wrote the code below to read data from JSON file and push it into Ignite distributed cache, this code works fine, however, the requirement of creating "ContainerAgg" class is a problem for me. Our data structure is not predefined extract gets generated dynamically based on the user selections.

I tried using BinaryObject however when I use the BinaryObject I am not able to run a SQL query, do you have any samples that use BinaryObject and do not use precompiled java class for the schema.

There is this "StreamVisitorExample", however, this uses a Precompiled Java Class (Instrument.class)

  public static void main(String[] args) {
    ObjectMapper mapper = new ObjectMapper();
    try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) {
      if (!ExamplesUtils.hasServerNodes(ignite))
        return;

      CacheConfiguration<String, ContainerAgg> config = new CacheConfiguration<>(MASSIVE_CACHE);

      config.setIndexedTypes(String.class, ContainerAgg.class);

      ignite.getOrCreateCache(config);


      try (BufferedReader br = new BufferedReader(new FileReader(args[0]))) {

        IgniteCache<String, ContainerAgg> cache = Ignition.ignite().cache(MASSIVE_CACHE);

        String line = null;
        Long cnt = 0L;
        while ((line = br.readLine()) != null) {
          ContainerAgg inst = mapper.readValue(line, ContainerAgg.class);
          cache.put(cnt.toString(), inst);
          cnt++;
        }


        long startTime = System.currentTimeMillis();
        String sql =
            "SELECT SFID, LABEL, PARTID, PROVIDERID, SUM(TOTALCNT) "
            + "FROM CONTAINERAGG GROUP BY SFID, LABEL, PARTID, PROVIDERID";

        QueryCursor<List<?>> cursor = cache.query(new SqlFieldsQuery(sql));
        long endTime = System.currentTimeMillis();

        for (List<?> row : cursor.getAll()){
          System.out.println(row);
        }
        System.out.println("Total Time: " + (endTime - startTime));

      } catch (FileNotFoundException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }
    }
  }

}

To give you more context on what I did with BinaryObject. I converted the JSON into a map and added each entry in BinaryObjectBuilder and created BinaryObject instance and stored it in IgniteCache<String, BinaryObject>

Answer 1

BinaryObject is the way to go. To run SQL queries you need to configure indexed fields via CacheConfiguration.QueryEntities (see https://apacheignite.readme.io/docs/indexes#queryentity-based-configuration).

However, you can only configure query entities once for a cache. So when your schema changes, you have to destroy the cache and create a new one with updated QueryEntity configuration.

There is no Java example for this use case. However, you can have a look at C# example, APIs are quite similar: https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Apache.Ignite.Examples/Datagrid/BinaryModeExample.cs