Cassandra spark connector write nested optional case class

Question

How to write optional case classes with cassandra spark connector ?

example :

case class User(name : String, address : Option[Address])
case class Address(street : String, city : String)

When I tried to save user to cassandra with rdd.saveToCassandra it's raise an error

Failed to get converter for field "address" of type scala.Option[Address] in User mapped to column "address" of "testspark.logs_raw"

I have tried to implement a TypeConverter but that has not worked.

However nested case classes are correctly converted to cassandra UDT and optional fields are accepted.

Any good way to deal with that without changing the data model?

Answer 1

Just for visibility. Everything works fine in the modern versions - there were a lot of changes in UDTs around SCC 1.4.0-1.6.0, plus many performance optimizations in the SCC 2.0.8. With SCC 2.5.1, RDD API correctly maps everything - for example, if we have following UDT & table:

cqlsh> create type test.address (street text, city text);
cqlsh> create table test.user(name text primary key, address test.address);
cqlsh> insert into test.user(name, address) values 
   ('with address', {street: 'street 1', city: 'city1'});
cqlsh> insert into test.user(name) values ('without address');
cqlsh> select * from test.user;

 name            | address
-----------------+-------------------------------------
    with address | {street: 'street 1', city: 'city1'}
 without address |                                null

(2 rows)

Then RDD API is able to correctly pull everything when reading data:

scala> import com.datastax.spark.connector._
import com.datastax.spark.connector._

scala> case class Address(street : String, city : String)
defined class Address

scala> case class User(name : String, address : Option[Address])
defined class User

scala> val data = sc.cassandraTable[User]("test", "user")
data: com.datastax.spark.connector.rdd.CassandraTableScanRDD[User] = CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:18

scala> data.collect
res0: Array[User] = Array(User(without address,None), 
   User(with address,Some(Address(street 1,city1))))