Slick codegen & tables with > 22 columns

Question

I'm new to Slick. I'm creating a test suite for a Java application with Scala, ScalaTest and Slick. I'm using slick to prepare data before the test and to do assertions on the data after the test. The database used has some tables with more than 22 columns. I use slick-codegen to generate my schema code.

For tables with more than 22 columns, slick-codegen does not generate a case class, but a HList-based custom type and a companion ‘constructor’ method. As I understand it, this is because the limitation that tuples and case classes can only have 22 fields. The way the code is generated, the fields of a Row-object can only be accessed by index.

I have a couple of questions about this:

1. For what I understand, the 22 fields restriction for case classes is already fixed in Scala 2.11, right?
2. If that's the case, would it be possible to customize slick-codegen to generate case classes for all tables? I looked into this: I managed to set override def hlistEnabled = false in an overridden SourceCodeGenerator. But this results in Cannot generate tuple for > 22 columns, please set hlistEnable=true or override compound. So I don’t get the point of being able to disbale HList. May be the catch is in the ‘or override compound’ part, but I don't understand what that means.
3. Searching the internet on slick and 22 columns, I came across some solutions based on nested tuples. Would it be possible to customize the codegen to use this approach?
4. If generating code with case classes with > 22 fields is not a viable option, I think it would be possible to generate an ordinary class, which has an ‘accessor’ function for each column, thus providing a ‘mapping’ from index-based access to name-based access. I’d be happy to implement the generation for this myself, but I think I need some pointers where to start. I think it should be able to override the standard codegen for this. I already use an overridden SourceCodeGenerator for some custom data types. But apart from this use case, the documentation of the code generator does not help me that much.

I would really appreciate some help here. Thanks in advance!

Answer 1

I ended up further customizing slick-codegen. First, I'll answer my own questions, then I'll post my solution.

Answers to questions

1. The 22 arity limit might be lifted for case classes, it is not for tuples. And slick-codegen also generates some tuples, which I was not fully aware of when I asked the question.
2. Not relevant, see answer 1. (This might become relevant if the 22 arity limit gets lifted for tuples as well.)
3. I chose not to investigate this further, so this question remains unanswered for now.
4. This is the approach I took, eventually.

Solution: the generated code

So, I ended up generating "ordinary" classes for tables with more than 22 columns. Let me give an example of what I generate now. (Generator code follows below.) (This example has less than 22 columns, for brevity and readability reasons.)

case class BigAssTableRow(val id: Long, val name: String, val age: Option[Int] = None)

type BigAssTableRowList = HCons[Long,HCons[String,HCons[Option[Int]]], HNil]

object BigAssTableRow {
  def apply(hList: BigAssTableRowList) = new BigAssTableRow(hlist.head, hList.tail.head, hList.tail.tail.head)
  def unapply(row: BigAssTableRow) = Some(row.id :: row.name :: row.age)
}

implicit def GetResultBoekingenRow(implicit e0: GR[Long], e1: GR[String], e2: GR[Optional[Int]]) = GR{
  prs => import prs._
  BigAssTableRow.apply(<<[Long] :: <<[String] :: <<?[Int] :: HNil)
}

class BigAssTable(_tableTag: Tag) extends Table[BigAssTableRow](_tableTag, "big_ass") {
  def * = id :: name :: age :: :: HNil <> (BigAssTableRow.apply, BigAssTableRow.unapply)

  val id: Rep[Long] = column[Long]("id", O.PrimaryKey)
  val name: Rep[String] = column[String]("name", O.Length(255,varying=true))
  val age: Rep[Option[Int]] = column[Option[Int]]("age", O.Default(None))
}

lazy val BigAssTable = new TableQuery(tag => new BigAssTable(tag))

The hardest part was finding out how the * mapping works in Slick. There's not much documentation, but I found this Stackoverflow answer rather enlightening.

I created the BigAssTableRow object to make the use of HList transparent for the client code. Note that the apply function in the object overloads the apply from the case class. So I can still create entities by calling BigAssTableRow(id: 1L, name: "Foo"), while the * projection can still use the apply function that takes an HList.

So, I can now do things like this:

// I left out the driver import as well as the scala.concurrent imports 
// for the Execution context.

val collection = TableQuery[BigAssTable]
val row = BigAssTableRow(id: 1L, name: "Qwerty") // Note that I leave out the optional age

Await.result(db.run(collection += row), Duration.Inf)

Await.result(db.run(collection.filter(_.id === 1L).result), Duration.Inf)

For this code it's totally transparent wether tuples or HLists are used under the hood.

Solution: how this is generated

I'll just post my entire generator code here. It's not perfect; please let me know if you have suggestions for improvement! Huge parts are just copied from the slick.codegen.AbstractSourceCodeGenerator and related classes and then slightly changed. There are also some things that are not directly related to this question, such as the addition of the java.time.* data types and the filtering of specific tables. I left them in, because they might be of use. Also note that this example is for a Postgres database.

import slick.codegen.SourceCodeGenerator
import slick.driver.{JdbcProfile, PostgresDriver}
import slick.jdbc.meta.MTable
import slick.model.Column

import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.Duration

object MySlickCodeGenerator {
  val slickDriver = "slick.driver.PostgresDriver"
  val jdbcDriver = "org.postgresql.Driver"
  val url = "jdbc:postgresql://localhost:5432/dbname"
  val outputFolder = "/path/to/project/src/test/scala"
  val pkg = "my.package"
  val user = "user"
  val password = "password"

  val driver: JdbcProfile = Class.forName(slickDriver + "$").getField("MODULE$").get(null).asInstanceOf[JdbcProfile]
  val dbFactory = driver.api.Database
  val db = dbFactory.forURL(url, driver = jdbcDriver, user = user, password = password, keepAliveConnection = true)

  // The schema is generated using Liquibase, which creates these tables that I don't want to use
  def excludedTables = Array("databasechangelog", "databasechangeloglock")

  def tableFilter(table: MTable): Boolean = {
    !excludedTables.contains(table.name.name) && schemaFilter(table.name.schema)
  }

  // There's also an 'audit' schema in the database, I don't want to use that one
  def schemaFilter(schema: Option[String]): Boolean = {
    schema match {
      case Some("public") => true
      case None => true
      case _ => false
    }
  }

  // Fetch data model
  val modelAction = PostgresDriver.defaultTables
    .map(_.filter(tableFilter))
    .flatMap(PostgresDriver.createModelBuilder(_, ignoreInvalidDefaults = false).buildModel)

  val modelFuture = db.run(modelAction)

  // customize code generator
  val codegenFuture = modelFuture.map(model => new SourceCodeGenerator(model) {

    // add custom import for added data types
    override def code = "import my.package.Java8DateTypes._" + "\n" + super.code

    override def Table = new Table(_) {
      table =>

      // Use different factory and extractor functions for tables with > 22 columns
      override def factory   = if(columns.size == 1) TableClass.elementType else if(columns.size <= 22) s"${TableClass.elementType}.tupled" else s"${EntityType.name}.apply"
      override def extractor = if(columns.size <= 22) s"${TableClass.elementType}.unapply" else s"${EntityType.name}.unapply"

      override def EntityType = new EntityTypeDef {
        override def code = {
          val args = columns.map(c =>
            c.default.map( v =>
              s"${c.name}: ${c.exposedType} = $v"
            ).getOrElse(
              s"${c.name}: ${c.exposedType}"
            )
          )
          val callArgs = columns.map(c => s"${c.name}")
          val types = columns.map(c => c.exposedType)

          if(classEnabled){
            val prns = (parents.take(1).map(" extends "+_) ++ parents.drop(1).map(" with "+_)).mkString("")
            s"""case class $name(${args.mkString(", ")})$prns"""
          } else {
            s"""
/** Constructor for $name providing default values if available in the database schema. */
case class $name(${args.map(arg => {s"val $arg"}).mkString(", ")})
type ${name}List = ${compoundType(types)}
object $name {
  def apply(hList: ${name}List): $name = new $name(${callArgs.zipWithIndex.map(pair => s"hList${tails(pair._2)}.head").mkString(", ")})
  def unapply(row: $name) = Some(${compoundValue(callArgs.map(a => s"row.$a"))})
}
          """.trim
          }
        }
      }

      override def PlainSqlMapper = new PlainSqlMapperDef {
        override def code = {
          val positional = compoundValue(columnsPositional.map(c => if (c.fakeNullable || c.model.nullable) s"<<?[${c.rawType}]" else s"<<[${c.rawType}]"))
          val dependencies = columns.map(_.exposedType).distinct.zipWithIndex.map{ case (t,i) => s"""e$i: GR[$t]"""}.mkString(", ")
          val rearranged = compoundValue(desiredColumnOrder.map(i => if(columns.size > 22) s"r($i)" else tuple(i)))
          def result(args: String) = s"$factory($args)"
          val body =
            if(autoIncLastAsOption && columns.size > 1){
              s"""
val r = $positional
import r._
${result(rearranged)} // putting AutoInc last
              """.trim
            } else {
              result(positional)
            }

              s"""
implicit def $name(implicit $dependencies): GR[${TableClass.elementType}] = GR{
  prs => import prs._
  ${indent(body)}
}
          """.trim
        }
      }

      override def TableClass = new TableClassDef {
        override def star = {
          val struct = compoundValue(columns.map(c=>if(c.fakeNullable)s"Rep.Some(${c.name})" else s"${c.name}"))
          val rhs = s"$struct <> ($factory, $extractor)"
          s"def * = $rhs"
        }
      }

      def tails(n: Int) = {
        List.fill(n)(".tail").mkString("")
      }

      // override column generator to add additional types
      override def Column = new Column(_) {
        override def rawType = {
          typeMapper(model).getOrElse(super.rawType)
        }
      }
    }
  })

  def typeMapper(column: Column): Option[String] = {
    column.tpe match {
      case "java.sql.Date" => Some("java.time.LocalDate")
      case "java.sql.Timestamp" => Some("java.time.LocalDateTime")
      case _ => None
    }
  }

  def doCodeGen() = {
    def generator = Await.result(codegenFuture, Duration.Inf)
    generator.writeToFile(slickDriver, outputFolder, pkg, "Tables", "Tables.scala")
  }

  def main(args: Array[String]) {
    doCodeGen()
    db.close()
  }
}