Using Opam to manage project dependencies

Question

I am a complete newbie to OCaml. Other languages I have used (for instance Scala, Clojure, Javascript on Node.js) have package managers that allow one to start a project as a clean slate that has a declared set of dependencies of known versions.

I am trying to do something of the sort with Opam. Ideally, I would like to have a file that lists dependencies (and possibly OCaml version) so that a collaborator can start a project with

git clone myproject
<magic opam command>
ocamlbuild

and have a working version, without having installed anything globally.

I understand the right way to do this would be to use Opam switches, but I am not sure what to do in practice. It seems that switches are tied to a compiler version, rather than per-project (altough there are aliases), and also I could not find whether there exists a sort of project Opam configuration file.

In short: say one wants to start a new project depending on Core and Yojson (for the sake of example). What would be the steps to have a clean reproducible build? Here, clean means that it won't interfere with existing installed libraries, and reproducible means that it will work on a clean machine with a freshly-cloned project.

Answer 1

It can't be really clean with respect to system libraries. Otherwise you need to start your own VM or some other container. But with respect to OCaml environment, you can achieve your goal with opam file in the root of your project. After you've described all your dependencies (including system one) in it, you can pin your project, and this will install all your dependencies, compile your project and deploy it to the opam stack. So the workflow is the following:

 $ # install opam 1.2
 $ # install aspcud (optionally, but highly recommended)
 $ opam switch install fresh -A 4.02.1
 $ opam pin add proj /path/to/proj -n
 $ opam depext --install proj
 # optional part:
 $ edit proj/src/main.ml # do the development
 $ opam upgrade proj

Now let's walk through this workflow in a step-by-step manner.

Install compiler

 $ opam switch install fresh -A 4.02.1 

This command creates a new compiler installation. Here fresh has no special meaning it is just an arbitrary name for the installation. Usually, I use date +"%y%m%d" command instead of fresh that creates a name consisting of current year, month and day.

Pin the project

$ opam pin add proj /path/to/proj -n

This command will introduce your project to OPAM system. It is like creating your own small repository of packages, containing only one package, the proj. The name proj is of course just a name of your project, whatever it is. pin uses opam file that describes your project to the OPAM system, you can create it manually using this instructions. Or you can allow pin command to create it for you. It will drive you gently through the process, asking some questions. You can read more about pinning in this blog post.

Installing package and dependencies

In the previous command we add -n flag, that will stop pin command from installing your package just after the pinning, because we want to move in small steps.

$ opam depext --install proj

This command will compute a transitive closure of your package dependencies and install them, including your system dependencies (if you're on ubuntu or fedora, and if you've specified your dependencies in opam file in a previous step).

Working on the project

Suppose you want to develop the code. There is an OPAM friendly workflow:

 $ edit proj/src/main.ml # do the development
 $ opam upgrade proj

This will reinstall your packages (and reinstall all dependents of your package, if they exist).

This workflow, on the other hand, has a severe drawback if your project is not tiny, as it will copy all the sources and compile them from scratch. But if you have a set of dependent packages on which you're working in parallel, then it is a way to do this.

Compiling and other stuff

But this is only about package management and inter-package dependencies. OPAM is absolutely agnostic to a particular build system, and don't give any favor to any of them (but still have tooling support for some of them). So, you can write a Makefile by yourself or have your own set of shell scripts invoking ocamlbuild this is absolutely up to you. But if I were in you, than I will use OASIS to manage my building process. OASIS is not a build system by itself, its purpose is to manage build systems in a cross-platform way. But by default it uses ocamlbuild and it integrates with it smoothly. Also, it has an integration with OPAM (in fact it is OPAM who has the integration with OASIS). There is a oasis2opam package, that will create an opam file from _oasis file. What about creating _oasis file, that describes building process of your project, then you can either create it by hand with your favorite text emacs, or you can allow OASIS to create it for you with oasis quickstart.

As a final note, I would suggest you to view existing projects and use them as a source of inspiration. You can start with BAP project, that I'm maintaining, but it has rather complex configuration. So, there might be a simpler examples.