How to force conan to build from source, but only if it is not in the cache?

Question

I am using conan in an enterprise environment where the operating system is rather old and has an old version of glibc (2.11). As a result, a lot of the pre-built binaries on conan.io do not end up working in my environment. However, conan doesn't know that and will happily download and install them on my system, resulting in link-time errors.

I have found that if I build from source I can get the libraries to work.

My desired behavior would be as follows:

• The first time using conan install to install the library (e.g. it is not in my cache) then conan will build from source and place it in my cache, then use it.
• On subsequent invocations of conan install, conan finds the cached library and uses that without having to rebuild from source.
• I am invoking conan install as part of an automated build script, so I would like to not have to modify the invocation depending on if this is the first time the library is installed or not (but modifying a configuration file is fine).

I have had troubles obtaining this behavior in practice. Here are the challenges I have run into:

• If I use conan install --build=thelibrary then conan will rebuild that library from source every time I invoke conan install --build=thelibrary, even if it is already present in my cache.
• If I use conan install --build=missing, then Ican trick conan into building the library by setting some build options that do not have a pre-built binary associated with them.
  • This is fragile, as it only works for projects with enough build options that it is not tractable to create pre-built options for all combinations.
  • It also doesn't work if all the build options I need correspond to a pre-built binary.

Here is what I am looking for (and I assume exists but am not able to find):

• Some setting I can place in my conanfile.txt (or some other configuration file) that tells conan to ignore pre-built binaries for a given library or libraries and instead build from source, but use the cached version if it is available.
  • This ideally should work without me having to tinker with build options.
• I don't necessarily want to build all libraries from source, just the ones that won't run on my ancient OS, but if I have to settle for "all-or-nothing" I will take "all".

Is this possible with conan?

Answer 1

glibc version is an old headache for Conan, because it's not part of settings, thus is not counted as part of package ID. The Conan Docker images are running Ubuntu, some of them are old, others are new. But there is a specific Docker image running CentOS6, which was created because of glibc 2.12 and could help with package generation.

For your specific case, we have few options:

• Add glibc as part of settings, so Conan won't replace your package because of its package ID. As you should have more coworkers, you can use conan config command for settings distribution.

  # ~/.conan/settings.yml
  glibc: [None, 2.11, ...]
  

  Adding it, you can update you profile too, making glibc=2.11 as a default setting.

• Another alternative is package revisions feature, where you can lock a specific binary package for usage, which means, you want use that specific package. You just need to upload your generated package with glibc and use its binary package revision, e.g. lib/1.0@conan/stable#RREV:PACKAGE_ID#PREV

Also, answering your question:

Some setting I can place in my conanfile.txt (or some other configuration file) that tells conan to ignore pre-built binaries for a given library or libraries and instead build from source, but use the cached version if it is available.

Your cache is Conan first option, it will look for a pre-built package there first, if it's not available, it will look into your remotes, following a sorted order. Your request is not possible, first, because conanfile.txt doesn't support build policies, second, because conanfile.py only supports build all from sources, or build only missing.

My propose is, install an Artifactory instance, build what you need, upload your custom packages, and make it as your default remote.

I don't necessarily want to build all libraries from source, just the ones that won't run on my ancient OS, but if I have to settle for "all-or-nothing" I will take "all".

You can associate some package reference to a remote, running conan remote command. Let's say you want to download zlib/1.2.11 built with glibc-2.11 and it's available only in your organization remote:

$ conan remote add_ref zlib/1.2.11@org/stable my_org_repo
$ conan remote list_ref # only to validate, not mandatory
zlib/1.2.11@org/stable: my_org_repo

Now your specific package is associated to your organization. Conan still will look for that package your local cache first, but when not found, it will try to find at your Artifactory.

As you can see, your case could be solved easier using a new setting, instead of trying to hack build policies. As another alternative, you can replace glib setting by distro and its version.