Physical layout on disk of large cross-platform C++ project with many third party dependencies

Question

I am in the process of reorganizing the physical (on disk) layout of a large cross-platform C++ project with many third party dependencies, built using CMake.

Since we need to support Windows, a platform on which there is no well-established package manager, we decided a long time ago to include the third party libraries we rely on in the source tree. However, on the other platforms we support such as Linux and Mac OS X, many of these third party libraries are available as packages or are already present in the system and are easily found by CMake.

The current project layout is as follow:

root/
    src/
        3rd-party-lib1/       (build system modified to output to build/)
        3rd-party-lib2/       (build system modified to output to build/)
        project-module1/      (our own code)
        project-module2/      (our own code)
    build/                    (CMake is invoked from here)
        3rd-party-lib1-bin/
        3rd-party-lib2-bin/

The third party libraries have been tweaked so that when built, they output their binaries to root/build/<lib>/.

The problems with this layout are multiple:

• The third party libraries are no longer 100% original. This makes updating them slightly more difficult than required.
• The src/ directory contains a mixture of our own code and third party code, which is confusing.
• The src/ directory is very large. Because src/ contains third party libraries, it is very large compared to the actual amount of original source code, making backuping our own code slightly more complicated than required (we can't just archive the whole src/ directory anymore).
• The project repository (Git) is very large, due to the inclusion of the third party libraries (which may contains lots of non-source files such as documentation, test data etc.), and it gets larger each time we update them. Unfortunately there is no way back here, unless we decide to restart with a fresh repository (unfortunately loosing the entire commit history).
• Many of those included third party libraries (e.g. zlib, libpng) are not needed at all by users building the project on Linux or Mac OS X, although they greatly simplify things for Windows users.

An alternative layout would be as follow:

root/
    3rdparty/
        3rd-party-lib1/       (100% original, contains built artifacts)
        3rd-party-lib2/       (100% original, contains built artifacts)
    src/
        project-module1/      (our own code)
        project-module2/      (our own code)
    build/                    (CMake is invoked from here)

Our CMake files would need to be modified to look for third party header files and libraries in the right place for each library.

What are the best practices regarding the handling of third party libraries in native, cross platform projects? Which layout would lead to the most unsurprising build experience for our developers on their respective platforms? Concrete examples of successful layouts from existing projects are also welcome.

Answer 1

My experience suggests the following as best practices:

• When a third-party open-source library is used completely as-is, commit a local copy of the downloaded compressed tarball within the main git repo, to avoid network connectivity issues from preventing software builds.

• When a third-party open-source library is almost used as-is, but needs to be tweaked (this is common when cross-compiling: many packages need slight adjustments to their configuration steps), store the compressed tarball and a "unified diff" patch file within the main git repo, and apply the patch within the PATCH_COMMAND step of the ExternalProject_Add.

• When a third-party open-source library is (to be) heavily modified or extended by your organization, use a separate git repository to hold pointers to the upstream repository (easiest when it also uses git, but upstream svn can be managed also). Commit your organization's changes to a distinct branch from the branch used to mirror upstream. If you wish, you can introduce a submodule relationship between the main git repo and this one, though because the DOWNLOAD_COMMAND can fetch directly from an arbitrary git repo, it is not technically necessary to do so.

• It is reasonable to treat small, less-frequently third-party proprietary binaries for a single target platform by archiving them within the main git repo also. However, third-party binaries which are available for a variety of platforms, are large, or frequently evolve should be stored in their own git repo and fetched in via DOWNLOAD_COMMAND, as above.