Package Manager — Dawn of Giants

Mathieu Ropert gave an excellent talk at ACCU 2019 about the state of package management in C++. He came up with a list of criteria to help select a package manager. Among the many candidates available, only Conan and Vcpkg made it through. The most relevant criteria to us is that the package manager be non-intrusive, i.e. the build system is totally independent of it.

In this post, we go through the different steps we took to build a Conan package for Sempervirens, and a test application that requires the package (We described our build system in a previous post). First of, we installed the python based Conan package with:

pip install conan

Then on the JFrog website, we applied for a free (cloud) subscription that includes access to JFrog Artifactory, the JFrog artifact repository manager. A server name (in our case dawnofgiantsstudios.jfrog.io) and some identification for access credentials are required. To create our first repository, we gave it a name (conan-dog) and added it to the list of remote repositories of our server with the commands:

conan tag REPO_NAME
conan remote add REPO_NAME SERVER_URL/artifactory/api/conan/REPO_NAME

with REPO_NAME and SERVER_URL replaced by ‘conan-dog’ and ‘dawnofgiantsstudios.jfrog.io‘ respectively. We then created our ‘Sempervirens’ package as follows:

mkdir Packages/Sempervirens && cd Packages/Sempervirens
conan new sempervirens/1.0
conan create . -s build_type=Debug

The conan new command will create a default template for what is called the ‘recipe’ file of the package. This is a python file named conanfile.py, that experienced Conan users would likely just create themselves from scratch. This is what our recipe looked like after we edited the default template:

from conans import ConanFile, CMake, tools

class SempervirensConan(ConanFile):
    name = "sempervirens"
    version = "1.0"
    license = ""
    author = " "
    url = ""
    description = ""
    topics = ("", "", "")
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False], "fPIC": [True, False]}
    default_options = {"shared": False, "fPIC": True}
    generators = "cmake_paths"

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def source(self):
        self.run("git clone https://gitlab.com/nico-dog/sempervirens.git")

    def build(self):
        cmake = CMake(self)
        cmake.configure(source_folder="sempervirens")
        cmake.build()

    def package(self):
        self.copy("*.hpp", dst="include/Sempervirens", src="sempervirens/include")
        self.copy("*.hpp", dst="include/Sempervirens/Application", src="sempervirens/src/Application")
        self.copy("*.hpp", dst="include/Sempervirens/Controller", src="sempervirens/src/Controller")
        self.copy("*.inl", dst="include/Sempervirens/Controller", src="sempervirens/src/Controller")
        self.copy("*.hpp", dst="include/Sempervirens/EventSystem", src="sempervirens/src/EventSystem")
        self.copy("*.inl", dst="include/Sempervirens/EventSystem", src="sempervirens/src/EventSystem")
        self.copy("*.hpp", dst="include/Sempervirens/Keyboard", src="sempervirens/src/Keyboard")
        self.copy("*.hpp", dst="include/Sempervirens/Keyboard/Platform/Linux", src="sempervirens/src/Keyboard/Platform/Linux")
        self.copy("*.hpp", dst="include/Sempervirens/Logging", src="sempervirens/src/Logging")
        self.copy("*.hpp", dst="include/Sempervirens/Macros", src="sempervirens/src/Macros")
        self.copy("*.hpp", dst="include/Sempervirens/MemoryAlloc", src="sempervirens/src/MemoryAlloc")
        self.copy("*.inl", dst="include/Sempervirens/MemoryAlloc", src="sempervirens/src/MemoryAlloc")
        self.copy("*.hpp", dst="include/Sempervirens/Mouse", src="sempervirens/src/Mouse")
        self.copy("*.hpp", dst="include/Sempervirens/Mouse/Platform/Linux", src="sempervirens/src/Mouse/Platform/Linux")
        self.copy("*.hpp", dst="include/Sempervirens/Timer", src="sempervirens/src/Timer")
        self.copy("*.hpp", dst="include/Sempervirens/Timer/Platform/Linux", src="sempervirens/src/Timer/Platform/Linux")
        self.copy("*.hpp", dst="include/Sempervirens/UnitTesting", src="sempervirens/src/UnitTesting")
        self.copy("*.inl", dst="include/Sempervirens/UnitTesting", src="sempervirens/src/UnitTesting")
        self.copy("*.hpp", dst="include/Sempervirens/Window", src="sempervirens/src/Window")
        self.copy("*.hpp", dst="include/Sempervirens/Window/Platform/Linux", src="sempervirens/src/Window/Platform/Linux")
        self.copy("*.hpp", dst="include/Sempervirens/WSI", src="sempervirens/src/WSI")

        self.copy("SempervirensConfigVersion.cmake", dst="lib/cmake/Sempervirens", src=".")
        self.copy("SempervirensConfig.cmake", dst="lib/cmake/Sempervirens", src=".")
        self.copy("SempervirensTargets.cmake", dst="lib/cmake/Sempervirens", src="cmake")

        self.copy("*.lib", dst="lib", keep_path=False)
        self.copy("*.dll", dst="bin", keep_path=False)
        self.copy("*.so", dst="lib", keep_path=False)
        self.copy("*.dylib", dst="lib", keep_path=False)
        self.copy("*.a", dst="lib", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["Sempervirens"]

The recipe declares a class based on your package name. The class members are meant to provide some package information, as well as a minimum set of parameters required to build the project. These include, in particular, the platform, compiler, architecture, and config the project is built for, whether we are building a shared or static library, and which build system (generator) is to be used. The latter is set by default to cmake, which is great as our build system is CMake! However, this setting will break the non-intrusive feature we selected Conan for, as this requires your top-level CMakeLists.txt file to include conan specific calls and variables. To remedy that, if need be, Conan provides what is referred to as ‘transparent integration’ with the cmake_paths generator. This has pros and cons, in particular as far as dealing with transitive dependencies is concerned, but we think it is a small price to pay for a complete decoupling between the package manager and the build system.

Next, the class defines five default methods: config_options, source, build, package, and package_info. Source is where one tells Conan where to get the source code from, in our case by cloning from our gitlab repo. In the build method, a CMake helper class is used to trigger the generation and build stages we defined in the CMakeLists.txt files of the project. For the install phase, there is definitely the possibility to delegate as well to that which is defined in the project, but, for some reason, we could not get this to work properly. So, here we basically replicated how our build system would package the project.

Once the recipe is ready, one can create the package with the conan create command. In our case, we passed the build configuration (Debug) as a setting parameter. One can search the local cache to check that the package was created with the conan search command (the ‘@’ after the package version is required):

conan search sempervirens/1.0@
Existing packages for recipe sempervirens/1.0:

    Package_ID: 038baac88f4c7bfa972ce5adac1616bed8fe2ef4
        [options]
            fPIC: True
            shared: False
        [settings]
            arch: x86_64
            build_type: Debug
            compiler: gcc
            compiler.libcxx: libstdc++11
            compiler.version: 9
            os: Linux
        Outdated from recipe: False

After that we uploaded the package to our remote repository with:

conan upload sempervirens/1.0 -r=conan-dog

This uploads both the recipe and the binary we created. This is one thing that Conan and Vcpkg do differently: Conan will never build from source if a binary with settings matching your requirements exists on the repo (If we understood correctly, the only time Conan would build from source, when not asked explicitly to do so, is when no binary matches your requirements neither in your local cache nor on the remote repository).

To test our Conan package, we created an application using Sempervirens. This is a simple CMake project where the Sempervirens::Sempervirens target is imported with find_package(Sempervirens 1.0 REQUIRED). To tell Conan that our application requires the Sempervirens package, we created a conanfile.txt file in the the top-level directory, with the following content:

[requires]
sempervirens/1.0

[generators]
cmake_paths

In addition to the package requirements, we also specify the generator to use for the build. One can also specify build options for the packages that are required if needed (using the [options] section header). Here is the short script we used to install the package:

#!/bin/sh
config=$1
platform=$2
if [ $platform != "Linux" ]
then
    if [ $platform != "apple" ]
    then
    echo "unrecognized platform"
    return
    fi
fi

builddir="build-Application-"$config'-'$platform
rm -rf ../$builddir
mkdir ../$builddir && cd ../$builddir
conan install ../Application -s build_type=$config

When running the conan install command, the cmake_paths generator will generate a conan_paths.cmake file, which provides the appropriate definitions for the CMAKE_MODULE_PATH and CMAKE_PREFIX_PATH variables necessary for the build system to find the Sempervirens config files, and locate the library. Here is another short script we used to pass that file as a toolchain file during the build stage:

#!/bin/sh
config=$1
platform=$2
if [ $platform != "Linux" ]
then
    if [ $platform != "apple" ]
    then
    echo "unrecognized platform"
    return
    fi
fi

builddir="build-Application-"$config'-'$platform
cd ../$builddir
cmake --log-context ../Application -DCMAKE_BUILD_TYPE=$config -DCMAKE_TOOLCHAIN_FILE=./conan_paths.cmake 
cmake --build . --config $config --parallel 8