tl;dr: we have started to use an internal Jenkins instance in combination with a public Copr – please update the URLs of DNF nightly builds repos and do the same with your project ;-D
I thought that it might be a good idea to incorporate all the ideas I had on my TODO list into our continuous integration process before I leave the DNF team. I would say that this effort was quite successful and that the process has improved a lot. I believe that it might be interesting (or even inspirational) for you to know how it works.
Originally, we started with a Jenkins job hosted by Fedora infrastructure which build RPMs on every new commit pushed upstream. Later, it turned out that it would be really nice if it would test submitted pull requests as well. There is a Jenkins plugin for almost anything you can come up with. That goes for GitHub pull requests as well. Unfortunately, you have to ask Fedora infrastructure to install any additional plugin you need. And what is worse, the plugin is designed so that the GitHub credentials must be configured globally. But we didn’t want to provide access to our repository to anyone who use the same Jenkins instance.
Since we were a bit impatient to wait whether the plugin can be changed, Michal succeeded to install Jenkins on our internal OpenStack instance. This change allowed us to structure (hence speed up) the continuous integration process a bit. I mean, we do not build just RPMs of DNF. We build also hawkey and the core DNF plugins and even two more dependencies – librepo and libcomps. Among other things, this allows us to develop against the most recent versions of these libraries and it also provides us with an additional assurance that a new version of these libraries will not break DNF. You can imagine that building RPMs of five projects for two architectures and for multiple versions of Fedora (sequentially) can take some time. With our own Jenkins instance, we don’t need to be ashamed to create 5 different Jenkins jobs where each is split to two sub-jobs, one for each architecture. Moreover, one job can be configured as an upstream of another job so that e.g. if a new build of hawkey was successful, Jenkins may trigger a build of DNF to test whether the new version of hawkey did not break DNF. A nice side effect of this split is also that e.g. a new change in DNF does not trigger a new build of hawkey – that means less builds at the same time, hence faster builds.
At the same time, something has happened with the hosted Jenkins. All DNF builds started to fail there. In the Job configuration, there is no “Multiple SCMs” option any more. I suspect, that the Multiple SCMs Plugin broke after an update or they have uninstalled this plugin. This proved to be another advantage of having an own Jenkins instance along with the fact that Fedora infrastructure does not guarantee availability of their Jenkins instance. This issue have caused another problem. Originally, with the launch of the continuous integration process, we also promised users to provide nightly builds of DNF. With a failing public Jenkins and a succeeding but private Jenkins, users have lost the access to the nightly builds.
Luckily, the people around Copr, have recently added the possibility to upload SRPMs to Copr. Copr is most likely the best place were to host RPMs of project snapshots. So, we decided to use Copr to build RPMs on every Git change (and the pull requests as well). This again sped up the build, allowed us to build on more architectures and we also got the best public hosting for our nightly builds.
To sum it up, our continuous integration have transformed a lot. Our internal Jenkins instance currently watches our GitHub repository (including the pull requests), on every change, it builds the SRPM of the appropriate component (hawkey, DNF, core DNF plugins, librepo, libcomps) using tito (in most cases), then it uploads the source RPM to Copr which builds the binary RPMs (using the RPMs from the previous builds), reports the result through emails and/or GitHub API and potentially triggers builds of the other dependant projects. If I gained your attention, I would be honoured, if you would like to take a look at the CI script here: https://github.com/rpm-software-management/ci-dnf-stack. You can find some instructions to set your own Jenkins job up there as well. Please note that the ability of tito to upload a SRPM to Copr is coming soon. Then you probably won’t need this script any more.
Since the new approach works very good, we are going to disable the job at http://jenkins.cloud.fedoraproject.org. If you are still interested in using nightly builds of DNF (at your own risk!), please, enable Copr rpmsoftwaremanagement/dnf-nightly, e.g. using:
dnf copr enable rpmsoftwaremanagement/dnf-nightly