All posts by Radek Holý

About Radek Holý

Associate Software Engineer at Red Hat

CI moved to Copr

tl;dr: we have started to use an internal Jenkins instance in combination with a public Copr – please update the URLs of DNF nightly builds repos and do the same with your project ;-D

I thought that it might be a good idea to incorporate all the ideas I had on my TODO list into our continuous integration process before I leave the DNF team. I would say that this effort was quite successful and that the process has improved a lot. I believe that it might be interesting (or even inspirational) for you to know how it works.

Originally, we started with a Jenkins job hosted by Fedora infrastructure which build RPMs on every new commit pushed upstream. Later, it turned out that it would be really nice if it would test submitted pull requests as well. There is a Jenkins plugin for almost anything you can come up with. That goes for GitHub pull requests as well. Unfortunately, you have to ask Fedora infrastructure to install any additional plugin you need. And what is worse, the plugin is designed so that the GitHub credentials must be configured globally. But we didn’t want to provide access to our repository to anyone who use the same Jenkins instance.

Since we were a bit impatient to wait whether the plugin can be changed, Michal succeeded to install Jenkins on our internal OpenStack instance. This change allowed us to structure (hence speed up) the continuous integration process a bit. I mean, we do not build just RPMs of DNF. We build also hawkey and the core DNF plugins and even two more dependencies – librepo and libcomps. Among other things, this allows us to develop against the most recent versions of these libraries and it also provides us with an additional assurance that a new version of these libraries will not break DNF. You can imagine that building RPMs of five projects for two architectures and for multiple versions of Fedora (sequentially) can take some time. With our own Jenkins instance, we don’t need to be ashamed to create 5 different Jenkins jobs where each is split to two sub-jobs, one for each architecture. Moreover, one job can be configured as an upstream of another job so that e.g. if a new build of hawkey was successful, Jenkins may trigger a build of DNF to test whether the new version of hawkey did not break DNF. A nice side effect of this split is also that e.g. a new change in DNF does not trigger a new build of hawkey – that means less builds at the same time, hence faster builds.

At the same time, something has happened with the hosted Jenkins. All DNF builds started to fail there. In the Job configuration, there is no “Multiple SCMs” option any more. I suspect, that the Multiple SCMs Plugin broke after an update or they have uninstalled this plugin. This proved to be another advantage of having an own Jenkins instance along with the fact that Fedora infrastructure does not guarantee availability of their Jenkins instance. This issue have caused another problem. Originally, with the launch of the continuous integration process, we also promised users to provide nightly builds of DNF. With a failing public Jenkins and a succeeding but private Jenkins, users have lost the access to the nightly builds.

Luckily, the people around Copr, have recently added the possibility to upload SRPMs to Copr. Copr is most likely the best place were to host RPMs of project snapshots. So, we decided to use Copr to build RPMs on every Git change (and the pull requests as well). This again sped up the build, allowed us to build on more architectures and we also got the best public hosting for our nightly builds.

To sum it up, our continuous integration have transformed a lot. Our internal Jenkins instance currently watches our GitHub repository (including the pull requests), on every change, it builds the SRPM of the appropriate component (hawkey, DNF, core DNF plugins, librepo, libcomps) using tito (in most cases), then it uploads the source RPM to Copr which builds the binary RPMs (using the RPMs from the previous builds), reports the result through emails and/or GitHub API and potentially triggers builds of the other dependant projects. If I gained your attention, I would be honoured, if you would like to take a look at the CI script here: https://github.com/rpm-software-management/ci-dnf-stack. You can find some instructions to set your own Jenkins job up there as well. Please note that the ability of tito to upload a SRPM to Copr is coming soon. Then you probably won’t need this script any more.

Since the new approach works very good, we are going to disable the job at http://jenkins.cloud.fedoraproject.org. If you are still interested in using nightly builds of DNF (at your own risk!), please, enable Copr rpmsoftwaremanagement/dnf-nightly, e.g. using:

dnf copr enable rpmsoftwaremanagement/dnf-nightly

Poll: How users use DNF

Dear users of YUM and DNF,

I’m writing to you regarding a request for your feedback. I would be very grateful if you could send me a brief description of how you use YUM or DNF currently or how would you like to use it. I am particularly interested in the occurrences of “dnf/yum install” calls in your scripts. What does these scripts do and what do they expect when they call the “install” command in different situations? Or even your own aliases, plugins, workarounds and the other hacks that you always need to do your job properly would be very interesting.

Please share with me the use cases, not the description of the “install” command. Think twice before you share something because I believe it’s not as easy as it might seem. As an example I think it might be something like:

  • “I call YUM install, because I want to get given packages into my system and I don’t care whether it requires an upgrade or downgrade or what.” or
  • “I want to get them there but it should protect me against dangerous operations like downgrades” or
  • “I often make typos, so I expect that the program knows what I mean” or
  • “it would be nice if it would literally perform the installation; if any of the packages cannot be installed because of any reason, it should fail”.

Not something like: “that’s obvious that the install command should never downgrade packages”.

Please focus on the use cases. The real (non-hypothetical) use cases. Not on the command’s name as it might also result in a new command (while preserving the well-known install command together with an appropriate behaviour).

Thank you very much in advance.

Nightly builds of DNF available

We are glad to announce the launch of continuous integration job for DNF. Each change now starts the bunch of DNF tests on top of upstream versions of hawkey, librepo and libcomps.

It also means that from now nightly builds of DNF and its dependencies are available to you. Download them at will from http://jenkins.cloud.fedoraproject.org/job/DNF/lastSuccessfulBuild/artifact/. To check presence of new builds please follow the RSS channel http://jenkins.cloud.fedoraproject.org/job/DNF/rssAll. You should also be able to use that source as a DNF/yum repository.

edit: Please note that we have moved the CI. Take a look at: http://dnf.baseurl.org/2015/09/21/ci-moved-to-copr/