The problem

No linux distro ever has exactly the set of software that you want use and install, and when it does it's too old or too new. Everyone thinks that their linux distro has the perfect or the best packaging system for managing packages. There are lots of good if not great packaging systems out there, but for a HPC sysadmin they usually suck when you need 3-4 versions of MPI compiled up with different compilers.

A good example is something like NAMD, if you wanted to build a customised version of NAMD such that it runs optimally on your cluster . The sysadmin usually needs to pick a compiler that works, pick an MPI implementation that either works or is optimised for the network that the sysadmin has, decide on whether to build fftw by hand or install it from the packaging system, charm++ and a few other things, decide whether to use MKL to back the FFTW implementation or not. Once all the decisions are made, the sysadmin configures and compiles things up and discovers there are updated versions of some of the software that he or she has built. Now the sysadmin goes back and changes some versions of software and re-links everything. Wash, rinse and repeat the said process till a usable and desired binary is produced.

Once a usable binary is produced, the sysadmin (me) usually goes back and tries to document what was done, and then repeats the whole build process on another machine for the user who asked for the software to be installed (of which there might be 2-3 versions of NAMD or whatever is installed). Or else the instructions are stored somewhere for another sysadmin to reproduce the said binary at a later point in time.

The possible solutions

Write shell scripts to repeat the build, write lots of documentation. Write you own packaging system and lay it on top of the distro of your choice. The guys at nixos did this, but it doesn't really allow users to sanely pick which versions of software they want to use and it is a little overly complicated to bootstrap and to allow a user to build stuff in the user land. The PETSc project and FEniCS project took the approach of building a build system for their software, since they require the usual MPI's, BLAS's, LAPACK's and FFTW's. It seems to work, but it's not too flexible to plug other stuff into the systems that they have built.

Some conclusions and a possible solution

It's a known and hard problem for package system builders to design systems that are smart and maintainable and has the capability of doing dependency checking, resolution and conflicts. It's often these design requirements that make the packaging system so smart that it makes them a little bit too complicated and useless for HPC sysadmins. That is these fancy systems often do not allow the HPC sysadmin to install multiple versions of the same software compiled up in different ways on a single system.

When you think about it, it's not a totally crazy idea that a HPC sysadmin would want to build their own set of scripts and framework to repeat a build. Most sysadmins think they have it bad when installing software for users, but HPC sysadmins have it much worse! But there's hope!

One particular build system had caught my eye a few months ago,

• mpkg - http://staff.e.kth.se/mpkg/index.html
• cports - http://www.caoslinux.org/ and http://caoswiki.infiscale.org/

mpkg kind of worked, but the cports developers took the concept and made it more useful, especially for HPC sysadmins, it generated environment module files and had been extended to allow different compilers to be used. It also had basic dependency checking so that when you installed a package it would also install its dependencies.

Sadly cports seems to be dead, but being free software I've taken a copy of the source code and have effectively forked the project. So far I have fixed a bunch of weird behavior/bugs in the build system. It's beginning to look promising as me and some colleagues of mine have plugged in a bunch of bio applications (for gene sequencing) for some users of our clusters in work.

cports is working out quite well for us in work, it's not really a build system, or a packaging system. It's really a building framework to make a HPC sysadmin's life much easier. We've been using it on our clusters to provide updated software in our module system. It's been refreshing to see our software stack installed in a pretty consistent way, and module files are automatically generated and tagged correctly for the user and that the builds could somewhat be repeated on similar systems. It's still a pretty dumb and semi manual system, but it works for us and it doesn't pretend to be clever.

It's not quite ready for prime time yet. It needs some more testing and probably way more polish before it can be published and advertised to other HPC sysadmins to play with.