How to handle config files with alibuild?

Hi,

Let’s imagine that I have an application that requires some configuration. In prod we will use a Configuration system, probably via an API provided by Fair/O2Device. During development, we want users to have a simple way to run on their laptop, and thus to use a config file instead. The default config file is provided with the code and is installed by alibuild.

My problem is that alibuild installs a fresh copy of the config file at every rebuild and the environment will point to it. Thus any change I do to the config file is kind of lost.

I could copy the config file in a safe place (home or etc) and point the binary to it (if at all possible) but it seems a bit weird to start moving or copying files outside alibuild.

My question is : how to handle, with alibuild, config files that would typically go to /etc ?

Thank you

@dberzano @eulisse @teo @awegrzyn @vmcb

I don’t have a good answer, but I too would be interested in the opportunity to have an aliBuild RPM install files in /etc without overwriting (or in a perfect world, have all aliBuild RPMs install stuff in standard /usr paths, avoiding /opt and modulefiles altogether).

What I currently do is an Ansible playbook run to copy the relevant files to /etc, with contents parametrized against the current package version (if needed).

Generally on the configuration side, ideally you would use the configuration library without having to care whether the data is coming from Consul or a file. If this is not yet the case, in the short- to mid-term we can try to figure out what are the missing pieces.

Why not using a developement package? I will be on holidays until next Monday

Hi Giulio,
This is not an urgent matter, enjoy your holidays !

To answer your question : this is a good idea. My use case is for the QCG (GUI for QC) and we could tell the users to do an init of qcg and quality_control in order to have the files at hand and not overwritten. I will think if there are cases not covered.

Concerning the RPMs, as noted by Teo, we would need to alter the way they are built I guess.

How about adding some custom bash code to the alibuild recipe that only conditionally overwrites your config file? E.g. let the build system install to etc/config.json.default or whatever and copy this file (after make install) to etc/config.json only if it is not yet existing (or with some more elaborated condition)?

This is how debian packages handle included default configs.

2 Likes

I like it !

@eulisse I have checked and the qcg package has no source so “initing” does not work for the time being.

Hi all, so - first off - I do not think this is an aliBuild matter, but it’s worth discussing it in any case, so thanks for starting this discussion!

TL;DR: do not edit default configuration files, leave them alone and managed by the package manager, and use override configuration files stored elsewhere, outside the installation prefix possibly.

Just to give you some context:

  • aliBuild installs every single package in its own prefix, at least for now
  • RPM generation is not part of aliBuild; it’s an independent system that relies on information provided by aliBuild to generate the proper packages

As for RPMs, we have two types of them ATM:

  • RPMs that install in separate prefixes (each one in a different prefix) - they are the so-called non-updateable RPMs and are the ones that allow several versions of the same package to be installed at the same time
  • RPMs that install under a single prefix (the updateable RPMs), which is - in any case - not /

In order not to complicate the matters, I would not add any special option that installs “exceptions” outside the given prefix. We are trying to keep a “convention over configuration” approach so keeping the concept simple as it is (isolated prefixes) is, in my opinion, the way to go, and it has proven to be successful (from aliBuild packages we can seamlessly create CVMFS ones, RPMs, DEBs, etc. with zero effort).

For what concerns my packages, if I am in need for a certain configuration I tend to create a list of priorities for configuration files. This is more or less what @deklein suggested. You won’t have a single configuration file, but you will have your program look in order for, say, the following:

  • /etc/config.json.default (installed by the RPM, do not modify it!)
  • $PREFIX/etc/config.json.default
  • ~/.config.json
  • /etc/config.json

where every configuration file loads the configuration variable, and the next overrides any doubly-defined value.

This is the approach followed by any decent package by now: as you know, modifying something like /etc/httpd/httpd.conf used to be a PITA in the past, as any new update created something like /etc/httpd/httpd.conf.rpmsave or similar things, and you had the duty to figure out what changed upstream and merge your configuration appropriately. Nowadays you have an “uneditable” file and overrides, where the latter are not managed by the package manager (you can either manage them manually or with tools like Ansible, WP8 has more insight on that.

The CVMFS package (if you’ve ever configured it) for instance has:

  • /etc/cvmfs/default.conf (the configuration you are not supposed to edit)
  • /etc/cvmfs/default.local (you add it and it overrides)

plus other repository-specific overrides, loaded in cascade.

Hope this helps!

Just to add to the picture, you can also combine the pattern with multiple ordered lookup locations described by @dberzano with putting multiple config sections into separate files at the same lookup level if your config format allows it:

  • /etc/my_package.d/default.conf
  • /etc/my_package.d/my_authentication.conf
  • /etc/my_package.d/arbitrary_file_name.conf

@deklein yes that’s what CVMFS does from my example above :slight_smile:

However for aliBuild packages let me stress, before we get into trouble, that it would be much better if we didn’t have overrides under the installation prefix. Remember, Convention over Configuration, so let’s not add this special use case otherwise we’ll have to handle it! :slight_smile:

So:

  • /etc/ is incidentally fine because / is never an installation prefix in our case
  • /opt/alisw/blah/blah/package/version/etc is not because we break the assumption that we could simply remove the whole directory to get rid of the package (this is for non-updateable RPMs)
  • /opt/alisw/etc: same thing as above for “updateable RPMs” (we break the assumption that it’s sufficient to remove /opt/alisw to get rid of the whole ALICE stuff and get your system back to a pristine state).

If one considers a lookup location under $HOME, please also consider being compatible to this: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

Thank you all for your input.

What I take out of it :

  • Copy the default config files, do not edit them. Default config files use a suffix (e.g. .default).
  • Copy them in a place different from the installation directory. Several locations are possible/allowed, they have different priorities.
  • Addition : It would help to have a common library/header to handle the location of files in O2 (not only config files). It should be platform-aware and be aware of the priority order of the various allowed directories.
1 Like

Personally I would have a somewhat different approach. I would have a top-level package (say O2-env) where we keep only configuration files and setup the actual full dependencies for the O2 environment. This has a few advantages:

  • Users could checkout O2-env as a development package and customise their configurations, without having to do a full rebuild of O2 every time.
  • Depending on what you want to do, the O2-env could bring in extra dependencies so that if you are deployed on your laptop you do not build the plugins which are instead required by the synchronous reconstruction environment.
  • It would allow us to have proper dependency inversion where code depends on the interface of a plugin, not on its implementation.

I would personally avoid suggesting anything which is not versioned, because that’s a recipe for hours of debugging (see how many times .rootrc caused us troubles).