Skip to main content

Dependency management

Requirements file structure​

The dependencies needed by Rucio are described in the rucio/requirements/ directory.

The requirements in this directory are divided based on the application:

  • requirements.server*: Dependencies needed to run the Rucio server and daemons
  • requirements.client*: Dependencies needed to run the Rucio client
  • requirements.dev*: Dependencies needed for testing and development

.in and .txt files​

  • .in files represent input files to pip-compile. These list the primary dependencies.
    • For client, the .in file is not necessary, as we do not pin client dependencies.
  • .txt files represent the actual requirements files used at installation time.
    • For server and dev, the .txt file is generated by pip-compile, pinning both primary and secondary dependencies.
    • For client, the .txt file is compiled manually, and only describes the primary dependencies.

Updating dependencies​

pip-compile and compiling dependencies​

We use pip-compile from pip-tools for server and dev in order to ensure we pin secondary dependencies to specific versions, to improve the stability of Rucio server, testing and development. See this issue for an example of a CI issue that was caused by an unpinned secondary dependency.

I want to add/remove/upgrade a server dependency. What should I do?​

  1. Make your changes in the requirements.server.in file
  2. Run pip-compile requirements.server.in
  3. Run pip-compile requirements.dev.in
  • (dev dependencies include dependencies from requirements.server.txt, so make sure to do these steps in order)

I want to add/remove/upgrade a dev dependency. What should I do?​

  1. Make your changes in the requirements.dev.in file
  2. Run pip-compile requirements.dev.in

I want to add/remove/upgrade a client dependency. What should I do?​

  1. Make your changes in the requirements.client.txt file

Major dependency upgrades​

Dependencies are generally upgraded to the latest possible version on every Rucio major release. This work is performed approximately a month prior to the major release, in order to address possible breaking changes and monitor test outcomes and runtime behaviour for errors.

For secondary dependencies, pip-compile --upgrade is used. This flag attempts to upgrade all secondary dependencies to their latest versions.

To perform this major dependency upgrade:

  1. Manually update all primary dependencies (where possible; be mindful of breaking changes) in:
    1. requirements.server.in
    2. requirements.dev.in
    3. requirements.client.txt
  2. Run pip-compile --upgrade requirements.server.in
  3. Run pip-compile --upgrade requirements.dev.in

Security updates​

For critical security updates, we rely on Dependabot to create alerts for dependencies listed in our requirements.

Dependabot supports pip-compile, and is able to automatically create PRs to ugprade both primary and secondary dependencies. When a primary dependency is upgraded in a .in file, Dependabot re-compiles that file into the .txt file as well.

FAQ

Why are client dependencies not pinned?​

In certain use cases, the Rucio client is used as a library in other applications (See this issue for an example). Because of this, client dependencies are left unpinned unless necessary.