Skip to main content

1.31.0rc3

Upgrade Instructions​

This feature release requires a database schema upgrade. Please consult the documentation about upgrading your database schema.

The following changes are necessary and are covered by the alembic upgrade head command:

  1. Adding REPLICAS_RSE_ID_TOMBSTONE_IDX to replicas table; CAUTION: If you add this index online, make sure the index is fully built before you deploy 1.31!

Important changes​

  • This release enables the usage of temporary tables per default. Previously, this was an opt-in, now it is an opt-out. If you notice any issues due to this change, you can turn the usage of temporary tables off via setting the [core]use_temp_tables=False setting in the config table.

Helm chart changes​

breaking change: Version 1.31.x of rucio-servers helm chart is incompatible with old values​

What changes? The rucio-server helm chart doesn't support anymore to deploy separate api, auth and trace servers. Only a single server deployment can be done by one helm release.

There are two possible cases. 1) If your helm values don't set any of the authReplicaCount or traceReplicaCount (or set them to 0), the impact of this release will be very limited on you.

- If you have any of the
  `authService`
  `traceService`
  `authIngress`
  `traceIngress`
  `authServerResources`
  `traceServerResources`
  `authServerResources`
  sections set in the values. They were already ignored anyway because of
  `authReplicaCount` and `traceReplicaCount` not being set. You can safely
  remove these sections

  • After that, you'll have to substitute any of the following constructs: 

    exposeErrorLogs:
      server: ... 
    useSSL:
      server: ...
    minReadySeconds:
      server: ...
    readinessProbe:
      server: 
        initialDelaySeconds: ...
        ...
    livenessProbe:
      server:
        initialDelaySeconds: ...
        ...
    serverType:
      server: ...
    logFormat:
      server: ...

    with 

    exposeErrorLogs: ... 
    useSSL: ...
    minReadySeconds: ...
    readinessProbe:
      initialDelaySeconds: ...
      ...
    livenessProbe:
      initialDelaySeconds: ...
      ...
    serverType: ...
    logFormat: ...

    if any of the sections have an authServer or traceServer attribute, it can be removed.

2) If your values set authReplicaCount or traceReplicaCount to any positive integer, you'll have to split your helm release into multiple helm releases.

For example. An initial helm release with values.yaml like following:

   replicaCount: 3
   authReplicaCount: 2
   traceReplicaCount: 1

   service: 
      something
   authService
      anotherThing
   traceService
      thirdThing

   useSSL:
      server: false
      authServer: true

Will have to become 3 different helm releases with following values:

  • api servers:

    replicaCount: 3
    service: 
       something
    useSSL: false

  • auth servers:

    replicaCount: 2
    service: 
       anotherThing
    useSSL: true
    config:
      api:
        endpoints: "ping, auth"

  • trace servers:

    replicaCount: 1
    service: 
       thirdThing
    config:
      api:
        endpoints: "ping, traces, nongrid_traces"

Why this change? The helm chart was difficult to maintain and had a lot of duplicate code to support an edge case for which a reasonable work-around exists. The behavior of the chart was also not consistent between trace and auth servers. Moreover, it didn't allow to set distinct parameters for those servers. Rather than trying to complicate things even further and fix inconsistent behavior, it was decided it's a better way forward to drop support for it.

What changes? Support for additionalSecrets will be dropped in a future major release. To be ready for that change, you can already migrate to the new format, for both rucio-servers and rucio-daemons helm charts. The old format will still be supported throughout the whole 1.31, but will be dropped in 1.32.

Substituted all occurrences of:

additionalSecrets:
  volumename:
    secretName: ...
    mountPath: ...
    subPath: ...

with:

secretMounts:
  - secretName: ...
    mountPath: ...
    subPath: ...
    volumeName: volumename  # volumeName is optional in most cases and will be inferred from secretName if ommited

Why this change? Avoiding user-input keys in the yaml object (here: the volume name) is more consistent with the overall kubernetes best practices. User input is always passed via yaml values, not via yaml keys. One of the consequences of this change is improved support for kubernetes kustomize. Secrets can now be referred via the fieldSpecs.path fields in kustomize.

Renaming additionalSecrets to secretMount is a cosmetic change. The new name seems to better reflect the function of this section.

What changes? The ftsRenewal sections in both rucio-servers and rucio-daemons helm charts suffered a major rework. The old format will still be supported throughout the whole 1.31 release line. There is no plan to drop support of the old format in 1.32, but we may decide to do so if the maintenance burden will be judged too high in the meantime.

In the new format, secrets and env variable have to be specified explicitly, rather than being selected implicitly depending on the 'vo' attribute. See the documentation: https://github.com/rucio/helm-charts/tree/master/charts/rucio-daemons#conveyor and helm values: https://github.com/rucio/helm-charts/blob/master/charts/rucio-daemons/values.yaml#L408 for example of new configuration.

Why this change? Two independent reasons: 1) the new format introduced support for multi-vo. 2) the source code of ftsRenewal cronjob became full of if/else to support vo-specific configurations. Configuring a new VO, with different requirements, was very difficult and required co-ordinated changes in multiple upstream rucio repositories. The new format is now much more flexible, but this comes at the price of increased verbosity of the configuration

Your opinion is welcomed: deprecation of implicit secrets​

What changes? Nothing, yet. But we are evaluating the possibility to deprecate implicit secrets required by helm charts. If the community will not be against removing support for implicit secrets, the support will be gradually removed in future major releases.

If you want to participate and provide your opinion, you'll have to set useDeprecatedImplicitSecrets: false in helm charts. This way, some secrets will not be automatically mounted into the containers. You'll have to explicitly mount corresponding secrets via helm values:

secretMounts:
- secretFullName: releasename-rucio-x509up
  mountPath: /opt/proxy
- secretFullName: releasename-rucio-ca-bundle
  mountPath: /opt/certs/

automatix:
  extraSecretMounts:
    - secretFullName: releasename-rucio-ca-bundle-reaper
      mountPath: /etc/grid-security/certificates/
reaper:
  extraSecretMounts:
    - secretFullName: releasename-rucio-ca-bundle-reaper
      mountPath: /etc/grid-security/certificates/
darkReaper:
  extraSecretMounts:
    - secretFullName: releasename-rucio-ca-bundle-reaper
      mountPath: /etc/grid-security/certificates/

Why this change? We consider that explicit declaration of dependencies in helm charts is a better practice. Requiring secrets with specific names to be created inside the cluster before running the helm chart is very confusing to new users.

This change also allows you to pick a different name for the secrets and share them between multiple helm releases.

General​

Features​

  • Metadata: Merge list_dids and list_dids_extended methods #5448
  • Recovery: Allow for file recovery policies based on scope #5928

Enhancements​

  • Core & Internals: Unicode support along all Core/API/REST API #4045
  • Core & Internals: switch ON temporary tables workflows by default #6084
  • Documentation: Fix broken Rest API docs view #4992
  • Messaging: Migrate to hermes-2 as default #6089
  • Use of bulk_insert_mappings in reaper causes issue with Hermes : Closes #6152 #6153
  • Release management: update dependencies for the 1.31 release #6110
  • Transfers: Remove full_mode switch from conveyor-receiver #5453

Bugs​

  • Messaging: Use of bulk_insert_mappings in reaper causes issue with Hermes #6152
  • REST & API: create_did_sample broken for DIDs with "/" #5608
  • Subscriptions: The transmogrifier updates the subscription history table everytime it runs #6109

Clients​

Enhancements​

  • Clients: Use GFAL’s create_parent transfer parameter #2973