Drupal 9 support, Lagoon metadata and Trivy scanning

• The main reason for this release is to ensure that our images and infrastructure are ready to support Drupal 9 on release day. We’ll also be releasing a blog post outlining what we’ve put in place.
• This release has also added customisable metadata fields to Lagoon projects, contributed by @stooit (and thanks to Salsa Digital) - we’ll expand more on what’s possible with this, and why it’s so important shortly.
• Also, in big ticket items, we have updated the image and vulnerability scanning component bundled with our Harbor install to use Trivy (instead of Clair). In our tests, Trivy not only detected more vulnerabilities, but was significantly less resource-heavy, and returned faster results - triple win!

Update to v1.6.0

With #1917 landed within Lagoon v1.6.0 we're now labeling all new created OpenShift and Kubernetes Namespaces with two new Labels: lagoon.sh/project and lagoon.sh/environment.
In order to label all existing namespaces please use this script:

#!/usr/bin/env bash

while read PROJECT; do
  echo "Handling lagoon project ${PROJECT}"
  while read LINE; do
    NAMESPACE=$(echo "$LINE" | awk '{print $5}')
    ENVIRONMENT=$(echo "$LINE" | awk '{print $2}')
    if kubectl get ns ${NAMESPACE} &> /dev/null; then
      PATCH="{\"metadata\": {\"labels\": {\"lagoon.sh/project\": \"${PROJECT}\", \"lagoon.sh/environment\": \"${ENVIRONMENT}\"}}}"
      echo "Updating namespace labels for ${NAMESPACE}"
      kubectl patch ns "${NAMESPACE}" -p "${PATCH}"
    else
      echo "${NAMESPACE} not found on this cluster"
    fi
  done < <(lagoon list environments --no-header --project="${PROJECT}")
done < <(lagoon list projects --no-header | awk '{print $2}')

this requires lagoon-cli >=v0.9.1 and kubectl, both fully authenticated against Lagoon and an OpenShift/Kubernetes cluster, run the script for each cluster that is managed by Lagoon.

Changes in this release

API & Authentication subsystem

• Re-implement JWT generation script using pyjwt @smlx (#1830)
• Added support to arbitrary project metadata key/value pairs @stooit (#1775)

Admin & User Interfaces subsystem

• Elasticsearch date bounds calculate @Schnitzel (#1874)

Build & Deploy subsystem

• Redeploy pods if the ConfigMap changes @shreddedbacon (#1923)
• Ability to temporarily disable pre or post rollout tasks @shreddedbacon (#1921)
• use lagoon.sh as label prefix @Schnitzel (#1835)
• Add Trivy Scanner to Harbor @cdchris12 (#1912)
• Ingress annotations @Schnitzel (#1903)
• Ensure project and environment labels exist on OpenShift projects. @nicksantamaria (#1917)
• ingress annotations support @Schnitzel (#1899)
• Disabling quota sync for harbor-core @cdchris12 (#1865)
• only run helm template if the folder actually exists @Schnitzel (#1862)

Logging & Reporting subsystem

• Lagoon logging helm chart @smlx (#1859)

Base Images & Testing subsystem

• Initial attempt at d9 tests @tobybellwood (#1904)
• Build Python3.7 images from Alpine 3.10 instead of 3.11 @tobybellwood (#1889)
• Moved es-curl to elasticsearch base images @vincenzodnp (#1886)
• Run mysql_upgrade on existing databases after upgrade @tobybellwood (#1891)
• Update MariaDB image to latest Alpine & remove Galera-based images @tobybellwood (#1866)

Documentation & Examples / DX subsystem

• Basic Authentication documentation update @seanhamlin (#1698)
• Fix markdown rendering for code blocks for container registries @seanhamlin (#1803)
• Reformat and update Lagoon service README files @rocketeerbkw (#1906)

Automation, Services & Helpers subsystem

• Alter the error message for webhook payloads that are not JSON. @seanhamlin (#1870)
• Automated Billing - Poly Site Calculations @justinlevi (#1914)
• Added support to arbitrary project metadata key/value pairs @stooit (#1775)
• k8s k8up backup fixes @Schnitzel (#1864)

Security subsystem

• Re-implement JWT generation script using pyjwt @smlx (#1830)

Release side notes

• Release drink: Old Fashioned - Maker's Mark 46
• Release song: You're the Conversation, I'm the Game by Chris Corner, Sue Denim