Garden 0.12

We are very pleased to release Garden 0.12. This release includes a number of new features, improvements, and fixes. Below we'll go through the highlights, and at the bottom you can find a full change log.

There are a handful of breaking changes in there, so be sure to check out the Migration Guide below, but upgrading should generally not be difficult.

Some of the new features and changes relate to our new enterprise offering, so you'll see references to that occasionally. Don't be alarmed though—the new product is purely an additional service layer on top of the open core, and the core will still be our primary development focus. Please read our announcement post for more on that.

We've also made several improvements to docs.garden.io. Those include some structural changes, among them a new landing page, which we hope will help new and existing users find the information they need more easily.

Thanks

First off, a big thank you to @swist, @mitchfriedman, @stevenmatthewt, @DanielKillenberger, @lucas-sonnabend and @abeyum for your code contributions, and to everyone who has posted issues or given us feedback on our Slack.

We're delighted to get an increasing number of contributions from the community, and gladly encourage you to reach out to us for assistance if you have a contribution in mind. Beyond code, it's super helpful to get your bug reports, feature requests and documentation improvements, so please keep those coming!

Release highlights

Namespaces

Garden now has a framework-native notion of namespaces. This provides a more structured way to partition individual environments, and is particularly useful for shared dev environments and clusters.

You can now prefix a namespace when specifying an environment on the command line, e.g. --env my-namespace.dev. You can also change the default namespace, which defaults to simply "default", and you can even use a template to set the default.

Note that this doesn't by default map directly to Kubernetes namespaces, unless explicitly configured that way. By default, the kubernetes provider will set namespaces to <project-name>-<garden-namespace>, adding the project name as a prefix. You can override this by setting the namespace field in the provider configuration, e.g. namespace: ${environment.namespace}.

See the Environments & Namespaces section in our docs for more information.

Workflows

Workflows make it possible to define and run a sequence of both Garden commands and custom scripts, such as how to connect to infrastructure before a deploy or what steps should be taken after a build is finished. Using workflows, you can run the same sequence of steps locally and in CI, making integration tests easy to run consistently both pre- and post-commit.

Workflows are also central to Garden Enterprise, where a workflow can be triggered directly by GitHub/GitLab events like pushing and updating PRs.

See the Workflows section in our docs for more.

Faster startup time

A common frustration in both day-to-day use of Garden, and when using Garden in scripts and CI, has been the slow startup time for each invocation of the CLI.

We've now implemented some caching to cut down the slowest part of the startup, namely resolving provider statuses. The cache has a default TTL of one hour (overridable with GARDEN_CACHE_TTL=<seconds>). To force a refresh—which may be necessary after cluster/config updates—you can run commands with --force-refresh.

Tools command

Many Garden providers automatically download specific binaries and libraries on-demand. For example, the kubernetes provider automatically fetches Helm and kubectl when needed. Until now, this hasn't been directly facilitated at the framework level, which made this behavior opaque and difficult to manage.

We now have native facilities for plugins to define tools they need, and a couple of new command to use and manage them.

garden tools allows users to run these provider tools (fetching them automatically if needed). For example, you could run:

garden tools kubectl -- get pods

garden util fetch-tools facilitates fetching all required tools for a project, which can be very helpful when building custom images with Garden, as well as to bootstrap CI workflows and make them faster.

Our automatically built docker images also use this feature to bundle tools at build-time now, which should improve execution times in CI considerably.

Templating improvements

We've made a few enhancements to our templating:

Nested expressions and map/array lookups.
In addition to dot-notation for key lookups, we also support bracketed lookups, e.g. ${some["key"]} and ${some-array[0]}.
This style offer nested template resolution, which is quite powerful, because you can use the output of one expression to choose a key in a parent expression.
For example, you can declare a mapping variable for your project, and look up values by another variable such as the current environment name. For example:

kind: Project
...
variables:
  - replicas:
      dev: 1
      prod: 3
  ...

---

kind: Module
type: container
...
services:
 # Use the variable defined above:
  replicas: ${var.replicas["${environment.name}"]}
  ...

See the docs for details and examples.

Environment-specific variables can now reference top-level variables.
Variables under environments[].variables in your project can now reference those defined in the top-level variables field (or in a project-level varfile), using ${var.*} template strings. This should help to avoid duplicating variables across your different environments.
Better error messages when keys are not found.
Previously, when a key in a template string couldn't be found, the error message didn't provide much helpful information. Now the error message will list which keys are available in place of the missing key.

Bug fixes and small improvements

We fixed an issue where hot reloading would not respect file exclusions and .gardenignore files, which made hot reloading unusable or very slow in some scenarios.
Note: This may be a breaking change in some situations, if you had been relying on the previous behavior. See the migration guide below for more on that.
Prior to 0.12, it wasn't possible to use ${runtime.*} templates for values in helm modules. This made it difficult or impossible to feed information from e.g. terraform modules into Helm charts. This has been fixed.
Sometimes it's necessary to run some external script before Garden becomes usable, e.g. to bootstrap your project, perform some authentication etc. You now have an option to define an "init script" via the exec provider. See here for details.

There are many other small fixes and improvements in the release. See the Changelog below for details.

Migration guide

Please review the following, to see if you need to make any changes ahead of upgrading to 0.12.

As mentioned above, we've changed the default namespace for the kubernetes and local-kubernetes providers. The default namespace in the kubernetes provider is now <project name>.<environment namespace>, whereas previously it was just the project name. To revert to the prior default, set namespace: ${project.name} in your provider configuration. If you had already explicitly set the namespace field in your provider configuration, no change is necessary.
Garden no longer respects .gitignore files by default now. This change is based on frequent user feedback, which was that this implicit behavior was unexpected and confusing—and we agree! If you'd like to retain the previous default behavior, you can explicitly set dotIgnoreFiles: [.gitignore, .gardenignore] in your project configs. Alternatively, you can copy the .gitignore contents to .gardenignore files in corresponding directories. If you already have dotIgnoreFiles set in your config, no change is necessary.
We've made the structured outputs from commands (when you use --output json|yaml) more explicitly defined and documented. Along the way, we have altered the command output structure for many commands, so you may need to update your scripts if you've been relying on these command outputs in them. The command reference now includes sections on the structure returned by most commonly used commands.
We no longer automatically detect and migrate from Helm 2.x and Tiller. This shouldn't affect users in most cases, but we thought we should let you know :)
All Terraform versions below 0.12.26 have now been removed from the terraform provider and are no longer supported. If you have explicitly set a Terraform version in your terraform provider config, you need to either update that to "0.12.26", or remove the field to automatically use the latest version. You may need to change your Terraform config files accordingly, here's a link to the Terraform release notes which describe how to migrate between versions.
The fix mentioned above for hot reloading means we no longer sync untracked files, ignored or excluded files when hot reloading. If you had been relying on those being sync'ed, you'll need to modify you include/exclude/ignore configuration accordingly.

Troubleshooting

On OSX, if this error error comes up when running the garden binary from inside your ~/Downloads directory:

"garden" cannot be opened because the developer cannot be verified

try moving it outside the ~/Downloads directory before running it again.

If you're still getting this error, a workaround is to find the garden binary in Finder, CTRL-click it and choose
Open. This should prevent this error message from coming up again.

Changelog

Note that several of the commits are fixes and improvements to features first introduced in this release.

Bug Fixes

enterprise: fix final log entry flush (996a2382)
enterprise: wait for event stream to flush (d50539de)
kaniko: correctly set container command when using kaniko w/o socat (419abef7)
properly connect to remote in proxy (2a8b748f)
add tests for config contexts (TBS) (48bb1afd)
build-sync: randomly choose pod for rsync (be816791)
cli: don't show analytics message when telemetry is disabled (642cfe8e)
container: extraFlags weren't used when building in-cluster (1bbaad6c)
container: install ssh to base image (#1890) (dbc601b9)
docker: remove default timeout, fix publish timeouts (71576de3)
docs: fix typo in FAQ (5062c9ec)
enterprise: use noPlatform for system Garden (496d2696)
enterprise: use correct body param when streaming events (e8afa003)
enterprise: fixes to login & secrets logic (22760447)
examples: fix local exec example (90d9b584)
helm: allow runtime templates in helm module values field (346e776f)
k8s: ensure rendered helm chart contain runtime values (#1882) (26a87b9f)
k8s: make hot reloading respect excludes (0b61ddd1)
k8s: make helm & k8s tasks respect timeouts (8b4a4b86)
workflows: abort if step cmd returns errors (2602122b)
workflows: fix some logging issues in run workflows command (eb4c341b)

Features

config: allow environment[].variables to reference top-level vars (#1910) (fec2b535)
enterprise: include log level when streaming (d301b2e5)
exec: add script option to exec provider configuration (fc8879d7)
template: support nested expressions, maps and numeric keys (597b38ed)
workflows: add duration to step events (c1e2ad88)
added minimist-based CLI arg + opt parsing (b6e950b6)
add --resolve=partial option to get config command (6c113071)
add garden-dev/garden-azure docker image (#1893) (1a08593e)
commands: add fields to get config output (26334f60)
conftest: add combine option to conftest module (9fdf6c73)
core: add projectId to get config cmd output (e6f0acf9)
core: support namespaces natively at the framework level (e5023f7e)
core: workflows for sequencing commands (bb304cfe)
core: allow custom SQLite db directory (ede0b69f)
core: pre-fetch provider tools and make tools a native feature (#1858) (95c2aea1)
core: emit runtime status events (841285ee)
k8s: support immutable build success (1d4dcd78)
kaniko: make kaniko image configurable in provider config (66e6974c)
kaniko: when using remote registry do not run the proxy (547d9ba8)
kaniko: pass extraFlags to kaniko builders (deaab079)
enterprise: support non-interactive auth (b0205936)
workflows: add support for arbitrary user scripts in workflows (d7b76a41)
workflows: support referencing outputs from previous steps (d637b362)
workflows: allow writing files ahead of workflow execution (12e8b247)
workflows: add namespacing support (7eefa2ea)
workflows: improved log & event streaming (418506c2)

Improvements

template: show available keys when key is not found (2c4a1af8)
core: always require a namespace and simplify env config (#1900) (a783adc0)
core: don't respect .gitignore files by default (c7ef4533)
k8s: upgrade kaniko to v0.22 (#1834) (b3eb25d5)
terraform: support v0.12.26 and remove old versions (c84564e2)

Performance Improvements

analytics: don't wait for event tracks (1ee05bef)
core: cache provider statuses for faster successive startup (db72f2a8)
k8s: remove Helm 2 support and migration flow (b9e5f74e)

garden-io/garden v0.12.0 on GitHub