grafana/loki v1.6.0 on GitHub

It's the second thursday of the eighth month of the year which means it's time for another Loki Release!!

Before we highlight important features and changes, congratulations to @adityacs, who is the newest member of the Loki team!
Aditya has been regularly contributing to the Loki project for the past year, with each contribution better than the last.
Many of the items on the following list were thanks to his hard work. Thank you, Aditya, and welcome to the team!

I think we might have set a new record with 189 PR's in this release!

Important Notes

Please Note There are several changes in this release which might require your attention!

The NET_BIND_SERVICE capability was removed from the Loki process in the docker image, it's no longer possible to run Loki with the supplied image on a port less than 1024
If you run microservices, there is an important rollout sequence to prevent query errors.
Scrape configs have changed for Promtail in both Helm and Ksonnet affecting two labels: instance -> pod and container_name -> container.
Almost all of the Loki Canary metrics were renamed.
A few command line flags where changed (although they are likely not commonly used)
If you use ksonnet and run on GCS and Bigtable you may see an error in your config as a default value was removed.
If you are using boltdb-shipper, you will likekly need to add a new schema_config entry.

Check the upgrade guide for detailed information on all these changes.

Notable Features and Fixes

Query language enhancements

2150 introduces bytes_rate, which calculates the per second byte rate of a log stream, and bytes_over_time, which returns the byte size of a log stream.
2182 introduces a long list of comparison operators, which will let you write queries like count_over_time({foo="bar"}[1m]) > 10. Check out the PR for a more detailed description.

Loki performance improvements

2216, 2218, and 2219 all improve how memory is allocated and reused for queries.
2239 is a huge improvement for certain cases in which a query covers a large number of streams that all overlap in time. Overlapping data is now internally cached while Loki works to sort all the streams into the proper time order.
2293 was a big refactor to how Loki internally processes log queries vs. metric queries, creating separate code paths to further optimize metric queries. Metric query performance is now 2 to 10 times faster.

If you are using the query-frontend:

2441 improves how label queries can be split and queried in parallel
2123 allows queries to the series API to be split by time and parallelized; and last but most significant
1927 allows for a much larger range of queries to be sharded and performed in parallel. Query sharding is a topic in itself, but as a rough summary, this type of sharding is not time dependent and leverages how data is already stored by Loki to be able to split queries up into 16 separate pieces to be queried at the same time.

Promtail

2296 allows Promtail to expose the Loki Push API. With this, you can push from any client to Promtail as if it were Loki, and Promtail can then forward those logs to another Promtail or to Loki. There are some good use cases for this with the Loki Docker Logging Driver; if you want an easier way to configure pipelines or expose metrics collection, point your Docker drivers at a Promtail instance.
2282 contains an example Amazon Lambda where you can use a fan-in approach and ingestion timestamping in Promtail to work around out of order issues with multiple Lambdas processing the same log stream. This is one way to get logs from a high-cardinality source without adding a high-cardinality label.
2060 introduces the Replace stage, which lets you find and replace or remove text inside a log line. Combined with 2422 and 2480, you can now find and replace sensitive data in a log line like a password or email address and replace it with ****, or hash the value to prevent readability, while still being able to trace the value through your logs. Last on the list of pipeline additions,
2496 adds a Drop pipeline stage, which lets you drop log lines based on several criteria options including regex matching content, line length, or the age of the log line. The last two are useful to prevent sending to Loki logs that you know would be rejected based on configured limits in the Loki server.

Logstash output plugin

1822 added a Logstash output plugin for Loki. If you have an existing Logstash install, you can now use this plugin to send your logs to Loki to make it easier to try out, or use Loki alongside an existing logging installation.

Loki Canary

2344 improved the canaries capabilities for checking for data integrity, including spot checking for logs over a longer time window and running metric queries to verify count_over_time accuracy.

Logcli

2470 allows you to color code your log lines based on their stream labels for a nice visual indicator of streams.
2497 expands on the series API query to Loki with the--analyze-labels flag, which can show you a detailed breakdown of your label key and value combinations. This is very useful for finding improper label usage in Loki or labels with high cardinality.
2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a --limit= far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the --batch= parameter (which defaults to 1000) until the requested limit is reached!

Misc

2453 improves the error messages when a query times out, as Context Deadline Exceeded wasn’t the most intuitive.
2336 provides two new flags that will print the entire Loki config object at startup. Be warned there are a lot of config options, and many won’t apply to your setup (such as storage configs you aren’t using), but this can be a really useful tool when troubleshooting. Sticking with the theme of best for last,
2224 and 2288 improve support for running Loki with a shared Ring using memberlist while not requiring Consul or Etcd. We need to follow up soon with some better documentation or a blog post on this!

All Changes

For a full list of changes, please checkout the CHANGELOG

Dependencies

Go Version: 1.14.2
Cortex Version: 7014ff11ed70d9d59ad29d0a95e73999c436c47c

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:1.6.0"
$ docker pull "grafana/promtail:1.6.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"