With 1.3.0 we are excited to announce several improvements focusing on performance!
First and most significant is the Query Frontend:
- 1442 cyriltovena: Loki Query Frontend
The query frontend allows for sharding queries by time and dispatching them in parallel to multiple queriers, giving true horizontal scaling ability for queries. Take a look at the jsonnet changes to see how we are deploying this in our production setup. Keep an eye out for a blog post with more information on how the frontend works and more information on this exciting new feature.
In our quest to improve query performance, we discovered that gzip, while good for compression ratio, is not the best for speed. So we introduced the ability to select from several different compression algorithms:
- 1411 cyriltovena: Adds configurable compression algorithms for chunks
We are currently testing out LZ4 and snappy, LZ4 seemed like a good fit however we found that it didn't always compress the same data to the same output which was causing some troubles for another important improvement:
- 1438 pstibrany: pkg/ingester: added sync period flags
Extending on the work done by @bboreham on Cortex, @pstibrany added a few new flags and code to synchronize chunks between ingesters, which reduces the number of chunks persisted to object stores and therefore also reduces the number of chunks loaded on queries and the amount of de-duplication work which needs to be done.
As mentioned above, LZ4 was in some cases compressing the same data with a different result which was interfering with this change, we are still investigating the cause of this issue (It may be in how we implemented something, or may be in the compression code itself). For now we have switched to snappy which has seen a reduction in data written to the object store from almost 3x the source data (with a replication factor of 3) to about 1.5x, saving a lot of duplicated log storage!
Another valuable change related to chunks:
- 1406 slim-bean: allow configuring a target chunk size in compressed bytes
With this change you can set a chunk_target_size and Loki will attempt to fill a chunk to approx that size before flushing (previously a chunk size was a hard coded 10 blocks where the default block size is 262144 bytes). Larger chunks are beneficial for a few reasons, mainly on reducing API calls to your object store when performing queries, but also in reducing overhead in a few places, especially when processing very high volume log streams.
Another big improvement is the introduction of accurate rate limiting when running microservices:
- 1486 pracucci: Add ingestion rate global limit support
Previously the rate limit was applied at each distributor, however with traffic split over many distributors the limit would need to be adjusted accordingly. This meant that scaling up distributors required changing the limit. Now this information is communicated between distributors such that the limit should be applied accurately regardless of the number of distributors.
And last but not least on the notable changes list is a new feature for Promtail:
- 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support
With this change Promtail can receive syslogs via TCP! Thanks to @bastjan for all the hard work on this submission!
Important things to note:
- 1519 Changes a core behavior in Loki regarding logs with duplicate content AND duplicate timestamps, previously Loki would store logs with duplicate timestamps and content, moving forward logs with duplicate content AND timestamps will be silently ignored. Mainly this change is to prevent duplicates that appear when a batch is retried (the first entry in the list would be inserted again, now it will be ignored). Logs with the same timestamp and different content will still be accepted.
- 1486 Deprecated
-distributor.limiter-reload-periodflag / distributor'slimiter_reload_periodconfig option.
As always, the full list of changes can be found in the CHANGELOG!
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:v1.3.0"
$ docker pull "grafana/promtail:v1.3.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.3.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"