2020-03-11 - v11.2
This release of pgBadger fix several issues reported by users since
past six months. It also adds some new features:
* Add support and autodetection of AWS redshift log format.
* Add support to pgbouncer 1.11 new log format.
* Handle zstd and lz4 compression format
* Allow to fully separate statistics build and HTML report build in
incremental mode without having to read a log file. For example
it is possible to run pgbadger each hours as follow:
pgbadger -I -O "/out-dir/data" --noreport /var/log/postgresql*.log
It just creates the data binary files in "/out-dir/data" then
for example you can make reports each night for the next day in
a separate directory `/out-dir/reports`:
pgbadger -I -l "/out-dir/data/LAST_PARSED" -H "/out-dir/reports" /out-dir/data/2020/02/19/*.bin
This require to set the path to the last parsed information, the
path where HTML reports will be written and the binary data file
of the day.
There is also new command line options:
* Add new command line option --explain-url used to override the url
of the graphical explain tool. Default URL is:
http://explain.depesz.com/?is_public=0&is_anon=0&plan=
If you want to use a local install of PgExplain or an other tool.
pgBadger will add the plan in text format escaped at the end of
the URL.
* Add new option --no-week to instruct pgbadger to not build weekly
reports in incremental mode. Useful if it takes too much time and
resources.
* Add new command line option --command to be able to set a command
that pgBadger will execute to retrieve log entries on stdin.
pgBadger will open a pipe to the command and parse log entries
generated by the command. For example:
pgbadger -f stderr --command 'cat /var/log/postgresql.log'
which is the same as executing pgbadger with the log file directly
as argument. The interest of this option is obvious if you have to
modify the log file on the fly or that log entries are extracted
from a program or generated from a database. For example:
pgbadger -f csv --command 'psql dbname -c "COPY jrn_log TO STDOUT (FORMAT CSV)"'
* Add new command line option --noexplain to prevent pgBadger to
parse and report explain plan written to log by auto_explain
extension. This is useful if you have a PostgreSQL version < 9.0
where pgBadger generate broken reports when there is explain plan
in log.
Backward compatibility:
- By default pgBadger will truncate queries up to 100000 characters.
This arbitrary value and can be adjusted using option --maxlength.
Previous behavior was to not truncate queries but this could
lead in excessive resources usage. Limiting default size is safer
and the size limit might allow no truncate in most cases. However
queries will not be beautified if they exceed 25000 characters.
Here is the complete list of changes and acknowledgments:
- Fix non working --exclude-client option. Thanks to John Walsh
for the report.
- Add regression test for RDS log parsing and --exclude-client.
- Fix progress bar for pgbouncer log file. The "queries" label is
changed in "stats" for pgbouncer log files.
- Add command line option --explain-url used to override the url
of the graphical explain tool. Thanks to Christophe Courtois for
the feature request.
- Add support to pgbouncer 1.11 new log format. Thanks to Dan
Aksenov for the report.
- Handle zstd and lz4 compression format. Thanks to Adrien Nayrat
for the patch.
- Add support and autodetection of AWS redshift log format. Thanks
to Bhuvanesh for the reature request.
- Update documentation about redshift log format.
- Add new option --no-week to instruct pgbadger to not build weekly
reports in incremental mode. Thanks to cleverKermit17 for the
feature request.
- Fix a pattern match on file path that breaks pgBadger on Windows.
- Fix #554 about cyrillic and other encoded statement parameters
that was not reported properly in the HTML report even with custom
charset. The regression was introduced with a fix to the well
known Perl error message "Wide character in print". The patch have
been reverted and a new command line option: --wide-char is
available to recover this behavior. Add this option to your
pgbadger command if you have message "Wide character in print".
Add a regression test with Cyrillic and french encoding. Thanks
to 4815162342lost and yethee for the report.
- Update documentation to inform that lc_messages = 'en_US.UTF-8'
is valid too. Thanks to nodje for the report.
- Update documentation about --maxlength which default truncate size
is 100000 and no more default to no truncate. Thanks to nodje for
the report.
- Fix retention calculation at year overlap. Thanks to Fabio Pereira
for the patch.
- Fix parsing of rds log file format. Thanks to Kadaffy Talavera for
the report.
- Prevent generating empty index file in incremental mode when there
is no new log entries. Thanks to Kadaffy Talavera for the report.
- Fix non up to date documentation. Thanks to Eric Hanson for the
patch.
- Fixes the command line parameter from -no-explain to -noexplain.
Thanks to Indrek Toom for the patch.
- Fall back to default file size when totalsize can not be found.
Thanks to Adrien Nayrat for the patch.
- Fix some dates in examples. Thanks to Greg Clough for the patch.
- Use compressed file extension regexp in remaining test and extract
.bin extension in a separate condition.
- Handle zstd and lz4 compression format. Thanks to Adrien Nayrat
for the patch.
- Fix remaining call of SIGUSR2 on Windows. Thanks to inrap for the
report.
- Fix progress bar with log file of indetermined size.
- Add new command line option --command to be able to set a command
that pgBadger will execute to retrieve log entries on stdin.
Thanks to Justin Pryzby for the feature request.
- Add new command line option --noexplain to prevent pgBadger to
parse and report explain plan written to log by auto_explain
extension. This is useful if you have a PostgreSQL version < 9.0
where pgBadger generate broken reports when there is explain plan
in log. Thanks to Massimo Sala for the feature request.
- Fix RDS log parsing when the prefix is set at command line. Thanks
to Bing Zhao for the report.
- Fix incremental mode with rds log format. Thanks to Bing Zhao for
the report.
- Fix possible rds log parsing. Thanks to James van Lommel and Simon
Dobner for the report.
- Fix statement classification and add regression test. Thanks to
alexanderlaw for the report.
- Fix anonymization of single characters in IN clause. Thanks to
Massimo Sala for the report.
- Fix RDS log parsing for rows without client/user/db information.
Thanks to Konrad for the report.