This release focuses on improvements to the repository format, stability and performance.
The new repository index format enables efficient append-only operation where delete permission is not required, removes the need for quick maintenance and enables internal content-level compression.
See https://kopia.io/docs/upgrade/ for upgrade instructions.
Kopia now also supports repository password change (new repositories only).
Big thanks to @pawitp for contributing major performance improvement to splitters (#1251), which improves large file upload performance by up to 30%.
Providers
- azure: added support for using SAS Tokens instead of storage keys (#1093)
- rclone: wait for rclone transfers to finish when closing (#1201)
- sftp: added automatic reconnect and connection pooling (#1195)
- sftp: ensure key file and known hosts are absolute paths (#1092)
- s3: point-in-time support for s3 versioned stores (#1259)
- webdav & and rclone support for atomic writes (#1206)
- webdav: disable server-side compression (#1100)
Repository features
- new index format supporting internal compression and does not require quick maintenance for performance reasons
- splitter: major performance improvements (#1251)
- implemented epoch-based index manager (#1174)
- internal logging - low-level logs are now stored in the repository.
- support for content-level compression (#1076)
- added 'lz4' and 'deflate' methods (#1077)
- added sharding of indexes on write (#1176)
- repository password change support (#1197)
- limit the duration of kopia.repository caching to 15 minutes (#1196)
- encryption: removed old, non-authenticated encryption methods (#979)
- big performance improvement for WriteContent with repo server (#1182)
- disabled quick maintenance for new repositories
- mount: fixed mounting of directories w/symlinks over WebDAV (#1358)
- localfs: fixed support for snapshotting UNC path root (\server\path) (#1362)
CLI
- 'kopia content verify' performance improvements (#1120)
- add a flag to not shallowrestore small files (#1126)
- added 'kopia repo set-parameters' to override mutable parameters (#1148)
- added 'kopia repository validate-provider` (#1205)
- additional 'index inspect' flags (#1231)
- changed 'kopia snapshot verify --verify-files-percent' to float (#1210)
- fixed ETA estimation of 'snapshot verify' (#1213)
- force-hash: Allow fractional percentages (#1209)
- include parameters in maintenance info JSON output (#981)
- optimized 'kopia index recover' by leveraging partial read and parallelism (#1094)
- support for shallow restore (#725)
- tagging of kopia snapshots and listing of snapshots by tag (#1030)
- logging: switched file log timestamps to be UTC by default, otherwise it's hard to correlate logs (#1240)
- only read first 128 MiB of provided file for compression benchmark (#1317)
- linux: read KOPIA_USE_KEYRING env (#1337)
Policy Changes
UI Changes
- do not attempt running maintenance if the current user is not the maintenance owner, to avoid producing error in the Tasks tab (#1010)
- fixed Estimate not honoring the defined policies (#1002)
- for read-only repositories start a read-only source manager (#1009)
- open .gitignore help in a new window (#1016)
- show a spinner when saving/deleting policy (#1018)
- fixed directory selector and browse buttons (#1310)
- fixed showing empty result pages (#1306)
- always show default repository at the top of the list (#1349)
- add simple page size selector (#1355)
Security
- security: switched to github.com/golang-jwt/jwt/v4 to fix upstream security issue (#1235)
Notable internal Changes
- automatically refresh indexes on read instead of relying on RefreshPeriodically() loop (#1243)
- ensure that we always consistently pick content.Info amongst entries with identical time and deleted flag (#1239)
- refactored content.Info to be an interface and switched index parsing to be lazy (#1008)
- refactored own writes cache and list cache into blob.Storage wrappers (#1133)
- few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
- reworked memory management to avoid allocating large chunks of memory for most buffers
- ci: Allow to compile Kopia cli on OpenBSD (#983)
- upgrade golang to 1.17
- testing: Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)
Changelog
8b2b91f content: fixed repo upgrade version (#1286)
06a9973 cli: include parameters in maintenance info JSON output (#981)
d98b0ed endurance: rewrote test to be more stable (#1285)
5941d63 ci: increased test timeout to 15 minutes
135c307 ci: fixed rclone installation
0d3c8cf testing: report reason why rclone test is being skipped
7e68d8e Consolidated format version flags (#1284)
87dbf44 Fix link in _index.md (#1134)
2721fba Upgraded remaining UI deps (#1283)
6c83cc6 ui: upgraded all NPM dependencies for htmlui/ and app/ (#1278)
296a519 build(deps): bump github.com/aws/aws-sdk-go from 1.40.34 to 1.40.37 (#1277)
562c596 build(deps): bump gocloud.dev from 0.23.0 to 0.24.0 (#1273)
1077cc2 blob/s3: point-in-time support for s3 versioned stores (#1259)
cc692c4 ci: delete golang installation on armhf before installing a new one (#1276)
0eedca8 build(deps-dev): bump electron-notarize from 1.0.1 to 1.1.0 in /app (#1262)
56edb04 content: switched defaults to use v2 index format and epoch-based indexes (#1275)
59b5a44 Makefile: refactored build tools installation (#1274)
06451aa upgrade golang to 1.17 + various dependencies (#1256)
082785a ci: run stress test on kopia/kopia repo (#1257)
310f451 ci: avoid double workflow executions on PRs (#1258)
9cebffc Fix endurance test (#1254)
7407419 maintenance: check for clock skew at the beginning of maintenance (#1253)
0578e2e splitter: optimize performance (#1251)
16aeb84 Fixed for rare stress test failure caused by use-after-free (#1250)
a203fa4 gather: MakeContiguous support for arbitrary chunk sizes (#1247)
9e182f1 linter: upgraded to 1.42.0 (#1246)
35d0f31 huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)
4676e9e content: automatically refresh indexes on read instead of relying on RefreshPeriodically() loop (#1243)
bece46d logging: switched file log timestamps to be UTC by default, otherwise it's hard to correlate logs (#1240)
1e10e85 content: ensure that we always consistently pick content.Info amongst entries with identical time and deleted flag (#1239)
0830de9 build(deps): bump github.com/klauspost/compress from 1.12.2 to 1.13.3 (#1237)
ff19caa build(deps): bump github.com/aws/aws-sdk-go from 1.40.12 to 1.40.17 (#1236)
4aacad2 server: switched to github.com/golang-jwt/jwt/v4 to fix upstream security issue (#1235)
e3e7cae content: removed EpochMarkerIndexBlobPrefix from allIndexBlobPrefixes (#1232)
1238a0d cli: additional 'index inspect' flags (#1231)
b96623f epoch: simplified TestIndexEpochManager_DeletionFailing test (#1230)
fcd79b4 build(deps): bump electron-log from 4.3.5 to 4.4.1 in /app (#1220)
eae7521 build(deps): bump google.golang.org/api from 0.50.0 to 0.52.0 (#1229)
67165ca build(deps): bump github.com/prometheus/client_golang (#1226)
17b4306 build(deps): bump postcss from 8.3.5 to 8.3.6 in /htmlui (#1217)
c04e3be build(deps-dev): bump electron-notarize from 1.0.0 to 1.0.1 in /app (#1219)
ecd9b02 build(deps): bump github.com/zeebo/blake3 from 0.1.1 to 0.2.0 (#1223)
140a86b build(deps): bump github.com/google/uuid from 1.2.0 to 1.3.0 (#1222)
5b8485b build(deps): bump github.com/aws/aws-sdk-go from 1.39.4 to 1.40.12 (#1224)
d6d9a1f Maintenance improvements for epoch-based index structures (#1225)
d77e771 provider test: fixed typo
cfdbc40 cli: Fixed ETA estimation of 'snapshot verify' (#1213)
6205b55 Fix weekly retention labels not being assigned correctly (#1211)
6c5bf42 cli: changed 'kopia snapshot verify --verify-files-percent' to float (#1210)
317d5c3 force-hash: Allow fractional percentages (#1209)
4f93a17 WebDAV and Rclone atomic writes (#1206)
e42cc6c Added 'kopia repository validate-provider` (#1205)
7dd9e33 ci: fixed syntax
e2e0cb3 ci: changed continue-on-error: true to if: succeeded() || failed()
bdd53f0 ci: install provider test dependencies separately
49d8a32 testing: split provider tests into separate steps
33cf322 ci: run provider test more often to get better stability signal
adbf213 build(deps): bump @testing-library/react in /htmlui (#1169)
9df5e5f rclone: updated wait logic
9fd6a89 testing: possible fix for TestS3StorageMinioSTS
8c2e4ff testing: simplified S3 test to fix error caused by minio behavior change
1a6f0ba testing: possible fix for TestS3StorageMinioSTS
4c9b9e5 Test cleanup improvements (#1203)
42f0278 s3: cleaned up test
4c2f52a Rclone and testing improvements (#1202)
928e1d3 rclone: wait for rclone transfers to finish when closing (#1201)
7b70446 ci: switch license NPM check to 'onlyAllow' instead of 'failOn'
42e5661 ci: added license check using github.com/frapposelli/wwhrd (#1200)
b2cc840 build(deps): bump github.com/pierrec/lz4 (#1188)
51c5403 build(deps): bump github.com/Azure/azure-storage-blob-go (#1194)
730ba7b Repository password change support (#1197)
f15c76e build(deps): bump google.golang.org/api from 0.48.0 to 0.50.0 (#1191)
bffe8b3 repo: limit the duration of kopia.repository caching to 15 minutes (#1196)
83b4dee serialize client auth (#232) (#1198)
7cabed8 sftp: added automatic reconnect and connection pooling (#1195)
f605f52 Upgraded bootstrap to v5 and react-bootstrap accordingly (#1192)
369dea9 build(deps): bump cloud.google.com/go/storage from 1.15.0 to 1.16.0 (#1193)
3d596a8 build(deps): bump google.golang.org/protobuf from 1.26.0 to 1.27.1 (#1189)
10aa214 build(deps): bump github.com/pkg/sftp from 1.13.0 to 1.13.2 (#1186)
ed88b77 build(deps): bump github.com/gofrs/flock from 0.8.0 to 0.8.1 (#1190)
6eb0064 build(deps): bump github.com/sanity-io/litter from 1.3.0 to 1.5.1 (#1171)
f14e865 build(deps): bump google.golang.org/grpc from 1.38.0 to 1.39.0 (#1172)
eb3b061 build(deps): bump github.com/aws/aws-sdk-go from 1.38.69 to 1.39.4 (#1180)
071a306 dependabot: switched to monthly cadence to reduce noise
f702e65 deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185)
f95ef88 testing: fixed typo
a1aa6ae tests: install minio mc tool (#1184)
1ef3d24 repo: big performance improvement for WriteContent with repo server (#1182)
61af639 cli: fixed profiling flags (#1181)
831ffcb Bump github.com/studio-b12/gowebdav (#1179)
47bc640 epoch: handling of canceled context on refresh (#1178)
62ad437 Implemented epoch-based index manager (#1174)
cc9b0e8 content: added sharding of indexes on write (#1176)
5642a8a refactor: extracted complete blob set functions to separate package (#1175)
a917027 build(deps): bump postcss from 8.2.15 to 8.3.5 in /htmlui (#1162)
4713148 build(deps): bump github.com/google/go-cmp from 0.5.5 to 0.5.6 (#1123)
7ac0cb3 build(deps): bump google.golang.org/api from 0.46.0 to 0.48.0 (#1125)
6d192fa build(deps): bump github.com/aws/aws-sdk-go from 1.38.41 to 1.38.69 (#1159)
9ceb302 build(deps): bump @testing-library/jest-dom in /htmlui (#1164)
e64d5b8 Fixed few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
d73e0d6 robustness: reduce memory consumption (#1155)
474ef1c Multiplaces (#1154)
d6d97de content: re-added missing lock accidentally removed by #1138 (#1156)
85cfbb1 cli: added 'kopia repo set-parameters' to override mutable parameters (#1148)
0756dee More epoch manager work (#1147)
9e059a1 upgraded linter to 1.41.0 (#1144)
5b04a1c Added documentation with storage tier advice (#1143)
8b0296c Misc index blob manager refactorings (#1138)
ede09c1 listcache: added missing FlushCaches() method which fixed test flakes
6277fa2 content: refactored own writes cache and list cache into blob.Storage wrappers (#1133)
e378229 site: fix privacy policy link in footer (#1132)
17ba77f Index manager refactor (#1131)
42c59cb Fixed bug in pattern matching for patterns containing a star. (#1129)
7735fcc epoch: added epoch manager + unit tests (#1128)
46a6cc3 blob: minor improvements + test coverage (#1127)
9397981 Add a flag to not shallowrestore small files (#1126)
6fc3bf1 Support for shallow restore (#725)
b7c8463 cli: 'kopia content verify' performance improvements (#1120)
c6accb8 Misc small and trivial changes (#1119)
53589d9 content: fixed minor data race
6b646f7 Content manager cleanups (#1118)
f094b97 build(deps-dev): bump concurrently from 6.1.0 to 6.2.0 in /app (#1108)
c01e732 build(deps): bump github.com/fatih/color from 1.11.0 to 1.12.0 (#1103)
87f1f52 cli: added tests for benchmark subcommands
d84c884 Added content manager internal logging (#1116)
4b251bd mechanical: added ctx parameter to repo.{Direct}WriteSession callback (#1114)
f5eb12e content: introduced content.Crypter (#1112)
3a15f66 webdav: disable server-side compression (#1100)
6811f1f cli: optimized 'kopia index recover' by leveraging partial read and parallelism (#1094)
e15a794 azure: added support for using SAS Tokens instead of storage keys (#1093)
76490dc sftp: ensure key file and known hosts are absolute paths (#1092)
40510c0 Support for content-level compression (#1076)
99b7a6e unrelated cleanups from PR #1076 (#1091)
ba5eb74 robustness: use testing.T for parallel robustness tests (#1051)
57c3ba5 testing: exclude additional test-only packages from code coverage
b5db8b0 trivial: fix typo
2faed2b sftp: run tests as part of unit tests (#1086)
227b2f1 ci: change codecov thresholds (#1085)
5179ad2 cli: test + misc improvements (#1083)
19ad808 build(deps): bump google.golang.org/grpc from 1.37.0 to 1.37.1 (#1080)
488ab94 build(deps): bump github.com/aws/aws-sdk-go from 1.38.36 to 1.38.41 (#1084)
5ed3a3a build(deps): bump gocloud.dev from 0.22.0 to 0.23.0 (#1082)
a699cc4 build(deps): bump github.com/fatih/color from 1.10.0 to 1.11.0 (#1079)
90a11d4 NPM package upgrades (#1078)
42be5cd build(deps): bump react-table from 7.6.3 to 7.7.0 in /htmlui (#1036)
d77149b build(deps): bump github.com/klauspost/compress from 1.11.13 to 1.12.2 (#1025)
56b5f6b build(deps): bump @testing-library/jest-dom in /htmlui (#1038)
79251b8 build(deps): bump react-bootstrap from 1.4.3 to 1.6.0 in /app (#1064)
a59ee47 build(deps): bump electron-updater from 4.3.5 to 4.3.8 in /app (#1041)
cb9ad45 build(deps-dev): bump electron from 11.2.1 to 12.0.7 in /app (#1063)
38d01d4 compression: added 'lz4' and 'deflate' methods (#1077)
34257ad testing: disable OneDrive test which is currently broken
c017019 rclone: fixed test cleanup (#1075)
aeca733 testing: added onedrive backend for rclone tests (#1074)
862298f testing: provider test fixes (#1073)
30ca3e2 Upgraded linter to 1.40.1 (#1072)
fcd507a Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)
41931f2 repo: refactored password persistence (#1065)
b844ce6 build(deps): bump electron-log from 4.3.4 to 4.3.5 in /app (#1039)
b5c3855 build(deps): bump github.com/aws/aws-sdk-go from 1.38.30 to 1.38.36 (#1061)
e5fecb1 build(deps): bump google.golang.org/api from 0.45.0 to 0.46.0 (#1060)
dec9d9a build(deps): bump github.com/pkg/profile from 1.5.0 to 1.6.0 (#1062)
281a7fc e2e test refactoring (#1058)
8a21677 cli: final steps to remove last global variables for password and globalPasswordFromToken (#1056)
9e861c9 Implemented index v2 builder and parser (#1028)
a461d76 cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)
6c5d58b build(deps): bump github.com/aws/aws-sdk-go from 1.38.26 to 1.38.30 (#1047)
d2288c4 cli: major refactoring (#1046)
a56e099 Benchmark: Remove superfluous = in printout (#1043)
1a7a016 Update full maintenance automation explanation (#1042)
dd41296 Tagging of kopia snapshots and listing of snapshots by tag (#1030)
d2b8169 robustness: add tests for kopia server repos (#1029)
5fcf389 Add documentation for nginx reverse proxy (#1033)
170d2a2 site: fixed Google indexing
cc0dd86 build(deps): bump cloud.google.com/go/storage from 1.14.0 to 1.15.0 (#1024)
11d764d build(deps): bump github.com/aws/aws-sdk-go from 1.38.25 to 1.38.26 (#1026)
df43037 Refactored content.Info to be an interface and switched index parsing to be lazy (#1008)
5dffad9 build(deps): bump google.golang.org/api from 0.44.0 to 0.45.0 (#1022)
2593cde build(deps): bump github.com/aws/aws-sdk-go from 1.38.22 to 1.38.25 (#1021)
dd3933f Grammar fix. (#1023)
0fb4ba4 ui: show a spinner when saving/deleting policy (#1018)
a5e1ecb ui: open .gitignore help in a new window (#1016)
d81f3f1 ci: do not upload artifacts from self-hosted runners, fixes build race (#1015)
d290c0a ui: do not attempt running maintenance if the current user is not the maintenance owner, to avoid producing error in the Tasks tab (#1010)
70a83b3 kopia-ui: for read-only repositories start a read-only source manager (#1009)
069299f Policy path validation (#1006)
62fab59 ui: fixed Estimate not honoring the defined policies (#1002)
3f29141 User must install npm manually on OpenBSD (#995)
bd5bdaa build(deps): bump github.com/aws/aws-sdk-go from 1.38.21 to 1.38.22 (#996)
74f926c content: added content.Info.OriginalLength (#989)
a2d31ad build(deps): bump github.com/aws/aws-sdk-go from 1.38.18 to 1.38.21 (#994)
28b21ac be consistent in $(MAKE) usage (#992)
d235672 Remove "KB" label from processed files count (#991)
2062c07 mechanical field renames (#988)
643c769 build(deps): bump electron-is-dev from 1.2.0 to 2.0.0 in /app (#984)
922bc7c ci: refactored credentials handling (#987)
e0e7f66 simple version of ospath to get default paths for openbsd systems. Based on the XDG version (#986)
2b6af55 Allow to compile Kopia cli on OpenBSD (#983)
a50d314 build(deps): bump @fortawesome/free-regular-svg-icons in /htmlui (#978)
bbb65e9 build(deps): bump uuid from 7.0.3 to 8.3.2 in /app (#977)
0d12cdc cleanup a few nits (#980)
2c3dfb7 encryption: removed old, non-authenticated encryption methods (#979)
85aa5e6 build: added freebsd build (untested) with FUSE mounting disabled (#964)
721c04c build(deps): bump google.golang.org/grpc from 1.36.1 to 1.37.0 (#965)
51ca018 build(deps): bump github.com/aws/aws-sdk-go from 1.38.17 to 1.38.18 (#973)
52240b7 build(deps): bump @fortawesome/free-regular-svg-icons in /htmlui (#971)
85e62e9 build(deps): bump electron-log from 4.3.2 to 4.3.4 in /app (#972)
2faf72c build(deps-dev): bump concurrently from 5.3.0 to 6.0.2 in /app (#975)
667d171 build(deps): bump google.golang.org/api from 0.43.0 to 0.44.0 (#966)
18c302e build(deps): bump github.com/aws/aws-sdk-go from 1.38.13 to 1.38.17 (#967)
a223b6f testing: run retrying_storage_test as part of regular unit tests
b59a113 ci: publish scoop and homebrew from tags