v4 RC2. This has only minor fixes vs RC1 (Fix off-by-one in max-failures and docs)
Converting from git-sync v3.x to v4.x
Git-sync v4 is a significant change from v3. It includes several flag changes
(though many of the old flags are kept for backwards compatibility), but more
importantly it fundamentally changes the way the internal sync-loop works.
It should be possible to upgrade a synced repo (e.g. in a volume) from git-sync
v3 to git-sync v4, but appropriate caution should be used for critical
deployments. We have a test which covers this, but there are many degrees of
config which we simply can't predict.
The v3 loop
The way git-sync v3.x works is sort of like how a human might work:
git clone <repo> <branch>
git fetch <remote>
git checkout <ref>
This made the code somewhat complicated, since it had to keep track of whether
this was the first pass (clone) or a subsequent pass (fetch). This led to a
number of bugs related to back-to-back runs of git-sync, and some race
conditions.
The v4 loop
In v4.x the loop is simpler - every pass is the same. This takes advantage of
some idempotent behaviors (e.g. git init
is safe to re-run) and uses git more
efficiently. Instead of cloning a branch, git-sync will now fetch exactly the
commit (by SHA) it needs. This transfers less data and closes the race
condition where a symbolic name can change after git ls-remote
but before
git fetch
.
Flags
The flag syntax parsing has changed in v4. git-sync v3 accept flags in Go's
own style: either -flag
or --flag
were accepted. git-sync v4 only accepts
long flag names in the more common two-dash style (--flag
), and accepts short
(single-character) flags in the one-dash style (-v 2
).
The following does not detail every flag available in v4 - just the one that
existed in v3 and are different in v4.
Verbosity: --v
-> -v
or --verbose
The change in flag parsing affects the old --v
syntax. To set verbosity
either use -v
or --verbose
. For backwards compatibility, --v
will be
used if it is specified.
Sync target: --branch
and --rev
-> --ref
The old --branch
and --rev
flags are deprecated in favor of the new --ref
flag. --ref
can be either a branch name, a tag name, or a commit hash (aka
SHA). For backwards compatibility, git-sync will still accept the old flags
and try to set --ref
from them.
|----------|---------|---------|------------------------------|
| --branch | --rev | --ref | meaning |
|----------|---------|---------|------------------------------|
| "" | "" | "HEAD" | remote repo's default branch |
| brname | "" | brname | remote branch `brname` |
| brname | "HEAD" | brname | remote branch `brname` |
| "" | tagname | tagname | remote tag `tagname` |
| other | other | "" | error |
|----------|---------|---------|------------------------------|
Log-related flags
git-sync v3 exposed a number of log-related flags (e.g. -logtostderr
). These
have all been removed. git-sync v4 always logs to stderr, and the only control
offered is the verbosity level (-v / --verbose
).
Symlink: --dest
-> --link
The old --dest
flag is deprecated in favor of --link
, which more clearly
conveys what it does. The allowed values remain the same, and for backwards
compatibility, --dest
will be used if it is specified.
Loop: --wait
-> --period
The old --wait
flag took a floating-point number of seconds as an argument
(e.g. "0.1" = 100ms). The new --period
flag takes a Go-style duration string
(e.g. "100ms" or "0.1s" = 100ms). For backwards compatibility, --wait
will
be used if it is specified.
Failures: --max-sync-failures
-> --max-failures
The new name of this flag is shorter and captures the idea that any
non-recoverable error in the sync loop counts as a failure. For backwards
compatibility, --max-sync-failures
will be used if it is specified.
Timeouts: --timeout
-> --sync-timeout
The old --timeout
flag took an integer number of seconds as an argument. The
new --sync-timeout
flag takes a Go-style duration string (e.g. "30s" or
"0.5m"). For backwards compatibility, --timeout
will be used if it is
specified.
Permissions: --change-permissions
-> --group-write
The old --change-permissions
flag was poorly designed and not able to express
the real intentions (e.g. "allow group write" does not mean "set everything to
0775"). The new --group-write
flag should cover what people ACTUALLY are
trying to do. The --change-permissions
flag is no longer supported.
Manual: --man
The new --man
flag prints a man-page style help document and exits.
Env vars
Most flags can also be configured by environment variables. In v3 the
variables all start with GIT_SYNC_
. In v4 they all start with GITSYNC_
,
though the old names are still accepted for compatibility.
Defaults
Depth
git-sync v3 would sync the entire history of the remote repo by default. v4
syncs just one commit, by default. This can be a significant performance and
disk-space savings for large repos. Users who want the full history can
specify --depth=0
.
Logs
The logging output for v3 was semi-free-form text. Log output in v4 is
structured and rendered as strict JSON.
Root dir
git-sync v3 container images defaulted --root
to "/tmp/git". In v4, that has
moved to "/git". Users who mount a volume and expect to use the default
--root
must mount it on "/git".
Hooks
git-sync v3 could "lose" exechook and webhook calls in the face of the app
restarting. In v4, app startup is treated as a sync, even if the correct hash
was already present, which means that hooks are always called.
Other changes
git-sync v3 would allow invalidly formatted env vars (e.g. a value that was
expected to be boolean holding an integer) and just ignore them with
a warning. v4 requires that they parse correctly.
Available at: registry.k8s.io/git-sync/git-sync:v4.0.0-rc2
Detailed changes
- Add help and manual flags, use pflag by @thockin in #298
- Change time-related flags to durations by @thockin in #299
- Add tests for other env funcs by @thockin in #301
- Change some flags (breaking) by @thockin in #300
- Add a test for tabs in manual by @thockin in #305
- Normalize the root path by @thockin in #302
- Clarify logging vs printf for fatal messages by @thockin in #303
- Add a "main struct" with methods by @thockin in #304
- Add the
--root
flag into the Usage section by @haiyanmeng in #307 - Change CONTRIBUTING.md to CNCF CLA by @thockin in #318
- Allow octal and hex values for int flags by @thockin in #322
- Log info about UID, GID, and HOME for debug by @thockin in #323
- Add an example pod YAML by @thockin in #328
- Default the git-sync root dir in container by @thockin in #330
- Use docker to run helper servers in e2e by @thockin in #332
- Makefile: Update base and build images by @justaugustus in #336
- Update git from backports by @thockin in #347
- Build container without cache by @thockin in #349
- Change the symlink targets to just the SHA (v4) by @thockin in #351
- Fix tests on master - reset needs "--" arg by @thockin in #353
- Add --git-config flag by @thockin in #342
- Fix exit non-zero exit codes when running as pid1 by @thockin in #344
- Don't require a TTY to build/test by @thockin in #362
- Copy all licenses into the container image (v4) by @thockin in #366
- Move sync-hook to after symlink flip by @thockin in #370
- Update to latest base image debian-base:buster-v1.6.0 (v4) by @thockin in #377
- Fix licenses for other arch'es by @thockin in #379
- Export the error details to an error file by @nan-yu in #375
- Sort the flags in the manual output by @thockin in #380
- Add support for sparse-checkout (v4 vbranch) by @thockin in #381
- Create the root directory if it doesn't exist by @nan-yu in #385
- Clean up global flRoot use (v4) by @thockin in #386
- move test cleanup to the end (v4) by @thockin in #391
- Grant read access for the error file to all users by @nan-yu in #390
- Small error string cleanup (v4) by @thockin in #394
- Make sure all code files have headers by @thockin in #397
- Downgrade libcurl to work around HTTP bug (v4) by @thockin in #399
- Change libcurl workaround to be less precise (v4) by @thockin in #401
- Update README to spec UID/GID by @thockin in #403
- Improve error message for askpass. by @briantkennedy in #407
- bump to logr v1.0.0-rc1 (v4) by @thockin in #406
- unbreak e2e for github (v4) by @thockin in #416
- Some collected cleanups on v4 by @thockin in #417
- Clean up worktree dirs during sync loop by @thockin in #418
- Handle a race between ls-remote and fetch by @thockin in #421
- Add Janet and Nan as approvers (master) by @thockin in #420
- Update RELEASING doc (v4 branch) by @thockin in #427
- Bump base image version (v4 branch) by @thockin in #433
- Remove broken analytics links by @xinydev in #434
- Add option to read password from file by @zhouhaibing089 in #431
- Small nits I found with password-file while porting #431 by @thockin in #436
- Make exechooks work like webhooks. by @thockin in #441
- use branch revision during worktree add (v4) by @thockin in #446
- E2E Overhaul (v4) by @thockin in #448
- v4: use repo default branch instead of master by @thockin in #451
- v4: Replace glogr with funcr by @thockin in #455
- Avoid writing to /etc/passwd unless needed by @thockin in #462
- Expose the ssh diagnostic message by @nan-yu in #465
- fix: resolve issue #463 by @ChrisERo in #466
- Followup to #466 - small cleanups for one-time hooks by @thockin in #468
- Don't try to remove the root if it appears corrupt (v4) by @thockin in #472
- Don't double-register the hook metric (v4) by @thockin in #476
- Allow --dest to be an absolute path (v4) by @thockin in #478
- Fix bug caused in previous PR (v4) by @thockin in #480
- Bump go to 1.17 by @thockin in #484
- small manual text fix (v4 branch) by @thockin in #486
- Fix tests to set missing arg (v4 only) by @thockin in #489
- Set $GITSYNC_HASH in exechook (v4) by @thockin in #491
- Set repoReady even when there wasn't a 1st clone (v4 branch) by @thockin in #494
- Add GC controls, e2e regexes (v4 branch) by @thockin in #496
- Update README to latest (master) by @thockin in #500
- update the base image to fix vulnerabilities (v4) by @thockin in #503
- Support repo change between invocations (v4) by @thockin in #505
- Only build test-tools when needed (v4) by @thockin in #513
- Use logger.WithName() for web/exechook (v4) by @thockin in #514
- Update RELEASING (v4) by @thockin in #512
- Pass the whole environment to exechooks (v4 branch) by @thockin in #521
- Fix Makefile to run tools on host OS/ARCH (v4) by @thockin in #526
- Pass CLEANUP=0 to test_e2e to leave logs (v4 branch) by @thockin in #528
- Make ssh e2e test not use --one-time by @thockin in #531
- upgrade base image to address vulnerabilities by @sdowell in #534
- v4: Fix inconsistency with build dotfile names by @thockin in #537
- v4: e2e: Reformat ncsvr scripts for readability by @thockin in #539
- v4: Clean up fail-count logging by @thockin in #540
- v4: Split password test to 2 - right and wrong passwd by @thockin in #542
- v4: Add -q to apt-get commands by @thockin in #544
- v4: e2e: Don't manually "docker kill" by @thockin in #538
- v4: Bump build image to go 1.18 by @thockin in #560
- v4: Bump go.mod to 1.18 by @thockin in #562
- v4: e2e with -v=6 by @thockin in #557
- V4: beef up askpass-url test by @thockin in #541
- V4: Rename e2e tests for easier partial runs by @thockin in #543
- v4: e2e: rebuild the container less often by @thockin in #564
- v4: Bring some Makefile cleanups from k/k by @thockin in #566
- Fix CVE-2022-2068 by @Liujingfang1 in #590
- v4: e2e: Fix ncsvr to wait for input on HTTP by @thockin in #568
- v4: e2e: $CLEANUP decides whether to rm containers by @thockin in #570
- v4: Clean up askpass_URL by @thockin in #572
- v4: e2e: don't set XDG_CONFIG_HOME by @thockin in #574
- v4: e2e: Make password test not use --one-time by @thockin in #576
- v4: update deps by @thockin in #587
- v4: Clean up credential init by @thockin in #578
- v4: Logging and error handling cleanup by @thockin in #580
- v4: log.V(9) md5sums of credentials by @thockin in #582
- v4: Change from "store" to "cache" for credentials by @thockin in #584
- v4: Don't set known_hosts to /dev/null by @thockin in #586
- Fix wrong env var name in docs by @thockin in #594
- e2e: also trap ERR by @thockin in #595
- Log a redacted form of environment at startup by @thockin in #596
- Update manifest-tool and get rid of hacks and old docs by @thockin in #597
- Document filesystem-volumes issue by @thockin in #599
- Log when exiting after --one-time by @thockin in #604
- Use os.Symlink() and os.Rename() instead of exec by @thockin in #605
- Reset --soft in the root by @thockin in #606
- e2e: basename the link when comparing by @thockin in #609
- Move tools into a tools/ subdir by @thockin in #608
- Better passwd redacting - including URLs by @thockin in #610
- Pass DBG=1 to
make
to build for debug by @thockin in #607 - Log before running command by @thockin in #612
- Set up buildx when building containers by @thockin in #614
- Retain user-provided GIT_SSH_COMMAND by @thockin in #616
- --max-sync-failures -> --max-failures by @thockin in #620
- Remove now-obsolete flag from struct by @thockin in #621
- Allow flags to have multiple env var names by @thockin in #618
- Support shallow sync when the rev is not in-range by @thockin in #615
- e2e: Allow N tests to fail and catch errexit by @thockin in #613
- Bump base image to 1.4.2 by @thockin in #625
- typo by @thockin in #626
- Disable local git configs by @thockin in #628
- e2e: Silence noise from git init by @thockin in #629
- Fix some of the manual help text by @thockin in #627
- e2e: move err tests and add bad-branch by @thockin in #630
- Change the sync loop to do a full sync by @thockin in #631
- Make --error-file allow abs paths by @thockin in #632
- use makeAbsPath for link by @thockin in #633
- Do print-and-exit flags before logging by @thockin in #638
- Add --touch-file flag and use it in tests by @thockin in #639
- refactor: move from io/ioutil to io and os packages by @Juneezee in #640
- V4: Allow quoted keys for --git-config by @thockin in #643
- V4 e2e: fix git submodules for file:// by @thockin in #647
- e2e: fix 2 broken tests by @thockin in #649
- Update Dockerfile docs and default for volumes by @thockin in #653
- Fix cases of syncing different SHAs back to back by @thockin in #656
- Update README on master to point to v3 by @thockin in #663
- v4: Update prometheus client (CVE) by @thockin in #669
- Fix e2e on v4 by @thockin in #670
- Log commands we run with original caller by @thockin in #675
- Require --http-bind when using other http flags by @thockin in #674
- v4: Default http-metrics to false by @thockin in #677
- v4: Deref tags on ls-remote by @thockin in #676
- v4: e2e improvements by @thockin in #673
- Allow sync wait period to be truncated by an HUP signal. by @trulede in #664
- Replace deprecated k8s registry references by @jmhbnz in #678
- e2e: move helper scripts to _test_tools/ by @thockin in #679
- Add logs when we use deprecated flags by @thockin in #681
- e2e: rename init tests, remove default-val args by @thockin in #682
- Add a test to exercise git by @thockin in #680
- fix typo by @KimHyeonwoo in #683
- Bump debian-base build image to v1.4.3 by @saschagrunert in #684
- e2e: s/REV/SHA by @thockin in #685
- e2e: compare numbers numerically, not lexically by @thockin in #686
- e2e: print failing line by @thockin in #687
- Add execution time to pkg/cmd runs by @thockin in #688
- Use a private gitconfig file by @thockin in #689
- Don't ignore invalid env vars by @thockin in #690
- Total overhaul of the fetch loop for v4 by @thockin in #691
- Move code (no edits), document methods by @thockin in #714
- Fine tune git command verbosity flags by @thockin in #713
- Change the default depth to 1 by @thockin in #712
- Tidy up log levels - logs read better now by @thockin in #710
- Add test for repo size and change default GC by @thockin in #711
- Makefile: set GOOS,GOARCH defaults to kill warning by @thockin in #721
- Make --webhook-success-status=0 the same as -1 by @thockin in #720
- Add a doc about changing from v3 to v4 by @thockin in #719
- Back-compat for --v usage by @thockin in #718
- e2e: handle log capture automatically by @thockin in #717
- Move worktrees to .worktrees/* by @thockin in #716
- Clean up all old worktrees by @thockin in #723
- Make all env var names GITSYNC_FOO by @thockin in #722
- Make flag defs easier to read by @thockin in #727
- Capture and simplify a git.Run() method by @thockin in #725
- Add an abspath type by @thockin in #728
- Update to go 1.20 by @thockin in #730
- Fetch just once per run and when hash changes by @thockin in #729
- Prevent git's 'dubious ownership' error (on v4 branch) by @thockin in #733
- Rename metric variables by @thockin in #734
- Support in-place updates from v3 by @thockin in #735
- Reorganize and rename deprecated flags vars by @thockin in #736
- e2e: Fix weirdly quoted strings by @thockin in #737
- EOL the
--change-permissions
flag by @thockin in #738 - Add --stale-worktree-timeout option by @sviscaino in #715
- Reorganize flag definitions by @thockin in #740
- Get rid of "must not start with ." logic by @thockin in #741
- Add a unit test for touch() by @thockin in #742
- Tighten up cleanup to be called once per loop by @thockin in #743
- Update README and other docs by @thockin in #744
- Move flag defs into main() by @thockin in #745
- Move main.go to root of repo by @thockin in #746
- Better logging around hooks by @thockin in #748
- Log actual flags rather than args+env by @thockin in #747
- Improve logging levels a bit by @thockin in #749
- Run hooks at startup by @thockin in #751
- Make the container image smaller by @thockin in #753
New Contributors
- @haiyanmeng made their first contribution in #307
- @justaugustus made their first contribution in #336
- @xinydev made their first contribution in #434
- @zhouhaibing089 made their first contribution in #431
- @ChrisERo made their first contribution in #466
- @sdowell made their first contribution in #534
- @Juneezee made their first contribution in #640
- @trulede made their first contribution in #664
- @jmhbnz made their first contribution in #678
- @KimHyeonwoo made their first contribution in #683
- @saschagrunert made their first contribution in #684
- @sviscaino made their first contribution in #715
Full Changelog: v3.2.1...v4.0.0-rc2