GitoxideLabs/gitoxide gix-odb-v0.85.0

gix-odb v0.85.0

on GitHub

Changed (BREAKING)

• replace forwarding errors with metadata contexts

  looked at the diff in detail, but went through quickly.
  Enough to give it my name, but really barely so. Some refactoring done as well,
  but nothing major.

  Return canonical Exn errors for loose and dynamic object lookup, alternate
  resolution, prefix lookup and integrity verification. Preserve original I/O,
  decoder, allocation, persistence and custom reader sources instead of forwarding
  them through operation-specific error enums.

  Use documented scalar Metadata contexts for native paths, object IDs, sizes,
  pack counts and recursion limits. Keep alternate::Cycle with its discovered
  directory chain, and preserve explicit retryability for interrupted verification
  or concurrent disk changes. An absent delta base remains not found, while a
  recursion limit alone implies neither absence nor corruption. Empty loose files
  are now classified as corruption.

  Remove empty error namespaces and redundant conversions in porcelain and CLI
  callers. Keep the genuine I/O boundary for store initialization and pack loading,
  using the existing adapter to retain both the I/O kind and the complete cause.

  Replace wrapper-construction tests with actual custom-reader, malformed-object,
  missing-delta and depth-limit failures. Validate metadata and classifications
  after conversion, including native path values and retained cycle details.

• migrate errors to gix-error

  rubberstamp, but looked at it more to understand why it's more code.
  Answer: downstream relies on better error classification.
  However, I think this can also be reduced a bit.

• raise MSRV to Rust 1.88

  The newly published dua-core 3.3 release used by linked-worktree removal
  requires Rust 1.88, so raise every workspace crate and the advertised badge
  together.

  Keep the MSRV checks buildable by selecting the latest sysinfo and rusqlite
  release lines that support Rust 1.88.

Performance

• avoid redundant in-memory object copies

  gix_object::Write::write() serialized an object into the trait buffer, then
  write_stream() copied that buffer again. Override it in the memory proxy so
  objects are serialized once, then hashed and stored directly.

  Keep the existing allocation when its content-addressed ID is already present.
  Valid callers promise that a known ID matches their bytes, so this does not
  change observable object contents.

  Compare against Git without filesystem I/O by including Git's notes.c
  directly, replacing object writes with its in-memory object path, and serving
  fixture trees from memory. Besides the already-materialized same-note case, the
  harness now parses a fresh root for each dispersed batch and replaces object IDs
  (00,00) through (0f,0f), (1f,1f), or (3f,3f) for batch sizes 16, 32,
  or 64. This matches the order and one-state lifetime of the gix-note
  Criterion benchmark.

  The same-note comparison remains:

  65,536-note fanout replacement	Time	Throughput
  gix-note before	250.45 µs/PUT	3,993 PUT/s
  gix-note after	231.43 µs/PUT	4,321 PUT/s
  Git in-memory baseline	148.71 µs/PUT	6,725 PUT/s

  The adapted harness was run as /tmp/git-notes-bench 20000 100. Each Git row
  contains 100 timed batches after one untimed warm-up. The gix-note batch-16
  rows are the Criterion means supplied for comparison; larger batches were only
  run for Git.

  Initial fanout	Batch	Implementation	Time/batch	Throughput	Tree hashes/PUT
  One level	16	gix-note	17.170 ms	931.84 PUT/s	—
  One level	16	Git	16.608 ms	963 PUT/s	2,185.50
  One level	32	Git	63.093 ms	507 PUT/s	4,241.50
  One level	64	Git	244.783 ms	261 PUT/s	8,353.50
  Two levels	16	gix-note	5.7305 ms	2,792.1 PUT/s	—
  Two levels	16	Git	5.863 ms	2,729 PUT/s	18.00
  Two levels	32	Git	21.691 ms	1,475 PUT/s	34.00
  Two levels	64	Git	80.637 ms	794 PUT/s	66.00

  Git exhibits the same batch-size sensitivity. With dispersed replacements,
  every write serializes all subtrees materialized by earlier replacements in the
  batch, so the average number of tree hashes per PUT grows linearly with the
  batch size.

  Here is the code for the Git baseline:

  /*
   * Disposable I/O-free comparison for gix-note replacement benchmarks.
   *
   * Build from git.git after `make libgit.a`:
   *
   *   cc -std=gnu23 -O3 -fno-common -I. -DNO_OPENSSL -DNO_GETTEXT \
   *     /tmp/git_notes_dispersed.c varint.c libgit.a -Wl,-dead_strip \
   *     -framework CoreServices -L/opt/homebrew/opt/gettext/lib \
   *     -lz -liconv -lpthread -o /tmp/git-notes-bench
   *
   * The first optional argument controls same-note iterations; the second
   * controls dispersed batches for each fanout and batch-size combination.
   */
  #define USE_THE_REPOSITORY_VARIABLE
  #define DISABLE_SIGN_COMPARE_WARNINGS
  
  #include "git-compat-util.h"
  #include "notes.h"
  #include "object-file.h"
  #include "odb/source.h"
  #include "repository.h"
  #include "tree-walk.h"
  
  static int bench_write_object(struct object_database *odb, const void *buf,
  			      unsigned long len, enum object_type type,
  			      struct object_id *oid);
  static void *bench_fill_tree_descriptor(struct repository *repo,
  					struct tree_desc *desc,
  					const struct object_id *oid);
  
  #define odb_write_object bench_write_object
  #define fill_tree_descriptor bench_fill_tree_descriptor
  #include "notes.c"
  #undef fill_tree_descriptor
  #undef odb_write_object
  
  static uint64_t object_writes;
  static struct strbuf root_one_level = STRBUF_INIT;
  static struct strbuf root_two_level = STRBUF_INIT;
  static struct strbuf one_level_subtree = STRBUF_INIT;
  static struct strbuf two_level_subtree = STRBUF_INIT;
  static struct strbuf leaf_tree = STRBUF_INIT;
  static struct object_id root_one_level_oid;
  static struct object_id root_two_level_oid;
  static struct object_id one_level_subtree_oid;
  static struct object_id two_level_subtree_oid;
  static struct object_id leaf_tree_oid;
  
  static int bench_write_object(struct object_database *odb, const void *buf,
  			      unsigned long len, enum object_type type,
  			      struct object_id *oid)
  {
  	object_writes++;
  	return odb_pretend_object(odb, (void *)buf, len, type, oid);
  }
  
  static const struct strbuf *fixture_tree(const struct object_id *oid)
  {
  	if (oideq(oid, &root_one_level_oid))
  		return &root_one_level;
  	if (oideq(oid, &root_two_level_oid))
  		return &root_two_level;
  	if (oideq(oid, &one_level_subtree_oid))
  		return &one_level_subtree;
  	if (oideq(oid, &two_level_subtree_oid))
  		return &two_level_subtree;
  	if (oideq(oid, &leaf_tree_oid))
  		return &leaf_tree;
  	die("unexpected fixture tree %s", oid_to_hex(oid));
  }
  
  static void *bench_fill_tree_descriptor(struct repository *repo UNUSED,
  					struct tree_desc *desc,
  					const struct object_id *oid)
  {
  	const struct strbuf *tree = fixture_tree(oid);
  	void *buf = xmemdupz(tree->buf, tree->len);
  
  	init_tree_desc(desc, oid, buf, tree->len);
  	return buf;
  }
  
  static struct leaf_node *leaf(const struct object_id *key,
  			      const struct object_id *value)
  {
  	struct leaf_node *out;
  
  	CALLOC_ARRAY(out, 1);
  	oidcpy(&out->key_oid, key);
  	oidcpy(&out->val_oid, value);
  	return out;
  }
  
  static struct object_id annotated_oid(unsigned first, unsigned second)
  {
  	struct object_id oid;
  
  	oidclr(&oid, the_repository->hash_algo);
  	oid.hash[0] = first;
  	oid.hash[1] = second;
  	return oid;
  }
  
  static struct object_id subtree_prefix(unsigned first)
  {
  	struct object_id oid;
  
  	oidclr(&oid, the_repository->hash_algo);
  	oid.hash[0] = first;
  	oid.hash[KEY_INDEX] = 1;
  	return oid;
  }
  
  static void hash_tree(const struct strbuf *tree, struct object_id *oid)
  {
  	hash_object_file(the_repository->hash_algo, tree->buf, tree->len,
  			 OBJ_TREE, oid);
  }
  
  static void build_fixture_trees(const struct object_id *note_oid)
  {
  	char path[GIT_MAX_HEXSZ + 1];
  
  	memset(path, '0', the_repository->hash_algo->hexsz);
  	path[the_repository->hash_algo->hexsz] = '\0';
  	write_tree_entry(&leaf_tree, 0100644, path + 4,
  			 the_repository->hash_algo->hexsz - 4, note_oid->hash);
  	hash_tree(&leaf_tree, &leaf_tree_oid);
  
  	for (unsigned second = 0; second < 256; second++) {
  		struct object_id annotated = annotated_oid(0, second);
  		const char *hex = oid_to_hex(&annotated);
  
  		write_tree_entry(&one_level_subtree, 0100644, hex + 2,
  				 the_repository->hash_algo->hexsz - 2,
  				 note_oid->hash);
  		xsnprintf(path, sizeof(path), "%02x", second);
  		write_tree_entry(&two_level_subtree, 040000, path, 2,
  				 leaf_tree_oid.hash);
  	}
  	hash_tree(&one_level_subtree, &one_level_subtree_oid);
  	hash_tree(&two_level_subtree, &two_level_subtree_oid);
  
  	for (unsigned first = 0; first < 256; first++) {
  		xsnprintf(path, sizeof(path), "%02x", first);
  		write_tree_entry(&root_one_level, 040000, path, 2,
  				 one_level_subtree_oid.hash);
  		write_tree_entry(&root_two_level, 040000, path, 2,
  				 two_level_subtree_oid.hash);
  	}
  	hash_tree(&root_one_level, &root_one_level_oid);
  	hash_tree(&root_two_level, &root_two_level_oid);
  }
  
  static void init_fixture_notes(struct notes_tree *notes,
  			       const struct object_id *root_oid)
  {
  	struct leaf_node root_tree = { 0 };
  
  	CALLOC_ARRAY(notes->root, 1);
  	notes->ref = "refs/notes/benchmark";
  	notes->combine_notes = combine_notes_overwrite;
  	notes->initialized = 1;
  	oidcpy(&root_tree.val_oid, root_oid);
  	load_subtree(notes, &root_tree, notes->root, 0);
  }
  
  static void release_notes(struct notes_tree *notes)
  {
  	note_tree_free(notes->root);
  	free(notes->root);
  }
  
  static void insert_reused_fixture(struct notes_tree *notes,
  				  const struct object_id *note_oid)
  {
  	for (unsigned first = 0; first < 256; first++) {
  		if (first == 0x80)
  			continue;
  		struct object_id key = subtree_prefix(first);
  		if (note_tree_insert(notes, notes->root, 0,
  				     leaf(&key, &one_level_subtree_oid),
  				     PTR_TYPE_SUBTREE, combine_notes_overwrite))
  			die("could not insert fixture subtree");
  	}
  	for (unsigned second = 0; second < 256; second++) {
  		struct object_id key = annotated_oid(0x80, second);
  		if (note_tree_insert(notes, notes->root, 0,
  				     leaf(&key, note_oid), PTR_TYPE_NOTE,
  				     combine_notes_overwrite))
  			die("could not insert fixture note");
  	}
  }
  
  static double seconds_since(const struct timespec *start,
  			    const struct timespec *end)
  {
  	return end->tv_sec - start->tv_sec +
  		(end->tv_nsec - start->tv_nsec) / 1000000000.0;
  }
  
  static void replace_and_write(struct notes_tree *notes,
  			      const struct object_id *annotated,
  			      const struct object_id *note,
  			      struct object_id *tree)
  {
  	if (add_note(notes, annotated, note, combine_notes_overwrite) ||
  	    write_notes_tree(notes, tree))
  		die("could not replace and write note");
  }
  
  static void run_same_note(uint64_t iterations,
  			  const struct object_id *note_oid,
  			  const struct object_id *replacement_oid)
  {
  	struct notes_tree notes = { 0 };
  	struct object_id annotated = annotated_oid(0x80, 0x80), tree;
  	struct timespec start, end;
  	uint64_t writes_before;
  	double elapsed;
  
  	CALLOC_ARRAY(notes.root, 1);
  	notes.ref = "refs/notes/benchmark";
  	notes.combine_notes = combine_notes_overwrite;
  	notes.initialized = 1;
  	insert_reused_fixture(&notes, note_oid);
  	replace_and_write(&notes, &annotated, replacement_oid, &tree);
  	writes_before = object_writes;
  
  	if (clock_gettime(CLOCK_MONOTONIC, &start))
  		die_errno("clock_gettime");
  	for (uint64_t i = 0; i < iterations; i++)
  		replace_and_write(&notes, &annotated,
  				  i & 1 ? replacement_oid : note_oid, &tree);
  	if (clock_gettime(CLOCK_MONOTONIC, &end))
  		die_errno("clock_gettime");
  
  	elapsed = seconds_since(&start, &end);
  	if (object_writes - writes_before != iterations * 258)
  		die("expected 258 tree writes per replacement, got %.2f",
  		    (double)(object_writes - writes_before) / iterations);
  	printf("fanout-expansion-1-level-fanout/replace/same-note/reused-state: "
  	       "%"PRIu64" PUTs in %.6f s = %.0f PUT/s (%.3f us/PUT); "
  	       "258 tree hashes/PUT; final tree %s\n",
  	       iterations, elapsed, iterations / elapsed,
  	       elapsed * 1000000.0 / iterations, oid_to_hex(&tree));
  	release_notes(&notes);
  }
  
  static void dispersed_batch(const struct object_id *root_oid,
  			    const struct object_id *replacement_oid,
  			    unsigned batch_size, struct object_id *tree)
  {
  	struct notes_tree notes = { 0 };
  
  	init_fixture_notes(&notes, root_oid);
  	for (unsigned i = 0; i < batch_size; i++) {
  		struct object_id annotated = annotated_oid(i, i);
  		replace_and_write(&notes, &annotated, replacement_oid, tree);
  	}
  	release_notes(&notes);
  }
  
  static void run_dispersed(const char *name, const struct object_id *root_oid,
  			  unsigned batch_size, uint64_t iterations,
  			  const struct object_id *replacement_oid)
  {
  	struct object_id tree;
  	struct timespec start, end;
  	uint64_t elements = iterations * batch_size;
  	uint64_t writes_before, writes_per_batch, writes;
  	double elapsed;
  
  	writes_before = object_writes;
  	dispersed_batch(root_oid, replacement_oid, batch_size, &tree);
  	writes_per_batch = object_writes - writes_before;
  
  	writes_before = object_writes;
  	if (clock_gettime(CLOCK_MONOTONIC, &start))
  		die_errno("clock_gettime");
  	for (uint64_t i = 0; i < iterations; i++)
  		dispersed_batch(root_oid, replacement_oid, batch_size, &tree);
  	if (clock_gettime(CLOCK_MONOTONIC, &end))
  		die_errno("clock_gettime");
  	writes = object_writes - writes_before;
  	if (writes != writes_per_batch * iterations)
  		die("tree write count changed between identical batches");
  
  	elapsed = seconds_since(&start, &end);
  	printf("%s/replace/dispersed-batch/%u/one-state: "
  	       "%"PRIu64" PUTs in %.6f s = %.0f PUT/s (%.3f us/PUT); "
  	       "%.2f tree hashes/PUT; final tree %s\n",
  	       name, batch_size, elements, elapsed, elements / elapsed,
  	       elapsed * 1000000.0 / elements, (double)writes / elements,
  	       oid_to_hex(&tree));
  }
  
  int main(int argc, const char **argv)
  {
  	const uint64_t same_iterations = argc > 1 ? strtoull(argv[1], NULL, 10) : 20000;
  	const uint64_t batch_iterations = argc > 2 ? strtoull(argv[2], NULL, 10) : 200;
  	const unsigned batch_sizes[] = { 16, 32, 64 };
  	struct object_id note_oid, replacement_oid;
  
  	if (!same_iterations || !batch_iterations)
  		die("iteration counts must be greater than zero");
  
  	the_repository->hash_algo = &hash_algos[GIT_HASH_SHA1];
  	the_repository->commondir = xstrdup("/tmp/git-notes-bench-no-objects");
  	the_repository->objects = odb_new(the_repository, 0);
  	odb_source_free(the_repository->objects->sources);
  	the_repository->objects->sources = NULL;
  	the_repository->objects->sources_tail = &the_repository->objects->sources;
  
  	hash_object_file(the_repository->hash_algo, "note", 4, OBJ_BLOB,
  			 &note_oid);
  	hash_object_file(the_repository->hash_algo, "replacement", 11, OBJ_BLOB,
  			 &replacement_oid);
  	build_fixture_trees(&note_oid);
  
  	run_same_note(same_iterations, &note_oid, &replacement_oid);
  	for (size_t i = 0; i < ARRAY_SIZE(batch_sizes); i++)
  		run_dispersed("fanout-expansion-1-level-fanout", &root_one_level_oid,
  			      batch_sizes[i], batch_iterations, &replacement_oid);
  	for (size_t i = 0; i < ARRAY_SIZE(batch_sizes); i++)
  		run_dispersed("steady-state-2-level-fanout", &root_two_level_oid,
  			      batch_sizes[i], batch_iterations, &replacement_oid);
  
  	strbuf_release(&root_one_level);
  	strbuf_release(&root_two_level);
  	strbuf_release(&one_level_subtree);
  	strbuf_release(&two_level_subtree);
  	strbuf_release(&leaf_tree);
  	odb_free(the_repository->objects);
  	free(the_repository->commondir);
  	return 0;
  }

Bug Fixes

• Keep rust workspace tests inside disposable repositories and isolated environments

  Direct Git launches inherited repository selectors and user configuration even
  when tests supplied a fixture working directory. Tests of default-environment
  APIs and local Git transports also shared the runner's environment. A few
  journey tests wrote beneath source directories or used the source checkout as
  the repository under test.

  Use the shared gix-testtools Git command builder for subprocess setup, isolated
  repository options for fixtures, and isolated child processes where the real
  environment-reading API must be exercised. Scope CWD changes, copy the fixture
  used by an object-write test, and run shell journeys through jtt run. Keep
  journey worktrees and example output within their disposable sandboxes and
  replace the attributes checkout test with a representative fixture repository.
  Prompt examples also run in isolated children and must build successfully; the
  old tests could ignore build failures and execute stale cached binaries.

  The affected Rust crate suites, internal test-tool build, and max-pure journey
  suite pass from a source copy without Git metadata. Signing and Git-daemon
  checks use only disposable keys, repositories, and local sockets.

Commit Statistics

• 13 commits contributed to the release over the course of 33 calendar days.
• 34 days passed between releases.
• 5 commits were understood as conventional.
• 0 issues like '(#ID)' were seen in commit messages

Commit Details