pack-objects: Create an alternative name hash algorithm (recreated) #1823

gitgitgadget · 2024-11-21T20:12:00Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote: > From: Derrick Stolee <[email protected]> > > The pack_name_hash() method has not been materially changed since it was > introduced in ce0bd64299a (pack-objects: improve path grouping > heuristics., 2006-06-05). The intention here is to group objects by path > name, but also attempt to group similar file types together by making > the most-significant digits of the hash be focused on the final > characters. > > Here's the crux of the implementation: > > /* > * This effectively just creates a sortable number from the > * last sixteen non-whitespace characters. Last characters > * count "most", so things that end in ".c" sort together. > */ > while ((c = *name++) != 0) { > if (isspace(c)) > continue; > hash = (hash >> 2) + (c << 24); > } Hah. I like that the existing implementation is small enough to fit (in its entirety!) into the commit message! > As the comment mentions, this only cares about the last sixteen > non-whitespace characters. This cause some filenames to collide more > than others. Here are some examples that I've seen while investigating > repositories that are growing more than they should be: > > * "/CHANGELOG.json" is 15 characters, and is created by the beachball > [1] tool. Only the final character of the parent directory can > differntiate different versions of this file, but also only the two s/differntiate/differentiate ;-). > most-significant digits. If that character is a letter, then this is > always a collision. Similar issues occur with the similar > "/CHANGELOG.md" path, though there is more opportunity for > differences in the parent directory. > > * Localization files frequently have common filenames but differentiate > via parent directories. In C#, the name "/strings.resx.lcl" is used > for these localization files and they will all collide in name-hash. > > [1] https://github.com/microsoft/beachball > > I've come across many other examples where some internal tool uses a > common name across multiple directories and is causing Git to repack > poorly due to name-hash collisions. > > It is clear that the existing name-hash algorithm is optimized for > repositories with short path names, but also is optimized for packing a > single snapshot of a repository, not a repository with many versions of > the same file. In my testing, this has proven out where the name-hash > algorithm does a good job of finding peer files as delta bases when > unable to use a historical version of that exact file. I'm not sure I entirely agree with the suggestion that the existing hash function is only about packing repositories with short pathnames. I think an important part of the existing implementation is that tries to group similar files together, regardless of whether or not they appear in the same tree. As you have shown, this can be a problem when the fact two files that happen to end in "CHANGELOG.json" end up in vastly different trees and *aren't* related. I don't think that nailing all of these details in the commit message is necessary, but I do think it's worth adjusting what the original commit message says in terms of what the existing algorithm is optimized for. > However, for repositories that have many versions of most files and > directories, it is more important that the objects that appear at the > same path are grouped together. > > Create a new pack_full_name_hash() method and a new --full-name-hash > option for 'git pack-objects' to call that method instead. Add a simple > pass-through for 'git repack --full-name-hash' for additional testing in > the context of a full repack, where I expect this will be most > effective. > > The hash algorithm is as simple as possible to be reasonably effective: > for each character of the path string, add a multiple of that character > and a large prime number (chosen arbitrarily, but intended to be large > relative to the size of a uint32_t). Then, shift the current hash value > to the right by 5, with overlap. The addition and shift parameters are > standard mechanisms for creating hard-to-predict behaviors in the bits > of the resulting hash. > > This is not meant to be cryptographic at all, but uniformly distributed > across the possible hash values. This creates a hash that appears > pseudorandom. There is no ability to consider similar file types as > being close to each other. I think you hint at this in the series' cover letter, but I suspect that this pseduorandom behavior hurts in some small number of cases and that the full-name hash idea isn't a pure win, e.g., when we really do want to delta two paths that both end in CHAGNELOG.json despite being in different parts of the tree. You have some tables here below that demonstrate a significant improvement with the full-name hash in use, which I think is good worth keeping in my own opinion. It may be worth updating those to include the new examples you highlighted in your revised cover letter as well. > In a later change, a test-tool will be added so the effectiveness of > this hash can be demonstrated directly. > > For now, let's consider how effective this mechanism is when repacking a > repository with and without the --full-name-hash option. Specifically, Is this repository publicly available? If so, it may be worth mentioning here. > let's use 'git repack -adf [--full-name-hash]' as our test. > > On the Git repository, we do not expect much difference. All path names > are short. This is backed by our results: > > | Stage | Pack Size | Repack Time | > |-----------------------|-----------|-------------| > | After clone | 260 MB | N/A | > | Standard Repack | 127MB | 106s | > | With --full-name-hash | 126 MB | 99s | Ahh. Here's a great example of it helping to a smaller extent. Thanks for including this as part of demonstrating the full picture (both the benefits and drawbacks). > This example demonstrates how there is some natural overhead coming from > the cloned copy because the server is hosting many forks and has not > optimized for exactly this set of reachable objects. But the full repack > has similar characteristics with and without --full-name-hash. Good. > However, we can test this in a repository that uses one of the > problematic naming conventions above. The fluentui [2] repo uses > beachball to generate CHANGELOG.json and CHANGELOG.md files, and these > files have very poor delta characteristics when comparing against > versions across parent directories. > > | Stage | Pack Size | Repack Time | > |-----------------------|-----------|-------------| > | After clone | 694 MB | N/A | > | Standard Repack | 438 MB | 728s | > | With --full-name-hash | 168 MB | 142s | > > [2] https://github.com/microsoft/fluentui > > In this example, we see significant gains in the compressed packfile > size as well as the time taken to compute the packfile. Amazing! > Using a collection of repositories that use the beachball tool, I was > able to make similar comparisions with dramatic results. While the > fluentui repo is public, the others are private so cannot be shared for > reproduction. The results are so significant that I find it important to > share here: > > | Repo | Standard Repack | With --full-name-hash | > |----------|-----------------|-----------------------| > | fluentui | 438 MB | 168 MB | > | Repo B | 6,255 MB | 829 MB | > | Repo C | 37,737 MB | 7,125 MB | > | Repo D | 130,049 MB | 6,190 MB | > > Future changes could include making --full-name-hash implied by a config > value or even implied by default during a full repack. > > It is important to point out that the name hash value is stored in the > .bitmap file format, so we must disable the --full-name-hash option when > bitmaps are being read or written. Later, the bitmap format could be > updated to be aware of the name hash version so deltas can be quickly > computed across the bitmapped/not-bitmapped boundary. Agreed. > Signed-off-by: Derrick Stolee <[email protected]> > --- > Documentation/git-pack-objects.txt | 3 ++- > builtin/pack-objects.c | 25 ++++++++++++++++++++----- > builtin/repack.c | 5 +++++ > pack-objects.h | 21 +++++++++++++++++++++ > t/t5300-pack-object.sh | 15 +++++++++++++++ > 5 files changed, 63 insertions(+), 6 deletions(-) > > diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt > index e32404c6aae..93861d9f85b 100644 > --- a/Documentation/git-pack-objects.txt > +++ b/Documentation/git-pack-objects.txt > @@ -15,7 +15,8 @@ SYNOPSIS > [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] > [--cruft] [--cruft-expiration=<time>] > [--stdout [--filter=<filter-spec>] | <base-name>] > - [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list> > + [--shallow] [--keep-true-parents] [--[no-]sparse] > + [--full-name-hash] < <object-list> OK, I see that --full-name-hash is now listed in the synopsis, but I don't see a corresponding description of what the option does later on in this file. I took a look through the remaining patches in this series and couldn't find any further changes to git-pack-objects(1) either. > DESCRIPTION > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 08007142671..85595dfcd88 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -266,6 +266,14 @@ struct configured_exclusion { > static struct oidmap configured_exclusions; > > static struct oidset excluded_by_config; > +static int use_full_name_hash; > + > +static inline uint32_t pack_name_hash_fn(const char *name) > +{ > + if (use_full_name_hash) > + return pack_full_name_hash(name); > + return pack_name_hash(name); > +} > > /* > * stats > @@ -1698,7 +1706,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type, > return 0; > } > > - create_object_entry(oid, type, pack_name_hash(name), > + create_object_entry(oid, type, pack_name_hash_fn(name), > exclude, name && no_try_delta(name), > found_pack, found_offset); > return 1; > @@ -1912,7 +1920,7 @@ static void add_preferred_base_object(const char *name) > { > struct pbase_tree *it; > size_t cmplen; > - unsigned hash = pack_name_hash(name); > + unsigned hash = pack_name_hash_fn(name); > > if (!num_preferred_base || check_pbase_path(hash)) > return; > @@ -3422,7 +3430,7 @@ static void show_object_pack_hint(struct object *object, const char *name, > * here using a now in order to perhaps improve the delta selection > * process. > */ > - oe->hash = pack_name_hash(name); > + oe->hash = pack_name_hash_fn(name); > oe->no_try_delta = name && no_try_delta(name); > > stdin_packs_hints_nr++; > @@ -3572,7 +3580,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type > entry = packlist_find(&to_pack, oid); > if (entry) { > if (name) { > - entry->hash = pack_name_hash(name); > + entry->hash = pack_name_hash_fn(name); > entry->no_try_delta = no_try_delta(name); > } > } else { > @@ -3595,7 +3603,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type > return; > } > > - entry = create_object_entry(oid, type, pack_name_hash(name), > + entry = create_object_entry(oid, type, pack_name_hash_fn(name), > 0, name && no_try_delta(name), > pack, offset); > } > @@ -4429,6 +4437,8 @@ int cmd_pack_objects(int argc, > OPT_STRING_LIST(0, "uri-protocol", &uri_protocols, > N_("protocol"), > N_("exclude any configured uploadpack.blobpackfileuri with this protocol")), > + OPT_BOOL(0, "full-name-hash", &use_full_name_hash, > + N_("optimize delta compression across identical path names over time")), > OPT_END(), > }; > > @@ -4576,6 +4586,11 @@ int cmd_pack_objects(int argc, > if (pack_to_stdout || !rev_list_all) > write_bitmap_index = 0; > > + if (write_bitmap_index && use_full_name_hash) { > + warning(_("currently, the --full-name-hash option is incompatible with --write-bitmap-index")); > + use_full_name_hash = 0; > + } > + Good, we determine this early on in the command, so we don't risk computing different hash functions within the same process. I wonder if it's worth guarding against mixing the hash functions within the pack_name_hash() and pack_full_name_hash() functions themselves. I'm thinking something like: static inline uint32_t pack_name_hash(const char *name) { if (use_full_name_hash) BUG("called pack_name_hash() with --full-name-hash") /* ... */ } and the inverse in pack_full_name_hash(). I don't think it's strictly necessary, but it would be a nice guard against someone calling, e.g., pack_full_name_hash() directly instead of pack_name_hash_fn(). The other small thought I had here is that we should use the convenience function die_for_incompatible_opt3() here, since it uses an existing translation string for pairs of incompatible options. (As an aside, though that function is actually implemented in the _opt4() variant, and it knows how to handle a pair, trio, and quartet of mutually incompatible options, there is no die_for_incompatible_opt2() function. It may be worth adding one here since I'm sure there are other spots which would benefit from such a function). > diff --git a/builtin/repack.c b/builtin/repack.c > index d6bb37e84ae..ab2a2e46b20 100644 > --- a/builtin/repack.c > +++ b/builtin/repack.c I'm surprised to see the new option plumbed into repack in this commit. I would have thought that it'd appear in the subsequent commit instead. The implementation below looks good, I just imagined it would be placed in the next commit instead of this one. The remaining parts of this change look good to me. Thanks, Taylor

On the Git mailing list, Taylor Blau wrote (reply to this):

On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote: > The remaining parts of this change look good to me. Oops, one thing I forgot (which reading Peff's message in [1] reminded me of) is that I think we need to disable full-name hashing when we're reusing existing packfiles as is the case with try_partial_reuse(). There we're always looking at classic name hash values, so mixing the two would be a mistake. I think that amounts to: --- 8< --- diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 762949e4c8..7e370bcfc9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs) if (!(bitmap_git = prepare_bitmap_walk(revs, 0))) return -1; + use_full_name_hash = 0; + if (pack_options_allow_reuse()) reuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfiles, --- >8 --- Thanks, Taylor [1]: https://lore.kernel.org/git/[email protected]/

On the Git mailing list, Junio C Hamano wrote (reply to this):

Taylor Blau <[email protected]> writes: > On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote: >> The remaining parts of this change look good to me. > > Oops, one thing I forgot (which reading Peff's message in [1] reminded > me of) is that I think we need to disable full-name hashing when we're > reusing existing packfiles as is the case with try_partial_reuse(). > > There we're always looking at classic name hash values, so mixing the > two would be a mistake. I think that amounts to: > > --- 8< --- > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 762949e4c8..7e370bcfc9 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs) > if (!(bitmap_git = prepare_bitmap_walk(revs, 0))) > return -1; > > + use_full_name_hash = 0; Hmph, is this early enough, or has some other code path already computed the name hashes for the paths for the files to be packed? ... Goes and looks ... This is called from get_object_list() which - is not called under --stdin-packs, - is not called in cruft mode, - is not called when reading object list from --stdin so we are looking at the bog-standard "objects to be packed are given in the form of rev-list command line options from our command line". And in the function, we walk the history near the end, which makes show_object calls that adds object-entry with the name-hash. So the call to get_object_list_from_bitmap() happens way before the first use of the name-hash function, so this is probably safe. And obviously get_object_list_from_bitmap() is the only place we select objects to be packed from an existing pack and a bitmap file, so even if we gain new callers in the future, it is very likely that the new callers would benefit from this change. OK. Nicely done. > if (pack_options_allow_reuse()) > reuse_partial_packfile_from_bitmap(bitmap_git, > &reuse_packfiles, > --- >8 --- > > Thanks, > Taylor > > [1]: https://lore.kernel.org/git/[email protected]/

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 11/21/24 4:35 PM, Taylor Blau wrote: > On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote: >> The remaining parts of this change look good to me. > > Oops, one thing I forgot (which reading Peff's message in [1] reminded > me of) is that I think we need to disable full-name hashing when we're > reusing existing packfiles as is the case with try_partial_reuse(). > > There we're always looking at classic name hash values, so mixing the > two would be a mistake. I think that amounts to: > > --- 8< --- > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 762949e4c8..7e370bcfc9 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs) > if (!(bitmap_git = prepare_bitmap_walk(revs, 0))) > return -1; > > + use_full_name_hash = 0; > + Thanks. I have applied this code change with a comment detailing the context around the bitmap file storing only the default name-hash (for now) but that it can change in the future. Thanks, -Stolee

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 11/21/24 3:08 PM, Taylor Blau wrote: > On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote: >> From: Derrick Stolee <[email protected]> >> It is clear that the existing name-hash algorithm is optimized for >> repositories with short path names, but also is optimized for packing a >> single snapshot of a repository, not a repository with many versions of >> the same file. In my testing, this has proven out where the name-hash >> algorithm does a good job of finding peer files as delta bases when >> unable to use a historical version of that exact file. > > I'm not sure I entirely agree with the suggestion that the existing hash > function is only about packing repositories with short pathnames. I > think an important part of the existing implementation is that tries to > group similar files together, regardless of whether or not they appear > in the same tree. I'll be more explicit about the design for "hash locality" earlier in the message, but also pointing out that the locality only makes sense as a benefit when there are not enough versions of a file in history, since it's nearly always better to choose a previous version of the same file instead of a different path with a name-hash collision. Directory renames are on place where this is a positive decision, but those are typically rare compared to the full history of a large repo. >> This is not meant to be cryptographic at all, but uniformly distributed >> across the possible hash values. This creates a hash that appears >> pseudorandom. There is no ability to consider similar file types as >> being close to each other. > > I think you hint at this in the series' cover letter, but I suspect that > this pseduorandom behavior hurts in some small number of cases and that > the full-name hash idea isn't a pure win, e.g., when we really do want > to delta two paths that both end in CHAGNELOG.json despite being in > different parts of the tree. I mention that this doesn't work well in all cases when operating under a 'git push' or in a shallow clone. Shallow clones are disabled in a later commit and we don't have the necessary implementation to make this hash function be selected within 'git push'. > You have some tables here below that demonstrate a significant > improvement with the full-name hash in use, which I think is good worth > keeping in my own opinion. It may be worth updating those to include the > new examples you highlighted in your revised cover letter as well. I'll try to remember to move the newer examples to the cover letter. >> In a later change, a test-tool will be added so the effectiveness of >> this hash can be demonstrated directly. >> >> For now, let's consider how effective this mechanism is when repacking a >> repository with and without the --full-name-hash option. Specifically, > > Is this repository publicly available? If so, it may be worth mentioning > here. Here, by "when repacking a repository" I mean "we are going to test repacking a number of example repositories, that will be listed in detail in the coming tables". >> Using a collection of repositories that use the beachball tool, I was >> able to make similar comparisions with dramatic results. While the >> fluentui repo is public, the others are private so cannot be shared for >> reproduction. The results are so significant that I find it important to >> share here: >> >> | Repo | Standard Repack | With --full-name-hash | >> |----------|-----------------|-----------------------| >> | fluentui | 438 MB | 168 MB | >> | Repo B | 6,255 MB | 829 MB | >> | Repo C | 37,737 MB | 7,125 MB | >> | Repo D | 130,049 MB | 6,190 MB | These repos B, C, and D are _not_ publicly available, though. >> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt >> index e32404c6aae..93861d9f85b 100644 >> --- a/Documentation/git-pack-objects.txt >> +++ b/Documentation/git-pack-objects.txt >> @@ -15,7 +15,8 @@ SYNOPSIS >> [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] >> [--cruft] [--cruft-expiration=<time>] >> [--stdout [--filter=<filter-spec>] | <base-name>] >> - [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list> >> + [--shallow] [--keep-true-parents] [--[no-]sparse] >> + [--full-name-hash] < <object-list> > > OK, I see that --full-name-hash is now listed in the synopsis, but I > don't see a corresponding description of what the option does later on > in this file. I took a look through the remaining patches in this series > and couldn't find any further changes to git-pack-objects(1) either. I'll fix that. Thanks. As well as moving the 'git repack' changes out of this patch. I'll adjust the commit message to say "packing all objects' instead of 'git repack' to be clear that this can be done with a direct call to 'git pack-objects' instead of needing 'git repack'. >> + if (write_bitmap_index && use_full_name_hash) { >> + warning(_("currently, the --full-name-hash option is incompatible with --write-bitmap-index")); >> + use_full_name_hash = 0; >> + } >> + > > Good, we determine this early on in the command, so we don't risk > computing different hash functions within the same process. > > I wonder if it's worth guarding against mixing the hash functions within > the pack_name_hash() and pack_full_name_hash() functions themselves. I'm > thinking something like: > > static inline uint32_t pack_name_hash(const char *name) > { > if (use_full_name_hash) > BUG("called pack_name_hash() with --full-name-hash") > /* ... */ > } > > and the inverse in pack_full_name_hash(). I don't think it's strictly > necessary, but it would be a nice guard against someone calling, e.g., > pack_full_name_hash() directly instead of pack_name_hash_fn(). I think this is interesting defensive programming for future contributions. We essentially want the methods to only be called by pack_name_hash_fn() and don't have method privacy. We could extract it to its own header file but then would need to modify the prototype to include the signal for which hash type to use, but that would cause us to lose our ability to check for a bug like this. It may be even better to store a static value for the value of use_full_name_hash when it first executes, so it can exit if it notices a different value. (This is becoming large enough for its own patch.) > The other small thought I had here is that we should use the convenience > function die_for_incompatible_opt3() here, since it uses an existing > translation string for pairs of incompatible options. > > (As an aside, though that function is actually implemented in the > _opt4() variant, and it knows how to handle a pair, trio, and quartet of > mutually incompatible options, there is no die_for_incompatible_opt2() > function. It may be worth adding one here since I'm sure there are other > spots which would benefit from such a function). Interesting. I've not considered these functions before. >> diff --git a/builtin/repack.c b/builtin/repack.c >> index d6bb37e84ae..ab2a2e46b20 100644 >> --- a/builtin/repack.c >> +++ b/builtin/repack.c > > I'm surprised to see the new option plumbed into repack in this commit. > I would have thought that it'd appear in the subsequent commit instead. > The implementation below looks good, I just imagined it would be placed > in the next commit instead of this one. Yes, I should delay that to patch 2. Thanks, -Stole

gitgitgadget · 2024-12-04T20:57:05Z

On the Git mailing list, karthik nayak wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: [snip] > diff --git a/builtin/repack.c b/builtin/repack.c > index d6bb37e84ae..05e13adb87f 100644 > --- a/builtin/repack.c > +++ b/builtin/repack.c > @@ -58,6 +58,7 @@ struct pack_objects_args { > int no_reuse_object; > int quiet; > int local; > + int full_name_hash; > struct list_objects_filter_options filter_options; > }; > This variable doesn't seem to be used anywhere in this commit, and the following commit replaces it. I'm guessing it is from the previous version. [snip]

gitgitgadget · 2025-01-22T22:20:30Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:48PM +0000, Derrick Stolee via GitGitGadget wrote: > From: Derrick Stolee <[email protected]> > > The previous change introduced a new pack_name_hash_v2() function that > intends to satisfy much of the hash locality features of the existing > pack_name_hash() function while also distinguishing paths with similar > final components of their paths. > > This change adds a new --name-hash-version option for 'git pack-objects' > to allow users to select their preferred function version. This use of > an integer version allows for future expansion and a direct way to later > store a name hash version in the .bitmap format. > > For now, let's consider how effective this mechanism is when repacking a > repository with different name hash versions. Specifically, we will > execute 'git pack-objects' the same way a 'git repack -adf' process > would, except we include --name-hash-version=<n> for testing. > > On the Git repository, we do not expect much difference. All path names > are short. This is backed by our results: > > | Stage | Pack Size | Repack Time | > |-----------------------|-----------|-------------| > | After clone | 260 MB | N/A | > | --name-hash-version=1 | 127 MB | 129s | > | --name-hash-version=2 | 127 MB | 112s | > > This example demonstrates how there is some natural overhead coming from > the cloned copy because the server is hosting many forks and has not > optimized for exactly this set of reachable objects. But the full repack > has similar characteristics for both versions. > > Let's consider some repositories that are hitting too many collisions > with version 1. First, let's explore the kinds of paths that are > commonly causing these collisions: > > * "/CHANGELOG.json" is 15 characters, and is created by the beachball > [1] tool. Only the final character of the parent directory can > differentiate different versions of this file, but also only the two > most-significant digits. If that character is a letter, then this is > always a collision. Similar issues occur with the similar > "/CHANGELOG.md" path, though there is more opportunity for > differences In the parent directory. > > * Localization files frequently have common filenames but > differentiates via parent directories. In C#, the name > "/strings.resx.lcl" is used for these localization files and they > will all collide in name-hash. > > [1] https://github.com/microsoft/beachball > > I've come across many other examples where some internal tool uses a > common name across multiple directories and is causing Git to repack > poorly due to name-hash collisions. > > One open-source example is the fluentui [2] repo, which uses beachball > to generate CHANGELOG.json and CHANGELOG.md files, and these files have > very poor delta characteristics when comparing against versions across > parent directories. > > | Stage | Pack Size | Repack Time | > |-----------------------|-----------|-------------| > | After clone | 694 MB | N/A | > | --name-hash-version=1 | 438 MB | 728s | > | --name-hash-version=2 | 168 MB | 142s | > > [2] https://github.com/microsoft/fluentui > > In this example, we see significant gains in the compressed packfile > size as well as the time taken to compute the packfile. > > Using a collection of repositories that use the beachball tool, I was > able to make similar comparisions with dramatic results. While the > fluentui repo is public, the others are private so cannot be shared for > reproduction. The results are so significant that I find it important to > share here: > > | Repo | --name-hash-version=1 | --name-hash-version=2 | > |----------|-----------------------|-----------------------| > | fluentui | 440 MB | 161 MB | > | Repo B | 6,248 MB | 856 MB | > | Repo C | 37,278 MB | 6,755 MB | > | Repo D | 131,204 MB | 7,463 MB | > > Future changes could include making --name-hash-version implied by a config > value or even implied by default during a full repack. > > It is important to point out that the name hash value is stored in the > .bitmap file format, so we must force --name-hash-version=1 when bitmaps > are being read or written. Later, the bitmap format could be updated to > be aware of the name hash version so deltas can be quickly computed > across the bitmapped/not-bitmapped boundary. > > Signed-off-by: Derrick Stolee <[email protected]> > --- > Documentation/git-pack-objects.txt | 32 ++++++++++++++++++- > builtin/pack-objects.c | 49 +++++++++++++++++++++++++++--- > t/t5300-pack-object.sh | 31 +++++++++++++++++++ > 3 files changed, 106 insertions(+), 6 deletions(-) > > diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt > index e32404c6aae..7f69ae4855f 100644 > --- a/Documentation/git-pack-objects.txt > +++ b/Documentation/git-pack-objects.txt > @@ -15,7 +15,8 @@ SYNOPSIS > [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] > [--cruft] [--cruft-expiration=<time>] > [--stdout [--filter=<filter-spec>] | <base-name>] > - [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list> > + [--shallow] [--keep-true-parents] [--[no-]sparse] > + [--name-hash-version=<n>] < <object-list> > > > DESCRIPTION > @@ -345,6 +346,35 @@ raise an error. > Restrict delta matches based on "islands". See DELTA ISLANDS > below. > > +--name-hash-version=<n>:: > + While performing delta compression, Git groups objects that may be > + similar based on heuristics using the path to that object. While > + grouping objects by an exact path match is good for paths with > + many versions, there are benefits for finding delta pairs across > + different full paths. Git collects objects by type and then by a > + "name hash" of the path and then by size, hoping to group objects > + that will compress well together. > ++ > +The default name hash version is `1`, which prioritizes hash locality by > +considering the final bytes of the path as providing the maximum magnitude > +to the hash function. This version excels at distinguishing short paths > +and finding renames across directories. However, the hash function depends > +primarily on the final 16 bytes of the path. If there are many paths in > +the repo that have the same final 16 bytes and differ only by parent > +directory, then this name-hash may lead to too many collisions and cause > +poor results. At the moment, this version is required when writing > +reachability bitmap files with `--write-bitmap-index`. > ++ > +The name hash version `2` has similar locality features as version `1`, > +except it considers each path component separately and overlays the hashes > +with a shift. This still prioritizes the final bytes of the path, but also > +"salts" the lower bits of the hash using the parent directory names. This > +method allows for some of the locality benefits of version `1` while > +breaking most of the collisions from a similarly-named file appearing in > +many different directories. At the moment, this version is not allowed > +when writing reachability bitmap files with `--write-bitmap-index` and it > +will be automatically changed to version `1`. > + > > DELTA ISLANDS > ------------- > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 08007142671..90ea19417bc 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -266,6 +266,28 @@ struct configured_exclusion { > static struct oidmap configured_exclusions; > > static struct oidset excluded_by_config; > +static int name_hash_version = 1; > + > +static void validate_name_hash_version(void) > +{ > + if (name_hash_version < 1 || name_hash_version > 2) > + die(_("invalid --name-hash-version option: %d"), name_hash_version); > +} > + > +static inline uint32_t pack_name_hash_fn(const char *name) > +{ > + switch (name_hash_version) > + { This is definitely a nitpick, but the opening curly brace should appear on the preceding line. I don't think our CodingGuidelines explicitly say this. But in my head the convention is that only function bodies have their enclosing braces on their own line, as opposed to: static inline uint32_t pack_name_hash_fn(const char *name) { > + case 1: > + return pack_name_hash(name); > + > + case 2: > + return pack_name_hash_v2((const unsigned char *)name); > + > + default: > + BUG("invalid name-hash version: %d", name_hash_version); > + } > +} Otherwise this function looks very reasonable. > @@ -4576,6 +4609,12 @@ int cmd_pack_objects(int argc, > if (pack_to_stdout || !rev_list_all) > write_bitmap_index = 0; > > + validate_name_hash_version(); > + if (write_bitmap_index && name_hash_version != 1) { > + warning(_("currently, --write-bitmap-index requires --name-hash-version=1")); > + name_hash_version = 1; > + } > + Hmm. validate_name_hash_version() is its own function, which seems good, but then we do further validation on it here. I wonder if we should either move the latter step into validate_name_hash_version(), which would require us to pass a pointer to name_hash_version, or assign it the return value of validate_name_hash_version() (assuming it were changed to return the appropriate type instead of void). I think that we are probably pretty far into bike-shedding territory, but figured I'd share as it jumped out to me while reviewing. > if (use_delta_islands) > strvec_push(&rp, "--topo-order"); > > diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh > index 3b9dae331a5..4270eabe8b7 100755 > --- a/t/t5300-pack-object.sh > +++ b/t/t5300-pack-object.sh > @@ -674,4 +674,35 @@ do > ' > done > > +test_expect_success 'valid and invalid --name-hash-versions' ' > + # Valid values are hard to verify other than "do not fail". > + # Performance tests will be more valuable to validate these versions. > + for value in 1 2 > + do > + git pack-objects base --all --name-hash-version=$value || return 1 > + done && > + > + # Invalid values have clear post-conditions. > + for value in -1 0 3 > + do > + test_must_fail git pack-objects base --all --name-hash-version=$value 2>err && > + test_grep "invalid --name-hash-version option" err || return 1 > + done > +' Looks great, and thanks for handling a few nonsensical values in the latter loop. > +# The following test is not necessarily a permanent choice, but since we do not > +# have a "name hash version" bit in the .bitmap file format, we cannot write the > +# hash values into the .bitmap file without risking breakage later. > +# > +# TODO: Make these compatible in the future and replace this test with the > +# expected behavior when both are specified. > +test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' ' > + git pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err && > + test_grep "currently, --write-bitmap-index requires --name-hash-version=1" err && > + > + # --stdout option silently removes --write-bitmap-index > + git pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err && > + ! test_grep "currently, --write-bitmap-index requires --name-hash-version=1" err > +' This is grea, too, I appreciate having the reminder here for the future (and it'll feel so satisfying to change/remove this test when the time comes) ;-). Thanks, Taylor

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 1/22/25 5:17 PM, Taylor Blau wrote: > On Fri, Dec 20, 2024 at 05:19:48PM +0000, Derrick Stolee via GitGitGadget wrote: >> From: Derrick Stolee <[email protected]> >> +static inline uint32_t pack_name_hash_fn(const char *name) >> +{ >> + switch (name_hash_version) >> + { > > This is definitely a nitpick, but the opening curly brace should appear > on the preceding line. I don't think our CodingGuidelines explicitly say > this. But in my head the convention is that only function bodies have > their enclosing braces on their own line, as opposed to: You're right, I messed up on the switch statement. > static inline uint32_t pack_name_hash_fn(const char *name) { but based on my expectations and your earlier comment I think that this suggestion is a copy/paste error and you meant to highlight the switch statement. >> + validate_name_hash_version(); >> + if (write_bitmap_index && name_hash_version != 1) { >> + warning(_("currently, --write-bitmap-index requires --name-hash-version=1")); >> + name_hash_version = 1; >> + } >> + > > Hmm. validate_name_hash_version() is its own function, which seems good, > but then we do further validation on it here. I wonder if we should > either move the latter step into validate_name_hash_version(), which > would require us to pass a pointer to name_hash_version, or assign it > the return value of validate_name_hash_version() (assuming it were > changed to return the appropriate type instead of void). > > I think that we are probably pretty far into bike-shedding territory, > but figured I'd share as it jumped out to me while reviewing. Sounds good. I'll try to minimize the uses of name_hash_version outside of the cluster of methods that implement its details. This may become more complicated later when the timing of these checks is more interesting. Thanks, -Stolee

gitgitgadget · 2025-01-22T22:38:34Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:54PM +0000, Derrick Stolee via GitGitGadget wrote: > Create a third name hash function and extend the '--name-hash-version' > option in 'git pack-objects' and 'git repack' to understand it. This > hash version abandons all efforts for locality and focuses on creating a > somewhat uniformly-distributed hash function to minimize collisions. > > We can observe the effect of this collision avoidance in a large > internal monorepo that suffered from collisions in the previous > versions. The updates to p5314-name-hash.sh show these results: > > Test this tree > -------------------------------------------------- > 5314.1: paths at head 227.3K > 5314.2: distinct hash value: v1 72.3K > 5314.3: maximum multiplicity: v1 14.4K > 5314.4: distinct hash value: v2 166.5K > 5314.5: maximum multiplicity: v2 138 > 5314.6: distinct hash value: v3 227.3K > 5314.7: maximum multiplicity: v3 2 > > These results demonstrate that of the 227,000+ paths, nearly all of them > find distinct hash values. The maximum multiplicity is 2, improved from > 138 in the v2 hash function. The v2 hash function also had only 166K > distinct values, so it had a wide spread of collisions. I had a little trouble reading this section of the commit message. I think the framing makes sense (v2 has collisions which can impact pack generation time and/or size), but this section explains v3 I think one level too deep. This comparison (and the one below it for v3) shows a reduction in distinct hash values and the maximum multiplicity (I'm assuming for colliding hash values, in which case I might suggest renaming it as "maximum collisions"). But I imagine that many readers will primarily care about the effect of the new hash function on pack generation time and size. You show that below, but I think that it should potentially appear earlier in the commit message. Alternatively, you could consider leaving the time/size table alone where it is, and devote an extra sentence or two to explaining the impact on repacking time/size that the two metrics above (distinct hash values, multiplicity/collisions) have on the repacking time/size. > A more modest improvement is available in the open source fluentui repo > [1] with these results: > > Test this tree > -------------------------------------------------- > 5314.1: paths at head 19.5K > 5314.2: distinct hash value: v1 8.2K > 5314.3: maximum multiplicity: v1 279 > 5314.4: distinct hash value: v2 17.8K > 5314.5: maximum multiplicity: v2 44 > 5314.6: distinct hash value: v3 19.5K > 5314.7: maximum multiplicity: v3 1 > > [1] https://github.com/microsoft/fluentui > > However, it is important to demonstrate the effectiveness of this > function in the context of compressing a repository. We can use > p5313-pack-objects.sh to measure these changes. I will use a simplified > table summarizing the output of that performance test. > > | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | > |-----------|---------|---------|---------|---------|---------|---------| > | Thin Pack | 0.37 s | 0.12 s | 0.07 s | 1.2 M | 22.0 K | 20.4 K | > | Big Pack | 2.04 s | 2.80 s | 1.40 s | 20.4 M | 25.9 M | 19.2 M | > | Shallow | 1.41 s | 1.77 s | 1.27 s | 34.4 M | 33.7 M | 34.8 M | > | Repack | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M | OK, now we get to the chart that I demonstrates the effects of each hash function on the most externally visible effects. Are these measurements taken from the fluentui repo, or somewhere else? In either case, it may be worth mentioning. > Here, there are some performance improvements on a time basis, and the > thin and big packs are somewhat smaller in v3. The shallow and repacked > packs are somewhat bigger, though, compared to v2. > > Two repositories that have very few collisions in the v1 name hash are > the Git and Linux repositories. Here are their stats for p5313: > > Git: > > | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | > |-----------|---------|---------|---------|---------|---------|---------| > | Thin Pack | 0.02 s | 0.02 s | 0.02 s | 1.1 K | 1.1 K | 15.3 K | > | Big Pack | 1.69 s | 1.95 s | 1.67 s | 13.5 M | 14.5 M | 14.9 M | > | Shallow | 1.26 s | 1.29 s | 1.16 s | 12.0 M | 12.2 M | 12.5 M | > | Repack | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M | > > Linux: > > | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | > |-----------|----------|----------|----------|---------|---------|---------| > | Thin Pack | 0.17 s | 0.07 s | 0.07 s | 4.6 K | 4.6 K | 6.8 K | > | Big Pack | 17.88 s | 12.35 s | 12.14 s | 201.1 M | 149.1 M | 160.4 M | > | Shallow | 11.05 s | 22.94 s | 22.16 s | 269.2 M | 273.8 M | 271.8 M | > | Repack | 727.39 s | 566.95 s | 539.33 s | 2.5 G | 2.5 G | 2.6 G | > > These repositories make good use of the cross-path deltas that come > about from the v1 name hash function, so they already had mixed results > with the v2 function. The v3 function is generally worse for these > repositories. I appreciate you sharing some counterexamples as well. > While the fluentui repo had an increase in size using the v3 name hash, > the others had modest improvements over the v2 name hash. But those > modest improvements are dwarfed by the difference from v1 to v2, so it > is unlikely that the regression seen in the other scenarios (packfiles > that are not from full repacks) will be worth using v3 over v2. That is, > unless there are enough collisions even with v2 that the full repack > scenario has larger improvements than these. This is the paragraph that I thought most about (both while reading the above sections, and then again after seeing my internal thoughts written down here). It seems like the general conclusion is that v2 is a strict improvement on v1 in almost all cases. v3 appears to be an improvement on v2 in some cases, and a regression (as you note) in others. But I think more importantly (again as you note) is that the improvement from v1 to v2 is so pronounced that it's unlikely that the regression from v2 to v3 will matter or even be noticeable in most cases. Are there easy ways to detect when v3 would be an improvement over v2? If so, then I think exposing those detection mechanisms to users (either as an automated tool or through documentation, perhaps in git-packing(7), which is perfect for this sort of discussion) would be worthwhile. Then users could make an informed decision about which hash function to use for their repositories. But if there isn't such a mechanism, then I wonder what would drive a user to choose v3 over v2. I suspect the answer is that curious users would try repacking both ways, and then stick with whichever one has a bigger impact on the metric(s) they care most about. If that's the case, I suspect that v2 will be the dominant choice, especially if we consider changing the default from 1 to 2 at some point in the future. Given all of that, I share your feeling that it may be worth dropping this patch entirely. It is true that some cases will be worse off (at least compared to v2) without this part of the series. But it gets us out of having to support v3 forever, or go through the process of deprecating it. I'd like the project to avoid both of those if possible, especially if we don't anticipate many users will select v3 over v2. Thanks, Taylor

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 1/22/25 5:37 PM, Taylor Blau wrote: > On Fri, Dec 20, 2024 at 05:19:54PM +0000, Derrick Stolee via GitGitGadget wrote: >> Create a third name hash function and extend the '--name-hash-version' >> option in 'git pack-objects' and 'git repack' to understand it. This >> hash version abandons all efforts for locality and focuses on creating a >> somewhat uniformly-distributed hash function to minimize collisions. >> >> We can observe the effect of this collision avoidance in a large >> internal monorepo that suffered from collisions in the previous >> versions. The updates to p5314-name-hash.sh show these results: >> >> Test this tree >> -------------------------------------------------- >> 5314.1: paths at head 227.3K >> 5314.2: distinct hash value: v1 72.3K >> 5314.3: maximum multiplicity: v1 14.4K >> 5314.4: distinct hash value: v2 166.5K >> 5314.5: maximum multiplicity: v2 138 >> 5314.6: distinct hash value: v3 227.3K >> 5314.7: maximum multiplicity: v3 2 >> >> These results demonstrate that of the 227,000+ paths, nearly all of them >> find distinct hash values. The maximum multiplicity is 2, improved from >> 138 in the v2 hash function. The v2 hash function also had only 166K >> distinct values, so it had a wide spread of collisions. > > I had a little trouble reading this section of the commit message. I > think the framing makes sense (v2 has collisions which can impact pack > generation time and/or size), but this section explains v3 I think one > level too deep. > > This comparison (and the one below it for v3) shows a reduction in > distinct hash values and the maximum multiplicity (I'm assuming for > colliding hash values, in which case I might suggest renaming it as > "maximum collisions"). > > But I imagine that many readers will primarily care about the effect of > the new hash function on pack generation time and size. You show that > below, but I think that it should potentially appear earlier in the > commit message. > > Alternatively, you could consider leaving the time/size table alone > where it is, and devote an extra sentence or two to explaining the > impact on repacking time/size that the two metrics above (distinct hash > values, multiplicity/collisions) have on the repacking time/size. > >> A more modest improvement is available in the open source fluentui repo >> [1] with these results: >> >> Test this tree >> -------------------------------------------------- >> 5314.1: paths at head 19.5K >> 5314.2: distinct hash value: v1 8.2K >> 5314.3: maximum multiplicity: v1 279 >> 5314.4: distinct hash value: v2 17.8K >> 5314.5: maximum multiplicity: v2 44 >> 5314.6: distinct hash value: v3 19.5K >> 5314.7: maximum multiplicity: v3 1 >> >> [1] https://github.com/microsoft/fluentui >> >> However, it is important to demonstrate the effectiveness of this >> function in the context of compressing a repository. We can use >> p5313-pack-objects.sh to measure these changes. I will use a simplified >> table summarizing the output of that performance test. >> >> | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | >> |-----------|---------|---------|---------|---------|---------|---------| >> | Thin Pack | 0.37 s | 0.12 s | 0.07 s | 1.2 M | 22.0 K | 20.4 K | >> | Big Pack | 2.04 s | 2.80 s | 1.40 s | 20.4 M | 25.9 M | 19.2 M | >> | Shallow | 1.41 s | 1.77 s | 1.27 s | 34.4 M | 33.7 M | 34.8 M | >> | Repack | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M | > > OK, now we get to the chart that I demonstrates the effects of each hash > function on the most externally visible effects. Are these measurements > taken from the fluentui repo, or somewhere else? In either case, it > may be worth mentioning. > >> Here, there are some performance improvements on a time basis, and the >> thin and big packs are somewhat smaller in v3. The shallow and repacked >> packs are somewhat bigger, though, compared to v2. >> >> Two repositories that have very few collisions in the v1 name hash are >> the Git and Linux repositories. Here are their stats for p5313: >> >> Git: >> >> | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | >> |-----------|---------|---------|---------|---------|---------|---------| >> | Thin Pack | 0.02 s | 0.02 s | 0.02 s | 1.1 K | 1.1 K | 15.3 K | >> | Big Pack | 1.69 s | 1.95 s | 1.67 s | 13.5 M | 14.5 M | 14.9 M | >> | Shallow | 1.26 s | 1.29 s | 1.16 s | 12.0 M | 12.2 M | 12.5 M | >> | Repack | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M | >> >> Linux: >> >> | Test | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size | >> |-----------|----------|----------|----------|---------|---------|---------| >> | Thin Pack | 0.17 s | 0.07 s | 0.07 s | 4.6 K | 4.6 K | 6.8 K | >> | Big Pack | 17.88 s | 12.35 s | 12.14 s | 201.1 M | 149.1 M | 160.4 M | >> | Shallow | 11.05 s | 22.94 s | 22.16 s | 269.2 M | 273.8 M | 271.8 M | >> | Repack | 727.39 s | 566.95 s | 539.33 s | 2.5 G | 2.5 G | 2.6 G | >> >> These repositories make good use of the cross-path deltas that come >> about from the v1 name hash function, so they already had mixed results >> with the v2 function. The v3 function is generally worse for these >> repositories. > > I appreciate you sharing some counterexamples as well. > >> While the fluentui repo had an increase in size using the v3 name hash, >> the others had modest improvements over the v2 name hash. But those >> modest improvements are dwarfed by the difference from v1 to v2, so it >> is unlikely that the regression seen in the other scenarios (packfiles >> that are not from full repacks) will be worth using v3 over v2. That is, >> unless there are enough collisions even with v2 that the full repack >> scenario has larger improvements than these. > > This is the paragraph that I thought most about (both while reading the > above sections, and then again after seeing my internal thoughts written > down here). > > It seems like the general conclusion is that v2 is a strict improvement > on v1 in almost all cases. v3 appears to be an improvement on v2 in some > cases, and a regression (as you note) in others. But I think more > importantly (again as you note) is that the improvement from v1 to v2 is > so pronounced that it's unlikely that the regression from v2 to v3 will > matter or even be noticeable in most cases. > > Are there easy ways to detect when v3 would be an improvement over v2? > If so, then I think exposing those detection mechanisms to users (either > as an automated tool or through documentation, perhaps in > git-packing(7), which is perfect for this sort of discussion) would be > worthwhile. Then users could make an informed decision about which hash > function to use for their repositories. > > But if there isn't such a mechanism, then I wonder what would drive a > user to choose v3 over v2. I suspect the answer is that curious users > would try repacking both ways, and then stick with whichever one has a > bigger impact on the metric(s) they care most about. > > If that's the case, I suspect that v2 will be the dominant choice, > especially if we consider changing the default from 1 to 2 at some point > in the future. Given all of that, I share your feeling that it may be > worth dropping this patch entirely. It is true that some cases will be > worse off (at least compared to v2) without this part of the series. But > it gets us out of having to support v3 forever, or go through the > process of deprecating it. I'd like the project to avoid both of those > if possible, especially if we don't anticipate many users will select v3 > over v2. Thank you for these detailed considerations. The most important one, in my opinion is this: > Are there easy ways to detect when v3 would be an improvement over v2? > If so, then I think exposing those detection mechanisms to users > ...would be worthwhile. I agree that having those detection mechanisms would be good. The test helpers can provide some of the information that helps make that decision, but doesn't form opinions or recommend thresholds for one over another. I agree with your overall thought that we should eject this patch (for now) and focus on the v2 as something that will help most users. We can learn from that and use that to inform any future iterations built on this framework. Thanks, -Stolee

gitgitgadget · 2024-11-21T20:20:12Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote: > From: Derrick Stolee <[email protected]> > > This also adds the '--full-name-hash' option introduced in the previous > change and adds newlines to the synopsis. I think "the previous change" is not quite accurate here, even if you move the implementation to pass through '--full-name-hash' via repack into the second patch. It would be nice to have the option added in 'repack' in the same commit as adjusts the documentation instead of splitting them apart. Thanks, Taylor

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 11/21/24 3:17 PM, Taylor Blau wrote: > On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote: >> From: Derrick Stolee <[email protected]> >> >> This also adds the '--full-name-hash' option introduced in the previous >> change and adds newlines to the synopsis. > > I think "the previous change" is not quite accurate here, even if > you move the implementation to pass through '--full-name-hash' via > repack into the second patch. Ah, I should definitely rearrange the commits. > It would be nice to have the option added in 'repack' in the same commit > as adjusts the documentation instead of splitting them apart. Part of the point of the split was that the synopsis in builtin/repack.c needs more than just the addition of the --full-name-hash option in order to make it match the Documentation synopsis. But you're right, the code change is small enough that these things can be combined. Thanks, -Stolee

gitgitgadget · 2024-12-04T21:18:19Z

On the Git mailing list, karthik nayak wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: [snip] > DESCRIPTION > ----------- > @@ -249,6 +251,36 @@ linkgit:git-multi-pack-index[1]). > Write a multi-pack index (see linkgit:git-multi-pack-index[1]) > containing the non-redundant packs. > > +--name-hash-version=<n>:: > + While performing delta compression, Git groups objects that may be > + similar based on heuristics using the path to that object. While > + grouping objects by an exact path match is good for paths with > + many versions, there are benefits for finding delta pairs across > + different full paths. Git collects objects by type and then by a > + "name hash" of the path and then by size, hoping to group objects > + that will compress well together. > ++ > +The default name hash version is `1`, which prioritizes hash locality by > +considering the final bytes of the path as providing the maximum magnitude > +to the hash function. This version excels at distinguishing short paths > +and finding renames across directories. However, the hash function depends > +primarily on the final 16 bytes of the path. If there are many paths in > +the repo that have the same final 16 bytes and differ only by parent > +directory, then this name-hash may lead to too many collisions and cause > +poor results. At the moment, this version is required when writing > +reachability bitmap files with `--write-bitmap-index`. > ++ > +The name hash version `2` has similar locality features as version `1`, > +except it considers each path component separately and overlays the hashes > +with a shift. This still prioritizes the final bytes of the path, but also > +"salts" the lower bits of the hash using the parent directory names. This > +method allows for some of the locality benefits of version `1` while > +breaking most of the collisions from a similarly-named file appearing in > +many different directories. At the moment, this version is not allowed > +when writing reachability bitmap files with `--write-bitmap-index` and it > +will be automatically changed to version `1`. > + > + Nit: I wonder if it'd be nicer to simply point to the documentation in 'Documentation/git-pack-objects.txt'. This would ensure we have consistent documentation and a single source of truth. > > CONFIGURATION > ------------- > > diff --git a/builtin/repack.c b/builtin/repack.c > index 05e13adb87f..5e7ff919c1a 100644 > --- a/builtin/repack.c > +++ b/builtin/repack.c > @@ -39,7 +39,9 @@ static int run_update_server_info = 1; > static char *packdir, *packtmp_name, *packtmp; > > static const char *const git_repack_usage[] = { > - N_("git repack [<options>]"), > + N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n" > + "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n" > + "[--write-midx] [--name-hash-version=<n>]"), > NULL > }; So this fixes the mismatch in t0450 which is seen below. Nit: might be worthwhile adding this in the commit message. [snip]

gitgitgadget · 2025-01-22T22:20:32Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:49PM +0000, Derrick Stolee via GitGitGadget wrote: > From: Derrick Stolee <[email protected]> > > The new '--name-hash-version' option for 'git repack' is a simple > pass-through to the underlying 'git pack-objects' subcommand. However, > this subcommand may have other options and a temporary filename as part > of the subcommand execution that may not be predictable or could change > over time. > > The existing test_subcommand method requires an exact list of arguments > for the subcommand. This is too rigid for our needs here, so create a > new method, test_subcommand_flex. Use it to check that the > --name-hash-version option is passing through. > > Since we are modifying the 'git repack' command, let's bring its usage > in line with the Documentation's synopsis. This removes it from the > allow list in t0450 so it will remain in sync in the future. > > Signed-off-by: Derrick Stolee <[email protected]> > --- > Documentation/git-repack.txt | 9 ++++++++- > builtin/repack.c | 9 ++++++++- > t/t0450/txt-help-mismatches | 1 - > t/t7700-repack.sh | 6 ++++++ > t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++ > 5 files changed, 48 insertions(+), 3 deletions(-) > > diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt > index c902512a9e8..5852a5c9736 100644 > --- a/Documentation/git-repack.txt > +++ b/Documentation/git-repack.txt > @@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository > SYNOPSIS > -------- > [verse] > -'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx] > +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] > + [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] > + [--write-midx] [--name-hash-version=<n>] I probably would have split this change into two separate patches (one to adjust the line wrapping in the synopsis, and another to introduce the new option). But regardless, I am really glad to see this change, since the long synopsis line has always bothered me, but I never got around to changing it. Thanks for deciding that enough is enough! > diff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches > index 28003f18c92..c4a15fd0cb8 100644 > --- a/t/t0450/txt-help-mismatches > +++ b/t/t0450/txt-help-mismatches > @@ -45,7 +45,6 @@ rebase > remote > remote-ext > remote-fd > -repack :-). Thanks, Taylor

gitgitgadget · 2024-11-21T20:44:24Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Tue, Nov 05, 2024 at 03:05:07AM +0000, Derrick Stolee via GitGitGadget wrote: > Test this tree > ----------------------------------------------------------------- > 5314.1: paths at head 4.5K > 5314.2: number of distinct name-hashes 4.1K > 5314.3: number of distinct full-name-hashes 4.5K > 5314.4: maximum multiplicity of name-hashes 13 > 5314.5: maximum multiplicity of fullname-hashes 1 > > Here, the maximum collision multiplicity is 13, but around 10% of paths > have a collision with another path. Neat. > diff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c > new file mode 100644 > index 00000000000..e4ecd159b76 > --- /dev/null > +++ b/t/helper/test-name-hash.c > @@ -0,0 +1,24 @@ > +/* > + * test-name-hash.c: Read a list of paths over stdin and report on their > + * name-hash and full name-hash. > + */ > + > +#include "test-tool.h" > +#include "git-compat-util.h" > +#include "pack-objects.h" > +#include "strbuf.h" > + > +int cmd__name_hash(int argc UNUSED, const char **argv UNUSED) > +{ > + struct strbuf line = STRBUF_INIT; > + > + while (!strbuf_getline(&line, stdin)) { > + uint32_t name_hash = pack_name_hash(line.buf); > + uint32_t full_hash = pack_full_name_hash(line.buf); > + > + printf("%10"PRIu32"\t%10"PRIu32"\t%s\n", name_hash, full_hash, line.buf); I'm definitely nitpicking, but having a tab to separate these two 32-bit values feels odd when we know already that they will be at most 10-characters wide. I probably would have written: printf("%10"PRIu32" %10"PRIu32"\t%s\n", name_hash, full_hash, line.buf); instead, but this is obviously not a big deal either way ;-). > diff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh > new file mode 100755 > index 00000000000..9fe26612fac > --- /dev/null > +++ b/t/perf/p5314-name-hash.sh > @@ -0,0 +1,41 @@ > +#!/bin/sh > + > +test_description='Tests pack performance using bitmaps' > +. ./perf-lib.sh > + > +GIT_TEST_PASSING_SANITIZE_LEAK=0 > +export GIT_TEST_PASSING_SANITIZE_LEAK Does this conflict with Patrick's series to remove these leak checking annotations? I think it might, which is not unexpected given this series was written before that one (and it's my fault for not reviewing it earlier). > +test_perf_large_repo > + > +test_size 'paths at head' ' > + git ls-tree -r --name-only HEAD >path-list && > + wc -l <path-list > +' > + > +test_size 'number of distinct name-hashes' ' > + cat path-list | test-tool name-hash >name-hashes && > + cat name-hashes | awk "{ print \$1; }" | sort -n | uniq -c >name-hash-count && In these two (and a handful of others lower down in this same script) the "cat ... |" is unnecessary. I think this one should be written as: test-tool name-hash <path-list >name-hashes && awk "{ print \$1; }" <name-hashes | sort | uniq -c >name-hash-count && (sort -n is unnecessary, since we just care about getting the list in sorted order so that "uniq -c" can count the number of unique values). > + wc -l <name-hash-count > +' > + > +test_size 'number of distinct full-name-hashes' ' > + cat name-hashes | awk "{ print \$2; }" | sort -n | uniq -c >full-name-hash-count && > + wc -l <full-name-hash-count > +' > + > +test_size 'maximum multiplicity of name-hashes' ' > + cat name-hash-count | \ > + sort -nr | \ > + head -n 1 | \ > + awk "{ print \$1; }" > +' > + > +test_size 'maximum multiplicity of fullname-hashes' ' > + cat full-name-hash-count | \ > + sort -nr | \ > + head -n 1 | \ > + awk "{ print \$1; }" Nitpicking again, but you could extract the "sort | head | awk" pipeline into a function. Thanks, Taylor

gitgitgadget · 2024-11-22T01:27:26Z

On the Git mailing list, Jonathan Tan wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: > From: Derrick Stolee <[email protected]> > > Add a new test-tool helper, name-hash, to output the value of the > name-hash algorithms for the input list of strings, one per line. I've looked at all 7 patches. I didn't really understand the concern with shallow in patch 6 (in particular, the documentation of "git pack-objects --shallow" seems to imply that it's for use by a server to a shallow client, but at the point that the server would need such a feature, it probably would already have bitmaps packed with the new hash algorithm). I didn't look at it further, though, since I had an algorithm that seemed to also do OK in the shallow test. So we might be able to drop patch 6. Other than that, and other than all my comments and Taylor's comments, this series looks good.

gitgitgadget · 2024-11-21T20:20:10Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote: > diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh > index 2e28d02b20f..75b40f07bbd 100755 > --- a/ci/run-build-and-tests.sh > +++ b/ci/run-build-and-tests.sh > @@ -30,6 +30,7 @@ linux-TEST-vars) > export GIT_TEST_NO_WRITE_REV_INDEX=1 > export GIT_TEST_CHECKOUT_WORKERS=2 > export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1 > + export GIT_TEST_FULL_NAME_HASH=1 > ;; > linux-clang) > export GIT_TEST_DEFAULT_HASH=sha1 Hmm. I appreciate what this new GIT_TEST_ variable is trying to do, but I am somewhat saddened to see this list in linux-TEST-vars growing rather than shrinking. I'm most definitely part of the problem here, but I think too often we add new entries to this list and let them languish without ever removing them after they have served their intended purpose. So I think the question is: what do we hope to get out of running the test suite in a mode where we use the full-name hash all of the time? I can't imagine any interesting breakage (other than individual tests' sensitivity to specific delta/base pairs) that would be caught by merely changing the hash function here. I dunno. Maybe there is some exotic behavior that this shook out for you during development which I'm not aware of. If that were the case, I think that keeping this variable around makes sense, since the appearance of that exotic behavior proves that the variable is useful at shaking out bugs. But assuming not, I think that I would just as soon avoid this test variable entirely, which I think in this case amounts to dropping this patch from the series. Thanks, Taylor

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 11/21/24 3:15 PM, Taylor Blau wrote: > On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote: >> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh >> index 2e28d02b20f..75b40f07bbd 100755 >> --- a/ci/run-build-and-tests.sh >> +++ b/ci/run-build-and-tests.sh >> @@ -30,6 +30,7 @@ linux-TEST-vars) >> export GIT_TEST_NO_WRITE_REV_INDEX=1 >> export GIT_TEST_CHECKOUT_WORKERS=2 >> export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1 >> + export GIT_TEST_FULL_NAME_HASH=1 >> ;; >> linux-clang) >> export GIT_TEST_DEFAULT_HASH=sha1 > > Hmm. I appreciate what this new GIT_TEST_ variable is trying to do, but > I am somewhat saddened to see this list in linux-TEST-vars growing > rather than shrinking. You make good points that this does not need to be here. It's enough that someone could manually check the test suite with this test variable to make sure that enough of the other options are tested with this feature. Thanks, -Stolee

gitgitgadget · 2024-11-22T01:15:22Z

On the Git mailing list, Jonathan Tan wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: > Second, there are two tests in t5616-partial-clone.sh that I believe are > actually broken scenarios. I took a look...this is a tricky one. > While the client is set up to clone the > 'promisor-server' repo via a treeless partial clone filter (tree:0), > that filter does not translate to the 'server' repo. Thus, fetching from > these repos causes the server to think that the client has all reachable > trees and blobs from the commits advertised as 'haves'. This leads the > server to providing a thin pack assuming those objects as delta bases. It is expected that the server sometimes sends deltas based on objects that the client doesn't have. In fact, this test tests the ability of Git to lazy-fetch delta bases. > Changing the name-hash algorithm presents new delta bases and thus > breaks the expectations of these tests. To be precise, the change resulted in no deltas being sent (before this change, one delta was sent). Here's what is meant to happen. The server has: commitB - treeB - file1 ("make the tree big\nanother line\n"), file2...file100 | commitA - treeA - file1...file100 ("make the tree big\n") The client only has commitA. (The client does not have treeA or any blob, since it was cloned with --filter=tree:0.) When GIT_TEST_FULL_NAME_HASH=0 (matching the current behavior), the server sends a non-delta commitB, a delta treeB (with base treeA), and a non-delta blob "make the tree big\nanother line\n". This triggers a lazy fetch of treeA, and thus treeB is inflated successfully. During the subsequent connectivity check (with --exclude-promisor-objects, see connected.c), it is noticed that the "make the tree big\n" blob is missing, but since it is a promisor object (referenced by treeA, which was fetched from the promisor remote), the connectivity check since passes. When GIT_TEST_FULL_NAME_HASH=1, the server sends a non-delta commitB, a non-delta treeB, and a non-delta blob "make the tree big\nanother line\n". No lazy fetch is triggered. During the subsequent connectivity check, the "make the tree big\n" blob (referenced by treeB) is missing. There is nothing that can vouch for it (the client does not have treeA, remember) so the client does not consider it a promisor object, and thus the connectivity check fails. Investigating this was made a bit harder due to a missing "git -C promisor-remote config --local uploadpack.allowfilter 1" in the test. The above behavior is after this is included in the test. I think the solution is to have an algorithm that preserves the property that treeB is sent as a delta object - if not, we need to find another way to test the lazy-fetch of delta bases. My proposal in [1] does do that. [1] https://lore.kernel.org/git/[email protected]/

On the Git mailing list, Junio C Hamano wrote (reply to this):

Jonathan Tan <[email protected]> writes: > ... During the subsequent connectivity > check, the "make the tree big\n" blob (referenced by treeB) is missing. > There is nothing that can vouch for it (the client does not have treeA, > remember) so the client does not consider it a promisor object, and thus > the connectivity check fails. It is sad that it is a (probably unfixable) flaw in the "promisor object" concept that the "promisor object"-ness of blobA depends on the lazy-fetch status of treeA. This is not merely a test failure, but it would cause blobA pruned if such a lazy fetch happens in the wild and then "git gc" triggers, no? It may not manifest as a repository corruption, since we would lazily fetch it again if the user requests to fully fetch what commitA and treeA need, but it does feel somewhat suboptimal. Thanks for a detailed explanation on what is going on.

On the Git mailing list, Jonathan Tan wrote (reply to this):

Junio C Hamano <[email protected]> writes: > It is sad that it is a (probably unfixable) flaw in the "promisor > object" concept that the "promisor object"-ness of blobA depends on > the lazy-fetch status of treeA. This is not merely a test failure, > but it would cause blobA pruned if such a lazy fetch happens in the > wild and then "git gc" triggers, no? Right now, it won't be pruned since we never prune promisor objects (we just concatenate all of them into one file). But in the future, we might only keep reachable promisor objects, in which case, yes, blobA will be pruned. In this case, though, I think blobA is like any other unreachable object in git. If a user memorizes a commit hash but does not point a ref to it (or point a ref to one of its descendants), that commit is still subject to being lost by GC. I think it's the same case here.

On the Git mailing list, Junio C Hamano wrote (reply to this):

Jonathan Tan <[email protected]> writes: > Junio C Hamano <[email protected]> writes: >> It is sad that it is a (probably unfixable) flaw in the "promisor >> object" concept that the "promisor object"-ness of blobA depends on >> the lazy-fetch status of treeA. This is not merely a test failure, >> but it would cause blobA pruned if such a lazy fetch happens in the >> wild and then "git gc" triggers, no? > > Right now, it won't be pruned since we never prune promisor objects > (we just concatenate all of them into one file). Sorry, but I am lost. In the scenario discussed, you have two commits A and B with their trees and blobs. You initially only have commit A because the partial clone is done with "tree:0". Then you fetch commit B (A's child), tree B in non-delta form, and blob B contained within tree B. Due to the tweak in the name hash function, we do not know of tree A (we used to learn about it because tree B was sent as a delta against it with the old name hash). If blob B was sent as a delta against blob A, lazy fetch would later materialize blob A even if you do not still have tree A, no? I thought the story was that we would not know who refers to blobA when treeA hasn't been lazily fetched, hence we cannot tell if blobA is a "promisor object" to begin with, no? The blob in such a scenario may be reclaimed by GC and we may still be able to refetch from the promisor, so it may not be the end of the world, but feels suboptimal.

On the Git mailing list, Jonathan Tan wrote (reply to this):

Junio C Hamano <[email protected]> writes: > Jonathan Tan <[email protected]> writes: > > > Junio C Hamano <[email protected]> writes: > >> It is sad that it is a (probably unfixable) flaw in the "promisor > >> object" concept that the "promisor object"-ness of blobA depends on > >> the lazy-fetch status of treeA. This is not merely a test failure, > >> but it would cause blobA pruned if such a lazy fetch happens in the > >> wild and then "git gc" triggers, no? > > > > Right now, it won't be pruned since we never prune promisor objects > > (we just concatenate all of them into one file). > > Sorry, but I am lost. In the scenario discussed, you have two > commits A and B with their trees and blobs. You initially only have > commit A because the partial clone is done with "tree:0". Then you > fetch commit B (A's child), tree B in non-delta form, and blob B > contained within tree B. Due to the tweak in the name hash > function, we do not know of tree A (we used to learn about it > because tree B was sent as a delta against it with the old name > hash). Yes, that's correct. > If blob B was sent as a delta against blob A, lazy fetch > would later materialize blob A even if you do not still have tree A, > no? Just to be clear, this is not happening right now (blob B is sent whole, not as a delta). But let's suppose that blob B was sent as a delta, then yes, the lazy fetch would materialize blob A... > I thought the story was that we would not know who refers to blobA > when treeA hasn't been lazily fetched, hence we cannot tell if blobA > is a "promisor object" to begin with, no? ...ah, in this case, blob A vouches for itself. Whenever we lazy fetch, all objects that are fetched go into promisor packs (packfiles with an associated .promisor file), so we know that they are promisor objects.

On the Git mailing list, Junio C Hamano wrote (reply to this):

Jonathan Tan <[email protected]> writes: > ...ah, in this case, blob A vouches for itself. Whenever we lazy fetch, > all objects that are fetched go into promisor packs (packfiles with an > associated .promisor file), so we know that they are promisor objects. Thanks.

gitgitgadget · 2024-12-04T21:24:03Z

On the Git mailing list, karthik nayak wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: > From: Derrick Stolee <[email protected]> > > Add a new environment variable to opt-in to differen values of the s/differen/different > --name-hash-version=<n> option in 'git pack-objects'. This allows for > extra testing of the feature without repeating all of the test > scenarios. Unlike many GIT_TEST_* variables, we are choosing to not add > this to the linux-TEST-vars CI build as that test run is already > overloaded. The behavior exposed by this test variable is of low risk > and should be sufficient to allow manual testing when an issue arises. > > But this option isn't free. There are a few tests that change behavior > with the variable enabled. > > First, there are a few tests that are very sensitive to certain delta > bases being picked. These are both involving the generation of thin > bundles and then counting their objects via 'git index-pack --fix-thin' > which pulls the delta base into the new packfile. For these tests, > disable the option as a decent long-term option. > > Second, there are two tests in t5616-partial-clone.sh that I believe are > actually broken scenarios. While the client is set up to clone the > 'promisor-server' repo via a treeless partial clone filter (tree:0), > that filter does not translate to the 'server' repo. Thus, fetching from > these repos causes the server to think that the client has all reachable > trees and blobs from the commits advertised as 'haves'. This leads the > server to providing a thin pack assuming those objects as delta bases. > Changing the name-hash algorithm presents new delta bases and thus > breaks the expectations of these tests. An alternative could be to set > up 'server' as a promisor server with the correct filter enabled. This > may also point out more issues with partial clone being set up as a > remote-based filtering mechanism and not a repository-wide setting. For > now, do the minimal change to make the test work by disabling the test > variable. > > Third, there are some tests that compare the exact output of a 'git > pack-objects' process when using bitmaps. The warning that ignores the > --name-hash-version=2 and forces version 1 causes these tests to fail. > Disable the environment variable to get around this issue. > The patch itself looked good to me. [snip]

gitgitgadget · 2024-12-09T23:15:18Z

On the Git mailing list, Jonathan Tan wrote (reply to this):

"Derrick Stolee via GitGitGadget" <[email protected]> writes: > t/t5616-partial-clone.sh | 26 ++++++++++++++++++++++++-- I believe the changes to this file are no longer needed.

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 12/9/24 6:12 PM, Jonathan Tan wrote: > "Derrick Stolee via GitGitGadget" <[email protected]> writes: >> t/t5616-partial-clone.sh | 26 ++++++++++++++++++++++++-- > > I believe the changes to this file are no longer needed. You're right. They are only needed in the last patch when the v3 name-hash is possible. (Unless maybe you fixed this issue in 'master' and I didn't see it) Thanks, -Stolee

gitgitgadget · 2025-01-22T22:23:29Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:50PM +0000, Derrick Stolee via GitGitGadget wrote: > @@ -209,6 +209,10 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' ' > ' > > test_expect_success 'out of order thresholds are rejected' ' > + # Disable this option to avoid stderr message > + GIT_TEST_NAME_HASH_VERSION=1 && > + export GIT_TEST_NAME_HASH_VERSION && > + > test_must_fail git \ > -c bitmapPseudoMerge.test.pattern="refs/*" \ > -c bitmapPseudoMerge.test.threshold=1.month.ago \ This is the only one that sets GIT_TEST_NAME_HASH_VERSION via an export. I suspect that this is to get around calling the shell function with a single-shot environment variable. But I think our convention for this is test_must_fail env GIT_TEST_NAME_HASH_VERSION=1 git ... Probably not a big deal, but I figured I'd mention it regardless in case you happen to reroll. Thanks, Taylor

gitgitgadget · 2025-01-22T22:23:32Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:53PM +0000, Derrick Stolee via GitGitGadget wrote: > --- > builtin/pack-objects.c | 8 ++++++++ > 1 file changed, 8 insertions(+) This is a very nice guard in my opinion, thanks for adding it! Thanks, Taylor

gitgitgadget · 2024-12-04T20:09:34Z

On the Git mailing list, karthik nayak wrote (reply to this):

"Jonathan Tan via GitGitGadget" <[email protected]> writes: [snip] > diff --git a/pack-objects.h b/pack-objects.h > index b9898a4e64b..15be8368d21 100644 > --- a/pack-objects.h > +++ b/pack-objects.h > @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name) > return hash; > } > > +static inline uint32_t pack_name_hash_v2(const char *name) > +{ > + uint32_t hash = 0, base = 0, c; > + > + if (!name) > + return 0; > + > + while ((c = *name++)) { > + if (isspace(c)) > + continue; > + if (c == '/') { > + base = (base >> 6) ^ hash; Just two questions about the implementation for my own knowledge. 1. Why use '6' here? I couldn't understand the reasoning for the specific value. 2. Generally in hashing algorithms the XOR is used to ensure that the output distribution is uniform which reduces collisions. Here, as you noted, we're more finding values for sorting rather than hashing in the traditional sense. So why use an XOR? > + hash = 0; > + } else { > + /* > + * 'c' is only a single byte. Reverse it and move > + * it to the top of the hash, moving the rest to > + * less-significant bits. > + */ > + c = (c & 0xF0) >> 4 | (c & 0x0F) << 4; > + c = (c & 0xCC) >> 2 | (c & 0x33) << 2; > + c = (c & 0xAA) >> 1 | (c & 0x55) << 1; > + hash = (hash >> 2) + (c << 24); > + } > + } > + return (base >> 6) ^ hash; > +} > + > static inline enum object_type oe_type(const struct object_entry *e) > { > return e->type_valid ? e->type_ : OBJ_BAD; > -- > gitgitgadget

On the Git mailing list, Junio C Hamano wrote (reply to this):

karthik nayak <[email protected]> writes: > 2. Generally in hashing algorithms the XOR is used to ensure that the > output distribution is uniform which reduces collisions. Here, as you > noted, we're more finding values for sorting rather than hashing in the > traditional sense. So why use an XOR? I am not Jonathan, but since the mixing-of-bits is done with XOR in the original that Linus and I wrote, I think the question applies to our version as well. We prefer not to lose entropy from input bytes, so we do not want to use OR or AND, which tend to paint everything with 1 or 0 as you mix in bits from more bytes. Anyway the question sounds like "generally when you take tests you write with a pencil, but right now you are merely taking notes and not taking any tests. why are you writing with a pencil?" There are multiple occasions that a pencil is a great fit as writing equipment.

On the Git mailing list, karthik nayak wrote (reply to this):

Junio C Hamano <[email protected]> writes: > karthik nayak <[email protected]> writes: > >> 2. Generally in hashing algorithms the XOR is used to ensure that the >> output distribution is uniform which reduces collisions. Here, as you >> noted, we're more finding values for sorting rather than hashing in the >> traditional sense. So why use an XOR? > > I am not Jonathan, but since the mixing-of-bits is done with XOR in > the original that Linus and I wrote, I think the question applies to > our version as well. We prefer not to lose entropy from input > bytes, so we do not want to use OR or AND, which tend to paint > everything with 1 or 0 as you mix in bits from more bytes. > Thanks for the answering. My reasoning at the start was more of my thoughts on why XOR could be used here and your explanation makes it clearer. > Anyway the question sounds like "generally when you take tests you > write with a pencil, but right now you are merely taking notes and > not taking any tests. why are you writing with a pencil?" There are > multiple occasions that a pencil is a great fit as writing equipment. I'd say my questions was more of "I know a pencil is used when taking tests because of XYZ reasons. On similar lines, why is a pencil used here?" :)

gitgitgadget · 2024-12-09T23:17:59Z

On the Git mailing list, Jonathan Tan wrote (reply to this):

"Jonathan Tan via GitGitGadget" <[email protected]> writes: > diff --git a/pack-objects.h b/pack-objects.h > index b9898a4e64b..15be8368d21 100644 > --- a/pack-objects.h > +++ b/pack-objects.h > @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name) > return hash; > } > > +static inline uint32_t pack_name_hash_v2(const char *name) > +{ > + uint32_t hash = 0, base = 0, c; > + > + if (!name) > + return 0; > + > + while ((c = *name++)) { > + if (isspace(c)) > + continue; > + if (c == '/') { > + base = (base >> 6) ^ hash; > + hash = 0; > + } else { > + /* > + * 'c' is only a single byte. Reverse it and move > + * it to the top of the hash, moving the rest to > + * less-significant bits. > + */ > + c = (c & 0xF0) >> 4 | (c & 0x0F) << 4; > + c = (c & 0xCC) >> 2 | (c & 0x33) << 2; > + c = (c & 0xAA) >> 1 | (c & 0x55) << 1; > + hash = (hash >> 2) + (c << 24); > + } > + } > + return (base >> 6) ^ hash; > +} This works because `c` is masked before any arithmetic is performed on it, but the cast from potentially signed char to uint32_t still makes me nervous - if char is signed, it behaves as if it was first cast to int32_t and only then to uint32_t, as you can see from running the code below: #include <stdio.h> int main() { signed char c = 128; unsigned int u = c; printf("hello %u\n", u); return 0; } I would declare `c` as uint8_t or unsigned char.

On the Git mailing list, Junio C Hamano wrote (reply to this):

Jonathan Tan <[email protected]> writes: > "Jonathan Tan via GitGitGadget" <[email protected]> writes: >> diff --git a/pack-objects.h b/pack-objects.h >> index b9898a4e64b..15be8368d21 100644 >> --- a/pack-objects.h >> +++ b/pack-objects.h >> @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name) >> return hash; >> } >> >> +static inline uint32_t pack_name_hash_v2(const char *name) >> +{ >> + uint32_t hash = 0, base = 0, c; >> + ... >> + while ((c = *name++)) { >> + ... >> + /* >> + * 'c' is only a single byte. Reverse it and move >> + * it to the top of the hash, moving the rest to >> + * less-significant bits. >> + */ >> + c = (c & 0xF0) >> 4 | (c & 0x0F) << 4; >> + ... > This works because `c` is masked before any arithmetic is performed on > it, but the cast from potentially signed char to uint32_t still makes > me nervous - if char is signed, it behaves as if it was first cast to > int32_t and only then to uint32_t, ... > I would declare `c` as uint8_t or unsigned char. I think you meant the type of "name", and your worry is that *name may pick up a negative integer from there when the platform char is signed? Here we are dealing with a pathname that has often UTF-8 encoded non-ASCII letters with 8th bit set, and I agree with you that being explicitly unsigned would certainly help reduce the cognitive load. Thanks.

gitgitgadget · 2025-01-22T22:11:42Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Fri, Dec 20, 2024 at 05:19:47PM +0000, Jonathan Tan via GitGitGadget wrote: > The first change is to be more careful about paths using non-ASCII > characters. With these characters in mind, reverse the bits in the byte > as the least-significant bits have the highest entropy and we want to > maximize their influence. This is done with some bit manipulation that > swaps the two halves, then the quarters within those halves, and then > the bits within those quarters. Makes sense, and seems quite reasonable. > The second change is to perform hash composition operations at every > level of the path. This is done by storing a 'base' hash value that > contains the hash of the parent directory. When reaching a directory > boundary, we XOR the current level's name-hash value with a downshift of > the previous level's hash. This perturbation intends to create low-bit > distinctions for paths with the same final 16 bytes but distinct parent > directory structures. Very clever, I love this idea. Thanks, Taylor

gitgitgadget · 2024-11-21T20:35:09Z

On the Git mailing list, Taylor Blau wrote (reply to this):

On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote: > From: Derrick Stolee <[email protected]> > > As custom options are added to 'git pack-objects' and 'git repack' to > adjust how compression is done, use this new performance test script to > demonstrate their effectiveness in performance and size. Nicely done, thank you for adding a perf test to allow readers to easily verify these changes themselves. > In the case of the Git repository, these numbers show some of the issues > with this approach: > > [...] > > The thin pack that simulates a push is much worse with --full-name-hash > in this case. The name hash values are doing a lot to assist with delta > bases, it seems. The big pack and shallow clone cases are slightly worse > with the --full-name-hash option. Only the full repack gains some > benefits in size. Not a problem with your patch, but just thinking aloud: do you think there is an easy/straightforward way to suggest when to use --full-name-hash or not? > --- > t/perf/p5313-pack-objects.sh | 94 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 94 insertions(+) > create mode 100755 t/perf/p5313-pack-objects.sh > > diff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh > new file mode 100755 > index 00000000000..dfa29695315 > --- /dev/null > +++ b/t/perf/p5313-pack-objects.sh > @@ -0,0 +1,94 @@ > +#!/bin/sh > + > +test_description='Tests pack performance using bitmaps' > +. ./perf-lib.sh > + > +GIT_TEST_PASSING_SANITIZE_LEAK=0 > +export GIT_TEST_PASSING_SANITIZE_LEAK > + > +test_perf_large_repo > + > +test_expect_success 'create rev input' ' > + cat >in-thin <<-EOF && > + $(git rev-parse HEAD) > + ^$(git rev-parse HEAD~1) > + EOF > + > + cat >in-big <<-EOF && > + $(git rev-parse HEAD) > + ^$(git rev-parse HEAD~1000) > + EOF > + > + cat >in-shallow <<-EOF > + $(git rev-parse HEAD) > + --shallow $(git rev-parse HEAD) > + EOF > +' I was going to comment that these could probably be moved into the individual perf test that cares about reading each of these inputs. But having them shared here makes sense since we are naturally comparing generating two packs with the same input (with and without --full-name-hash). So the shared setup here makes sense to me. > + > +test_perf 'thin pack' ' > + git pack-objects --thin --stdout --revs --sparse <in-thin >out > +' > + > +test_size 'thin pack size' ' > + test_file_size out > +' Nice. I always forget about this and end up writing 'wc -c <out'. > +test_perf 'thin pack with --full-name-hash' ' > + git pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out > +' > + > +test_size 'thin pack size with --full-name-hash' ' > + test_file_size out > +' > + > +test_perf 'big pack' ' > + git pack-objects --stdout --revs --sparse <in-big >out > +' > + > +test_size 'big pack size' ' > + test_file_size out > +' > + > +test_perf 'big pack with --full-name-hash' ' > + git pack-objects --stdout --revs --sparse --full-name-hash <in-big >out > +' > + > +test_size 'big pack size with --full-name-hash' ' > + test_file_size out > +' > + > +test_perf 'shallow fetch pack' ' > + git pack-objects --stdout --revs --sparse --shallow <in-shallow >out > +' > + > +test_size 'shallow pack size' ' > + test_file_size out > +' > + > +test_perf 'shallow pack with --full-name-hash' ' > + git pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out > +' > + > +test_size 'shallow pack size with --full-name-hash' ' > + test_file_size out > +' > + > +test_perf 'repack' ' > + git repack -adf > +' > + > +test_size 'repack size' ' > + pack=$(ls .git/objects/pack/pack-*.pack) && > + test_file_size "$pack" Here and below, I think it's fine to inline this as in: test_file_size "$(ls .git/objects/pack/pack-*.pack)" ...but I wonder: will using ".git" break this test in bare repositories? Should we write instead: pack="$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)" && test_file_size ? Thanks, Taylor

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 11/21/24 3:31 PM, Taylor Blau wrote: > On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote: >> From: Derrick Stolee <[email protected]> >> The thin pack that simulates a push is much worse with --full-name-hash >> in this case. The name hash values are doing a lot to assist with delta >> bases, it seems. The big pack and shallow clone cases are slightly worse >> with the --full-name-hash option. Only the full repack gains some >> benefits in size. > > Not a problem with your patch, but just thinking aloud: do you think > there is an easy/straightforward way to suggest when to use > --full-name-hash or not? The kinds of heuristics I would use are: 1. Are there enough commits that enough files have enough versions across history that it's very important to keep deltas within a path? 2. Is the repository at least 500MB such that there is actually room for a "meaningful" change in size? 3. Are there a lot of name-hash collisions? (The last patch in the series helps do this through a test-helper, but isn't something we can expect end users to check themselves.) >> + cat >in-shallow <<-EOF >> + $(git rev-parse HEAD) >> + --shallow $(git rev-parse HEAD) >> + EOF >> +' > > I was going to comment that these could probably be moved into the > individual perf test that cares about reading each of these inputs. But > having them shared here makes sense since we are naturally comparing > generating two packs with the same input (with and without > --full-name-hash). So the shared setup here makes sense to me. I also wanted to avoid having these commands be part of the time measurement, even if they are extremely small. >> + >> +test_perf 'thin pack' ' >> + git pack-objects --thin --stdout --revs --sparse <in-thin >out >> +' >> + >> +test_size 'thin pack size' ' >> + test_file_size out >> +' > > Nice. I always forget about this and end up writing 'wc -c <out'. I believe this is a Junio recommendation from an earlier version. >> +test_size 'repack size' ' >> + pack=$(ls .git/objects/pack/pack-*.pack) && >> + test_file_size "$pack" > > Here and below, I think it's fine to inline this as in: > > test_file_size "$(ls .git/objects/pack/pack-*.pack)" Generally I prefer to split things into stages so the verbose output provides a clear definition of the value when calling the Git command. > ...but I wonder: will using ".git" break this test in bare repositories? > Should we write instead: > > pack="$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)" && > test_file_size > > ? While this would break a bare repo, the perf lib makes a bare repo be copied into a non-bare repo as follows: test_perf_copy_repo_contents () { for stuff in "$1"/* do case "$stuff" in */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*) ;; *) cp -R "$stuff" "$repo/.git/" || exit 1 ;; esac done } I'll still add the `git rev-parse` suggestion because it's safest. Thanks, -Stolee

-Original file line number
+Diff line change
@@ Expand Up / @@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o @@
     TEST_BUILTINS_OBJS += test-match-trees.o
     TEST_BUILTINS_OBJS += test-mergesort.o
     TEST_BUILTINS_OBJS += test-mktemp.o
+    TEST_BUILTINS_OBJS += test-name-hash.o
     TEST_BUILTINS_OBJS += test-online-cpus.o
     TEST_BUILTINS_OBJS += test-pack-mtimes.o
     TEST_BUILTINS_OBJS += test-parse-options.o
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -39,7 +39,9 @@ static int run_update_server_info = 1; @@
     static char *packdir, *packtmp_name, *packtmp;
     static const char *const git_repack_usage[] = {
-    	N_("git repack [<options>]"),
+    	N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n"
+    	   "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n"
+    	   "[--write-midx] [--name-hash-version=<n>]"),
     	NULL
     };
@@ Expand All / @@ -58,6 +60,7 @@ struct pack_objects_args { @@
     	int no_reuse_object;
     	int quiet;
     	int local;
+    	int name_hash_version;
     	struct list_objects_filter_options filter_options;
     };
@@ Expand Down Expand Up / @@ -306,6 +309,8 @@ static void prepare_pack_objects(struct child_process *cmd, @@
     		strvec_pushf(&cmd->args, "--no-reuse-delta");
     	if (args->no_reuse_object)
     		strvec_pushf(&cmd->args, "--no-reuse-object");
+    	if (args->name_hash_version)
+    		strvec_pushf(&cmd->args, "--name-hash-version=%d", args->name_hash_version);
     	if (args->local)
     		strvec_push(&cmd->args,  "--local");
     	if (args->quiet)
@@ Expand Down Expand Up / @@ -1203,6 +1208,8 @@ int cmd_repack(int argc, @@
     				N_("pass --no-reuse-delta to git-pack-objects")),
     		OPT_BOOL('F', NULL, &po_args.no_reuse_object,
     				N_("pass --no-reuse-object to git-pack-objects")),
+    		OPT_INTEGER(0, "name-hash-version", &po_args.name_hash_version,
+    				N_("specify the name hash version to use for grouping similar objects by path")),
     		OPT_NEGBIT('n', NULL, &run_update_server_info,
     				N_("do not run git-update-server-info"), 1),
     		OPT__QUIET(&po_args.quiet, N_("be quiet")),
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up @@
     sure the expected tests are executed and not silently skipped when their
     dependency breaks or is simply not present in a new environment.
+    GIT_TEST_NAME_HASH_VERSION=<int>, when set, causes 'git pack-objects' to
+    assume '--name-hash-version=<n>'.
     Naming Tests
     ------------
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -44,6 +44,7 @@ static struct test_cmd cmds[] = { @@
     	{ "match-trees", cmd__match_trees },
     	{ "mergesort", cmd__mergesort },
     	{ "mktemp", cmd__mktemp },
+    	{ "name-hash", cmd__name_hash },
     	{ "online-cpus", cmd__online_cpus },
     	{ "pack-mtimes", cmd__pack_mtimes },
     	{ "parse-options", cmd__parse_options },
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -37,6 +37,7 @@ int cmd__lazy_init_name_hash(int argc, const char **argv); @@
     int cmd__match_trees(int argc, const char **argv);
     int cmd__mergesort(int argc, const char **argv);
     int cmd__mktemp(int argc, const char **argv);
+    int cmd__name_hash(int argc, const char **argv);
     int cmd__online_cpus(int argc, const char **argv);
     int cmd__pack_mtimes(int argc, const char **argv);
     int cmd__parse_options(int argc, const char **argv);
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -15,7 +15,8 @@ SYNOPSIS @@
     	[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
     	[--cruft] [--cruft-expiration=<time>]
     	[--stdout [--filter=<filter-spec>] | <base-name>]
-    	[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
+    	[--shallow] [--keep-true-parents] [--[no-]sparse]
+    	[--name-hash-version=<n>] < <object-list>
     DESCRIPTION
@@ Expand Down Expand Up / @@ -345,6 +346,35 @@ raise an error. @@
     	Restrict delta matches based on "islands". See DELTA ISLANDS
     	below.
+    --name-hash-version=<n>::
+    	While performing delta compression, Git groups objects that may be
+    	similar based on heuristics using the path to that object. While
+    	grouping objects by an exact path match is good for paths with
+    	many versions, there are benefits for finding delta pairs across
+    	different full paths. Git collects objects by type and then by a
+    	"name hash" of the path and then by size, hoping to group objects
+    	that will compress well together.
+    +
+    The default name hash version is `1`, which prioritizes hash locality by
+    considering the final bytes of the path as providing the maximum magnitude
+    to the hash function. This version excels at distinguishing short paths
+    and finding renames across directories. However, the hash function depends
+    primarily on the final 16 bytes of the path. If there are many paths in
+    the repo that have the same final 16 bytes and differ only by parent
+    directory, then this name-hash may lead to too many collisions and cause
+    poor results. At the moment, this version is required when writing
+    reachability bitmap files with `--write-bitmap-index`.
+    +
+    The name hash version `2` has similar locality features as version `1`,
+    except it considers each path component separately and overlays the hashes
+    with a shift. This still prioritizes the final bytes of the path, but also
+    "salts" the lower bits of the hash using the parent directory names. This
+    method allows for some of the locality benefits of version `1` while
+    breaking most of the collisions from a similarly-named file appearing in
+    many different directories. At the moment, this version is not allowed
+    when writing reachability bitmap files with `--write-bitmap-index` and it
+    will be automatically changed to version `1`.
     DELTA ISLANDS
     -------------
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository @@
     SYNOPSIS
     --------
     [verse]
-    'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]
+    'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]
+    	[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]
+    	[--write-midx] [--name-hash-version=<n>]
     DESCRIPTION
     -----------
@@ Expand Down Expand Up / @@ -249,6 +251,11 @@ linkgit:git-multi-pack-index[1]). @@
     	Write a multi-pack index (see linkgit:git-multi-pack-index[1])
     	containing the non-redundant packs.
+    --name-hash-version=<n>::
+    	Provide this argument to the underlying `git pack-objects` process.
+    	See linkgit:git-pack-objects[1] for full details.
     CONFIGURATION
     -------------
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -266,6 +266,43 @@ struct configured_exclusion { @@
     static struct oidmap configured_exclusions;
     static struct oidset excluded_by_config;
+    static int name_hash_version = -1;
+    /**
+     * Check whether the name_hash_version chosen by user input is apporpriate,
+     * and also validate whether it is appropriate with other features.
+     */
+    static void validate_name_hash_version(void)
+    {
+    	if (name_hash_version < 1 || name_hash_version > 2)
+    		die(_("invalid --name-hash-version option: %d"), name_hash_version);
+    	if (write_bitmap_index && name_hash_version != 1) {
+    		warning(_("currently, --write-bitmap-index requires --name-hash-version=1"));
+    		name_hash_version = 1;
+    	}
+    }
+    static inline uint32_t pack_name_hash_fn(const char *name)
+    {
+    	static int seen_version = -1;
+    	if (seen_version < 0)
+    		seen_version = name_hash_version;
+    	else if (seen_version != name_hash_version)
+    		BUG("name hash version changed from %d to %d mid-process",
+    		    seen_version, name_hash_version);
+    	switch (name_hash_version) {
+    	case 1:
+    		return pack_name_hash(name);
+    	case 2:
+    		return pack_name_hash_v2((const unsigned char *)name);
+    	default:
+    		BUG("invalid name-hash version: %d", name_hash_version);
+    	}
+    }
     /*
      * stats
@@ Expand Down Expand Up @@
     		return 0;
     	}
-    	create_object_entry(oid, type, pack_name_hash(name),
+    	create_object_entry(oid, type, pack_name_hash_fn(name),
     			    exclude, name && no_try_delta(name),
     			    found_pack, found_offset);
     	return 1;
@@ Expand Down Expand Up / @@ -1912,7 +1949,7 @@ static void add_preferred_base_object(const char *name) @@
     {
     	struct pbase_tree *it;
     	size_t cmplen;
-    	unsigned hash = pack_name_hash(name);
+    	unsigned hash = pack_name_hash_fn(name);
     	if (!num_preferred_base || check_pbase_path(hash))
     		return;
@@ Expand Down Expand Up @@
     	 * here using a now in order to perhaps improve the delta selection
     	 * process.
     	 */
-    	oe->hash = pack_name_hash(name);
+    	oe->hash = pack_name_hash_fn(name);
     	oe->no_try_delta = name && no_try_delta(name);
     	stdin_packs_hints_nr++;
@@ Expand Down Expand Up @@
     	entry = packlist_find(&to_pack, oid);
     	if (entry) {
     		if (name) {
-    			entry->hash = pack_name_hash(name);
+    			entry->hash = pack_name_hash_fn(name);
     			entry->no_try_delta = no_try_delta(name);
     		}
     	} else {
@@ Expand All @@
     			return;
     		}
-    		entry = create_object_entry(oid, type, pack_name_hash(name),
+    		entry = create_object_entry(oid, type, pack_name_hash_fn(name),
 , name && no_try_delta(name),
     					    pack, offset);
     	}
@@ Expand Down Expand Up @@
     	if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
     		return -1;
+    	/*
+    	 * For now, force the name-hash version to be 1 since that
+    	 * is the version implied by the bitmap format. Later, the
+    	 * format can include this version explicitly in its format,
+    	 * allowing readers to know the version that was used during
+    	 * the bitmap write.
+    	 */
+    	name_hash_version = 1;
     	if (pack_options_allow_reuse())
     		reuse_partial_packfile_from_bitmap(bitmap_git,
     						   &reuse_packfiles,
@@ Expand Down Expand Up / @@ -4429,6 +4475,8 @@ int cmd_pack_objects(int argc, @@
     		OPT_STRING_LIST(0, "uri-protocol", &uri_protocols,
     				N_("protocol"),
     				N_("exclude any configured uploadpack.blobpackfileuri with this protocol")),
+    		OPT_INTEGER(0, "name-hash-version", &name_hash_version,
+    			 N_("use the specified name-hash function to group similar objects")),
     		OPT_END(),
     	};
@@ Expand Down Expand Up / @@ -4576,6 +4624,11 @@ int cmd_pack_objects(int argc, @@
     	if (pack_to_stdout || !rev_list_all)
     		write_bitmap_index = 0;
+    	if (name_hash_version < 0)
+    		name_hash_version = (int)git_env_ulong("GIT_TEST_NAME_HASH_VERSION", 1);
+    	validate_name_hash_version();
     	if (use_delta_islands)
     		strvec_push(&rp, "--topo-order");
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name) @@
     	return hash;
     }
+    static inline uint32_t pack_name_hash_v2(const unsigned char *name)
+    {
+    	uint32_t hash = 0, base = 0, c;
+    	if (!name)
+    		return 0;
+    	while ((c = *name++)) {
+    		if (isspace(c))
+    			continue;
+    		if (c == '/') {
+    			base = (base >> 6) ^ hash;
+    			hash = 0;
+    		} else {
+    			/*
+    			 * 'c' is only a single byte. Reverse it and move
+    			 * it to the top of the hash, moving the rest to
+    			 * less-significant bits.
+    			 */
+    			c = (c & 0xF0) >> 4 | (c & 0x0F) << 4;
+    			c = (c & 0xCC) >> 2 | (c & 0x33) << 2;
+    			c = (c & 0xAA) >> 1 | (c & 0x55) << 1;
+    			hash = (hash >> 2) + (c << 24);
+    		}
+    	}
+    	return (base >> 6) ^ hash;
+    }
     static inline enum object_type oe_type(const struct object_entry *e)
     {
     	return e->type_valid ? e->type_ : OBJ_BAD;
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,23 @@
+    /*
+     * test-name-hash.c: Read a list of paths over stdin and report on their
+     * name-hash and full name-hash.
+     */
+    #include "test-tool.h"
+    #include "git-compat-util.h"
+    #include "pack-objects.h"
+    #include "strbuf.h"
+    int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)
+    {
+    	struct strbuf line = STRBUF_INIT;
+    	while (!strbuf_getline(&line, stdin)) {
+    		printf("%10u ", pack_name_hash(line.buf));
+    		printf("%10u ", pack_name_hash_v2((unsigned const char *)line.buf));
+    		printf("%s\n", line.buf);
+    	}
+    	strbuf_release(&line);
+    	return 0;
+    }

-Original file line number
+Diff line change
@@ -0,0 +1,70 @@
+    #!/bin/sh
+    test_description='Tests pack performance using bitmaps'
+    . ./perf-lib.sh
+    GIT_TEST_PASSING_SANITIZE_LEAK=0
+    export GIT_TEST_PASSING_SANITIZE_LEAK
+    test_perf_large_repo
+    test_expect_success 'create rev input' '
+    	cat >in-thin <<-EOF &&
+    	$(git rev-parse HEAD)
+    	^$(git rev-parse HEAD~1)
+    	EOF
+    	cat >in-big <<-EOF &&
+    	$(git rev-parse HEAD)
+    	^$(git rev-parse HEAD~1000)
+    	EOF
+    	cat >in-shallow <<-EOF
+    	$(git rev-parse HEAD)
+    	--shallow $(git rev-parse HEAD)
+    	EOF
+    '
+    for version in 1 2
+    do
+    	export version
+    	test_perf "thin pack with version $version" '
+    		git pack-objects --thin --stdout --revs --sparse \
+    			--name-hash-version=$version <in-thin >out
+    	'
+    	test_size "thin pack size with version $version" '
+    		test_file_size out
+    	'
+    	test_perf "big pack with version $version" '
+    		git pack-objects --stdout --revs --sparse \
+    			--name-hash-version=$version <in-big >out
+    	'
+    	test_size "big pack size with version $version" '
+    		test_file_size out
+    	'
+    	test_perf "shallow fetch pack with version $version" '
+    		git pack-objects --stdout --revs --sparse --shallow \
+    			--name-hash-version=$version <in-shallow >out
+    	'
+    	test_size "shallow pack size with version $version" '
+    		test_file_size out
+    	'
+    	test_perf "repack with version $version" '
+    		git repack -adf --name-hash-version=$version
+    	'
+    	test_size "repack size with version $version" '
+    		gitdir=$(git rev-parse --git-dir) &&
+    		pack=$(ls $gitdir/objects/pack/pack-*.pack) &&
+    		test_file_size "$pack"
+    	'
+    done
+    test_done

pack-objects: Create an alternative name hash algorithm (recreated) #1823

Are you sure you want to change the base?

pack-objects: Create an alternative name hash algorithm (recreated) #1823

Diff view

Diff view

There are no files selected for viewing

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 4, 2024

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Jan 24, 2025

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Jan 24, 2025

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 4, 2024

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 25, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 25, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 26, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 4, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 9, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 20, 2024

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Dec 4, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 4, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 5, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 9, 2024

Choose a reason for hiding this comment

gitgitgadget bot Dec 10, 2024

Choose a reason for hiding this comment

gitgitgadget bot Jan 22, 2025

Choose a reason for hiding this comment

gitgitgadget bot Nov 21, 2024

Choose a reason for hiding this comment

gitgitgadget bot Nov 22, 2024

Choose a reason for hiding this comment