Add support for an Index alongside the Catalog by chennes · Pull Request #332 · FreeCAD/AddonManager

chennes · 2026-02-08T18:45:18Z

Also implement sparse checkout of the huge repos, and change the data source location for the cache.

Copilot

Pull request overview

This PR updates the addon cache generation flow to consume the new Addons Index.json, generate both an index cache and a curated-only catalog cache for backward compatibility, and adds sparse git cloning for very large repos.

Changes:

Switch cache source from the legacy AddonCatalog.json URL to the Addons Data/Index.json.
Generate a full addon_index_cache.zip plus a curated-only addon_catalog_cache.zip for older Addon Manager versions.
Add sparse-clone support and clone timeout/error reporting for large repositories.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`AddonCatalogCacheCreator.py`	Fetch from `Index.json`, write new index+catalog cache artifacts, add sparse clone + timeout/error tracking.
`AddonCatalog.py`	Add `curated` / `sparse_cache` fields and use curated flag on the instantiated `Addon`.
`Addon.py`	Add `Addon.curated` property for UI/logic to consume.
`AddonCatalog.schema.json`	Update schema to include the new `curated` boolean field.
`AddonManagerTest/app/test_addon_catalog_cache_creator.py`	Adjust tests for the updated cache writer behavior (currently inconsistent with implementation).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-08T18:49:59Z

AddonCatalogCacheCreator.py

            ]
-            completed_process = subprocess.run(command)
+            try:
+                completed_process = subprocess.run(command, CLONE_TIMEOUT)


In clone_or_update, subprocess.run(command, CLONE_TIMEOUT) is passing CLONE_TIMEOUT as a positional arg (it becomes bufsize), so the clone will not time out. Use timeout=CLONE_TIMEOUT (and consider check=True to raise on non-zero exit) so oversized repos can be handled as intended.

Suggested change

completed_process = subprocess.run(command, CLONE_TIMEOUT)

completed_process = subprocess.run(command, timeout=CLONE_TIMEOUT)

Copilot · 2026-02-08T18:50:00Z

AddonCatalogCacheCreator.py

+        # Write the entire index for versions of the Addon Manager after 2026-01-24
+        with zipfile.ZipFile(
+            os.path.join(self.cwd, "addon_index_cache.zip"), "w", zipfile.ZIP_DEFLATED
+        ) as zipf:
+            zipf.writestr(
+                "addon_index_cache.json",
+                json.dumps(recursive_serialize(self.catalog.get_catalog()), indent="  "),
+            )


This adds a new addon_index_cache.zip artifact, but unlike the other caches it does not generate a corresponding .sha256. addonmanager_workers_startup.new_cache_available() expects a <cache_name>_cache_url + '.sha256' endpoint, so the index cache may not be refreshable/detectable consistently. Consider writing addon_index_cache.zip.sha256 as well (and updating the final log message accordingly).

Copilot · 2026-02-08T18:50:00Z

AddonManagerTest/app/test_addon_catalog_cache_creator.py

        writer = accc.CacheWriter()
-        writer.catalog = MagicMock()
+        writer.index = MagicMock()
        writer.cwd = os.path.abspath(os.path.join("home", "cache"))
        writer.create_local_copy_of_single_addon("TestMod", catalog_entries)


These tests now set writer.index = MagicMock(), but CacheWriter uses self.catalog (e.g., self.catalog.add_metadata_to_entry(...) in create_local_copy_of_single_addon). With catalog left as None, the test will raise AttributeError. Update the test to set writer.catalog (or mock the add_metadata_to_entry call path) instead of writer.index.

Copilot · 2026-02-08T18:50:00Z

AddonManagerTest/app/test_addon_catalog_cache_creator.py

        writer = accc.CacheWriter()
-        writer.catalog = MagicMock()
+        writer.index = MagicMock()
        writer.cwd = os.path.abspath(os.path.join("home", "cache"))
        writer.create_local_copy_of_single_addon("TestMod", catalog_entries)


Same issue as above: writer.index is not used by CacheWriter, but create_local_copy_of_single_addon() relies on writer.catalog being set to something that has add_metadata_to_entry. Setting writer.index here will cause failures when the test hits the metadata/zip steps.

Copilot · 2026-02-08T18:50:00Z

AddonManagerTest/app/test_addon_catalog_cache_creator.py

-        mock_fetch_catalog.return_value = MockCatalog()
        writer = accc.CacheWriter()
+        writer.index = MockCatalog()
        writer.create_local_copy_of_addons()


create_local_copy_of_addons() iterates self.catalog.get_catalog(), but this test sets writer.index and never sets writer.catalog, so it will fail with AttributeError. Also, the test still references accc.EXCLUDED_REPOS, which no longer exists (and the production code no longer skips those repos). Update the test data/expectations to match the new behavior (e.g., set writer.catalog = MockCatalog() and adjust the skipped-repo assertion).

Copilot · 2026-02-08T18:50:01Z

AddonManagerTest/app/test_addon_catalog_cache_creator.py

+    def test_generate_cache_entry_with_approval(self):
+        """If the addon appears in the catalog (as opposed to just the index), it gets marked as approved."""
+


test_generate_cache_entry_with_approval currently has no assertions and will always pass, so it doesn't actually validate the new curated/approved behavior described in the docstring. Either implement the test logic (assert curated filtering/flagging) or remove the placeholder to avoid giving a false sense of coverage.

Copilot · 2026-02-08T18:50:01Z

AddonCatalogCacheCreator.py

+    def write(self, addon_id: Optional[str] = None) -> None:
        original_working_directory = os.getcwd()
        os.makedirs(self.cwd, exist_ok=True)
        os.chdir(self.cwd)
-        self.create_local_copy_of_addons()

+        fetcher = CatalogFetcher()
+        self.catalog = fetcher.catalog
+
+        if addon_id is None:
+            self.create_local_copy_of_addons()
+        else:
+            catalog = self.catalog.get_catalog()
+            if addon_id not in catalog:
+                raise RuntimeError(f"ERROR: Addon {addon_id} not in index")
+            catalog_entries = catalog[addon_id]
+            self.create_local_copy_of_single_addon(addon_id, catalog_entries)
+


write() changes the process working directory with os.chdir(self.cwd) but does not restore it if an exception occurs (e.g., addon_id not in index, fetch failure, zip write error). Wrap the body in try/finally and always os.chdir(original_working_directory) to avoid leaking state to callers/tests.

Copilot · 2026-02-08T18:50:01Z

AddonCatalogCacheCreator.py

+            if addon_id in FORCE_SPARSE_CLONE:
+                if catalog_entry.repository is None:
+                    print(
+                        f"ERROR: Cannot use sparse clone for {addon_id} because it has no git repo."
+                    )
+                    return


In the FORCE_SPARSE_CLONE path, if an entry has no repository you return early, which skips caching any remaining entries (including ones that might have a usable zip_url). Prefer handling that entry with the normal zip path (or continue) rather than aborting the whole addon.

Suggested change

if addon_id in FORCE_SPARSE_CLONE:

if catalog_entry.repository is None:

print(

f"ERROR: Cannot use sparse clone for {addon_id} because it has no git repo."

)

return

use_sparse_clone = addon_id in FORCE_SPARSE_CLONE and catalog_entry.repository is not None

if use_sparse_clone:

Copilot · 2026-02-08T18:50:01Z

AddonCatalog.py

+            else:
+                # TODO: Try to generate the expected URL form based on the repo location
+                raise RuntimeError(f"Sparse cache entry {addon_id} has no zip_url")


instantiate_addon() raises RuntimeError when sparse_cache is true but zip_url is missing. Since CacheWriter can set sparse_cache=True without ensuring zip_url exists, this can cause addons/branches to be skipped at runtime. Consider falling back to repository when zip_url is absent (or validate/enforce zip_url at cache-generation time).

Suggested change

else:

# TODO: Try to generate the expected URL form based on the repo location

raise RuntimeError(f"Sparse cache entry {addon_id} has no zip_url")

elif self.repository:

# Fallback: use the repository URL when a sparse cache entry has no explicit zip_url

url = self.repository

else:

raise RuntimeError(

f"Sparse cache entry {addon_id} has neither zip_url nor repository URL"

)

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

AddonCatalogCacheCreator.py:416

CacheWriter.clone_or_update was changed from a @staticmethod to an instance method, but other code still calls it as a static utility (e.g. MacroCacheCreator.py imports CacheWriter specifically to call CacheWriter.clone_or_update(...)). This change will raise a TypeError at runtime. Consider keeping clone_or_update as a static/class method (and passing an optional error sink/timeout), or update all external callers accordingly as part of this PR.

    def clone_or_update(self, name: str, url: str, branch: str) -> None:
        """If a directory called "name" exists, and it contains a subdirectory called .git,
        then 'git fetch' is called; otherwise we use 'git clone' to make a bare, shallow
        copy of the repo (in the normal case where minimal is True), or a normal clone,
        if minimal is set to False."""

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-08T19:15:20Z

AddonCatalogCacheCreator.py

+                catalog_entry.sparse_cache = True
+                self.create_local_copy_of_single_addon_with_git_sparse(
+                    addon_id, index, catalog_entry
+                )


When addon_id is in FORCE_SPARSE_CLONE, the code unconditionally sets catalog_entry.sparse_cache = True but does not ensure catalog_entry.zip_url is present. Downstream, AddonCatalogEntry.instantiate_addon() raises if sparse_cache is true and zip_url is missing, so entries that only provide a git repo/ref will break. Either require/derive zip_url before setting sparse_cache, or fall back to the normal git path when zip_url is absent.

Suggested change

catalog_entry.sparse_cache = True

self.create_local_copy_of_single_addon_with_git_sparse(

addon_id, index, catalog_entry

)

if catalog_entry.zip_url is not None:

catalog_entry.sparse_cache = True

self.create_local_copy_of_single_addon_with_git_sparse(

addon_id, index, catalog_entry

)

else:

print(

f"WARNING: Cannot use sparse clone for {addon_id} because it has no zip_url; "

"falling back to normal git clone."

)

self.create_local_copy_of_single_addon_with_git(

addon_id, index, catalog_entry

)

Copilot · 2026-02-08T19:15:21Z

AddonCatalog.py

+        if self.sparse_cache:
+            if self.zip_url:
+                url = self.zip_url
+            else:
+                # TODO: Try to generate the expected URL form based on the repo location
+                raise RuntimeError(f"Sparse cache entry {addon_id} has no zip_url")
+        elif self.repository:
+            url = self.repository
+        else:
+            url = self.zip_url


instantiate_addon() raises a RuntimeError if sparse_cache is true but zip_url is missing. Since zip_url is optional for git-based entries (and sparse_cache is set automatically during cache generation for some repos), this can cause Addon Manager to crash while building the addon list. Consider a safe fallback (e.g. use repository when zip_url is absent) or make cache generation guarantee zip_url for sparse entries.

Also implement sparse checkout of the huge repos, and change the data source location for the cache.

github-actions · 2026-02-08T23:53:24Z

Backport failed for main, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin main
git worktree add -d .worktree/backport-332-to-main origin/main
cd .worktree/backport-332-to-main
git switch --create backport-332-to-main
git cherry-pick -x d787321a42f4f3c2d16e0151095817a6db501c44

github-actions · 2026-02-09T00:01:09Z

Backport failed for main, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin main
git worktree add -d .worktree/backport-332-to-main origin/main
cd .worktree/backport-332-to-main
git switch --create backport-332-to-main
git cherry-pick -x d787321a42f4f3c2d16e0151095817a6db501c44

Copilot AI review requested due to automatic review settings February 8, 2026 18:45

Copilot started reviewing on behalf of chennes February 8, 2026 18:46 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

chennes force-pushed the supportIndexAndCatalog branch from ed1fdae to 73635be Compare February 8, 2026 19:09

chennes requested a review from Copilot February 8, 2026 19:09

Copilot started reviewing on behalf of chennes February 8, 2026 19:10 View session

chennes force-pushed the supportIndexAndCatalog branch from 73635be to 105ff4e Compare February 8, 2026 19:10

Copilot AI reviewed Feb 8, 2026

View reviewed changes

chennes force-pushed the supportIndexAndCatalog branch from 105ff4e to df3ef56 Compare February 8, 2026 19:39

Add support for an Index alongside the Catalog

d787321

Also implement sparse checkout of the huge repos, and change the data source location for the cache.

chennes force-pushed the supportIndexAndCatalog branch from df3ef56 to d787321 Compare February 8, 2026 20:44

chennes mentioned this pull request Feb 8, 2026

Support large addons via sparse checkout of cache #329

Closed

chennes merged commit a4855d4 into FreeCAD:dev Feb 8, 2026
10 of 11 checks passed

chennes deleted the supportIndexAndCatalog branch February 8, 2026 23:51

chennes added the release to main Trigger an Action to create a PR backporting to `main` label Feb 8, 2026

	completed_process = subprocess.run(command, CLONE_TIMEOUT)
	completed_process = subprocess.run(command, timeout=CLONE_TIMEOUT)

		def test_generate_cache_entry_with_approval(self):
		"""If the addon appears in the catalog (as opposed to just the index), it gets marked as approved."""

Conversation

chennes commented Feb 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Feb 8, 2026

Uh oh!

github-actions bot commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant