Skip to content

Switch to using the local IPFS Kubo daemon for IPFS files #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hoh
Copy link
Member

@hoh hoh commented Mar 12, 2025

IPFS Kubo is now present on all CRNs but unused.

Downloading files from this service would be more reliable than using
third party and provide content integrity checks.

This bypasses the vm-connector when fetching resources from IPFS.

Related Jira tickets : ALEPH-36

Self proofreading checklist

  • The new code clear, easy to read and well commented.
  • New code does not duplicate the functions of builtin or popular libraries.
  • An LLM was used to review the new code and look for simplifications.
  • New classes and functions contain docstrings explaining what they provide.
  • All new code is covered by relevant tests.
  • Documentation has been updated regarding these changes.
  • Dependencies update in the project.toml have been mirrored in the Debian package build script packaging/Makefile

Changes

  • Fixes how the configuration of Kubo is applied (previous version did not work anymore).
  • Adds setting IPFS_SERVER in aleph-vm.
  • Refactors relevant code to download from the IPFS gateway instead of the vm-connector

How to test

Stop the services, purge all application data, then restart the services.

sudo systemctl stop aleph-vm-supervisor
sudo systemctl stop ipfs
sudo rm -fr /var/lib/aleph/vm/ /var/cache/aleph/ /var/lib/ipfs/
sudo systemctl start ipfs
sudo systemctl start aleph-vm-supervisor

Then check that the index page of the CRN.

@hoh hoh requested a review from olethanh March 12, 2025 16:56
@hoh hoh self-assigned this Mar 18, 2025
@@ -133,6 +135,32 @@ async def download_file(url: str, local_path: Path) -> None:
tmp_path.unlink(missing_ok=True)


async def download_file_from_ipfs_or_connector(ref: str, cache_path: Path, filetype: str) -> None:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
async def download_file_from_ipfs_or_connector(ref: str, cache_path: Path, filetype: str) -> None:
async def download_file_from_ipfs_or_connector(item_hash: ItemHash, cache_path: Path, filetype: str) -> None:

Copy link

codecov bot commented Mar 19, 2025

Codecov Report

Attention: Patch coverage is 88.13559% with 7 lines in your changes missing coverage. Please review.

Project coverage is 63.41%. Comparing base (73013a2) to head (dadad88).

Files with missing lines Patch % Lines
src/aleph/vm/storage.py 84.21% 5 Missing and 1 partial ⚠️
tests/supervisor/views/test_operator.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #779      +/-   ##
==========================================
+ Coverage   63.33%   63.41%   +0.07%     
==========================================
  Files          77       77              
  Lines        6854     6877      +23     
  Branches      576      578       +2     
==========================================
+ Hits         4341     4361      +20     
- Misses       2325     2326       +1     
- Partials      188      190       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hoh added 2 commits March 19, 2025 15:49
IPFS Kubo is now present on all CRNs but unused.

Downloading files from this service would be more reliable than using
third party and provide content integrity checks.

This bypasses the vm-connector when fetching resources from IPFS.
Kubo service could also use unlimited memory.
@hoh
Copy link
Member Author

hoh commented Mar 19, 2025

@olethanh I moved the resource limits into this branch.

@hoh hoh removed their assignment May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant