feat: publish index files for published packages to the registry

[kayagokalp]

Disclaimer

For now this branch is pointing to FuelLabs/sway@f4477cf for forc-pkg. This will be the version that is going to be released soon from sway repo side. Before merging we should move it to the released instance, either 0.67.1 or 0.68.0 whichever comes next.

Description

This PR connects forc, registry fetching and actual publishing step in forc.pub. Basically for each published package we are generating an index file, and corresponding entry to that file to describe the package. This index information is stored in a registry. Backend actually commits these files once the publishing is completed.

This file then used by forc to resolve version and package name tuple to an actual IPFS CID that can be fetched. This is done in FuelLabs/sway#7038.

How to e2e test

Following steps outline some way to test this change e2e with new forc that can build registry packages. The steps assume this PR is pointing to the dummy repo, if not first point to dummy repo manually by changing the repo name and org in src/handlers/publish.rs.

1. Accept my invitation request to https://github.com/kayagokalp/dummy-forc.pub-index from github
2. Generate a PAT, with enough permissions to be able to push code
3. Fetch forc.pub and come to kayagokalp/index-file branch
4. cp .env .env.local
   5.Place IPFS, GITHUB client you fetched from the vault into .env.local and also place your GITHUB_TOKEN/PAT there.
5. run ./scripts/start_local_db.sh (make sure docker daemon is running)
6. go to app/ folder and run npm i then npm start
7. Step 7 will open a browser if not go to localhost:3000 login with github
8. Click on your profile and generate an api key
9. Go to the package you want to publish and run ../target/debug/forc-publish --registry-url http://0.0.0.0:8080/ --token <TOKEN> (I published std so forc-publish location was like this from sway-lib-std folder)
10. Generate a new forc project and depend on this new dependency by version

[project]
authors = ["kayagokalp"]
entry = "lib.sw"
license = "Apache-2.0"
name = "testing"
implicit-std = false

[dependencies]
std = "0.12.0"

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
forc-pub	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 18, 2025 8:23am

[sdankel]

In terms of merging this, I think we might need to split the sway PR into two: One with the forc-pkg changes needed for this PR, and a separate one for reading from the index. That way we can merge this, and test the forc-build (fetching from the index) against the real repo. So the order could be:

1. merge forc-pkg PR with library changes (IndexFile, PackageEntry, etc)
2. release sway
3. merge this PR pointing at the sway release
4. test & merge the forc-build PR that fetches from the real registry repo

What do you think?

[kayagokalp]

In terms of merging this, I think we might need to split the sway PR into two: One with the forc-pkg changes needed for this PR, and a separate one for reading from the index. That way we can merge this, and test the forc-build (fetching from the index) against the real repo. So the order could be:

1. merge forc-pkg PR with library changes (IndexFile, PackageEntry, etc)
2. release sway
3. merge this PR pointing at the sway release
4. test & merge the forc-build PR that fetches from the real registry repo

What do you think?

We could do that but i was thinking we keep this aside for the time being and only get the sway PR merged first. Sway repo PR is self containing at the moment as building things is not connected to publishing things. One can still publish without this PR. Just that packages needs to be manually added with a PR to the index repo. So i am seeing this as more like an automation, more than a thing that unblocks sway repo PR.

That being said If you prefer to keep the flow you described i can also do that too.

[kayagokalp]

In memory ssh key authentication works now. But there is a caveat. For git2 distribution (not the rust crate but system dependency) the key should be in OpenSSL PEM format which can be generated like the following:

ssh-keygen -t rsa -b 4096 -m PEM -f ~/id_rsa

From OpenSSH v7.8 onwards, ssh-keygen will write private keys in the OpenSSH format by default. By using -m PEM we ensure that the key pair is in the OpenSSL PEM format, regardless of the OpenSSH version. -m PEM can be omitted as long as the -o flag is also omitted if OpenSSH is older than v7.8.

The env variable SSH_KEY should include the generated file content fully, with the headers

> echo $SSH_KEY
-----BEGIN RSA PRIVATE KEY-----
...
...
-----END RSA PRIVATE KEY-----

[sdankel]

Great work on this!