feat: PDP

[magik6k]

Reading:

• Proposed curio api storacha/RFC#34
• https://github.com/FILCAT/pdp/blob/main/contracts/src/PDPService.sol

Roadmap:

• Database schema
• PieceCID version correctness
• HTTP API
  • Basic structure
  • Auth
  • Data ingest
    • POST /piece
    • PUT /piece/upload/..
    • Call notify
      • make optional
    • Store in long-term storage
  • Data retrieval through /piece
    • Auth
  • PDPTool
    • Creating secret/tokens
    • Ping tool
    • Upload tool
    • Retrieve tool
    • Piece set manipulation
  • Piece sets
    • Create
    • Delete
    • Add root (w subroots)
    • Del Root
    • Get root subroots
  • Proof-set prover
• UI

[hannahhoward]

I've mostly put in comments to check for understanding before I start writing code. But generally LGTM -- looks like it's compliant with the API we agreed on :)

[hannahhoward]

It looks like you're not requiring piececid v2 here, not in the put handler. Long term, I think we should require this. From the standpoint of the SP, I imagine them to want to filter at the piece level, and I imagine one of the main filter critieria to be size. They should know exactly what size piece they're about to accept before they accept it.

Unless I'm wrong, a PieceCID v1 doesn't specify that, and absent a size parameter, they could be accepting anything from 32 bytes to 256 MB.

[magik6k]

Yes, PieceCIDv2 is still todo, can quickly update if that's a blocker

[hannahhoward]

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

[hannahhoward]

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

[magik6k]

Github thinks I have an unsubmitted review here and can't discard it..

[magik6k]

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

Yes, correct, retrievals work already. No ACL mechanism yet, but it's not going to be hard to implement once we have a spec (you can see that all retrieval requests quite quickly become aware what piece they are reading from + ACLs should be quite cachable)

[magik6k]

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

Other than delete all the things should be there. Operations on PDP proofsets (other than create) you should probably watch through eth events. I can also make the message-sending endpoints output the message hash somehow, but you still want to watch for all the events to not miss any contract changes made by the SP

[hannahhoward]

Upon further review, it's possible we have a design issue here that we need a solution for.

Please let me know if I am understanding the sequence of events correctly when a user uploads a pdp piece:

1. The piece is put in the “Stash”
2. A parked piece record is created
3. Asychronously, the piece moves from the stash to permaneant storage (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/piece/task_park_piece.go)
4. After that, again asynchronously, the notify task is called, which hits the notify URL. (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/pdp/notify_task.go)

Why this matters: a basic gaurantee of all hot storage, which storacha tries to offer, is read-on-write. The basic expectation is from the users point of view is that when an upload finishes, it should be retrievable.

One thing that is not clear I'd like to clarify: are steps 3 & 4 likely to run in a couple seconds or much longer? If it's only a few seconds, we can work around this. Fortunately our upload process includes a confirmation step AFTER the upload, and what we can do is essentially block the confirmation until step 4 completes. But that only works if it's a short period of time before the piece is fully retrievable.

If not, we really need to revisit the architecture and find a way to shorten the process.

[hannahhoward]

Follow-up: a pull based approach for checking for piece cid -- #304

[magik6k]

Could use an index on this table like https://github.com/filecoin-project/curio/pull/392/files

[magik6k]

Should drop the verbosity here

[magik6k]

There is also the pdp_piecerefs table, and I'm not entirely sure what's the best way to approach this because:

• On upload the piece is only referred in pdp_piecerefs (+the piecepark tables)
• Only after upload is complete and data is verified the PDP service can add the piece to a proofset
• Technically one piece can be in multiple proofsets / roots (as a subroot)
• So probably:
  • Add a flag to pdp_piecerefs which is optionally settable through the delete api which tells curio to drop the piece reference when all pdp_proofset_roots are gone
  • Possibly add an endpoint which lists all pdp_proofset_roots for a given piece (so the service can know why a given piece may be "locked")

[magik6k]

Did a deeper review pass with focus on whether this will impact other curio functionality - looks like this should be safe to merge, generally most invasive changes are in piece_park, but default behaviors don't change there, and in retrieval paths, but there PDP is plugged in as a fallback, so it shouldn't impact normal operation.

[rvagg]

Suggested change

	more = len(proofSets) == 2
	more = len(proofSets) > 1

?