Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bittorrent Integration [draft] #1137

Draft
wants to merge 64 commits into
base: master
Choose a base branch
from
Draft

Bittorrent Integration [draft] #1137

wants to merge 64 commits into from

Conversation

marcinczenko
Copy link
Contributor

@marcinczenko marcinczenko commented Feb 28, 2025 •
edited
Loading

Draft of the BitTorrent Codex integration.

Should be enough to see the overall idea in code.

What is there:

  • storeTorrent is equivalent of Codex store - this is the entry point when seeding BitTorrent content into Codex network
  • retrieveTorrent is equivalent of Codex retrieve - entry point when requesting BitTorrent content having only info hash as the input.

To be functional still needs:

  • b-encoding and the actual info-hash validation (now there is only a placeholder around the place where it most probably will happen)
  • streaming pieces and validating them
  • REST API (download)
  • REST API (upload)
  • REST API (fetch torrent manifest only for the given info hash)
  • Try to make the Cid for BitTorrent Manifest a valid Cid - investigation done - not technically feasible and does not add any more security.
  • update docs
  • a bit more testing

See also codex-storage/codex-pm#211.

@marcinczenko marcinczenko self-assigned this Feb 28, 2025
@marcinczenko marcinczenko marked this pull request as draft February 28, 2025 17:11
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 3 times, most recently from bf800ac to 8a608e8 Compare March 5, 2025 14:45
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 1b28c03 to b531696 Compare March 11, 2025 01:05
@marcinczenko marcinczenko mentioned this pull request Mar 13, 2025
Open
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 2 times, most recently from aa9989c to e19065b Compare March 23, 2025 00:44
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 6 times, most recently from 73591d5 to 820709c Compare March 31, 2025 15:57
@marcinczenko
updates torrent streaming to take advantage of the new interface of t…
4c7b7cf 
…he piece validator
@marcinczenko
adds piece validator tests to codex tests
31e7623
@marcinczenko
updates bittorrent tests to use unittest2
6728016
@marcinczenko
ignore "./data*" folders from git
9c50539
@marcinczenko
fixing problems after rebasing
37e5b06
@marcinczenko
uses MultiHash as a type for piece hashes (instead alias)
d36cd54
@marcinczenko
fixes checking block results
17b1997
@marcinczenko
removes unused import
7bf9655
@marcinczenko
adds push raises to a couple of new files
9cfad38
@marcinczenko
refactor piece validator tests
271a1e1
@marcinczenko
adds torrent downloader abstraction
1bd6f1b
@marcinczenko
adds testing setup for torrentdownloader
2956b3c
@marcinczenko
do not randomize piece fetching sequence for now
8bc9b34
@marcinczenko
adds some good weather tests for torrentdownloader
668213a
@marcinczenko
adds more tests and ability to retrieve the downloaded blocks
ab58de4
@marcinczenko
updates and integrates torrentdownloader into api and node
c351306
@marcinczenko
updates copyright
5a65c1a
@marcinczenko
checked exceptions in stores
f4bb87a
@marcinczenko
better exception handling
f54e15d
@marcinczenko
better exception handling in node
ea422a7
@marcinczenko
checked exceptions in stores
5a464a4
@marcinczenko
introduce "SafeAsyncIter" that uses Results and limits exceptions to …
b2afd7e 
…cancellations
@marcinczenko
uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"
057b64a
@marcinczenko
uses SafeAsyncIter to stream the blocks in api
8dccd66
@marcinczenko
fix failing test in torrentdownloader
0ead230
@marcinczenko
simplifies safeasynciter (magic of auto)
da5851e
@marcinczenko
gets rid of ugly casts
fa5959a
@marcinczenko
adds more tests to torrentdownloader
4237e8b
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 820709c to 4237e8b Compare March 31, 2025 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@marcinczenko marcinczenko

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@marcinczenko