Dataset health

[benbierens]

Context

During the all-Codex in Trogir, we discussed how best to support altruistic mode. This task followed from that session. Supporting this feature would allow users to detect the "health" of their datasets in the network. This enables altruistic use-cases to implement automatic mechanisms to organize repair/dispersal of altruistic datasets. We discussed and agreed that the disemination of the altruistic-dataset-CIDs would be handled outside of Codex. This feature would allow the code that controls this to take action when datasets are at risk.

Tasks

• Understand how to measure the "health" of a dataset. Is it enough to count the nodes who contain the manifest? Should there be some kind of block-sampling? (Keep in mind: We should NOT attempt to provide solid/highly reliable durability information or guarantees this way. For use cases that require high durability guarantees we should direct them to use the marketplace.)
• Design the changes necessary to support this feature at the network level
• Design the API to access the dataset health information
• Buidl!

[gmega]

Yeah, it's tricky. You can have a bunch of nodes that have the manifest and advertise themselves in the DHT; you count them and they're plenty, but none of them actually has the file, which would make it unhealthy. I guess the first point is understanding what guarantees we want, then design a mechanism; e.g. bittorrent has that info centralized on the tracker and takes node's self-reports as truthtful, which makes it a lot simpler.

If we want real proofs, it starts to get more complicated.

[benbierens]

Everyone I spoke to about this (outside of our team) understood that guarantees/proof/certainty is going to be near impossible unless we bring in the marketplace as a way to actually incentivize nodes to be honest about their data. So the people designing use-cases that would consume this health data are taking this signal/noise into account.

How about this:

• We count the nodes for the manifest in the DHT.
• We randomly dial a few of them and ask if they have a block at a random index. (Actually getting the block becomes risky when bandwidth incentives come into play.)
• Then we know: number of nodes with manifest * rough ratio of those who have a block of the dataset = estimate of dataset presence is in the network.

The problem with this:
In order to pick a random block index, you need to know how many blocks in the dataset. Which means nodes trying to measure the health of something will have the manifest. Which means of all the nodes who have the manifest, a percentage of them just have it so they can measure. They won't have blocks. Which in turn throws off the measurement. I really like puzzles of this kind.