Some good news: Aljoscha and I have been awarded another grant by the NGI Zero Commons fund. We’ll use this to work on “Bab”, a new hash function we want to use with Willow. It will give Willow torrent-like parallel downloads of chunks of large downloads, safe resumption of downloads after connection failures, and integrity-verified propagation of partial data.
Aljoscha has written more about Bab here: https://aljoscha-meyer.de/posts/bab_funding/index.html#bab_funding
@gwil I came here to ask how this is different from just using a Merkle tree for the hashes of the chunks. Then I thought I'd better read further so that I can see if there's any details published that I can read about myself. And when I finally clicked through enough links to find it, I saw that it said that it uses a Merkele tree 😄
So say I wanted to "sign" a Bab stream while I hash it, say with ed25519. Would I just sign the top merkle tree nodes every few MB or so?
@gwil I assumed that signing / verifying each chunk individually would be too computationally expensive.
@forestjohnson Aljoscha is not on fedi so I’ll patch through his response:
“That depends on what you are trying to achieve. The most straight-foward way would be to prepend a signature to each actual data chunk that is being transmitted. In most scenarios I can come up with, there is little need for signing intermediate nodes, since those are derived deterministically from the (signed) leaf chunks anyway”