Some good news: Aljoscha and I have been awarded another grant by the NGI Zero Commons fund. We’ll use this to work on “Bab”, a new hash function we want to use with Willow. It will give Willow torrent-like parallel downloads of chunks of large downloads, safe resumption of downloads after connection failures, and integrity-verified propagation of partial data.
Aljoscha has written more about Bab here: https://aljoscha-meyer.de/posts/bab_funding/index.html#bab_funding
@gwil I came here to ask how this is different from just using a Merkle tree for the hashes of the chunks. Then I thought I'd better read further so that I can see if there's any details published that I can read about myself. And when I finally clicked through enough links to find it, I saw that it said that it uses a Merkele tree 😄
So say I wanted to "sign" a Bab stream while I hash it, say with ed25519. Would I just sign the top merkle tree nodes every few MB or so?
@gwil I assumed that signing / verifying each chunk individually would be too computationally expensive.
@forestjohnson > Then it might indeed make sense to sign only (some) inner hashes. But now the answer really starts to depend on *why* you are signing things in the first place. The default setting would be one where the requesting peer already trusts the hash it requests to a certain degree. Which seems to be a setting that does not require additional signatures.
>
> Which additional guarantees do you want to obtain by signing things? Also, do you want signatures to be reflected in the hash? If so, could you achieve the effect by preprocessing the string-to-be-hashed (for example by inserting signatures over the preceding data at regular intervals) instead of modifying the response stream?
(also, we have a discord [i know] where it might be easier to have this discussion: https://discord.gg/6NtYzQC2G4)
@forestjohnson Aljoscha is not on fedi so I’ll patch through his response:
“That depends on what you are trying to achieve. The most straight-foward way would be to prepend a signature to each actual data chunk that is being transmitted. In most scenarios I can come up with, there is little need for signing intermediate nodes, since those are derived deterministically from the (signed) leaf chunks anyway”