Re: [EN] NNCP 7.0.0 release announcement

John Goerzen
2021-06-30T13:30:11Z

Hi Sergey,

I need to read up more on these algorithms but this all sounds
very nice!

I have a question about memory usage:

On Wed, Jun 30 2021, Sergey Matveev wrote:
> * Merkle Tree-based Hashing with BLAKE3 (MTH) is used instead of
> BLAKE2b. Because of that, there are backward *incompatible*
> changes of encrypted files (everything laying in the spool
> directory) and ".meta" files of chunked transfer.
>
> Current implementation is far from being optimal: it lacks
> parallelizable calculations and has higher memory consumption:
> nearly 512 KiB for each 1 GiB of file’s data. Future
> performance
> and memory size optimizations should not lead to packet’s
> format
> change. But it is still several times faster than BLAKE2b.

I'm not familiar with MTH, but I'm guessing that's what leads to
the memory consumption that scales with the size of the data? I
would assume that BLAKE3, like other hashing algorithms, would be
constant-space?

I often process packets up to about 256GB in size, so I assume
that would lead to about 128MB of RAM usage. That's not ideal on
some of the lower-powered systems I use, particularly Raspberry
Pis, but I can probably live with it in general (the
lowest-powered ones don't tend to process data at that size).
Would I be correct in assuming that this is used in pretty much
all utilities? nncp-file/exec, call, daemon, toss?

I have been meaning to write a message about packet redirection as
well. I don't know if this makes it easier - I don't think so -
but I thought I might mention.

So I have a use case like this.

My node "laptop" sends backups to a machine on the LAN, I'll call
it "baksvr". Usually I only want it to send backups over the LAN
(in case I'm tethered to 4G or something), but occasionally I want
it to send backups remotely.

Now the machine on my home LAN isn't reachable from the internet,
but it does have an NNCP link to a VPS out there, which my laptop
could reach.

So I have packets queued up in my tx directory on the laptop for
baksvr, but I want to redirect them via the VPS.

I eventually wound up using nncp-bundle to bundle them up, sent
that bundle across NNCP, and unpacked it on the destination. That
can even be automated with nncp-exec -- but it might be nice to
have something like "nncp-redir" that could redirect queued
packets, adding additional onion routing layers around what's
there already.

- John