Subject: Re: [EN] NNCP 7.0.0 release announcement

From: John Goerzen <[email-masked]>
Date: Wed, 30 Jun 2021 06:28:58 -0700

Hi Sergey,

I need to read up more on these algorithms but this all sounds 
very nice!

I have a question about memory usage:

On Wed, Jun 30 2021, Sergey Matveev wrote:
> * Merkle Tree-based Hashing with BLAKE3 (MTH) is used instead of
>   BLAKE2b.  Because of that, there are backward *incompatible*
>   changes of encrypted files (everything laying in the spool
>   directory) and ".meta" files of chunked transfer.
>
>   Current implementation is far from being optimal: it lacks
>   parallelizable calculations and has higher memory consumption:
>   nearly 512 KiB for each 1 GiB of file’s data.  Future 
>   performance
>   and memory size optimizations should not lead to packet’s 
>   format
>   change.  But it is still several times faster than BLAKE2b.

I'm not familiar with MTH, but I'm guessing that's what leads to 
the memory consumption that scales with the size of the data?  I 
would assume that BLAKE3, like other hashing algorithms, would be 
constant-space?

I often process packets up to about 256GB in size, so I assume 
that would lead to about 128MB of RAM usage.  That's not ideal on 
some of the lower-powered systems I use, particularly Raspberry 
Pis, but I can probably live with it in general (the 
lowest-powered ones don't tend to process data at that size). 
Would I be correct in assuming that this is used in pretty much 
all utilities?  nncp-file/exec, call, daemon, toss?

I have been meaning to write a message about packet redirection as 
well.  I don't know if this makes it easier - I don't think so - 
but I thought I might mention.

So I have a use case like this.

My node "laptop" sends backups to a machine on the LAN, I'll call 
it "baksvr".  Usually I only want it to send backups over the LAN 
(in case I'm tethered to 4G or something), but occasionally I want 
it to send backups remotely.

Now the machine on my home LAN isn't reachable from the internet, 
but it does have an NNCP link to a VPS out there, which my laptop 
could reach.

So I have packets queued up in my tx directory on the laptop for 
baksvr, but I want to redirect them via the VPS.

I eventually wound up using nncp-bundle to bundle them up, sent 
that bundle across NNCP, and unpacked it on the destination.  That 
can even be automated with nncp-exec -- but it might be nice to 
have something like "nncp-redir" that could redirect queued 
packets, adding additional onion routing layers around what's 
there already.

- John