Subject: Re: Streaming data of unknown length

From: Sergey Matveev <[email-masked]>
Date: Wed, 9 Nov 2022 22:23:51 +0300

Greetings!

*** Jonathan Hemingway [2022-11-08 23:03]:
>In this case, what is the best way to use NNCP? A file-transfer with chunking?

Personally I have never tried real streaming use-cases with NNCP,
moreover only relatively recently it was able to send data without
storing it in temporary file (to determine resulting packet's size),
but seems that the only friendly way is indeed just to use chunked
transfer and to dump packets to remote side in background.

Although you won't be able to automatically reassemble (nncp-reass)
those chunks to the single file, because .nncp.meta (that holds all
information needed for reliable reassembling) is sent only when stdin is
dried up -- but .nncp.chunk files are numbered and hardly can be damaged
if successfully already tossed. You can just literally concatenate them
(preserving the order). .nncp.meta file additionally holds just chunk's
checksums.

But pay attention that while nncp-file is generating new chunks, it
accumulates their checksums (for resulting .nncp.meta) in memory.
32-byte for each chunk, but if it is long-running process with thousands
of chunks already generated, then it could noticeably grow.

Another problem I see is that you can fill your spool of encrypted
queued packets quicker than you transfer them. And soon you (nncp-file)
will just file because of lack of disk space. Another problem is that
each time nncp-call connects and online-protocol is activated -- there
is no particular order of packets delivery. None of NNCP component's
look at mtime/ctime/whatever and have got any kind of date-based
ordering. You can queue hundred of packets, but remote side will
download them in random order. But if amount of generated data is
definitely not a problem for available disk space and outgoing online
bandwidth, then seems it should work. However it is definitely not the
very good solution, taking in account that NNCP never tried to preserve
any kind of packets transmission ordering.

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A