Subject: Re: Issues with very large packets

From: John Goerzen <[email-masked]>
Date: Sat, 20 Feb 2021 22:31:12 -0600

On Sat, Feb 20 2021, Sergey Matveev wrote:

> much more linearly on the disk, much more denser, because of 
> sequential
> writings of checkpoints on ZFS. And after that nncp-stat works a
> magnitude faster:
>         1.62 real         0.01 user         0.04 sys

Impressive!

> Also I have made support of ".nock" (non-checksummed) files. 
> After
> online daemons receives the file, it is renamed from ".part" to 
> ".nock".
> By default it is checksummed in the background and then renamed 
> to
> ordinary fully verified packet. But with -nock 
> command-line/configuration
> option you can disable its checksumming at all, doing that after
> daemon/caller invocations with nncp-check command.

Interesting.  How do these different options interact with 
removing the packet from the remote end?

I gather that, right now, the packet is preserved on the remote 
end until the recipient has checked and verified the checksum. 
With background checksumming, I assume that could happen later, 
possibly even after the call drops.  Will the sender still keep 
the packet around until the recipient confirms the checksum?  If 
not, that could result in data loss.  And if so, how exactly would 
that happen, especially since the call my drop by then?  (eg, 
onlinedeadline of 20s and it takes 200s to checksum a big file)  I 
worry there may be a race condition there where nncp-toss may see 
and receive a successfully-processed file before this can be 
indicated back to the sender.

> There is only one exception: if the file is received from the 
> very
> beginning (from zero offset), then it is hashed immediately in 
> the
> background during receiving. And if it is received till the end, 
> then no
> ".nock" is created and file is immediately renamed to ordinary 
> received
> packet (of course if checksum is good). If it is interrupted and 
> resumed
> later, then ".nock" is used anyway.

That's great!

> All of that is in the "develop" branch in git and everything 
> seems
> working. But currently I did not test it much, so there should 
> be some
> bugs. I will work a couple of days under that updated versions 
> and will
> make a release.

Thanks for all your work on this!

- John