Subject: NNCP at scale report

From: John Goerzen <[email-masked]>
Date: Fri, 01 Jan 2021 21:09:50 -0600

Hi folks,

I thought I would summarize how this is working for me!  (tl;dr: 
VERY WELL!)

I have 5 machines backing up ZFS over NNCP every hour.  This 
generates one packet per ZFS dataset, or around 100 packets each 
hour.  They are sent to a spooling machine, which queues them for 
transmission to the backup host.  (Not airgapped yet, but may be 
soon.)

I have two drives that can be plugged to the backup host, and only 
one is plugged in at a time; the idea is that one can go offsite. 
There is a separate NNCP installation for each one.  The spooling 
machine receives the packets, and then distributes them, queueing 
one for each target drive.  The one for the online drive will be 
picked up nearly immediately, but the others may way some days.  I 
could have had the sending machines just queue the data twice, but 
that would have resulted in twice the size of transfer (possibly 
over the Internet) and, well, we can't have that, right? :-)

So far I have observed over 2000 packets totaling about 70GB in 
that spool when I swapped drives.  I observe occasional "i/o 
timeout" on the daemon side and "panic: send on closed channel" on 
the caller side, but I have systemd restarting it.  I also 
observed the "data exceeds max slice limit" issue.

However - I wouldn't have even noticed those if not for watching 
logs very closely.  Every packet was delivered correctly and 
intact.

The machines that send the backups generally rush through their 
processing in a few seconds - much faster than in a synchronous 
setup.  For whatever reason, the Debian machines all run their 
cron.hourly jobs at the same time each hour, so the nncp-daemon on 
the spooler gets inundated with backups.  Not a problem.

The packets range in size from a few hundred bytes to 7-10GB or 
more.  I had to switch away from using nncp-exec due to this (due 
to RAM consumption).  nncp-file is working fine, and then I have a 
job that scans the incoming directories to process the packets. 
This also has an advantage of providing more precise control over 
processing order.

Some reflections on this setup:

- I really appreciate the NNCP security model.  I could have run 
  this all through UUCP as well, but then I'd have to deal with a 
  bunch more security questions.  Also I'm not sure UUCP deals 
  super well with enormous packets.

- The communications seem to be a bit prone to disconnections, but 
  no data is ever lost.

- One of the machines is the spooler.  I want to use the same code 
  as everything else, so I use nncp-file to send to self. 
  Unfortunately nncp-toss doesn't see that.  The workaround is to 
  call nncp-xfer /var/spool/nncptmp twice.  It moves the files out 
  of tx, then into rx, and then it works :-)

- There are scenarios in which it may be helpful to redirect 
  packets.  eg, A is sending to B but B is down, so A could wrap 
  the outbound packets to B in another layer of encryption to send 
  to C, and then C could send them on when it's up...  but 
  admittedly this is somewhat niche.

- I wonder if it might be nice to have a nncp-maint - it could 
  rotate the log file, run some checks for very old packets, 
  remove old .seen files, etc.  This can all be done with 
  conventional tools for those familiar with Unix administration, 
  but not everyone is.

Anyhow, thanks again for NNCP!  Hard to believe I'm just now 
getting into it, but it pretty much took vacation time for me to 
have the time to redo my backups :-)

- John