public inbox for nncp-devel@lists.stargrave.org
Atom feed
From: John Goerzen <jgoerzen@complete•org>
To: nncp-devel@lists.cypherpunks.ru
Subject: NNCP at scale report
Date: Fri, 01 Jan 2021 21:09:50 -0600 [thread overview]
Message-ID: <875z4gm3up.fsf@complete.org> (raw)
Hi folks,
I thought I would summarize how this is working for me! (tl;dr:
VERY WELL!)
I have 5 machines backing up ZFS over NNCP every hour. This
generates one packet per ZFS dataset, or around 100 packets each
hour. They are sent to a spooling machine, which queues them for
transmission to the backup host. (Not airgapped yet, but may be
soon.)
I have two drives that can be plugged to the backup host, and only
one is plugged in at a time; the idea is that one can go offsite.
There is a separate NNCP installation for each one. The spooling
machine receives the packets, and then distributes them, queueing
one for each target drive. The one for the online drive will be
picked up nearly immediately, but the others may way some days. I
could have had the sending machines just queue the data twice, but
that would have resulted in twice the size of transfer (possibly
over the Internet) and, well, we can't have that, right? :-)
So far I have observed over 2000 packets totaling about 70GB in
that spool when I swapped drives. I observe occasional "i/o
timeout" on the daemon side and "panic: send on closed channel" on
the caller side, but I have systemd restarting it. I also
observed the "data exceeds max slice limit" issue.
However - I wouldn't have even noticed those if not for watching
logs very closely. Every packet was delivered correctly and
intact.
The machines that send the backups generally rush through their
processing in a few seconds - much faster than in a synchronous
setup. For whatever reason, the Debian machines all run their
cron.hourly jobs at the same time each hour, so the nncp-daemon on
the spooler gets inundated with backups. Not a problem.
The packets range in size from a few hundred bytes to 7-10GB or
more. I had to switch away from using nncp-exec due to this (due
to RAM consumption). nncp-file is working fine, and then I have a
job that scans the incoming directories to process the packets.
This also has an advantage of providing more precise control over
processing order.
Some reflections on this setup:
- I really appreciate the NNCP security model. I could have run
this all through UUCP as well, but then I'd have to deal with a
bunch more security questions. Also I'm not sure UUCP deals
super well with enormous packets.
- The communications seem to be a bit prone to disconnections, but
no data is ever lost.
- One of the machines is the spooler. I want to use the same code
as everything else, so I use nncp-file to send to self.
Unfortunately nncp-toss doesn't see that. The workaround is to
call nncp-xfer /var/spool/nncptmp twice. It moves the files out
of tx, then into rx, and then it works :-)
- There are scenarios in which it may be helpful to redirect
packets. eg, A is sending to B but B is down, so A could wrap
the outbound packets to B in another layer of encryption to send
to C, and then C could send them on when it's up... but
admittedly this is somewhat niche.
- I wonder if it might be nice to have a nncp-maint - it could
rotate the log file, run some checks for very old packets,
remove old .seen files, etc. This can all be done with
conventional tools for those familiar with Unix administration,
but not everyone is.
Anyhow, thanks again for NNCP! Hard to believe I'm just now
getting into it, but it pretty much took vacation time for me to
have the time to redo my backups :-)
- John
next reply other threads:[~2021-01-02 3:11 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-02 3:09 John Goerzen [this message]
2021-01-02 14:17 ` NNCP at scale report Sergey Matveev
2021-01-02 20:39 ` John Goerzen
2021-01-02 21:48 ` Sergey Matveev
2021-01-03 6:08 ` Shawn K. Quinn
2021-01-03 11:47 ` Sergey Matveev