public inbox for nncp-devel@lists.stargrave.org
Atom feed
* NNCP at scale report
@ 2021-01-02 3:09 John Goerzen
2021-01-02 14:17 ` Sergey Matveev
0 siblings, 1 reply; 6+ messages in thread
From: John Goerzen @ 2021-01-02 3:09 UTC (permalink / raw)
To: nncp-devel
Hi folks,
I thought I would summarize how this is working for me! (tl;dr:
VERY WELL!)
I have 5 machines backing up ZFS over NNCP every hour. This
generates one packet per ZFS dataset, or around 100 packets each
hour. They are sent to a spooling machine, which queues them for
transmission to the backup host. (Not airgapped yet, but may be
soon.)
I have two drives that can be plugged to the backup host, and only
one is plugged in at a time; the idea is that one can go offsite.
There is a separate NNCP installation for each one. The spooling
machine receives the packets, and then distributes them, queueing
one for each target drive. The one for the online drive will be
picked up nearly immediately, but the others may way some days. I
could have had the sending machines just queue the data twice, but
that would have resulted in twice the size of transfer (possibly
over the Internet) and, well, we can't have that, right? :-)
So far I have observed over 2000 packets totaling about 70GB in
that spool when I swapped drives. I observe occasional "i/o
timeout" on the daemon side and "panic: send on closed channel" on
the caller side, but I have systemd restarting it. I also
observed the "data exceeds max slice limit" issue.
However - I wouldn't have even noticed those if not for watching
logs very closely. Every packet was delivered correctly and
intact.
The machines that send the backups generally rush through their
processing in a few seconds - much faster than in a synchronous
setup. For whatever reason, the Debian machines all run their
cron.hourly jobs at the same time each hour, so the nncp-daemon on
the spooler gets inundated with backups. Not a problem.
The packets range in size from a few hundred bytes to 7-10GB or
more. I had to switch away from using nncp-exec due to this (due
to RAM consumption). nncp-file is working fine, and then I have a
job that scans the incoming directories to process the packets.
This also has an advantage of providing more precise control over
processing order.
Some reflections on this setup:
- I really appreciate the NNCP security model. I could have run
this all through UUCP as well, but then I'd have to deal with a
bunch more security questions. Also I'm not sure UUCP deals
super well with enormous packets.
- The communications seem to be a bit prone to disconnections, but
no data is ever lost.
- One of the machines is the spooler. I want to use the same code
as everything else, so I use nncp-file to send to self.
Unfortunately nncp-toss doesn't see that. The workaround is to
call nncp-xfer /var/spool/nncptmp twice. It moves the files out
of tx, then into rx, and then it works :-)
- There are scenarios in which it may be helpful to redirect
packets. eg, A is sending to B but B is down, so A could wrap
the outbound packets to B in another layer of encryption to send
to C, and then C could send them on when it's up... but
admittedly this is somewhat niche.
- I wonder if it might be nice to have a nncp-maint - it could
rotate the log file, run some checks for very old packets,
remove old .seen files, etc. This can all be done with
conventional tools for those familiar with Unix administration,
but not everyone is.
Anyhow, thanks again for NNCP! Hard to believe I'm just now
getting into it, but it pretty much took vacation time for me to
have the time to redo my backups :-)
- John
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP at scale report
2021-01-02 3:09 NNCP at scale report John Goerzen
@ 2021-01-02 14:17 ` Sergey Matveev
2021-01-02 20:39 ` John Goerzen
2021-01-03 6:08 ` Shawn K. Quinn
0 siblings, 2 replies; 6+ messages in thread
From: Sergey Matveev @ 2021-01-02 14:17 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3455 bytes --]
Greetings!
*** John Goerzen [2021-01-01 21:09]:
>I thought I would summarize how this is working for me!
Thanks for sharing your interesting experience! I have got much lower
amounts of data transferred (but still many gigabytes) and less nodes.
All email is passing through NNCP for many years. "chunked" transfers is
done very often. But do not remember when I used nncp-bundle the last
time (but I like that creation :-))
>- One of the machines is the spooler. I want to use the same code as
>everything else, so I use nncp-file to send to self. Unfortunately nncp-toss
>doesn't see that. The workaround is to call nncp-xfer /var/spool/nncptmp
>twice. It moves the files out of tx, then into rx, and then it works :-)
Or you can just "mv path/to/self/tx/* path/to/self/rx/".
>- There are scenarios in which it may be helpful to redirect packets.
Technically this is possible: just wrap the existing packet to
transition wrapper and encrypt it. But current tools do not do that.
Think this is small niche, but it is not hard to to.
Also there is ability to override "via" via "-via" command line option,
ignoring hard-coded configuration. But of course it is done only for
newly created packets.
>- I wonder if it might be nice to have a nncp-maint - it could rotate the
>log file, run some checks for very old packets, remove old .seen files, etc.
>This can all be done with conventional tools for those familiar with Unix
>administration, but not everyone is.
Well, I think that it is preferrable people to learn more of Unix
administration, because it will bring much more benefit. When I have
added chunked transfer, that can be done manually with split+nncp-file
and somehow metainformation transfer -- it will reduce requirements for
temporary files or by burden of using dd with some kind of seeks. So it
was not so Unixy way, but very convenient. But adding completely the
same and similar kind of log-rotation utility, that already exists out
of box in every distribution... I do not see any benefits. I am sure
that every Unix user have to know how to rotate his logfiles. In
newsyslog it is just single line for NNCP log files rotation. If
nncp-maint will rotate the files, then anywat it will be completely the
same thing that you will have to learn how to configure and use. The
same applies to .seen packets removing, that is just single "find -delete"
call.
Of course it is hard to draw a line between that kind of decisions. For
example some time ago I wrote goredo (http://www.goredo.cypherpunks.ru/)
that has exactly that kind of command: removing temporary files,
captured logs and states. But, in goredo it can not be done easily with
single simple "find" invocation, because there are various different
files. All of them can be replaced with shell scripts (with "find"
commands), but goredo aims to be the single binary for convenience.
Moreover, goredo is a tool actively used interactively and temporary
files removing is not done as a cron-job. So in that case it is better
to create that kind of maintenance utility.
But NNCP has only single log file. Cleaning up is done with relatively
new nncp-rm utility, that can remove .seen files. Seems that adding
-anewer kind of "find" options will be useful however. Will look at this.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP at scale report
2021-01-02 14:17 ` Sergey Matveev
@ 2021-01-02 20:39 ` John Goerzen
2021-01-02 21:48 ` Sergey Matveev
2021-01-03 6:08 ` Shawn K. Quinn
1 sibling, 1 reply; 6+ messages in thread
From: John Goerzen @ 2021-01-02 20:39 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Sat, Jan 02 2021, Sergey Matveev wrote:
> Greetings!
>
> *** John Goerzen [2021-01-01 21:09]:
>>I thought I would summarize how this is working for me!
>
> Thanks for sharing your interesting experience! I have got much
> lower
> amounts of data transferred (but still many gigabytes) and less
> nodes.
> All email is passing through NNCP for many years. "chunked"
> transfers is
> done very often. But do not remember when I used nncp-bundle the
> last
> time (but I like that creation :-))
Yes, nncp-bundle is a very nice idea. I haven't used it much but
it has definite potential. Streaming formats are always great.
I'd be interested to hear more about your use case for chunked
transfers. I guess their main use would be if individual files
are too large for sneakernet? With the resumable transfers over
the network, though, I haven't bothered.
>>- One of the machines is the spooler. I want to use the same
>>code as
>>everything else, so I use nncp-file to send to self.
>>Unfortunately nncp-toss
>>doesn't see that. The workaround is to call nncp-xfer
>>/var/spool/nncptmp
>>twice. It moves the files out of tx, then into rx, and then it
>>works :-)
>
> Or you can just "mv path/to/self/tx/* path/to/self/rx/".
I wondered, but then I wasn't sure if there would be race
conditions with temporary files, half-written files, etc., and I
didn't want to risk it.
>>- There are scenarios in which it may be helpful to redirect
>>packets.
>
> Technically this is possible: just wrap the existing packet to
> transition wrapper and encrypt it. But current tools do not do
> that.
> Think this is small niche, but it is not hard to to.
>
> Also there is ability to override "via" via "-via" command line
> option,
> ignoring hard-coded configuration. But of course it is done only
> for
> newly created packets.
My thought was this... when I travel (hopefully we can do that
again this year!) my laptop would, when on a good network,
continue to send backups to the spooling server at my house.
However, sometimes something happens - power outage, Internet
outage, etc. In that case I would rather have the backups --
including the ones previously generated -- go spool on my server
where they could sit until the home network comes back up.
I also had the thought -- well I could probably just manually copy
them over to the appropriate tx queue on the server. Or if
nncp-xfer or nncp-bundle would have a feature where they would not
ignore, but instead ingest, packets whose next hop is any node
that the machine knows about, I could always just send them down a
nncp-bundle pipe.
> Well, I think that it is preferrable people to learn more of
> Unix
> administration, because it will bring much more benefit. When I
> have
As I think about this, you're probably right as far as whether
it's worth it to spend precious dev time on. I mean, it is very
nice to have a self-contained "maintenance free" program, but
people using NNCP are unlikely to be newbies at all this anyhow.
What might be useful would be a "maintenance" page in the manual,
just listing things that a person might do: rotate the log file,
look for old packets, look for old seen files.
Anyhow, if I knew Go I would totally be contributing patches at
this point.
If I had a few more weeks of vacation, I might try to learn it
just for this :-)
- John
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP at scale report
2021-01-02 20:39 ` John Goerzen
@ 2021-01-02 21:48 ` Sergey Matveev
0 siblings, 0 replies; 6+ messages in thread
From: Sergey Matveev @ 2021-01-02 21:48 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3415 bytes --]
*** John Goerzen [2021-01-02 14:39]:
>I'd be interested to hear more about your use case for chunked transfers. I
>guess their main use would be if individual files are too large for
>sneakernet? With the resumable transfers over the network, though, I haven't
>bothered.
Either they are too large for storages used in the sneakernet, or spool
directory does not have enough space for very big files. My main storage
on one of the servers is on the encrypted pool, but the spool itself is
not on encrypted disk (thus it is available after the reboot) and has
zfs set quota=2G on it. By placing "freq: {chunked: ...}" configuration
option for that server, all my nncp-file transfers automatically make
512MiB chunks if file is greater. When my spool is filled up (for
example when tossing is slower than network/whatever transfers) everyone
"waits" until there will be enough space. nncp-xfer checks for free
space and won't even try to copy files. nncp-daemon won't ask for file
download if there is not enough space too. And incoming NNCP directory
(for tossed files) is placed on big encrypted pool. But not simply on
the pool, but on the separate dataset with ZFS deduplication turned on.
That prevents writing on the disk when files are reassembled. Because
chunks are just simply concatenated together, and when NNCP and ZFS
block boundaries are "in sync", then deduplication detects that
reassembled file's block contents are identical and won't write copy on
the disk (because the same block are already here).
>I wondered, but then I wasn't sure if there would be race conditions with
>temporary files, half-written files, etc., and I didn't want to risk it.
Temporary files are placed in tmp/ subdirectory. .part-files can not
exist in tx/ directory. And any partly written file either has .part
suffix (in rx/ directories), or it is placed in tmp/ and then atomically
renamed to necessary file in tx/ directory. So if there is a file in
tx/, then it is complete fsync-ed fully written file with checksum used
its filename.
>My thought was this... when I travel (hopefully we can do that again this
>year!) my laptop would, when on a good network, continue to send backups to
>the spooling server at my house. However, sometimes something happens - power
>outage, Internet outage, etc. In that case I would rather have the backups
>-- including the ones previously generated -- go spool on my server where
>they could sit until the home network comes back up.
I see. Well, currently only manual interaction can help with that
situations.
>I also had the thought -- well I could probably just manually copy them over
>to the appropriate tx queue on the server. Or if nncp-xfer or nncp-bundle
>would have a feature where they would not ignore, but instead ingest, packets
>whose next hop is any node that the machine knows about, I could always just
>send them down a nncp-bundle pipe.
I will think what can be done easily. Probably some kind of additional
nncp- utility for transitional packets.
>What might be useful would be a "maintenance" page in the manual, just
>listing things that a person might do: rotate the log file, look for old
>packets, look for old seen files.
That is good idea. Definitely will write it.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP at scale report
2021-01-02 14:17 ` Sergey Matveev
2021-01-02 20:39 ` John Goerzen
@ 2021-01-03 6:08 ` Shawn K. Quinn
2021-01-03 11:47 ` Sergey Matveev
1 sibling, 1 reply; 6+ messages in thread
From: Shawn K. Quinn @ 2021-01-03 6:08 UTC (permalink / raw)
To: nncp-devel
On 1/2/21 08:17, Sergey Matveev wrote:
> [John Goerzen originally wrote:]
>> - I wonder if it might be nice to have a nncp-maint - it could rotate the
>> log file, run some checks for very old packets, remove old .seen files, etc.
>> This can all be done with conventional tools for those familiar with Unix
>> administration, but not everyone is.
> Well, I think that it is preferrable people to learn more of Unix
> administration, because it will bring much more benefit.
Is getting NNCP working under Windows not a long-term goal? If it is, we
need some simple tool like an nncp-maint that's not going to require a
whole bunch of command line song and dance just to do basic cleanup.
It would be nice to have a GUI front-end to NNCP as well, especially for
possible use on phones (making it much easier to relay packets to/from
some airgapped machines) but also to reduce tedium even on GNU/Linux and
other Unix-like operatings systems. The command-line tools are great for
use on laptops and desktops with suitable Unix-like operating systems
and work well for automation, but definitely restrict the audience to
those comfortable typing in commands, which is unfortunately an
ever-decreasing percentage of computer users.
I admit that I resisted the GUI revolution for many years, but today I'd
say I do at least 90% of my work using GUI software, though it's rare to
see me without at least one terminal window open. I have found it's
really not the GUI I hate, it's Windows and/or being forced to use the
GUI even for tasks where it adds tedium instead of reducing it.
--
Shawn K. Quinn <skquinn@rushpost•com>
http://www.rantroulette.com
http://www.skqrecordquest.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP at scale report
2021-01-03 6:08 ` Shawn K. Quinn
@ 2021-01-03 11:47 ` Sergey Matveev
0 siblings, 0 replies; 6+ messages in thread
From: Sergey Matveev @ 2021-01-03 11:47 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]
*** Shawn K. Quinn [2021-01-03 00:08]:
>Is getting NNCP working under Windows not a long-term goal?
Personally I am absolutely not interested in that. I am sure that people
must use free software. Period. If one chooses non-free software, then...
let it be, but for me that person does not "exist" at all. That is
completely unacceptable life choice, path of struggle and permanent pain
(in my opinion :-)). The same applies to users of mobile devices too.
If someone wish to use NNCP on that proprietary platforms, I have
nothing against. It is free software -- do as you wish, it is your
choice. But I won't help anyhow (spend my time) in those tasks, because
they are completely useless and even harmful (people should turn to free
software), as all proprietary software. Yes, I am in the "Church" of
free software movement :-), no offence to other people's choices.
>I admit that I resisted the GUI revolution for many years, but today I'd
>say I do at least 90% of my work using GUI software, though it's rare to
>see me without at least one terminal window open. I have found it's
>really not the GUI I hate, it's Windows and/or being forced to use the
>GUI even for tasks where it adds tedium instead of reducing it.
I have nothing against GUI too, *if* it is really good choice for the
task. In (my) practice, very few tasks in *my* life required GUI tools.
I always run X.org, but only because it is the easiest way to configure
fonts and use multiple languages (I need cyrillic). But only seldom
(when I need to watch pages with images or possibly use Gitlab interface
once per week) I run GUI Web-browser (most of time I use Lynx) and, of
course except for video and images watching and videogames, no GUI
programs anymore, because there are no tasks that can be conveniently
done in GUI.
Personally I am not interested in users of proprietary
software/platforms at all (mobile ones are included here too, because in
practice all that mobile platforms are completely unworkable without
non-free software). I am also not interested in users of OS/2, MS-DOS,
BeOS, TempleOS systems for various reasons. Windows/macOS/iOS/Android is
the same category for me -- parallel road.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-01-03 11:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-02 3:09 NNCP at scale report John Goerzen
2021-01-02 14:17 ` Sergey Matveev
2021-01-02 20:39 ` John Goerzen
2021-01-02 21:48 ` Sergey Matveev
2021-01-03 6:08 ` Shawn K. Quinn
2021-01-03 11:47 ` Sergey Matveev