public inbox for nncp-devel@lists.stargrave.org
Atom feed
From: Sergey Matveev <stargrave@stargrave•org>
To: Leah Neukirchen <leah@vuxu•org>
Cc: nncp-devel@lists.cypherpunks.ru
Subject: Re: [nncp-devel] Four questions on nncp
Date: Thu, 27 Apr 2017 22:53:11 +0300 [thread overview]
Message-ID: <20170427195311.GA12868@stargrave.org> (raw)
In-Reply-To: <87r30gpqgp.fsf@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3083 bytes --]
Greetings, again!
*** Leah Neukirchen <leah@vuxu•org> [2017-04-25 16:27]:
>1) AFAICS, nncp-xfer will transfer the whole packet at once, even if a
>partial nncp-call succeeded, is that how it's supposed to be? I'm not
>sure nncp-xfer can do partial transmissions?
>2) nncp-xfer with a quota would be useful... (say, I partition 10G of
>my notebook disk space for nncp transfers, so transfering a 20G file
>will take two roundtrips.)
As I said in previous mail, I decided to implement automatic splitting
of files. Its draft initial version is already somehow working:
https://git.cypherpunks.ru/cgit.cgi/nncp.git/commit/?h=develop&id=66a1f121c12a73e49848e7946ceda8b75095b32e
If you specify -chunk XXX (where XXX is chunk size in KiBs) option to
nncp-file, then it will transmit not the whole file, but:
FILE.nncp.meta, FILE.nncp.part0, FILE.nncp.part1, .... Meta file
contains full FILE's size and all hashes of each of its part. nncp-toss
does NOT automatically do anything with them, except just saving as an
ordinary files. I thought about that and it seems very cumbersome to do
something at once. It is much more simpler to reassemble the whole file
in a separate process.
So there appeared nncp-reass utility that searches for those meta-files,
determines do we have enough chunks available, verifies their integrity
and reassembles the file (all those parts can be reassembled manually
just doing cat FILE.part0 FILE.part1 > FILE).
This is rather simple changes, but they allow to prepare huge quantity
of data for transferring through small storage devices. When all parts
are available -- you can reassemble the whole file.
Some notes:
* creating of outbound packets (partXXX, .meta) is done streamingly
* reassembling process, after all integrity tests are passed, simply
copies all chunks in the destination file, simultaneously removing
them at once (some kind of: cat partXXX >> file ; rm partXXX ; ...),
so you have to have free space only for single chunk additionally. If
you have got 100 GB filesystem and want to transmit 99 GB file with
1GB chunks, then you must only have 99+1 GB of free space
* if you have got filesystem with deduplication capabilities, then all
those cat * >> file should be done very efficiently, without any
additional space requirements
I will write more documentation about that feature soon.
So, with your usecase when you have got relatively small transferable
media: you just
* nncp-file -chunk XXX HUGEFILE
* nncp-xfer to transferable device until disk space allows to hold at
least some of packets (you can always use -nice for prioritizing
packets)
* carry device to remote side
* nncp-xfer on remote side to copy that packets from device
* nncp-toss to process encrypted packets and save some of HUGEFILE's chunks
* repeat that process with nncp-xfer-s until all remaining are done
* nncp-reass on remote side to reassemble HUGEFILE
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2017-04-27 19:53 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <87r30gpqgp.fsf@gmail.com>
2017-04-25 14:31 ` [nncp-devel] Four questions on nncp Sergey Matveev
2017-04-27 19:53 ` Sergey Matveev [this message]
2017-04-27 20:05 ` Sergey Matveev
2017-04-27 20:18 ` Shawn K. Quinn
2017-04-27 20:40 ` Sergey Matveev
2017-04-28 10:35 ` Sergey Matveev