* Re: [nncp-devel] Four questions on nncp
[not found] <87r30gpqgp.fsf@gmail.com>
@ 2017-04-25 14:31 ` Sergey Matveev
2017-04-27 19:53 ` Sergey Matveev
2017-04-28 10:35 ` Sergey Matveev
2 siblings, 0 replies; 6+ messages in thread
From: Sergey Matveev @ 2017-04-25 14:31 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 5757 bytes --]
Greetings!
*** Leah Neukirchen <leah@vuxu•org> [2017-04-25 16:27]:
>first of all, thanks for writing NNCP, I've been dreaming of such a
>tool for quite some time, and found it quite easy to set up.
Thanks for appreciation!
>1) AFAICS, nncp-xfer will transfer the whole packet at once, even if a
>partial nncp-call succeeded, is that how it's supposed to be?
>2) nncp-xfer with a quota would be useful... (say, I partition 10G of
>my notebook disk space for nncp transfers, so transfering a 20G file
>will take two roundtrips.)
Well, honestly, I have never thought about partial transmission through
removable media device. I assumed that those devices are big enough to
hold all necessary data... well, at least some packets. I have never
assumed that single packet won't fit on media device (unlike unreliable
network connection).
Seems I lied a little: personally I had several times situation when my
hard drive's volume could not hold all the data at once. But I just
split (using "split" command) input data on several files and send them
sequentially.
You can always manually call (tar+)split command, but I understand that
it is not convenient. Possibly it is useful to built-in some kind of
that functionality to NNCP itself. For example to tell it to split data
on chunks automatically and then join them together somehow.
I think that really partial xfer-ing is not to so trivial to do. You
have to do robust atomic transactions and I do not know to do them
easily. It is easier to split packet/file on chunks. Seems that packet's
format will be altered, holding that chunks information.
I will about that that more deeply sooner. It is really good idea!
>3) It would be nice if nncp-file could read from stdin (- doesn't
>work, and /dev/stdin results in size 0).
Unfortunately I see big problems with ability to stream encrypted
packets (http://www.nncpgo.org/Encrypted.html). To be able to just read
an encrypted stream and process it on the fly, you have to know the size
you must read from the socket/descriptor/whatever. You can not just read
the stream until (for example) you found some authenticated encrypted
data in it -- this will leak information to the adversary from
cryptographic point of view. That kind of attack was in first version of
OpenSSH protocol -- by determining how many bytes is read from the
socket you can determine some information about an encrypted size
packet. So it can not be at the end of the stream. It also can not be at
the beginning of the stream, by not knowing it in advance, because you
have to stream out that encrypted packet.
Well, possibly you can generate some random delimiter (for example
256-bits of random), put somewhere in the beginning of encrypted packet
and then put it at the end. Then, when you read input encrypted stream,
you just search for that delimiter, that is not encrypted. And
encrypted/authenticated information about the size will precede that
delimiter. But, again, you have to save that streamed data somewhere
temporarily and make packet processing two-pass: at first we just parse
the header, get delimiter and save the stream until we find delimiter in
the stream; and then we decrypt/authenticate that temporarily saved
stream. But, again, single pass streaming processing is beautiful.
Another possibility is just to save stdin to temporary file and then
process it after it is saved and stdin is empty. This is acceptable, but
of course anyway requires disk space to store the whole stream and then
to store an encrypted packet. That temporary file could be encrypted on
the fly, so possibly it is useful not to require user to save input data
manually on either encrypted drive or memory tmpfs. This is the only
good reason I could see to store stdin in temporary file, encrypting it
on the fly. Will it be useful, what is your opinion?
>4) Say I didn't want to expose the port for nncp-daemon, but tunnel
>over SSH instead, how would you do it?
Well, it could be tunneled quite easily, because it uses single simple
TCP connection. For example you could create listening TCP port on
localhost and connect with nncp-call to it:
% ssh -N -L 5400:localhost:5400 remote.example.com
# remote.example.com is running nncp-daemon on localhost:5400
% nncp-call remote_node_name localhost:5400
# to forcefully call remote_node_name with specified address:port
Remote localhost's 5400 port will be forwarded to local 5400 one through
the tunnel.
But pay attention that so-called Sync protocol (http://www.nncpgo.org/Sync.html)
is explicitly created with security in mind: it uses Noise-IK protocol
to encrypt/authenticate all packets and authenticate peers. Personally I
created it to omit SSH tunnels usage at all, that I used with UUCP
previously. And I trust its security (from cryptographic point of view)
more, comparing to OpenSSH.
Possible idea is to run as rsync: by talking to stdin/stdout, to deal
with SSH connections without an additional encryption/authentication
overhead, completely trusting SSH. I am not sure about that, because
personally I would run rsync+ssh from time to time to sync
inbound/outbound directories. In my opinion, if you have got working SSH
connection, then rsync is working is good enough on it too, so no reason
to use NNCP's Sync protocol, except to lower traffic usage. But if
traffic usage is important, then possibly it is better to make
nncp-daemon (it has very compact packets) visible to outer world? Just
not sure about its usefulness ("raw" protocol without Noise encryption).
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [nncp-devel] Four questions on nncp
[not found] <87r30gpqgp.fsf@gmail.com>
2017-04-25 14:31 ` [nncp-devel] Four questions on nncp Sergey Matveev
@ 2017-04-27 19:53 ` Sergey Matveev
2017-04-27 20:05 ` Sergey Matveev
2017-04-27 20:18 ` Shawn K. Quinn
2017-04-28 10:35 ` Sergey Matveev
2 siblings, 2 replies; 6+ messages in thread
From: Sergey Matveev @ 2017-04-27 19:53 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3083 bytes --]
Greetings, again!
*** Leah Neukirchen <leah@vuxu•org> [2017-04-25 16:27]:
>1) AFAICS, nncp-xfer will transfer the whole packet at once, even if a
>partial nncp-call succeeded, is that how it's supposed to be? I'm not
>sure nncp-xfer can do partial transmissions?
>2) nncp-xfer with a quota would be useful... (say, I partition 10G of
>my notebook disk space for nncp transfers, so transfering a 20G file
>will take two roundtrips.)
As I said in previous mail, I decided to implement automatic splitting
of files. Its draft initial version is already somehow working:
https://git.cypherpunks.ru/cgit.cgi/nncp.git/commit/?h=develop&id=66a1f121c12a73e49848e7946ceda8b75095b32e
If you specify -chunk XXX (where XXX is chunk size in KiBs) option to
nncp-file, then it will transmit not the whole file, but:
FILE.nncp.meta, FILE.nncp.part0, FILE.nncp.part1, .... Meta file
contains full FILE's size and all hashes of each of its part. nncp-toss
does NOT automatically do anything with them, except just saving as an
ordinary files. I thought about that and it seems very cumbersome to do
something at once. It is much more simpler to reassemble the whole file
in a separate process.
So there appeared nncp-reass utility that searches for those meta-files,
determines do we have enough chunks available, verifies their integrity
and reassembles the file (all those parts can be reassembled manually
just doing cat FILE.part0 FILE.part1 > FILE).
This is rather simple changes, but they allow to prepare huge quantity
of data for transferring through small storage devices. When all parts
are available -- you can reassemble the whole file.
Some notes:
* creating of outbound packets (partXXX, .meta) is done streamingly
* reassembling process, after all integrity tests are passed, simply
copies all chunks in the destination file, simultaneously removing
them at once (some kind of: cat partXXX >> file ; rm partXXX ; ...),
so you have to have free space only for single chunk additionally. If
you have got 100 GB filesystem and want to transmit 99 GB file with
1GB chunks, then you must only have 99+1 GB of free space
* if you have got filesystem with deduplication capabilities, then all
those cat * >> file should be done very efficiently, without any
additional space requirements
I will write more documentation about that feature soon.
So, with your usecase when you have got relatively small transferable
media: you just
* nncp-file -chunk XXX HUGEFILE
* nncp-xfer to transferable device until disk space allows to hold at
least some of packets (you can always use -nice for prioritizing
packets)
* carry device to remote side
* nncp-xfer on remote side to copy that packets from device
* nncp-toss to process encrypted packets and save some of HUGEFILE's chunks
* repeat that process with nncp-xfer-s until all remaining are done
* nncp-reass on remote side to reassemble HUGEFILE
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread