Subject: Re: Issues with very large packets

From: John Goerzen <[email-masked]>
Date: Fri, 19 Feb 2021 13:18:29 -0600

On Fri, Feb 19 2021, Sergey Matveev wrote:

> Greetings!
>
> *** John Goerzen [2021-02-18 15:35]:
>>First, nncp-stat was extremely slow on a system that had packets 
>>like that
>>queued for transmission.  I'm wondering if it is trying to read 
>>all the
>>packets, even when called with no parameters?
>>It makes me wonder if nncp-daemon was doing some sort of 
>>expensive scan at
>>the beginning of the call, and either it or the OS cached the 
>>results?
>
> Both of them, as all other commands uses (very simple) 
> src/jobs.go code
> to get the list of available encrypted packets in the spool. It 
> does
> I/O, but I would not call it expensive:

Indeed that doesn't seem like it should be very slow.  I will see 
if I can do some experimentation with strace next week (when I'll 
next have packets of this size) and see if I can validate it.

I do also sometimes have about a thousand packets in the rx 
directory.  That slows it down too, but to the point of taking a 
few dozen seconds, not to the point of taking a minute+ as this 
is.  Though of course to read an entire file of this size would 
take a lot more than a minute, so it wasn't that.  I am sure I saw 
this when there were way less than 1000 packets in the directory, 
and also reasonably sure that I saw it when there were 3.

Random other question: do .seen files impact this at all?

> metainformation getting. I am talking much about ZFS there only 
> because
> reading of that little piece of information will be more 
> expensive on
> it, comparing to other filesystems (and I think it is ok, 
> because in
> most real world use-cases you wish to read the whole file after 
> that).

It is indeed ZFS here.

> If we will keep that metainformation nearby, then it should help 
> much:
>
> * Keeping separate database/cache-file of course is not an 
> option,
>   because of complexity and raised consistency questions.

Agreed.  This use case is not worth a lot of optimizations, it 
makes sense not to have it in the filename.  .meta could work but 
again this one little thing may not justify it.  (Of course it 
should still be encrypted if it's in .meta or xattrs)

> * That kind of information can also be kept in filesystem's 
> extended
>   attributes. Honestly I have never ever worked with xattrs at 
>   all.
>   Today was my first setextattr/getextattr command invocations. 
>   But
>   seems that it should be even more optimal and faster access
>   information than separate file on any filesystem. It xattrs 
>   are
>   missing (disabled?), then fallback to ordinary file reading
> * But as I can see, OpenBSD does not support xattrs at all. So 
> fallback
>   to separate ".meta" could be valuable anyway

I'd discourage that path.  xattrs are really add-ons in POSIXland. 
The support for them is often not present, sometimes broken, 
sometimes has weird compatibility issues.  Probably not worth the 
hassle.

> So seems I am going to write that header keeping in xattrs with 
> fallback
> to separate file storage.

Hey, if you want to <grin>  But I don't think it's really worth 
doing this for my (or really most any) use case.  If indeed you're 
reading just a few bytes from the start of the file, this wouldn't 
make much difference (or at least shouldn't) - and may even hurt, 
because now things like ls -l could have 2000 files to look at 
instead of 1000.

Thanks again!

- John