Subject: Re: redo-ifchange hangs when called with many arguments

From: goredo <[email-masked]>
Date: Tue, 19 Oct 2021 14:56:27 +0000 (UTC)

> Well, basically it should depend solely on OS. Of course except for
> maximal command line arguments number, that is the main limiting factor
> of redo. I wonder why DJB decided to pass dependencies as arguments (as
> tar), instead of reading them from stdin (as cpio/pax), that would not
> artificially limit their number.

I would have expected that, too. I use xargs to pass unknown number of arguments to redo-ifchange, as it manages maximum arg len.

> I do not clearly remember (possibly I am mistaken), but I think I met
> that kind of behaviour last year when I run several thousand targets at
> once in parallel. I do not remember how I debugged and traced it, but
> the problem was in touching limits of opened file descriptors, that I
> fixed in 1.2.0 release (it was non-optimal resources usage, but everything
> got stuck at OS level limits). I also noticed that my FreeBSD has nearly
> two million opened files limit, but some version of Ubuntu just only one
> thousand.

So, I may have hit the bug again? Resource exhaustion should have led to the OS killing the redo-ifchange process or would opening another file just fail and redo does not always check if that succeeded? Because it really looks like it's waiting for something that does never happen. In case it hits the open file limit. Redo could just wait for already spawned jobs to finish or serialise its dependency checking.

> According to the code (and I checked it in practice), it should not
> terminate any of its processes at all :-). I added "infanticide" to it
> in 1.17.0 release.

:) Thanks! I may have never noticed, because the other targets usually run quickly.