Subject: Memory exhaustion issues

From: spacefrogg <[email-masked]>
Date: Mon, 19 Aug 2024 12:12:59 +0200

Hi,

In a fairly large project in the order of several 100k build targets I 
face memory exhaustion issues.  The build closure has roughly the 
following shape:
- ca. 50 immediate dependencies max. in a single redo-ifchange call at 
several levels of the dependency graph
- max dependency chain length of 14 levels
- 1 Python process per active target

The build is run with a single active job `-j 1`.

The following happens: For each target, the Python process eventually 
waits on a redo-ifchange call to finish its sub targets.  One could 
expect that with a single job, only one process per dependency depth 
level (+1 for each redo-if*) would be created until reaching the end of 
the chain.  But this is not what happens.  Whenever a target calls 
redo-if*, redo activates a next-in-line target, which is not necessarily 
in the chain of the same process.  Instead it is likely some other 
target from an earlier redo-if* call up the dependency chain.  This 
leads to a large tree of processes, because the current chain is not 
finished. Instead it is waiting on a redo-if* call while another chain 
opens up – also eventually waiting on a redo-if* call and so forth.  
This creates hundreds of new processes effectively sleeping all the time 
(which is fine).  Due to them being active processes, they consume a lot 
of memory until exhaustion (which is not fine).

I believe the high fan out of 50 direct dependencies accommodates the 
growth of the active process tree, because it makes selecting the next 
target on a deeper level more unlikely.

It is obvious that using Python immediately contributes to the severity 
of the issue but a) a change is out of the question for several reasons 
and b) it is just uncovering a more fundamental issue which would have 
hit the next guy on a project 3 times the size or so.

There is no way, using the current redo-if* interface or any of its 
command-line options to control this development.

My suggestion would be that, if in any way possible, redo should try to 
follow the dependency graph in depth-first-order to minimize the number 
of half-executed targets waiting on redo-if* calls.

Thanks for reading!
–Michael