public inbox for goredo-devel@lists.stargrave.org
Atom feed
From: spacefrogg <spacefrogg-goredo@spacefrogg•net>
To: goredo-devel@lists.cypherpunks.su
Subject: Memory exhaustion issues
Date: Mon, 19 Aug 2024 12:12:59 +0200 [thread overview]
Message-ID: <3ed9ebbd49e87a5babd7bf9e5b0984ab@spacefrogg.net> (raw)
Hi,
In a fairly large project in the order of several 100k build targets I
face memory exhaustion issues. The build closure has roughly the
following shape:
- ca. 50 immediate dependencies max. in a single redo-ifchange call at
several levels of the dependency graph
- max dependency chain length of 14 levels
- 1 Python process per active target
The build is run with a single active job `-j 1`.
The following happens: For each target, the Python process eventually
waits on a redo-ifchange call to finish its sub targets. One could
expect that with a single job, only one process per dependency depth
level (+1 for each redo-if*) would be created until reaching the end of
the chain. But this is not what happens. Whenever a target calls
redo-if*, redo activates a next-in-line target, which is not necessarily
in the chain of the same process. Instead it is likely some other
target from an earlier redo-if* call up the dependency chain. This
leads to a large tree of processes, because the current chain is not
finished. Instead it is waiting on a redo-if* call while another chain
opens up – also eventually waiting on a redo-if* call and so forth.
This creates hundreds of new processes effectively sleeping all the time
(which is fine). Due to them being active processes, they consume a lot
of memory until exhaustion (which is not fine).
I believe the high fan out of 50 direct dependencies accommodates the
growth of the active process tree, because it makes selecting the next
target on a deeper level more unlikely.
It is obvious that using Python immediately contributes to the severity
of the issue but a) a change is out of the question for several reasons
and b) it is just uncovering a more fundamental issue which would have
hit the next guy on a project 3 times the size or so.
There is no way, using the current redo-if* interface or any of its
command-line options to control this development.
My suggestion would be that, if in any way possible, redo should try to
follow the dependency graph in depth-first-order to minimize the number
of half-executed targets waiting on redo-if* calls.
Thanks for reading!
–Michael
next reply other threads:[~2024-08-19 10:39 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-19 10:12 spacefrogg [this message]
2024-09-05 18:49 ` Memory exhaustion issues Sergey Matveev
2024-09-05 19:49 ` goredo
2024-09-08 10:11 ` Sergey Matveev
2024-09-09 10:49 ` spacefrogg
2024-09-10 6:50 ` Sergey Matveev