Subject: Dependency collection takes very long

From: spacefrogg <[email-masked]>
Date: Thu, 28 Sep 2023 12:05:08 +0000

Hi!

I have a not-so-small project. The final target has about 5700 
dependencies boiling down to 1300 unique dependencies, both numbers 
include transient dependencies. redo-targets reports about 270 targets, 
so the rest are source files (≈1050).

The first build runs very smooth (2min single job, 50secs with 11 jobs). 
Starting with the second build, though, goredo takes tens of seconds of 
busy CPU work after logging "dbg collecting deps" in the debug output. 
`iotop` shows that it is busy doing I/O.

N.B. `time` records about 2m50s real time while redo only counts 1m23s. 
So, some significant code executes before redo starts counting the time.

As a comparison, apenwarr/redo needs a fraction of a second to check all 
dependencies. (It does no source file hashing, though.)

redo-targets returns instantly, but redo-sources gets stuck as well, 
thrashing the disk at a rate of 12MB/s taking almost 11m to finish. This 
means it touched almost 8GB!  So, I assume that something is off when 
redo encounters a source file.

I did not change the default REDO_INODE_TRUST. So, I assume redo is 
using ctime checking. My ctimes are reliable. I tested with goredo 1.30 
and 1.32.

By the looks of it, redo always hashes source files. But even if, this 
takes way too much time for the actual amount of data (≈120MB).

–Michael