public inbox for goredo-devel@lists.stargrave.org
Atom feed
* Potentially unnecessary redoing of targets
@ 2026-01-12 9:02 Niklas Böhm
2026-01-12 12:53 ` spacefrogg
0 siblings, 1 reply; 6+ messages in thread
From: Niklas Böhm @ 2026-01-12 9:02 UTC (permalink / raw)
To: goredo-devel
Hello,
I noticed that there is some behavior with ood detection that can lead
to unnecessary redoing of a target.
If you have the two simple files:
1.do:
echo 1
2.do:
redo-ifchange 1
sleep 3
echo 2
then calling `redo-ifchange 2` will run them, as intended. When adding
a comment to 1.do and again calling `redo-ifchange 2`, it will correctly
redo the file `1` and afterwards redo itself. This is kind of the
crucial point. If you Ctrl-C the redoing process while it's sleeping
during execution of `2.do`, then it will (correctly, in my opinion) not
redo the file `2` in a subsequent call to `redo-ifchange 2` since all of
the dependencies are unchanged.
To me, this behavior could also be extended to be performed during
execution of redo itself. In this example, this would mean to check,
again, if the dependencies have differing output before then starting
the process of redoing `2`.
I acknowledge that this is not a trivial change in itself, though the
additional computation time should be negligible, as the hash of `1` is
already computed. It would mean to change the ood detection to not
declare `2` ood only based on `1.do` differing, but instead to check
this after redoing `1`.
Let me know what you think about this proposed change!
Greetings
Nik
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potentially unnecessary redoing of targets
2026-01-12 9:02 Potentially unnecessary redoing of targets Niklas Böhm
@ 2026-01-12 12:53 ` spacefrogg
2026-01-12 15:12 ` Niklas Böhm
0 siblings, 1 reply; 6+ messages in thread
From: spacefrogg @ 2026-01-12 12:53 UTC (permalink / raw)
To: goredo-devel
You are describing two different cases.
In the first, the execution of `2.do` has already started (due to OOD).
This could result in anything, even in dropping `1` as a dependency
entirely. The assumption that `redo-ifchange 1` resulting in an
unchanged output ALSO results in not changing the output of `2` is just
wrong. Anything could happen depending on what the content of `1` means
in the context of executing `2.do`. It could even result in never
running `1.do`. The (implicit) contract is that ONLY `2.do` may decide
what the content of `1` means. In addition, there is also the
(implicit) assumption that the meaning of `1` should only depend on the
current content of `1` and not on whether/how it has changed from the
previous run.
In the second case, OOD has just determined that there is nothing to do.
You only reached that case by (nearly) compromising the build flow,
which is (or should be) out of scope for the design of redo. Actually,
you would have reached the same case by running `redo-ifchange 1;
redo-ifchange 2`. There, the first `redo-ifchange` also "clears" the OOD
state of `1`.
So, to make the distinction clear: In your case, which was running
`redo-ifchange 2`, `2` being OOD because of `1` just results in running
`2.do`. That's it. Whether that means to also run `1.do` is entirely
up to the program in `2.do`. Running `1.do` is not the result of `1`
being determined OOD but of the command `redo-ifchange 1` being part of
`2.do`.
The two cases have nothing in common, really.
Kind regards,
Michael
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potentially unnecessary redoing of targets
2026-01-12 12:53 ` spacefrogg
@ 2026-01-12 15:12 ` Niklas Böhm
2026-01-12 16:26 ` spacefrogg
2026-01-13 8:23 ` Sergey Matveev
0 siblings, 2 replies; 6+ messages in thread
From: Niklas Böhm @ 2026-01-12 15:12 UTC (permalink / raw)
To: goredo-devel
Thanks for your response. In general, I want to say that `2` is not ood
because `1` did not change after completion of `1.do`, hence we would
never have to start it. The state when `2.do` is executing is in fact
not ood, but because we determined `1` to be ood (as 1.do that changed),
we run `2.do` anyways. What I would suggest is to check, once more, if
the targets actually did change their content before launching `2.do`.
I see that currently the problem is that the ood checking goes as
follows (with redo-ifchange -d):
[48985] dbg ood: 2 checking
[48985] dbg ood: 2 -> 2.do: checking
[48985] dbg ood: 2 -> 2.do: same inode
[48985] dbg ood: 2 -> 2.do: is source
[48985] dbg ood: 2 -> 1: checking
[48985] dbg ood: 2 -> 1: inode differs
[48985] dbg ood: 2 -> 1: same hash
[48985] dbg ood: . 1 checking
[48985] dbg ood: . 1 -> 1.do: checking
[48985] dbg ood: . 1 -> 1.do: inode differs
[48985] dbg ood: . 1 -> 1.do: hash differs
[48985] dbg ood: . 1: true
[48985] dbg ood: 2 -> 1: ood
[48985] dbg ood: 2: true
I want to make the claim that `2` is not necessarily ood at that point,
as we could first redo `1` and then check if the output differs before
determining `2` to be ood. As the build file `2.do` did not change, we
know that its dependencies do not change; hence in my opinion it is a
forgone conclusion to say that `2` is ood only because `1.do` has a
differing hash, but actually `1` itself does not.
I'll admit that it might be an involved change in the code base and will
also make redo itself change its behavior (as you lose the transitivity
property for ood detection), but in my eyes it would be correct to check
the direct targets of the .do file, instead of cascading the ood
property. This would also make the build system minimal
Anyways, thanks for reading all of this text, I hope my point became a
little clearer.
Cheers
Nik
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potentially unnecessary redoing of targets
2026-01-12 15:12 ` Niklas Böhm
@ 2026-01-12 16:26 ` spacefrogg
2026-01-13 8:23 ` Sergey Matveev
1 sibling, 0 replies; 6+ messages in thread
From: spacefrogg @ 2026-01-12 16:26 UTC (permalink / raw)
To: goredo-devel
On 2026-01-12 15:12, Niklas Böhm wrote:
> Thanks for your response. In general, I want to say that `2` is not
> ood because `1` did not change after completion of `1.do`, hence we
> would never have to start it. The state when `2.do` is executing is in
> fact not ood, but because we determined `1` to be ood (as 1.do that
> changed), we run `2.do` anyways. What I would suggest is to check,
> once more, if the targets actually did change their content before
> launching `2.do`.
Thanks, I see the case you are constructing. But you are misjudging the
task that you've given redo to do in the first place.
What I was trying to convey in my previous message was, in essence,
that:
1) `redo-ifchange 1; redo-ifchange 2` and
2) `redo-ifchange 2`
are two separate cases that you are conflating with each other. You are
trying to turn case 2) into case 1), which is not what redo is about.
> I see that currently the problem is that the ood checking goes as
> follows (with redo-ifchange -d):
> [48985] dbg ood: 2 checking
> [48985] dbg ood: 2 -> 2.do: checking
> [48985] dbg ood: 2 -> 2.do: same inode
> [48985] dbg ood: 2 -> 2.do: is source
> [48985] dbg ood: 2 -> 1: checking
> [48985] dbg ood: 2 -> 1: inode differs
> [48985] dbg ood: 2 -> 1: same hash
> [48985] dbg ood: . 1 checking
> [48985] dbg ood: . 1 -> 1.do: checking
> [48985] dbg ood: . 1 -> 1.do: inode differs
> [48985] dbg ood: . 1 -> 1.do: hash differs
> [48985] dbg ood: . 1: true
> [48985] dbg ood: 2 -> 1: ood
> [48985] dbg ood: 2: true
This is exactly as it should go, because what you are trying to execute
is case 2). In case 2), the only decision redo has to make is whether
or not to execute `2.do`. It does NOT decide whether to execute `1.do`.
As I explained in my last message, executing `2.do` could result in
removing `1` (and by transitivity `1.do`) from its dependency graph
entirely. So, executing `1.do` just to find out whether you have to
execute `2.do` or, even worse, cancelling the execution of `2.do` after
the line `redo-ifchange 1` just because you've determined it no longer
to be OOD, is the wrong move.
Prematurely executing `1.do` before `2.do` (which is precisely turning
case 2) into case 1)) can be wrong for the following reasons:
- it could be removed from `2`'s dependency graph on the next run of
`2.do`. So it would be a waste of effort.
- it could be an after-dependency that is only to be executed after most
of `2.do` has run (I use after-dependencies to collect dependencies from
.tex files processes, for example). So running it eagerly would
potentially operate on old data. And even if not, it is up to the user
to place `redo-ifchange` calls in .do files. By running them early, you
virtually move them all to the front of the file, which could change the
results in non-obvious ways. This "feature" would also be very hard to
communicate.
> I want to make the claim that `2` is not necessarily ood at that point,
> as we could first redo `1` and then check if the output differs before
> determining `2` to be ood. As the build file `2.do` did not change, we
> know that its dependencies do not change; hence in my opinion it is a
> forgone conclusion to say that `2` is ood only because `1.do` has a
> differing hash, but actually `1` itself does not.
This is where you are mixing things up. Let me say the concluding
statement of my last message again: The call that produced above trace
was `redo-ifchange 2`. It's purpose is to determine, whether or not to
run `2.do`. That's it. For that decision, it uses the currently known
dependency graph of `2` as a starting point. According to this graph,
`2` IS OOD and `2.do` must be run. This does not mean anything with
regards to `1` or `1.do`, even if the alteration of `1.do` was the
cause. But the only way to get rid of this outdated knowledge (that `1`
is still a dependency of `2`) is to execute `2.do` and to find out.
> I'll admit that it might be an involved change in the code base and
> will also make redo itself change its behavior (as you lose the
> transitivity property for ood detection), but in my eyes it would be
> correct to check the direct targets of the .do file, instead of
> cascading the ood property. This would also make the build system
> minimal
As I was trying to clarify, such a system would be very different from
redo and would not allow for free placement of `redo-ifchange ...` calls
throughout .do files. Or, at least, these calls would have a different
meaning, depending on whether you run the .do file for the first time or
because it was determined to be OOD.
Believe me, I see, what you are trying to achieve, but with that change,
the semantics of redo and redo-ifchange calls become weird and
non-obvious.
Kind regards,
Michael
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potentially unnecessary redoing of targets
2026-01-12 15:12 ` Niklas Böhm
2026-01-12 16:26 ` spacefrogg
@ 2026-01-13 8:23 ` Sergey Matveev
2026-01-13 19:04 ` Niklas Böhm
1 sibling, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2026-01-13 8:23 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 971 bytes --]
Greetings!
*** Niklas Böhm [2026-01-12 16:12]:
>What I would suggest is to check, once more, if the targets
>actually did change their content before launching `2.do`.
redo just do not work that way. If it determines that "2" is
OOD (because it depends on "1", that depends on altered "1.do"),
then it runs 2.do. It does not run 1.do, but 2.do, that *may*
run 1.do. Even if we saw that result of "1" target was not
changed, we already executing 2.do anyway, maybe somewhere in
the middle of its code, with various steps already done before
encountering "redo-ifchange 1" in it.
spacefrogg perfectly explained all of that (and other issues,
like premature "1.do" execution) in details indeed.
Here is the crucial difference in behaviour between Make and
redo: Make, if 2 is OOD, will run 1(.do). Redo will run 2.do,
that may run 1.do.
--
Sergey Matveev (http://www.stargrave.org/)
LibrePGP: 12AD 3268 9C66 0D42 6967 FD75 CB82 0563 2107 AD8A
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potentially unnecessary redoing of targets
2026-01-13 8:23 ` Sergey Matveev
@ 2026-01-13 19:04 ` Niklas Böhm
0 siblings, 0 replies; 6+ messages in thread
From: Niklas Böhm @ 2026-01-13 19:04 UTC (permalink / raw)
To: goredo-devel
Hi again,
alright, I think I see the point now. I agree that the behavior would
definitely not seem intuitive and I suppose I am just not thinking of
some corner cases where this could be very confusing.
My point is that `2.do` specifically did not change; hence the
dependencies should stay the same, depending only on whether `1` changes
or not. If the hash stays the same, then there is no need to even start
the execution of `2.do`. In case the contents of `1` do change then all
bets are off, so I agree that it does not make sense to change the
behavior of redo.
Personally, I actually think that it would be nice to have
`redo-ifchange 1; redo-ifchange 2` and only `redo-ifchange 2` work the
same, although that would also lead to trouble when you rely on one of
those to produce files that are not properly tracked by redo.
In any case, thanks for the discussion and explanations.
Cheers
Nik
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-01-13 19:05 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-01-12 9:02 Potentially unnecessary redoing of targets Niklas Böhm
2026-01-12 12:53 ` spacefrogg
2026-01-12 15:12 ` Niklas Böhm
2026-01-12 16:26 ` spacefrogg
2026-01-13 8:23 ` Sergey Matveev
2026-01-13 19:04 ` Niklas Böhm