Potentially unnecessary redoing of targets

public inbox for goredo-devel@lists.stargrave.org
Atom feed

* Potentially unnecessary redoing of targets
@ 2026-01-12  9:02 Niklas Böhm
  2026-01-12 12:53 ` spacefrogg
  0 siblings, 1 reply; 6+ messages in thread
From: Niklas Böhm @ 2026-01-12  9:02 UTC (permalink / raw)
  To: goredo-devel

Hello,

I noticed that there is some behavior with ood detection that can lead 
to unnecessary redoing of a target.

If you have the two simple files:

1.do:
echo 1

2.do:
redo-ifchange 1
sleep 3
echo 2

then calling `redo-ifchange 2` will run them, as intended.  When adding 
a comment to 1.do and again calling `redo-ifchange 2`, it will correctly 
redo the file `1` and afterwards redo itself.  This is kind of the 
crucial point.  If you Ctrl-C the redoing process while it's sleeping 
during execution of `2.do`, then it will (correctly, in my opinion) not 
redo the file `2` in a subsequent call to `redo-ifchange 2` since all of 
the dependencies are unchanged.

To me, this behavior could also be extended to be performed during 
execution of redo itself.  In this example, this would mean to check, 
again, if the dependencies have differing output before then starting 
the process of redoing `2`.

I acknowledge that this is not a trivial change in itself, though the 
additional computation time should be negligible, as the hash of `1` is 
already computed.  It would mean to change the ood detection to not 
declare `2` ood only based on `1.do` differing, but instead to check 
this after redoing `1`.

Let me know what you think about this proposed change!

Greetings
Nik

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Potentially unnecessary redoing of targets
  2026-01-12  9:02 Potentially unnecessary redoing of targets Niklas Böhm
@ 2026-01-12 12:53 ` spacefrogg
  2026-01-12 15:12   ` Niklas Böhm
  0 siblings, 1 reply; 6+ messages in thread
From: spacefrogg @ 2026-01-12 12:53 UTC (permalink / raw)
  To: goredo-devel

You are describing two different cases.

In the first, the execution of `2.do` has already started (due to OOD).  
This could result in anything, even in dropping `1` as a dependency 
entirely.  The assumption that `redo-ifchange 1` resulting in an 
unchanged output ALSO results in not changing the output of `2` is just 
wrong.  Anything could happen depending on what the content of `1` means 
in the context of executing `2.do`.  It could even result in never 
running `1.do`.  The (implicit) contract is that ONLY `2.do` may decide 
what the content of `1` means.  In addition, there is also the 
(implicit) assumption that the meaning of `1` should only depend on the 
current content of `1` and not on whether/how it has changed from the 
previous run.

In the second case, OOD has just determined that there is nothing to do. 
  You only reached that case by (nearly) compromising the build flow, 
which is (or should be) out of scope for the design of redo.  Actually, 
you would have reached the same case by running `redo-ifchange 1; 
redo-ifchange 2`. There, the first `redo-ifchange` also "clears" the OOD 
state of `1`.

So, to make the distinction clear: In your case, which was running 
`redo-ifchange 2`, `2` being OOD because of `1` just results in running 
`2.do`.  That's it.  Whether that means to also run `1.do` is entirely 
up to the program in `2.do`.  Running `1.do` is not the result of `1` 
being determined OOD but of the command `redo-ifchange 1` being part of 
`2.do`.

The two cases have nothing in common, really.

Kind regards,
Michael

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Potentially unnecessary redoing of targets
  2026-01-12 12:53 ` spacefrogg
@ 2026-01-12 15:12   ` Niklas Böhm
  2026-01-12 16:26     ` spacefrogg
  2026-01-13  8:23     ` Sergey Matveev
  0 siblings, 2 replies; 6+ messages in thread
From: Niklas Böhm @ 2026-01-12 15:12 UTC (permalink / raw)
  To: goredo-devel

Thanks for your response.  In general, I want to say that `2` is not ood 
because `1` did not change after completion of `1.do`, hence we would 
never have to start it.  The state when `2.do` is executing is in fact 
not ood, but because we determined `1` to be ood (as 1.do that changed), 
we run `2.do` anyways.  What I would suggest is to check, once more, if 
the targets actually did change their content before launching `2.do`.

I see that currently the problem is that the ood checking goes as 
follows (with redo-ifchange -d):
[48985] dbg  ood: 2 checking
[48985] dbg  ood: 2 -> 2.do: checking
[48985] dbg  ood: 2 -> 2.do: same inode
[48985] dbg  ood: 2 -> 2.do: is source
[48985] dbg  ood: 2 -> 1: checking
[48985] dbg  ood: 2 -> 1: inode differs
[48985] dbg  ood: 2 -> 1: same hash
[48985] dbg  ood: . 1 checking
[48985] dbg  ood: . 1 -> 1.do: checking
[48985] dbg  ood: . 1 -> 1.do: inode differs
[48985] dbg  ood: . 1 -> 1.do: hash differs
[48985] dbg  ood: . 1: true
[48985] dbg  ood: 2 -> 1: ood
[48985] dbg  ood: 2: true

I want to make the claim that `2` is not necessarily ood at that point, 
as we could first redo `1` and then check if the output differs before 
determining `2` to be ood.  As the build file `2.do` did not change, we 
know that its dependencies do not change; hence in my opinion it is a 
forgone conclusion to say that `2` is ood only because `1.do` has a 
differing hash, but actually `1` itself does not.

I'll admit that it might be an involved change in the code base and will 
also make redo itself change its behavior (as you lose the transitivity 
property for ood detection), but in my eyes it would be correct to check 
the direct targets of the .do file, instead of cascading the ood 
property.  This would also make the build system minimal

Anyways, thanks for reading all of this text, I hope my point became a 
little clearer.

Cheers
Nik

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Potentially unnecessary redoing of targets
  2026-01-12 15:12   ` Niklas Böhm
@ 2026-01-12 16:26     ` spacefrogg
  2026-01-13  8:23     ` Sergey Matveev
  1 sibling, 0 replies; 6+ messages in thread
From: spacefrogg @ 2026-01-12 16:26 UTC (permalink / raw)
  To: goredo-devel

On 2026-01-12 15:12, Niklas Böhm wrote:
> Thanks for your response.  In general, I want to say that `2` is not 
> ood because `1` did not change after completion of `1.do`, hence we 
> would never have to start it.  The state when `2.do` is executing is in 
> fact not ood, but because we determined `1` to be ood (as 1.do that 
> changed), we run `2.do` anyways.  What I would suggest is to check, 
> once more, if the targets actually did change their content before 
> launching `2.do`.

Thanks, I see the case you are constructing.  But you are misjudging the 
task that you've given redo to do in the first place.

What I was trying to convey in my previous message was, in essence, 
that:
1) `redo-ifchange 1; redo-ifchange 2` and
2) `redo-ifchange 2`

are two separate cases that you are conflating with each other.  You are 
trying to turn case 2) into case 1), which is not what redo is about.

> I see that currently the problem is that the ood checking goes as 
> follows (with redo-ifchange -d):
> [48985] dbg  ood: 2 checking
> [48985] dbg  ood: 2 -> 2.do: checking
> [48985] dbg  ood: 2 -> 2.do: same inode
> [48985] dbg  ood: 2 -> 2.do: is source
> [48985] dbg  ood: 2 -> 1: checking
> [48985] dbg  ood: 2 -> 1: inode differs
> [48985] dbg  ood: 2 -> 1: same hash
> [48985] dbg  ood: . 1 checking
> [48985] dbg  ood: . 1 -> 1.do: checking
> [48985] dbg  ood: . 1 -> 1.do: inode differs
> [48985] dbg  ood: . 1 -> 1.do: hash differs
> [48985] dbg  ood: . 1: true
> [48985] dbg  ood: 2 -> 1: ood
> [48985] dbg  ood: 2: true

This is exactly as it should go, because what you are trying to execute 
is case 2).  In case 2), the only decision redo has to make is whether 
or not to execute `2.do`.  It does NOT decide whether to execute `1.do`. 
  As I explained in my last message, executing `2.do` could result in 
removing `1` (and by transitivity `1.do`) from its dependency graph 
entirely.  So, executing `1.do` just to find out whether you have to 
execute `2.do` or, even worse, cancelling the execution of `2.do` after 
the line `redo-ifchange 1` just because you've determined it no longer 
to be OOD, is the wrong move.

Prematurely executing `1.do` before `2.do` (which is precisely turning 
case 2) into case 1)) can be wrong for the following reasons:
- it could be removed from `2`'s dependency graph on the next run of 
`2.do`.  So it would be a waste of effort.
- it could be an after-dependency that is only to be executed after most 
of `2.do` has run (I use after-dependencies to collect dependencies from 
.tex files processes, for example).  So running it eagerly would 
potentially operate on old data.  And even if not, it is up to the user 
to place `redo-ifchange` calls in .do files.  By running them early, you 
virtually move them all to the front of the file, which could change the 
results in non-obvious ways.  This "feature" would also be very hard to 
communicate.

> I want to make the claim that `2` is not necessarily ood at that point, 
> as we could first redo `1` and then check if the output differs before 
> determining `2` to be ood.  As the build file `2.do` did not change, we 
> know that its dependencies do not change; hence in my opinion it is a 
> forgone conclusion to say that `2` is ood only because `1.do` has a 
> differing hash, but actually `1` itself does not.

This is where you are mixing things up.  Let me say the concluding 
statement of my last message again: The call that produced above trace 
was `redo-ifchange 2`.  It's purpose is to determine, whether or not to 
run `2.do`.  That's it.  For that decision, it uses the currently known 
dependency graph of `2` as a starting point.  According to this graph, 
`2` IS OOD and `2.do` must be run.  This does not mean anything with 
regards to `1` or `1.do`, even if the alteration of `1.do` was the 
cause.  But the only way to get rid of this outdated knowledge (that `1` 
is still a dependency of `2`) is to execute `2.do` and to find out.

> I'll admit that it might be an involved change in the code base and 
> will also make redo itself change its behavior (as you lose the 
> transitivity property for ood detection), but in my eyes it would be 
> correct to check the direct targets of the .do file, instead of 
> cascading the ood property.  This would also make the build system 
> minimal

As I was trying to clarify, such a system would be very different from 
redo and would not allow for free placement of `redo-ifchange ...` calls 
throughout .do files. Or, at least, these calls would have a different 
meaning, depending on whether you run the .do file for the first time or 
because it was determined to be OOD.

Believe me, I see, what you are trying to achieve, but with that change, 
the semantics of redo and redo-ifchange calls become weird and 
non-obvious.

Kind regards,
Michael

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Potentially unnecessary redoing of targets
  2026-01-12 15:12   ` Niklas Böhm
  2026-01-12 16:26     ` spacefrogg
@ 2026-01-13  8:23     ` Sergey Matveev
  2026-01-13 19:04       ` Niklas Böhm
  1 sibling, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2026-01-13  8:23 UTC (permalink / raw)
  To: goredo-devel

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

Greetings!

*** Niklas Böhm [2026-01-12 16:12]:
>What I would suggest is to check, once more, if the targets
>actually did change their content before launching `2.do`.

redo just do not work that way. If it determines that "2" is
OOD (because it depends on "1", that depends on altered "1.do"),
then it runs 2.do. It does not run 1.do, but 2.do, that *may*
run 1.do. Even if we saw that result of "1" target was not
changed, we already executing 2.do anyway, maybe somewhere in
the middle of its code, with various steps already done before
encountering "redo-ifchange 1" in it.

spacefrogg perfectly explained all of that (and other issues,
like premature "1.do" execution) in details indeed.

Here is the crucial difference in behaviour between Make and
redo: Make, if 2 is OOD, will run 1(.do). Redo will run 2.do,
that may run 1.do.

-- 
Sergey Matveev (http://www.stargrave.org/)
LibrePGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Potentially unnecessary redoing of targets
  2026-01-13  8:23     ` Sergey Matveev
@ 2026-01-13 19:04       ` Niklas Böhm
  0 siblings, 0 replies; 6+ messages in thread
From: Niklas Böhm @ 2026-01-13 19:04 UTC (permalink / raw)
  To: goredo-devel

Hi again,

alright, I think I see the point now.  I agree that the behavior would 
definitely not seem intuitive and I suppose I am just not thinking of 
some corner cases where this could be very confusing.

My point is that `2.do` specifically did not change; hence the 
dependencies should stay the same, depending only on whether `1` changes 
or not.  If the hash stays the same, then there is no need to even start 
the execution of `2.do`.  In case the contents of `1` do change then all 
bets are off, so I agree that it does not make sense to change the 
behavior of redo.

Personally, I actually think that it would be nice to have 
`redo-ifchange 1; redo-ifchange 2` and only `redo-ifchange 2` work the 
same, although that would also lead to trouble when you rely on one of 
those to produce files that are not properly tracked by redo.

In any case, thanks for the discussion and explanations.

Cheers
Nik

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-01-13 19:05 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-01-12  9:02 Potentially unnecessary redoing of targets Niklas Böhm
2026-01-12 12:53 ` spacefrogg
2026-01-12 15:12   ` Niklas Böhm
2026-01-12 16:26     ` spacefrogg
2026-01-13  8:23     ` Sergey Matveev
2026-01-13 19:04       ` Niklas Böhm