public inbox for goredo-devel@lists.stargrave.org
Atom feed
* Source files with matching .do file keep targets ood
@ 2026-08-07 6:25 Nik Boehm
2026-08-07 8:03 ` Sergey Matveev
0 siblings, 1 reply; 5+ messages in thread
From: Nik Boehm @ 2026-08-07 6:25 UTC (permalink / raw)
To: goredo-devel
Hello,
I noticed that if there is a source file (without a .dep file in .redo/) but a .do file with a matching suffix exists, then targets depending on the source file will be perpetually out of date. Is this intended behavior or can this be circumvented somehow?
I have a large collection of binary files that I have downloaded and that will not change. I initially downloaded them with a script until I switched to a .do file. Since the contents will not change, I would prefer not having to download them again.
Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work? Is the build uuid that is at the top of the .dep file significant or can I write there whatever I please? And is there a way to get the BLAKE3 hash function out of goredo?
In case there is an easier way to fix the issue, I would also be happy to hear that.
Greetings,
Nik
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood
2026-08-07 6:25 Source files with matching .do file keep targets ood Nik Boehm
@ 2026-08-07 8:03 ` Sergey Matveev
2026-08-07 8:36 ` Nik Boehm
0 siblings, 1 reply; 5+ messages in thread
From: Sergey Matveev @ 2026-08-07 8:03 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 2526 bytes --]
Greetings!
*** Nik Boehm [2026-08-07 08:25]:
>I noticed that if there is a source file (without a .dep file in .redo/) but a .do file with a matching suffix exists, then targets depending on the source file will be perpetually out of date. Is this intended behavior or can this be circumvented somehow?
Do I clearly understand, that you have got "foo" file, and a "foo.do"
appeared, which leads to "foo" being an out-of-date? Yes, that is
intended, because "foo" is missing the "record" in .redo/ directory, it
was not built under redo control and has no information to determine is
it OOD. For goredo it is safer to rebuild it.
>I have a large collection of binary files that I have downloaded and that will not change. I initially downloaded them with a script until I switched to a .do file. Since the contents will not change, I would prefer not having to download them again.
I see. I met the same problem in my BASS build system:
http://www.bass.stargrave.org/Build/Distfiles/Cache.html
http://www.git.stargrave.org/?p=bass.git&f=/build/distfiles/dl/default.do
http://www.git.stargrave.org/?p=bass.git&f=/build/distfiles/dl/default.rc
The .do checks if cache/$target exists and hardlinks it
(ln cache/$target $3). Actually target is rebuilt, but at least it just
finishes with non expensive ln call. Same is done with prebuilt packages
(which should not be rebuild, just used as is):
http://www.git.stargrave.org/?p=bass.git&f=/build/skel/common.rc
>Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work?
Yes, it will. Hash will be optional in that case.
$ mkdir .redo
$ {
echo Build: 00000000-0000-0000-0000-000000000000
redo-inode foo | perl -npe 'print "Type: ifchange\n" if /^Target/'
} >.redo/foo.rec
$ redo-depfix foo
>Is the build uuid that is at the top of the .dep file significant or can I write there whatever I please?
It is only used to determine if the target is build during the current
running redo build "session". Can be any valid UUID.
>And is there a way to get the BLAKE3 hash function out of goredo?
Currently no such option.
>In case there is an easier way to fix the issue, I would also be happy to hear that.
None that I can think out more.
--
* Origin: Stargrave's homepage http://www.stargrave.org/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood
2026-08-07 8:03 ` Sergey Matveev
@ 2026-08-07 8:36 ` Nik Boehm
2026-08-07 8:56 ` Sergey Matveev
0 siblings, 1 reply; 5+ messages in thread
From: Nik Boehm @ 2026-08-07 8:36 UTC (permalink / raw)
To: goredo-devel
Thanks for the quick answer!
> On 7 Aug 2026, at 10:03, Sergey Matveev <stargrave@stargrave•org> wrote:
>
> Do I clearly understand, that you have got "foo" file, and a "foo.do"
> appeared, which leads to "foo" being an out-of-date? Yes, that is
> intended, because "foo" is missing the "record" in .redo/ directory, it
> was not built under redo control and has no information to determine is
> it OOD. For goredo it is safer to rebuild it.
Almost, I have a file a.ext and default.ext.do, but I still get the warning. I agree that if there is a file called a.ext.do then it would make sense. With a default.ext.do file the usefulness is more debatable. I would tend to say that there shouldn't be a reason to consider targets that depend on the source file a.ext to always be OOD.
Right now the behavior is as follows IIUC:
- a.ext.do warning/ood
- default.ext.do warning/ood
- default.do nothing
Surely the last line has to stay since otherwise every source file would always be OOD. I don’t know if it makes sense to keep the second line with its current behavior. What do you think? I realize that it can be surprising that a target is not redone even though it has a .do file simply because it’s recorded as a source file. But that is partly a user error. Would it be possible to have a file depending on a.ext not be OOD?
> I see. I met the same problem in my BASS build system:
> The .do checks if cache/$target exists and hardlinks it
> (ln cache/$target $3). Actually target is rebuilt, but at least it just
> finishes with non expensive ln call. Same is done with prebuilt packages
> (which should not be rebuild, just used as is):
> http://www.git.stargrave.org/?p=bass.git&f=/build/skel/common.rc
Good to see that you have faced a similar problem before.
>
>> Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work?
>
> Yes, it will. Hash will be optional in that case.
>
> $ mkdir .redo
> $ {
> echo Build: 00000000-0000-0000-0000-000000000000
> redo-inode foo | perl -npe 'print "Type: ifchange\n" if /^Target/'
> } >.redo/foo.rec
> $ redo-depfix foo
Why is the hash optional in those cases?
>> And is there a way to get the BLAKE3 hash function out of goredo?
>
> Currently no such option.
Okay, it’s easy enough to get a reference implementation, so no problem.
Thanks for the detailed answer to my points!
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood
2026-08-07 8:36 ` Nik Boehm
@ 2026-08-07 8:56 ` Sergey Matveev
2026-08-20 15:01 ` spacefrogg
0 siblings, 1 reply; 5+ messages in thread
From: Sergey Matveev @ 2026-08-07 8:56 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]
*** Nik Boehm [2026-08-07 10:36]:
>I agree that if there is a file called a.ext.do then it would make sense. With a default.ext.do file the usefulness is more debatable.
default*.do should not be any way different from non-default .do files.
It is just like "wildcard" .do.
>I would tend to say that there shouldn't be a reason to consider targets that depend on the source file a.ext to always be OOD.
Either I am missing something, or there is misuse of terms.
"source file" is a file, which is not generated by redo, which lacks any
corresponding .do file. For example having default.do means that there
can be no source files in the directory with it (or in subdirectories).
Either the target is generated, or it is "source file". Having foo.ext
and default.ext.do means that foo.ext is not treated like a source file,
but a valid target which redo must generate if it is OOD. If it lacks
dependency (.dep file) information, then redo treats it like "target has
not been done", it is non existent, so must be rebuilt.
>Surely the last line has to stay since otherwise every source file would always be OOD. I don’t know if it makes sense to keep the second line with its current behavior. What do you think? I realize that it can be surprising that a target is not redone even though it has a .do file simply because it’s recorded as a source file. But that is partly a user error. Would it be possible to have a file depending on a.ext not be OOD?
If there is default.ext.do and a.ext was not build under redo (so it
misses the corresponding .redo/a.ext.dep) -- then it will be OOD, no
other choices.
>> $ redo-depfix foo
>
>Why is the hash optional in those cases?
Because redo-depfix will recalculate it anyway.
--
* Origin: Stargrave's homepage http://www.stargrave.org/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood
2026-08-07 8:56 ` Sergey Matveev
@ 2026-08-20 15:01 ` spacefrogg
0 siblings, 0 replies; 5+ messages in thread
From: spacefrogg @ 2026-08-20 15:01 UTC (permalink / raw)
To: goredo-devel
On 2026-08-07 10:56, Sergey Matveev wrote:
> *** Nik Boehm [2026-08-07 10:36]:
>> I agree that if there is a file called a.ext.do then it would make
>> sense. With a default.ext.do file the usefulness is more debatable.
>
> default*.do should not be any way different from non-default .do files.
> It is just like "wildcard" .do.
>
>> I would tend to say that there shouldn't be a reason to consider
>> targets that depend on the source file a.ext to always be OOD.
>
> Either I am missing something, or there is misuse of terms.
> "source file" is a file, which is not generated by redo, which lacks
> any
> corresponding .do file. For example having default.do means that there
> can be no source files in the directory with it (or in subdirectories).
> Either the target is generated, or it is "source file". Having foo.ext
> and default.ext.do means that foo.ext is not treated like a source
> file,
> but a valid target which redo must generate if it is OOD. If it lacks
> dependency (.dep file) information, then redo treats it like "target
> has
> not been done", it is non existent, so must be rebuilt.
>
>> Surely the last line has to stay since otherwise every source file
>> would always be OOD. I don’t know if it makes sense to keep the
>> second line with its current behavior. What do you think? I realize
>> that it can be surprising that a target is not redone even though it
>> has a .do file simply because it’s recorded as a source file. But
>> that is partly a user error. Would it be possible to have a file
>> depending on a.ext not be OOD?
>
> If there is default.ext.do and a.ext was not build under redo (so it
> misses the corresponding .redo/a.ext.dep) -- then it will be OOD, no
> other choices.
I think, I have argued that before and I still think this behaviour is
very wrong.
As we all agree, redo is a stateful build system. So, what are the
useful states a target <A> can be in?
unknown: redo does know about <A>, ie. has no record of it
source: redo has no record (or is recorded as a "source" to keep extra
data to determine OOD) of <A> but <A> exists. So, it is a source.
target: redo has a record of <A> and <A> either does not exist or <A>'s
state is consistent with the record
locked: redo has a record of <A> and <A> exists but is inconsistent with
the record
As we can see, for <A> itself, OOD is not a factor. OOD only comes into
play in the relation between <A> and its dependencies. All targets have
at least one dependency, their producing .do file.
Now to the transitions between the states for target <A>:
unknown -> source: redo is called on <A>, has no prior knowledge about
it, and <A> exists.
unknown -> target: redo is called on <A>, has no prior knowledge about
it, <A> doesn't exist and a .do file to produce <A> can be found.
source -> unknown: redo is called on some <B> which used to depend on
<A> but no longer does. So redo forgets about <A> entirely.
target -> locked: redo is called on <A>, <A> exists and does not match
redo's record of it.
locked -> target: redo is called on <A>, <A> no longer exists, which
resolves the mismatch with the record and <A> is rebuilt.
Now, where I think goredo currently takes the wrong turn:
It should not consider <A> OOD just because there is some .do file to
produce <A>. If <A> exists before redo has a record of it, it should
just treat <A> as a source and be happy. Obviously, redo will never
rebuild <A>. So, if a user mistakenly has removed the .redo database,
all current targets turn into sources and redo won't rebuild anything
anymore. But hey, so is life. Complex project just have cleanup rules
anyhow which the user can call and start over again.
Expecting that a directory that contains a default.do file must
otherwise be empty (because all other files, including in
subdirectories, would be in conflict now) is completely useless.
default.do files are supposed to be catchall rules that make sure that
build instructions never escape a subtree, because they always terminate
at the nearest default.do.
I hope my ramblings made any sense.
–Michael
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-08-20 15:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-08-07 6:25 Source files with matching .do file keep targets ood Nik Boehm
2026-08-07 8:03 ` Sergey Matveev
2026-08-07 8:36 ` Nik Boehm
2026-08-07 8:56 ` Sergey Matveev
2026-08-20 15:01 ` spacefrogg