public inbox for goredo-devel@lists.stargrave.org Atom feed
* Source files with matching .do file keep targets ood @ 2026-08-07 6:25 Nik Boehm 2026-08-07 8:03 ` Sergey Matveev 0 siblings, 1 reply; 5+ messages in thread From: Nik Boehm @ 2026-08-07 6:25 UTC (permalink / raw) To: goredo-devel Hello, I noticed that if there is a source file (without a .dep file in .redo/) but a .do file with a matching suffix exists, then targets depending on the source file will be perpetually out of date. Is this intended behavior or can this be circumvented somehow? I have a large collection of binary files that I have downloaded and that will not change. I initially downloaded them with a script until I switched to a .do file. Since the contents will not change, I would prefer not having to download them again. Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work? Is the build uuid that is at the top of the .dep file significant or can I write there whatever I please? And is there a way to get the BLAKE3 hash function out of goredo? In case there is an easier way to fix the issue, I would also be happy to hear that. Greetings, Nik ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood 2026-08-07 6:25 Source files with matching .do file keep targets ood Nik Boehm @ 2026-08-07 8:03 ` Sergey Matveev 2026-08-07 8:36 ` Nik Boehm 0 siblings, 1 reply; 5+ messages in thread From: Sergey Matveev @ 2026-08-07 8:03 UTC (permalink / raw) To: goredo-devel [-- Attachment #1: Type: text/plain, Size: 2526 bytes --] Greetings! *** Nik Boehm [2026-08-07 08:25]: >I noticed that if there is a source file (without a .dep file in .redo/) but a .do file with a matching suffix exists, then targets depending on the source file will be perpetually out of date. Is this intended behavior or can this be circumvented somehow? Do I clearly understand, that you have got "foo" file, and a "foo.do" appeared, which leads to "foo" being an out-of-date? Yes, that is intended, because "foo" is missing the "record" in .redo/ directory, it was not built under redo control and has no information to determine is it OOD. For goredo it is safer to rebuild it. >I have a large collection of binary files that I have downloaded and that will not change. I initially downloaded them with a script until I switched to a .do file. Since the contents will not change, I would prefer not having to download them again. I see. I met the same problem in my BASS build system: http://www.bass.stargrave.org/Build/Distfiles/Cache.html http://www.git.stargrave.org/?p=bass.git&f=/build/distfiles/dl/default.do http://www.git.stargrave.org/?p=bass.git&f=/build/distfiles/dl/default.rc The .do checks if cache/$target exists and hardlinks it (ln cache/$target $3). Actually target is rebuilt, but at least it just finishes with non expensive ln call. Same is done with prebuilt packages (which should not be rebuild, just used as is): http://www.git.stargrave.org/?p=bass.git&f=/build/skel/common.rc >Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work? Yes, it will. Hash will be optional in that case. $ mkdir .redo $ { echo Build: 00000000-0000-0000-0000-000000000000 redo-inode foo | perl -npe 'print "Type: ifchange\n" if /^Target/' } >.redo/foo.rec $ redo-depfix foo >Is the build uuid that is at the top of the .dep file significant or can I write there whatever I please? It is only used to determine if the target is build during the current running redo build "session". Can be any valid UUID. >And is there a way to get the BLAKE3 hash function out of goredo? Currently no such option. >In case there is an easier way to fix the issue, I would also be happy to hear that. None that I can think out more. -- * Origin: Stargrave's homepage http://www.stargrave.org/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 265 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood 2026-08-07 8:03 ` Sergey Matveev @ 2026-08-07 8:36 ` Nik Boehm 2026-08-07 8:56 ` Sergey Matveev 0 siblings, 1 reply; 5+ messages in thread From: Nik Boehm @ 2026-08-07 8:36 UTC (permalink / raw) To: goredo-devel Thanks for the quick answer! > On 7 Aug 2026, at 10:03, Sergey Matveev <stargrave@stargrave•org> wrote: > > Do I clearly understand, that you have got "foo" file, and a "foo.do" > appeared, which leads to "foo" being an out-of-date? Yes, that is > intended, because "foo" is missing the "record" in .redo/ directory, it > was not built under redo control and has no information to determine is > it OOD. For goredo it is safer to rebuild it. Almost, I have a file a.ext and default.ext.do, but I still get the warning. I agree that if there is a file called a.ext.do then it would make sense. With a default.ext.do file the usefulness is more debatable. I would tend to say that there shouldn't be a reason to consider targets that depend on the source file a.ext to always be OOD. Right now the behavior is as follows IIUC: - a.ext.do warning/ood - default.ext.do warning/ood - default.do nothing Surely the last line has to stay since otherwise every source file would always be OOD. I don’t know if it makes sense to keep the second line with its current behavior. What do you think? I realize that it can be surprising that a target is not redone even though it has a .do file simply because it’s recorded as a source file. But that is partly a user error. Would it be possible to have a file depending on a.ext not be OOD? > I see. I met the same problem in my BASS build system: > The .do checks if cache/$target exists and hardlinks it > (ln cache/$target $3). Actually target is rebuilt, but at least it just > finishes with non expensive ln call. Same is done with prebuilt packages > (which should not be rebuild, just used as is): > http://www.git.stargrave.org/?p=bass.git&f=/build/skel/common.rc Good to see that you have faced a similar problem before. > >> Alternatively, I could also write a .dep file for each of the binary files manually to get goredo to treat them as not ood. Is there anything I need to be aware of for that? My idea is to write a .rec file and then use redo-depfix, will that work? > > Yes, it will. Hash will be optional in that case. > > $ mkdir .redo > $ { > echo Build: 00000000-0000-0000-0000-000000000000 > redo-inode foo | perl -npe 'print "Type: ifchange\n" if /^Target/' > } >.redo/foo.rec > $ redo-depfix foo Why is the hash optional in those cases? >> And is there a way to get the BLAKE3 hash function out of goredo? > > Currently no such option. Okay, it’s easy enough to get a reference implementation, so no problem. Thanks for the detailed answer to my points! ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood 2026-08-07 8:36 ` Nik Boehm @ 2026-08-07 8:56 ` Sergey Matveev 2026-08-20 15:01 ` spacefrogg 0 siblings, 1 reply; 5+ messages in thread From: Sergey Matveev @ 2026-08-07 8:56 UTC (permalink / raw) To: goredo-devel [-- Attachment #1: Type: text/plain, Size: 1834 bytes --] *** Nik Boehm [2026-08-07 10:36]: >I agree that if there is a file called a.ext.do then it would make sense. With a default.ext.do file the usefulness is more debatable. default*.do should not be any way different from non-default .do files. It is just like "wildcard" .do. >I would tend to say that there shouldn't be a reason to consider targets that depend on the source file a.ext to always be OOD. Either I am missing something, or there is misuse of terms. "source file" is a file, which is not generated by redo, which lacks any corresponding .do file. For example having default.do means that there can be no source files in the directory with it (or in subdirectories). Either the target is generated, or it is "source file". Having foo.ext and default.ext.do means that foo.ext is not treated like a source file, but a valid target which redo must generate if it is OOD. If it lacks dependency (.dep file) information, then redo treats it like "target has not been done", it is non existent, so must be rebuilt. >Surely the last line has to stay since otherwise every source file would always be OOD. I don’t know if it makes sense to keep the second line with its current behavior. What do you think? I realize that it can be surprising that a target is not redone even though it has a .do file simply because it’s recorded as a source file. But that is partly a user error. Would it be possible to have a file depending on a.ext not be OOD? If there is default.ext.do and a.ext was not build under redo (so it misses the corresponding .redo/a.ext.dep) -- then it will be OOD, no other choices. >> $ redo-depfix foo > >Why is the hash optional in those cases? Because redo-depfix will recalculate it anyway. -- * Origin: Stargrave's homepage http://www.stargrave.org/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 265 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Source files with matching .do file keep targets ood 2026-08-07 8:56 ` Sergey Matveev @ 2026-08-20 15:01 ` spacefrogg 0 siblings, 0 replies; 5+ messages in thread From: spacefrogg @ 2026-08-20 15:01 UTC (permalink / raw) To: goredo-devel On 2026-08-07 10:56, Sergey Matveev wrote: > *** Nik Boehm [2026-08-07 10:36]: >> I agree that if there is a file called a.ext.do then it would make >> sense. With a default.ext.do file the usefulness is more debatable. > > default*.do should not be any way different from non-default .do files. > It is just like "wildcard" .do. > >> I would tend to say that there shouldn't be a reason to consider >> targets that depend on the source file a.ext to always be OOD. > > Either I am missing something, or there is misuse of terms. > "source file" is a file, which is not generated by redo, which lacks > any > corresponding .do file. For example having default.do means that there > can be no source files in the directory with it (or in subdirectories). > Either the target is generated, or it is "source file". Having foo.ext > and default.ext.do means that foo.ext is not treated like a source > file, > but a valid target which redo must generate if it is OOD. If it lacks > dependency (.dep file) information, then redo treats it like "target > has > not been done", it is non existent, so must be rebuilt. > >> Surely the last line has to stay since otherwise every source file >> would always be OOD. I don’t know if it makes sense to keep the >> second line with its current behavior. What do you think? I realize >> that it can be surprising that a target is not redone even though it >> has a .do file simply because it’s recorded as a source file. But >> that is partly a user error. Would it be possible to have a file >> depending on a.ext not be OOD? > > If there is default.ext.do and a.ext was not build under redo (so it > misses the corresponding .redo/a.ext.dep) -- then it will be OOD, no > other choices. I think, I have argued that before and I still think this behaviour is very wrong. As we all agree, redo is a stateful build system. So, what are the useful states a target <A> can be in? unknown: redo does know about <A>, ie. has no record of it source: redo has no record (or is recorded as a "source" to keep extra data to determine OOD) of <A> but <A> exists. So, it is a source. target: redo has a record of <A> and <A> either does not exist or <A>'s state is consistent with the record locked: redo has a record of <A> and <A> exists but is inconsistent with the record As we can see, for <A> itself, OOD is not a factor. OOD only comes into play in the relation between <A> and its dependencies. All targets have at least one dependency, their producing .do file. Now to the transitions between the states for target <A>: unknown -> source: redo is called on <A>, has no prior knowledge about it, and <A> exists. unknown -> target: redo is called on <A>, has no prior knowledge about it, <A> doesn't exist and a .do file to produce <A> can be found. source -> unknown: redo is called on some <B> which used to depend on <A> but no longer does. So redo forgets about <A> entirely. target -> locked: redo is called on <A>, <A> exists and does not match redo's record of it. locked -> target: redo is called on <A>, <A> no longer exists, which resolves the mismatch with the record and <A> is rebuilt. Now, where I think goredo currently takes the wrong turn: It should not consider <A> OOD just because there is some .do file to produce <A>. If <A> exists before redo has a record of it, it should just treat <A> as a source and be happy. Obviously, redo will never rebuild <A>. So, if a user mistakenly has removed the .redo database, all current targets turn into sources and redo won't rebuild anything anymore. But hey, so is life. Complex project just have cleanup rules anyhow which the user can call and start over again. Expecting that a directory that contains a default.do file must otherwise be empty (because all other files, including in subdirectories, would be in conflict now) is completely useless. default.do files are supposed to be catchall rules that make sure that build instructions never escape a subtree, because they always terminate at the nearest default.do. I hope my ramblings made any sense. –Michael ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-08-20 15:26 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-08-07 6:25 Source files with matching .do file keep targets ood Nik Boehm 2026-08-07 8:03 ` Sergey Matveev 2026-08-07 8:36 ` Nik Boehm 2026-08-07 8:56 ` Sergey Matveev 2026-08-20 15:01 ` spacefrogg