[Scons-users] ParseDepends and duplicate with variant directories

[Gaurav Juvekar]

Hi,

I'm using this (https://bitbucket.org/scons/scons/wiki/ParseDepends) to use the compiler generated dependencies without requiring two passes. However, the problem is that when using it with variant directories, SCons first copies over the files that it detects a dependencies using the internal scanner, and then the compiler is run (in the variant dir) to generate dependencies which fails if some required files weren't found by the SCons scanner, defeating the point of using ParseDepends in the first place.


Is there a way to get the ParseDepends Builder to run in the source directory, and then copy over all of the compiler generated dependencies.



ParseDepends implemented as a tool https://gist.github.com/gauravjuvekar/9aed8d1ac117dc30ef34b28f4933e192

[Bill Deegan]

Any logic in the wiki is community supported.
You might consider contributing your logic to the scons-contrib repo.

The copying of files is only true with variant dir if duplicate = 1.
If duplicate =0, then they are not copied.

It's probably simpler to get such to work in the case where duplicate=0.

That said (as you can see) the implementation in the wiki is not
"production ready" for all scons usages, so the rest is an exercise for the
user.

You might consider picking up the pull request still on bitbucket for a
more complete C scanner implementation which does much of the preprocessor
logic.

What's driving you to use the compiler itself to generate the dependencies?

Post by Gaurav Juvekar
Hi,
I'm using this (https://bitbucket.org/scons/scons/wiki/ParseDepends) to
use the compiler generated dependencies without requiring two passes.
However, the problem is that when using it with variant directories, SCons
first copies over the files that it detects a dependencies using the
internal scanner, and then the compiler is run (in the variant dir) to
generate dependencies which fails if some required files weren't found by
the SCons scanner, defeating the point of using ParseDepends in the first
place.
Is there a way to get the ParseDepends Builder to run in the source
directory, and then copy over all of the compiler generated dependencies.
ParseDepends implemented as a tool https://gist.github.com/gauravjuvekar/
9aed8d1ac117dc30ef34b28f4933e192
--
Regards,
Gaurav Juvekar
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Bill Deegan]

Hmm... you might be able to provide the compiler with the source directory
for each variantdir path?

Look at the manpage and search for:
http://scons.org/doc/production/HTML/scons-man.html

rsrcdir or rsrcpath

That should work.
You'll have to do something similar to _concat, but to replace the values
with the .rsrcpath.

Then you may have to transpose those back to the variant dir paths
depending if you expect the variantdir to have generated header files or
not..

Post by Bill Deegan
Any logic in the wiki is community supported.
You might consider contributing your logic to the scons-contrib repo.
The copying of files is only true with variant dir if duplicate = 1.
If duplicate =0, then they are not copied.
It's probably simpler to get such to work in the case where duplicate=0.
That said (as you can see) the implementation in the wiki is not
"production ready" for all scons usages, so the rest is an exercise for the
user.
You might consider picking up the pull request still on bitbucket for a
more complete C scanner implementation which does much of the preprocessor
logic.
What's driving you to use the compiler itself to generate the dependencies?

Post by Gaurav Juvekar
Hi,
I'm using this (https://bitbucket.org/scons/scons/wiki/ParseDepends) to
use the compiler generated dependencies without requiring two passes.
However, the problem is that when using it with variant directories, SCons
first copies over the files that it detects a dependencies using the
internal scanner, and then the compiler is run (in the variant dir) to
generate dependencies which fails if some required files weren't found by
the SCons scanner, defeating the point of using ParseDepends in the first
place.
Is there a way to get the ParseDepends Builder to run in the source
directory, and then copy over all of the compiler generated dependencies.
ParseDepends implemented as a tool https://gist.github.com/gaurav
juvekar/9aed8d1ac117dc30ef34b28f4933e192
--
Regards,
Gaurav Juvekar
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Gaurav Juvekar]

Hi,

Post by Bill Deegan
Hmm... you might be able to provide the compiler with the source directory
for each variantdir path?
http://scons.org/doc/production/HTML/scons-man.html
rsrcdir or rsrcpath
That should work.
You'll have to do something similar to _concat, but to replace the values
with the .rsrcpath.
Then you may have to transpose those back to the variant dir paths
depending if you expect the variantdir to have generated header files or
not..

Thanks, I'll try doing that.

[Gaurav Juvekar]

Hi,

Post by Bill Deegan
You might consider picking up the pull request still on bitbucket for a
more complete C scanner implementation which does much of the preprocessor
logic.

Instead of improving the C scanner implementation, would a patch be accepted for a scanner that actually invokes the compiler with all the flags and then uses ParseDepends? It just seems "more correct" to use the compiler generated dependencies instead of parsing the file using a scanner like it's currently done. This way, the compiler can handle any crazy include combinations (even those that rely on compiler builtins such as #include_next). I know that this could be way slower than using a parser in Python.

Post by Bill Deegan
What's driving you to use the compiler itself to generate the dependencies?

Crazy organization in a third-party project whose source I can't change.

[Bill Deegan]

Any such patch would have to work with all currently supported compilers to
replace the existing scanner.
(Or be enablable on a compiler by compiler basis)
Also it's performance would have to be comparable to the current C/C++
scanner.

Then there would be a period of community review.
I'd suggest if you want to go down that path to get what you have
functional (with tests) and push to scons-contrib so others can try it out.

Post by Bill Deegan
Hi,

Post by Bill Deegan
You might consider picking up the pull request still on bitbucket for a
more complete C scanner implementation which does much of the

preprocessor

Post by Bill Deegan
logic.

Instead of improving the C scanner implementation, would a patch be
accepted for a scanner that actually invokes the compiler with all the
flags and then uses ParseDepends? It just seems "more correct" to use the
compiler generated dependencies instead of parsing the file using a scanner
like it's currently done. This way, the compiler can handle any crazy
include combinations (even those that rely on compiler builtins such as
#include_next). I know that this could be way slower than using a parser in
Python.

Post by Bill Deegan
What's driving you to use the compiler itself to generate the

dependencies?
Crazy organization in a third-party project whose source I can't change.
--
Regards,
Gaurav Juvekar
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Bill Deegan]

re why use compiler instead of existing scanner..
not sure why the organization of the third party source code would matter.
Can you explain?

Post by Bill Deegan
Any such patch would have to work with all currently supported compilers
to replace the existing scanner.
(Or be enablable on a compiler by compiler basis)
Also it's performance would have to be comparable to the current C/C++
scanner.
Then there would be a period of community review.
I'd suggest if you want to go down that path to get what you have
functional (with tests) and push to scons-contrib so others can try it out.

Post by Bill Deegan
Hi,

Post by Bill Deegan
You might consider picking up the pull request still on bitbucket for a
more complete C scanner implementation which does much of the

preprocessor

Post by Bill Deegan
logic.

Instead of improving the C scanner implementation, would a patch be
accepted for a scanner that actually invokes the compiler with all the
flags and then uses ParseDepends? It just seems "more correct" to use the
compiler generated dependencies instead of parsing the file using a scanner
like it's currently done. This way, the compiler can handle any crazy
include combinations (even those that rely on compiler builtins such as
#include_next). I know that this could be way slower than using a parser in
Python.

Post by Bill Deegan
What's driving you to use the compiler itself to generate the

dependencies?
Crazy organization in a third-party project whose source I can't change.
--
Regards,
Gaurav Juvekar
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Gaurav Juvekar]

Hi,

Post by Bill Deegan
not sure why the organization of the third party source code would matter.
Can you explain?

I have files (anonymised) like this, where different variants of the build can be selected by -DSELECT_CONFIG=config{1,2,3}

module/configurable/config.h: #include <SELECT_CONFIG/SELECT_CONFIG.h>
module/configurable/config1/init.c: #include <config.h>
module/configurable/config1/config1.h
module/configurable/config2/init.c: #include <config.h>
module/configurable/config2/config2.h
module/configurable/config3/init.c: #include <config.h>
module/configurable/config3/config3.h

I can't really modify these files or the structure. Right now, I have a module/configurable/SConscript which checks CPPDEFINES for SELECT_CONFIG, and manually adds SELECT_CONFIG/SELECT_CONFIG.h as a dependency of config.h

[Bill Deegan]

Gotcha. So the rewritten c scanner in the bitbucket.org pull request should
handle these..

Post by Bill Deegan
Hi,

Post by Bill Deegan
not sure why the organization of the third party source code would

matter.

Post by Bill Deegan
Can you explain?

I have files (anonymised) like this, where different variants of the build
can be selected by -DSELECT_CONFIG=config{1,2,3}
module/configurable/config.h: #include <SELECT_CONFIG/SELECT_CONFIG.h>
module/configurable/config1/init.c: #include <config.h>
module/configurable/config1/config1.h
module/configurable/config2/init.c: #include <config.h>
module/configurable/config2/config2.h
module/configurable/config3/init.c: #include <config.h>
module/configurable/config3/config3.h
I can't really modify these files or the structure. Right now, I have a
module/configurable/SConscript which checks CPPDEFINES for SELECT_CONFIG,
and manually adds SELECT_CONFIG/SELECT_CONFIG.h as a dependency of config.h
--
Regards,
Gaurav Juvekar
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[s***@triad.rr.com]

This post might be inappropriate. Click to display it.

[Bill Deegan]

Are you Glob()'ing files in your SCons logic?

Do you see the above only when some build step fails?

Can you try 3.0.1 or the master branch?

Post by s***@triad.rr.com
I am getting these cycle errors. Any clues, tip or tricks I can try?
Anecdotally it seems to occur more frequently (though its not common) if
the workspace has a lot of packages/files. Usually they go away if I just
run the build again. What prompted this email is it occurred three times
before it went away, which is unusual.
By the way I tried two different versions of SCons:(see below)
Internal Error: no cycle found for node /home/XXXXXXXX/debBuild2/opt/
XXXXXX/genesis/lib/libmainmonitorapp.so.1.1.1 (<SCons.Node.FS.File object
at 0x214bd80>) in state pending
Internal Error: no cycle found for node /home/XXXXXXXX/debBuild2/opt/
XXXXXX/genesis/lib/libutildb.so.1.1.1 (<SCons.Node.FS.File object at
0x21393a0>) in state pending
Internal Error: no cycle found for node /home/XXXXXXXX/debBuild2/opt/
XXXXXX/genesis/lib/liblocale.so.1.1.1 (<SCons.Node.FS.File object at
0x20f7d20>) in state pending
script: v2.4.1.rel_2.4.1:3453:73fefd3ea0b0, 2015/11/09 03:25:05,
by bdbaddog on ubuntu1404-32bit
engine: v2.4.1.rel_2.4.1:3453:73fefd3ea0b0, 2015/11/09 03:25:05,
by bdbaddog on ubuntu1404-32bit
script: v3.0.0.rel_3.0.0:4395:8972f6a2f699, 2017/09/18 12:59:24,
by bdbaddog on ubuntu-16
engine: v3.0.0.rel_3.0.0:4395:8972f6a2f699, 2017/09/18 12:59:24,
by bdbaddog on ubuntu-16
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Post by Bill Deegan
Are you Glob()'ing files in your SCons logic?
Do you see the above only when some build step fails?
Can you try 3.0.1 or the master branch?

Post by s***@triad.rr.com
I am getting these cycle errors. Any clues, tip or tricks I can try?
Anecdotally it seems to occur more frequently (though its not common) if
the workspace has a lot of packages/files. Usually they go away if I just
run the build again. What prompted this email is it occurred three times
before it went away, which is unusual.

seems like the last few times I ran into this, a "retry build" didn't
help, but a "retry build after 'rm -fr .scon*'" did help. Not sure
that's worth anything, but it's how it's been here.

[Spencer Yost]

Thanks Mats and Bill. Have not tried 3.0.1, and yes: I am globbing all source specification in builders (most packages contain too many source files to individually ennumerate). I have not tried removing the dblite file. I'll try that tonight.

Thanks!

Spencer Yost

18

[Bill Deegan]

Are you glob'ing files in a directory(s) where generated files might be
picked up?

Post by Spencer Yost
Thanks Mats and Bill. Have not tried 3.0.1, and yes: I am globbing all
source specification in builders (most packages contain too many source
files to individually ennumerate). I have not tried removing the dblite
file. I'll try that tonight.
Thanks!
Spencer Yost

18

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[s***@triad.rr.com]

I do, but think they occur before and after the creation of the target. First, a packaging prep step *may* create a single file in a subdir below the source directory , but only if it doesn't exist (if a package is new and a changelog doesn't exist I create an initial file for the developer). This file is also modified for each build. However, this all happens prior to the SConscript file being read. Some files are created after the compile and link phases and they are created in that same directory(a debian package for example) but the errors seem to occur well before that.

Couple of other notes:
Typical glob specification:

finalDeliverable = env.SharedLibrary('propertytree',
[Glob('*.c*'), Glob('config/platform/*.c*' )],
LIBS=['utility',
'stringalgo'],
LIBPATH=[mySConscriptLibPath],
RPATH=[mySConscriptLibPath]
)

I use a variant dir.
I duplicate files

Hope this helps,

Spencer

Post by Bill Deegan
Are you glob'ing files in a directory(s) where generated files might be
picked up?

Post by Spencer Yost
Thanks Mats and Bill. Have not tried 3.0.1, and yes: I am globbing all
source specification in builders (most packages contain too many source
files to individually ennumerate). I have not tried removing the dblite
file. I'll try that tonight.
Thanks!
Spencer Yost

18

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users