[Scons-users] cleaning and surprises

[Mats Wichmann]

In general, software tools should not "surprise" you.

In the big project I've been working on, one of the surprises is how
much work happens when you really didn't ask to do much work. Like the
"clean" operation... yes it's easy to agree, after you think about it,
that you have to do enough work to figure out what targets have been (or
could be) built so you know what to remove. In many build systems
that's quite explicit - the information is in the scripts and the tool
just removes that and doesn't do a lot of calculation. But as I say, I
get surprised by how much does happen in our build. I know that this is
the result of errors (rather, bad assumptions) by the authors of the
various bits of this setup, but these are smart developers and somehow
it doesn't come clear how to set things up. To be a little less obtuse:

- configure checks get run
Checking for C++ header file arpa/inet.h... yes
Checking for C++ header file fcntl.h... yes
...

- external-library stuff gets run, sometimes many times:
Checking for C++ library boost_thread... yes
Checking for C++ library boost_system... yes

and in the end, there's been a flood of output, and some files in the
variant directory - which for the sake of amusement is completely
removed before running the clean operation - has ended up with some
/built/ files!!! So it's not that they were missed by the clean, they
were actually built during the clean process (remember, started
completely empty).

out/linux/x86_64/release/liblogger.a
out/linux/x86_64/release/libc_common.a
out/linux/x86_64/release/resource/csdk/logger/liblogger.a
out/linux/x86_64/release/resource/csdk/logger/src/logger.o
out/linux/x86_64/release/resource/csdk/logger/src/trace.o
... 13 more in current setup

It's not that this makes builds not work, but it tells me something just
isn't right in the script setup.

In fact the same thing happens in an even less work-expected situation,
asking for help. We also had a bit of this discussion recently on list
over dry-run invocations, another "surprise".


Do we have documentation in scons of "things not to do" that could be
helpful in avoiding falling into these kinds of traps?


Note:

Even scons itself has a bit of this:

$ scons -c
scons: Reading SConscript files ...
Mkdir("/home/mats/Work/scons/build/doc/design")
Mkdir("/home/mats/Work/scons/build/doc/design/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/user")
Mkdir("/home/mats/Work/scons/build/doc/user/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/reference")
Mkdir("/home/mats/Work/scons/build/doc/reference/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/man")
Mkdir("/home/mats/Work/scons/build/doc/man/titlepage")
scons: done reading SConscript files.
scons: Cleaning targets ...
scons: done cleaning targets.

and, not in the messages, those directories were populated, so after
starting with a clean git checkout - no "build" directory at all -
"scons -c" leaves me with a build directory with 113 files (.xml, .xsl,
SConstruct, and a few others). did cleaning really need to create
directories and populate them?


any thoughts on how to make this less surprising? or should I just suck
it up and treat this as normal?

[Bill Deegan]

Mats,

If your SConscripts are doing a lot of miscellaneous python logic which
create those files and/or use Execute(), then that an issue with your build
setup.
If SCons is doing that, I'd be quite surprised with the exception of any
configure context files.

-Bill

Post by Mats Wichmann
In general, software tools should not "surprise" you.
In the big project I've been working on, one of the surprises is how
much work happens when you really didn't ask to do much work. Like the
"clean" operation... yes it's easy to agree, after you think about it,
that you have to do enough work to figure out what targets have been (or
could be) built so you know what to remove. In many build systems
that's quite explicit - the information is in the scripts and the tool
just removes that and doesn't do a lot of calculation. But as I say, I
get surprised by how much does happen in our build. I know that this is
the result of errors (rather, bad assumptions) by the authors of the
various bits of this setup, but these are smart developers and somehow
- configure checks get run
Checking for C++ header file arpa/inet.h... yes
Checking for C++ header file fcntl.h... yes
...
Checking for C++ library boost_thread... yes
Checking for C++ library boost_system... yes
and in the end, there's been a flood of output, and some files in the
variant directory - which for the sake of amusement is completely
removed before running the clean operation - has ended up with some
/built/ files!!! So it's not that they were missed by the clean, they
were actually built during the clean process (remember, started
completely empty).
out/linux/x86_64/release/liblogger.a
out/linux/x86_64/release/libc_common.a
out/linux/x86_64/release/resource/csdk/logger/liblogger.a
out/linux/x86_64/release/resource/csdk/logger/src/logger.o
out/linux/x86_64/release/resource/csdk/logger/src/trace.o
... 13 more in current setup
It's not that this makes builds not work, but it tells me something just
isn't right in the script setup.
In fact the same thing happens in an even less work-expected situation,
asking for help. We also had a bit of this discussion recently on list
over dry-run invocations, another "surprise".
Do we have documentation in scons of "things not to do" that could be
helpful in avoiding falling into these kinds of traps?
$ scons -c
scons: Reading SConscript files ...
Mkdir("/home/mats/Work/scons/build/doc/design")
Mkdir("/home/mats/Work/scons/build/doc/design/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/user")
Mkdir("/home/mats/Work/scons/build/doc/user/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/reference")
Mkdir("/home/mats/Work/scons/build/doc/reference/titlepage")
Mkdir("/home/mats/Work/scons/build/doc/man")
Mkdir("/home/mats/Work/scons/build/doc/man/titlepage")
scons: done reading SConscript files.
scons: Cleaning targets ...
scons: done cleaning targets.
and, not in the messages, those directories were populated, so after
starting with a clean git checkout - no "build" directory at all -
"scons -c" leaves me with a build directory with 113 files (.xml, .xsl,
SConstruct, and a few others). did cleaning really need to create
directories and populate them?
any thoughts on how to make this less surprising? or should I just suck
it up and treat this as normal?
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Post by Bill Deegan
Mats,
If your SConscripts are doing a lot of miscellaneous python logic which
create those files and/or use Execute(), then that an issue with your build
setup.
If SCons is doing that, I'd be quite surprised with the exception of any
configure context files.

okay, so let me clarify further.

I know some of what my project does wrong - like one of the scripts that
does do configure stuff being called repeatedly from different places
instead of saving the information once and reusing it. Yes, some of it
is "just doing things in Python" as we've discussed elsewhere. I can
even fix some of it fairly cheaply, if I can ever get people to review
the script changes (certainly not scons' problem). There's not actually
much calling of Execute.

The question was more from the scons viewpoint: it seems like it's hard
for people to discover the best practices, or alternatively, the worst
practices. It seems like the folks who set this project up found a lot
of suboptimal alleys to drive down. We seem not to be alone in that.

[Bill Deegan]

Common problems:
1 - Trying to force SCons order of execution rather than specifying
dependencies (which is the proper usage of any build tool IMHO)
2 - After realizing SConscripts are python.. doing too much python and
using SConscripts as general scripting for your build. Move that logic out
into standalone scripts and run via builders or Command. Not popen. Not
system... etc. The added bonus is that now you can easily test that logic,
which should always be done if it's non-trivial ..
3 - Not reading the docs.. including the manpage when trying to figure out
how to do something.
4 - Thinking SCons should do things a certain way and operating as if that
was true, when the initial presumption was wrong.. (back to #3 is the usual
cause for this)
5 - Reaching to far into SCons when there are simpler, documented, and
supported ways for doing things.

These are mostly broad generalities, but I think cover most of the issues
I've seen working with SCons for more than 10 years..

Maybe worthy of a page and/or blog post..

-Bill

Post by Mats Wichmann

Post by Bill Deegan
Mats,
If your SConscripts are doing a lot of miscellaneous python logic which
create those files and/or use Execute(), then that an issue with your

build

Post by Bill Deegan
setup.
If SCons is doing that, I'd be quite surprised with the exception of any
configure context files.

okay, so let me clarify further.
I know some of what my project does wrong - like one of the scripts that
does do configure stuff being called repeatedly from different places
instead of saving the information once and reusing it. Yes, some of it
is "just doing things in Python" as we've discussed elsewhere. I can
even fix some of it fairly cheaply, if I can ever get people to review
the script changes (certainly not scons' problem). There's not actually
much calling of Execute.
The question was more from the scons viewpoint: it seems like it's hard
for people to discover the best practices, or alternatively, the worst
practices. It seems like the folks who set this project up found a lot
of suboptimal alleys to drive down. We seem not to be alone in that.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users