Discussion:
[Scons-users] Clone effectiveness
Pierre-Luc Boily
2018-09-13 18:27:07 UTC
Permalink
Hello!

Just read on the forum :

/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./

Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.

We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?

Thx



--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
Damien Hocking
2018-09-13 18:52:34 UTC
Permalink
We do this too. Most dlls and sos have a cloned environment.
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Mats Wichmann
2018-09-13 19:20:27 UTC
Permalink
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.

Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?

* - recent build timings:
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds

15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Hua Yanghao
2018-09-14 11:55:48 UTC
Permalink
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Gary Oberbrunner
2018-09-14 12:02:49 UTC
Permalink
No. It may create an override internally, if you pass some keyword args,
but that's not a clone. So you can call Object() thousands of times for
sure.

-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
On Sep 13, 2018, at 12:27 PM, Pierre-Luc Boily <
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start
seeing
Post by Mats Wichmann
Post by Damien Hocking
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800,
300 are
Post by Mats Wichmann
Post by Damien Hocking
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning
of
Post by Mats Wichmann
Post by Damien Hocking
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating
of the
Post by Mats Wichmann
Post by Damien Hocking
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
Hua Yanghao
2018-09-14 12:15:24 UTC
Permalink
Thanks Gary.

In this case I think I am mostly using overrides, and only one env per
software image build.

If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
No. It may create an override internally, if you pass some keyword args, but that's not a clone. So you can call Object() thousands of times for sure.
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Gary Oberbrunner
2018-09-14 13:11:56 UTC
Permalink
Most people use them in a sub-SConscript so they can make changes to the
env in that script that don't propate up to the calling SConstruct. Not the
only use case of course, but a popular one.

-- Gary
Post by Hua Yanghao
Thanks Gary.
In this case I think I am mostly using overrides, and only one env per
software image build.
If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
Post by Gary Oberbrunner
No. It may create an override internally, if you pass some keyword args,
but that's not a clone. So you can call Object() thousands of times for
sure.
Post by Gary Oberbrunner
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not
getting
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
killed on memory; I can build easily enough in a 2gb
Vagrant-provisioned
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that
exists
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
On Sep 13, 2018, at 12:27 PM, Pierre-Luc Boily <
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may
start seeing
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800,
300 are
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the
beginning of
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
*each *sconscript. So, if I understand well, it might not be the
best
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the
creating of the
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
Hua Yanghao
2018-09-14 13:41:35 UTC
Permalink
But seems passing flags also does similar thing with override?

What must be changed in the env that cannot be done with ovrride? as
majority of the builds are based on variable substitution.
Most people use them in a sub-SConscript so they can make changes to the env in that script that don't propate up to the calling SConstruct. Not the only use case of course, but a popular one.
-- Gary
Post by Hua Yanghao
Thanks Gary.
In this case I think I am mostly using overrides, and only one env per
software image build.
If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
No. It may create an override internally, if you pass some keyword args, but that's not a clone. So you can call Object() thousands of times for sure.
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Gary Oberbrunner
2018-09-14 14:18:43 UTC
Permalink
Overrides aren't expensive. Go ahead & use them, pass keyword flags to
builders etc.
But let's say you have a sub-SConscript that must use a different compiler
for everything in that subdir. Typical use is to import the main env, clone
it, and modify for all the targets in that sub-SConscript.
Post by Hua Yanghao
But seems passing flags also does similar thing with override?
What must be changed in the env that cannot be done with ovrride? as
majority of the builds are based on variable substitution.
Post by Gary Oberbrunner
Most people use them in a sub-SConscript so they can make changes to the
env in that script that don't propate up to the calling SConstruct. Not the
only use case of course, but a popular one.
Post by Gary Oberbrunner
-- Gary
Post by Hua Yanghao
Thanks Gary.
In this case I think I am mostly using overrides, and only one env per
software image build.
If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
Post by Gary Oberbrunner
No. It may create an override internally, if you pass some keyword
args, but that's not a clone. So you can call Object() thousands of times
for sure.
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of
environment?
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not
getting
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
killed on memory; I can build easily enough in a 2gb
Vagrant-provisioned
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that
exists
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%,
I'd
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
On Sep 13, 2018, at 12:27 PM, Pierre-Luc Boily <
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may
start seeing
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
memory and speed impacted./
Our project branch contains 800 sconscript files. From those
800, 300 are
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the
beginning of
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
*each *sconscript. So, if I understand well, it might not be
the best
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
approach. It would be better to do a Clone only when it is
needed?
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
Reducing the number of clone could potentially speed up the
creating of the
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
Hua Yanghao
2018-09-14 16:56:19 UTC
Permalink
I see. I solve this problem by defining and creating only new env for
a build at top level, and never creating new env in SConscript.
e.g. all SConscript is using a common pre-defined env, but one
SConscript can be used by multiple env.

This way my number of env is equal to the number of build target I have.
Overrides aren't expensive. Go ahead & use them, pass keyword flags to builders etc.
But let's say you have a sub-SConscript that must use a different compiler for everything in that subdir. Typical use is to import the main env, clone it, and modify for all the targets in that sub-SConscript.
Post by Hua Yanghao
But seems passing flags also does similar thing with override?
What must be changed in the env that cannot be done with ovrride? as
majority of the builds are based on variable substitution.
Most people use them in a sub-SConscript so they can make changes to the env in that script that don't propate up to the calling SConstruct. Not the only use case of course, but a popular one.
-- Gary
Post by Hua Yanghao
Thanks Gary.
In this case I think I am mostly using overrides, and only one env per
software image build.
If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
No. It may create an override internally, if you pass some keyword args, but that's not a clone. So you can call Object() thousands of times for sure.
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
Post by Pierre-Luc Boily
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start seeing
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800, 300 are
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning of
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating of the
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Bill Deegan
2018-09-14 21:30:10 UTC
Permalink
Yes. That's generally how I go about it.
An Environment per type of build. Passed via SConscript's export argument
generally to all SConscripts but that my not make sense depending on your
build.
That way each SConscript could check if the env it's been passed is one it
"wants" to build.
Post by Hua Yanghao
I see. I solve this problem by defining and creating only new env for
a build at top level, and never creating new env in SConscript.
e.g. all SConscript is using a common pre-defined env, but one
SConscript can be used by multiple env.
This way my number of env is equal to the number of build target I have.
Post by Gary Oberbrunner
Overrides aren't expensive. Go ahead & use them, pass keyword flags to
builders etc.
Post by Gary Oberbrunner
But let's say you have a sub-SConscript that must use a different
compiler for everything in that subdir. Typical use is to import the main
env, clone it, and modify for all the targets in that sub-SConscript.
Post by Gary Oberbrunner
Post by Hua Yanghao
But seems passing flags also does similar thing with override?
What must be changed in the env that cannot be done with ovrride? as
majority of the builds are based on variable substitution.
Post by Gary Oberbrunner
Most people use them in a sub-SConscript so they can make changes to
the env in that script that don't propate up to the calling SConstruct. Not
the only use case of course, but a popular one.
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
-- Gary
Post by Hua Yanghao
Thanks Gary.
In this case I think I am mostly using overrides, and only one env
per
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
software image build.
If we have overrides what are the scenarios that a Clone() of the env
is a must to have?
On Fri, Sep 14, 2018 at 2:03 PM Gary Oberbrunner <
Post by Gary Oberbrunner
No. It may create an override internally, if you pass some keyword
args, but that's not a clone. So you can call Object() thousands of times
for sure.
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
-- Gary
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of
environment?
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts
contain a
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're
not getting
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
killed on memory; I can build easily enough in a 2gb
Vagrant-provisioned
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
(Linux) image. So while I'd like to *know* the numbers, it's
mostly
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
curiosity.
Maybe we can resurrect some of the benchmarking/timing code
that exists
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by
50%, I'd
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
On Sep 13, 2018, at 12:27 PM, Pierre-Luc Boily <
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you
may start seeing
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
memory and speed impacted./
Our project branch contains 800 sconscript files. From
those 800, 300 are
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the
beginning of
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
*each *sconscript. So, if I understand well, it might not
be the best
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
approach. It would be better to do a Clone only when it is
needed?
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
Reducing the number of clone could potentially speed up the
creating of the
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
tree?
Thx
--
http://scons.1086193.n5.nabble.com/Users-f16930.html
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Gary Oberbrunner
Post by Hua Yanghao
Post by Mats Wichmann
Post by Damien Hocking
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
--
Gary
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Mats Wichmann
2018-09-16 00:42:32 UTC
Permalink
Post by Bill Deegan
Yes. That's generally how I go about it.
An Environment per type of build. Passed via SConscript's export argument
generally to all SConscripts but that my not make sense depending on your
build.
That way each SConscript could check if the env it's been passed is one it
"wants" to build.
so it seems there's one specific case for a new cloned env: if a
subsidiary script is going to call env.ParseConfig that's going to make
changes to the big env we're sharing now. Maybe not in a bad way, but
still, it's a little less predictable than explicit additions.
Bill Deegan
2018-09-14 21:28:16 UTC
Permalink
It does not.
It uses the DefaultEnvironment object.
Post by Hua Yanghao
Is every call to "Object()" implicitly creates a clone of environment?
I seems like so just want to confirm.
Post by Mats Wichmann
Post by Damien Hocking
We do this too. Most dlls and sos have a cloned environment.
As I said earlier, we clone lots (>90% of our sconscripts contain a
Clone() call), but the actual build time dwarfs the "process the
SConscripts" phase (*) so I don't worry much about it. We're not getting
killed on memory; I can build easily enough in a 2gb Vagrant-provisioned
(Linux) image. So while I'd like to *know* the numbers, it's mostly
curiosity.
Maybe we can resurrect some of the benchmarking/timing code that exists
in the codebase now to get more info?
Total build time: 360.533189 seconds (using scons -j4)
Total SConscript file execution time: 14.785102 seconds
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
Post by Damien Hocking
On Sep 13, 2018, at 12:27 PM, Pierre-Luc Boily <
Hello!
/Here's the caveat.
Don't make a gazillion Clone's they're not lightweight.
If you make 100, it won't kill you, if you make 1000's you may start
seeing
Post by Mats Wichmann
Post by Damien Hocking
memory and speed impacted./
Our project branch contains 800 sconscript files. From those 800,
300 are
Post by Mats Wichmann
Post by Damien Hocking
for windows compilation, 500 are for unix compilation.
We systematically create a clone of the environment at the beginning
of
Post by Mats Wichmann
Post by Damien Hocking
*each *sconscript. So, if I understand well, it might not be the best
approach. It would be better to do a Clone only when it is needed?
Reducing the number of clone could potentially speed up the creating
of the
Post by Mats Wichmann
Post by Damien Hocking
tree?
Thx
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Pierre-Luc Boily
2018-09-18 14:31:04 UTC
Permalink
I agree 15 seconds is not a lot compare to the whole. But when doing
iterative build, 15 seconds it's a lot! At least we have the --interactive
as a good work around (on unix).

So, beside having longer tree creation with many Clone, could this lead to
other problems? Is it better to avoid Clone where I can?
Post by Mats Wichmann
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
Hua Yanghao
2018-09-18 14:46:18 UTC
Permalink
Fully agree.

My approach is that I have a keyword parameter from the command line
to chose which target to build and then only the pattern-matched
builds are actually created, essentially only one env is created. I do
it only because I want to avoid the initial startup latency ... if
your env is not conditionally created or for a built that really have
many clones then bad luck. Of course it would definitely be great if
the we cannot feel the existence of scons's runtime.
On Tue, Sep 18, 2018 at 4:31 PM Pierre-Luc Boily
Post by Pierre-Luc Boily
I agree 15 seconds is not a lot compare to the whole. But when doing
iterative build, 15 seconds it's a lot! At least we have the --interactive
as a good work around (on unix).
So, beside having longer tree creation with many Clone, could this lead to
other problems? Is it better to avoid Clone where I can?
Post by Mats Wichmann
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Bill Deegan
2018-09-18 15:59:36 UTC
Permalink
Don't forget what Donald Knuth said "*premature optimization is the root of
all evil (or at least most of it) in programming"..*

Is your startup time (null incremental build time) an issue?
If not then don't worry about creating extra clones.

Is it an issue? The run --debug=count and see how many environments you
have. If that number is LARGE then do something to reduce the number of
Clones you're making.

There's no hard and fast rule here. Anyone saying "Clone() is bad" is more
times than not wrong.
Anyone saying "Clone() a gazillion times and don't worry about it" is more
times than not wrong as well..

I've done some benchmarking on some larger SCons build systems.
The top two null incremental (or when rebuilding a small percentage of
targets due to some small change) build time users are often:
1. Subst() calls
2. Reading and writing sconsign.

Hey.. guess what I've been working on for a while... A Subst rewrite to
hopefully address some performance issues. Guess what's also on the list..
sconsign rework..

Hope this helps,
Bill
Post by Hua Yanghao
Fully agree.
My approach is that I have a keyword parameter from the command line
to chose which target to build and then only the pattern-matched
builds are actually created, essentially only one env is created. I do
it only because I want to avoid the initial startup latency ... if
your env is not conditionally created or for a built that really have
many clones then bad luck. Of course it would definitely be great if
the we cannot feel the existence of scons's runtime.
On Tue, Sep 18, 2018 at 4:31 PM Pierre-Luc Boily
Post by Pierre-Luc Boily
I agree 15 seconds is not a lot compare to the whole. But when doing
iterative build, 15 seconds it's a lot! At least we have the
--interactive
Post by Pierre-Luc Boily
as a good work around (on unix).
So, beside having longer tree creation with many Clone, could this lead
to
Post by Pierre-Luc Boily
other problems? Is it better to avoid Clone where I can?
Post by Mats Wichmann
15 seconds is slow, but if I sped up the sconscript time by 50%, I'd
still make no more than a 2% impact on the overall time.
--
Sent from: http://scons.1086193.n5.nabble.com/Users-f16930.html
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Loading...