[Scons-users] SCons Python Builder not building in parallel?

Discussion:

Hua Yanghao

2018-09-22 17:16:14 UTC

Dear Scons Users,
I recently noticed that with -jN my CPU utilization is less than 100%
(4 cores) during some phases (e.g. when executing a python builder)
while reaches 400% when compiling C codes. I have a python builder
that generates quite some .h files, which all the C files depend on.
Those .h files has no relationship/dependencies among themselves, and
running with --debug=explain gives below information:

./run configs/qemu_arm_vexpress.py -j8 --debug=explain
Running : ./tools/scons -Q config=configs/qemu_arm_vexpress.py -j8
--debug=explain
USW INFO: usw_config completed in 1.60 seconds.
scons: rebuilding
`build/configs/qemu_arm_vexpress/bios_bin/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/bios_bin/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/bios_elf/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/bios_elf/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core0/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core0/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core1/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core1/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core2/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core2/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core3/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core3/config/config.h

Any hints why these are not building in parallel? Is it because scons
builds python with multithreads where the GIL eats all the CPU
intensive workloads performance? Or should I re-implement the builder
in such a way that it is not using scons tools but rather running as a
dedicated executable and then wrapped into scons tools?

Thanks,
Yanghao Hua

Bill Deegan

2018-09-22 20:00:57 UTC

Permalink

So you have a builder which is implemented in python and does non-trivial
amount of work in the builder logic itself?
If so, then yes move the logic to a script and have your builder call the
script with the appropriate arguments.

When scons is running your builders python logic the GIL comes into play.

-Bill

Post by Hua Yanghao
Dear Scons Users,
I recently noticed that with -jN my CPU utilization is less than 100%
(4 cores) during some phases (e.g. when executing a python builder)
while reaches 400% when compiling C codes. I have a python builder
that generates quite some .h files, which all the C files depend on.
Those .h files has no relationship/dependencies among themselves, and
./run configs/qemu_arm_vexpress.py -j8 --debug=explain
Running : ./tools/scons -Q config=configs/qemu_arm_vexpress.py -j8
--debug=explain
USW INFO: usw_config completed in 1.60 seconds.
scons: rebuilding
`build/configs/qemu_arm_vexpress/bios_bin/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/bios_bin/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/bios_elf/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/bios_elf/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core0/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core0/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core1/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core1/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core2/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core2/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/core3/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/core3/config/config.h
Any hints why these are not building in parallel? Is it because scons
builds python with multithreads where the GIL eats all the CPU
intensive workloads performance? Or should I re-implement the builder
in such a way that it is not using scons tools but rather running as a
dedicated executable and then wrapped into scons tools?
Thanks,
Yanghao Hua
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-10-05 20:32:33 UTC

Permalink

Thanks Bill.

I was thinking why scons does not fork some process to handle the
python-based buliders. But now I realized that this cannot be done
reliably as arbitrary python code could be executed and change the
current python execution context and this can only be done in the same
address space e.g. only threads can be used.

It is not so easy to move the logic to a independent script based
builder due to exactly the same reason ... because I do not have
control of what users might have add in their configuration python
code. So I decided to add a few more prints to ease the feeling of a
none-response period ... until I got some time to really bench marking
my configuration code and find out the hot spots ... :-)

So you have a builder which is implemented in python and does non-trivial amount of work in the builder logic itself?
If so, then yes move the logic to a script and have your builder call the script with the appropriate arguments.
When scons is running your builders python logic the GIL comes into play.
-Bill

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users