[Scons-users] Scons none-deterministic behavior for incremental build

Hi,

Not sure if this could be related, but a similar issue happened to me where
I get a relink of a shared library without changing anything in the
project. It happened that I was setting the environment LIBS and LIBPATH
after calling the builder of the library.

Post by Hua Yanghao
Dear Scons Experts,
I recently hit on a very strange problem where scons incremental
builds sometimes OK, sometimes the entire project is re-built.
scons: 3.0.1
python: 3.6.7
scons: rebuilding
`build/configs/qemu_arm_vexpress/arm3/config/config.h' because
AlwaysBuild() is specified
__CONFIG build/configs/qemu_arm_vexpress/arm3/config/config.h
scons: rebuilding
`build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'
`firmware/arch/armv7/arch_init.c' is no longer a dependency
`firmware/arch/armv7/arch_init.c' is a new dependency
It seems some how the C file itself is removed and re-added as the
Object() dependency, and causes rebuild. However this is *NOT*
*always* the case. Sometime the rebuild doesn't happen.
I failed to create a minimal example that can show this issue so far,
but in the production environment it is about 50% full-rebuild and 50%
incremental re-build.
Any thoughts or hints?
Thanks,
Yanghao
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-04 20:23:04 UTC

I does that a lot ... but a quick try to make sure no
LIBS/LIBPATH/CPPPATH etc. is touched after Object() is being called,
still shows the same 50~70% rebuild of the entire project.
I am not using any parallel build, and somehow scons is having a very
none-deterministic behavior.

Actually I fixed partially the issue (e.g. in simpler build the
incremental builds now works fine), due to python set() is having a
random order for every run. I am thinking maybe scons is somehow also
depends on the ordering of a set()? And when it happens to be the same
order then no rebuild?

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Bill Deegan

2018-12-04 21:15:30 UTC

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file?
Or checked in?

It's very unlikely to be set ordering. set's should only be used for
membership checks not iteration in the SCons codebase.

We're actively debugging similar issue.

-Bill

Post by Hua Yanghao
I does that a lot ... but a quick try to make sure no
LIBS/LIBPATH/CPPPATH etc. is touched after Object() is being called,
still shows the same 50~70% rebuild of the entire project.
I am not using any parallel build, and somehow scons is having a very
none-deterministic behavior.
Actually I fixed partially the issue (e.g. in simpler build the
incremental builds now works fine), due to python set() is having a
random order for every run. I am thinking maybe scons is somehow also
depends on the ordering of a set()? And when it happens to be the same
order then no rebuild?

Post by Oscar Anson
Hi,
Not sure if this could be related, but a similar issue happened to me

where I get a relink of a shared library without changing anything in the
project. It happened that I was setting the environment LIBS and LIBPATH
after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-04 21:36:51 UTC

Hi Bill,
config.h is generated file, not checked in, it is generated in the
build folder only.
Well at least I see scons is very odering-sensitive. For example, if
the order of SConscript processing changes (in the case of a set()),
even if there is no dependency actually changed, scons thinks the
target need to be rebuilt.

any progress on these similar issues already? :)

Best Regards,
Yanghao

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Bill Deegan

2018-12-04 22:24:32 UTC

Generated by logic in your sconstruct/sconscripts? or do you shell out and
run a script/program to generate it.

Post by Hua Yanghao
Hi Bill,
config.h is generated file, not checked in, it is generated in the
build folder only.
Well at least I see scons is very odering-sensitive. For example, if
the order of SConscript processing changes (in the case of a set()),
even if there is no dependency actually changed, scons thinks the
target need to be rebuilt.
any progress on these similar issues already? :)
Best Regards,
Yanghao

Post by Bill Deegan
Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated

file? Or checked in?

Post by Bill Deegan
It's very unlikely to be set ordering. set's should only be used for

membership checks not iteration in the SCons codebase.

Post by Bill Deegan
We're actively debugging similar issue.
-Bill

Post by Oscar Anson
Hi,
Not sure if this could be related, but a similar issue happened to me

where I get a relink of a shared library without changing anything in the
project. It happened that I was setting the environment LIBS and LIBPATH
after calling the builder of the library.

`build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'

Post by Hua Yanghao
`firmware/arch/armv7/arch_init.c' is no longer a

dependency

Post by Hua Yanghao
`firmware/arch/armv7/arch_init.c' is a new dependency
It seems some how the C file itself is removed and re-added as the
Object() dependency, and causes rebuild. However this is *NOT*
*always* the case. Sometime the rebuild doesn't happen.
I failed to create a minimal example that can show this issue so far,
but in the production environment it is about 50% full-rebuild and

50%

Post by Hua Yanghao
incremental re-build.
Any thoughts or hints?
Thanks,
Yanghao
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-04 22:54:30 UTC

That is generated by sconstruct, not from shell. Any potential issue
here? (It was working for a very long time before and only breaks
recently when I added some new features).

In the meantime I have isolated the issue to be in Node.Node.changed()
call, where scons old & new signature for the C files seems not
matching (print("###### 3.1") is executed in error case):
1463 if self.has_builder():
|3
1464 print("####### 3")
|3
1465 import SCons.Util
|3
1466 newsig = SCons.Util.MD5signature(contents)
|3
1467 if bi.bactsig != newsig:
|3
1468 if t: Trace(': bactsig %s != newsig %s' %
(bi.bactsig, newsig)) |3
1469 print("####### 3.1")
|3
1470 result = True

Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-04 23:00:27 UTC

Apparently the MD5 signature is taking *turns* some how.

You can see below the old md5 and new md5 checksum is somehow
alternating between two fixed value ...
How is scons calculating the md5 value? is it only depends on the file content?
The md5sum of the "arch_init.c" is:
md5sum firmware/arch/armv7/arch_init.c
db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c

First build:
e494890ea1b6e36acf275f3698a3f3f9
c706a016770a7c85ac42f16f79456797
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Second build:
c706a016770a7c85ac42f16f79456797
e494890ea1b6e36acf275f3698a3f3f9
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Post by Hua Yanghao
That is generated by sconstruct, not from shell. Any potential issue
here? (It was working for a very long time before and only breaks
recently when I added some new features).
In the meantime I have isolated the issue to be in Node.Node.changed()
call, where scons old & new signature for the C files seems not
|3
1464 print("####### 3")
|3
1465 import SCons.Util
|3
1466 newsig = SCons.Util.MD5signature(contents)
|3
|3
1468 if t: Trace(': bactsig %s != newsig %s' %
(bi.bactsig, newsig)) |3
1469 print("####### 3.1")
|3
1470 result = True

Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Bill Deegan

2018-12-04 23:07:41 UTC

Does this happen if you use python 2.7?

Post by Hua Yanghao
Apparently the MD5 signature is taking *turns* some how.
You can see below the old md5 and new md5 checksum is somehow
alternating between two fixed value ...
How is scons calculating the md5 value? is it only depends on the file content?
md5sum firmware/arch/armv7/arch_init.c
db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c
e494890ea1b6e36acf275f3698a3f3f9
c706a016770a7c85ac42f16f79456797
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
______CC
build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
c706a016770a7c85ac42f16f79456797
e494890ea1b6e36acf275f3698a3f3f9
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
______CC
build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Post by Bill Deegan
Generated by logic in your sconstruct/sconscripts? or do you shell out

and run a script/program to generate it.

Post by Bill Deegan
Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a

generated file? Or checked in?

Post by Bill Deegan
It's very unlikely to be set ordering. set's should only be used

for membership checks not iteration in the SCons codebase.

Post by Bill Deegan
We're actively debugging similar issue.
-Bill

Post by Hua Yanghao
I does that a lot ... but a quick try to make sure no
LIBS/LIBPATH/CPPPATH etc. is touched after Object() is being

called,

Post by Hua Yanghao
still shows the same 50~70% rebuild of the entire project.
I am not using any parallel build, and somehow scons is having a

very

Post by Hua Yanghao
none-deterministic behavior.
Actually I fixed partially the issue (e.g. in simpler build the
incremental builds now works fine), due to python set() is having a
random order for every run. I am thinking maybe scons is somehow

also

Post by Hua Yanghao
depends on the ordering of a set()? And when it happens to be the

same

Post by Hua Yanghao
order then no rebuild?

Post by Oscar Anson
Hi,
Not sure if this could be related, but a similar issue happened

to me where I get a relink of a shared library without changing anything in
the project. It happened that I was setting the environment LIBS and
LIBPATH after calling the builder of the library.

`build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'

Post by Hua Yanghao
`firmware/arch/armv7/arch_init.c' is no longer a

dependency

Post by Hua Yanghao
`firmware/arch/armv7/arch_init.c' is a new dependency
It seems some how the C file itself is removed and re-added as

the

Post by Hua Yanghao
Object() dependency, and causes rebuild. However this is *NOT*
*always* the case. Sometime the rebuild doesn't happen.
I failed to create a minimal example that can show this issue

so far,

Post by Hua Yanghao
but in the production environment it is about 50% full-rebuild

and 50%

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-05 08:23:35 UTC

It is impossible to run on python2.7 within a reasonable time as a
huge list of python3-only syntax/modules needs to be adapted ...

Post by Bill Deegan
Does this happen if you use python 2.7?

Post by Hua Yanghao
Apparently the MD5 signature is taking *turns* some how.
You can see below the old md5 and new md5 checksum is somehow
alternating between two fixed value ...
How is scons calculating the md5 value? is it only depends on the file content?
md5sum firmware/arch/armv7/arch_init.c
db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c
e494890ea1b6e36acf275f3698a3f3f9
c706a016770a7c85ac42f16f79456797
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
c706a016770a7c85ac42f16f79456797
e494890ea1b6e36acf275f3698a3f3f9
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-05 08:38:27 UTC

Alright, I have figured out a way to bypass/stub most of the python3
code and can confirm this *ONLY* happens in python3. And python3.6.x
and python3.7.x doesn't make a difference.

But I think the problem is clearly that the previously stored md5sum
and currently computed md5sum is not reliable, seems scons computation
of md5sum of a dependency does not only depends on the file content.

Post by Hua Yanghao
It is impossible to run on python2.7 within a reasonable time as a
huge list of python3-only syntax/modules needs to be adapted ...

Post by Bill Deegan
Does this happen if you use python 2.7?

Post by Hua Yanghao
Apparently the MD5 signature is taking *turns* some how.
You can see below the old md5 and new md5 checksum is somehow
alternating between two fixed value ...
How is scons calculating the md5 value? is it only depends on the file content?
md5sum firmware/arch/armv7/arch_init.c
db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c
e494890ea1b6e36acf275f3698a3f3f9
c706a016770a7c85ac42f16f79456797
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
c706a016770a7c85ac42f16f79456797
e494890ea1b6e36acf275f3698a3f3f9
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Hua Yanghao

2018-12-05 09:50:15 UTC

I finally nailed it down.

It is because the ordering of SConscript files processing, where in
SConscript I allow users/modules to provide additional CFLAGS.
I used a set() to eliminate duplicated SConscript, but in Python3
set() does not have a stable run-time ordering.

The fix is very simple to convert the set() back to list and sort it out.

So scons is behaving all correctly and it is my user code problem.

Thanks Oscar and Bill for all the hints!

Best Regards,
Yanghao

Post by Hua Yanghao
Alright, I have figured out a way to bypass/stub most of the python3
code and can confirm this *ONLY* happens in python3. And python3.6.x
and python3.7.x doesn't make a difference.
But I think the problem is clearly that the previously stored md5sum
and currently computed md5sum is not reliable, seems scons computation
of md5sum of a dependency does not only depends on the file content.

Post by Hua Yanghao
It is impossible to run on python2.7 within a reasonable time as a
huge list of python3-only syntax/modules needs to be adapted ...

Post by Bill Deegan
Does this happen if you use python 2.7?

Post by Hua Yanghao
Apparently the MD5 signature is taking *turns* some how.
You can see below the old md5 and new md5 checksum is somehow
alternating between two fixed value ...
How is scons calculating the md5 value? is it only depends on the file content?
md5sum firmware/arch/armv7/arch_init.c
db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c
e494890ea1b6e36acf275f3698a3f3f9
c706a016770a7c85ac42f16f79456797
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
c706a016770a7c85ac42f16f79456797
e494890ea1b6e36acf275f3698a3f3f9
Returning True
in self.changed ... 2
return not r: False
['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
<SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o

Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.

Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
We're actively debugging similar issue.
-Bill

Hi,
Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Mats Wichmann

2018-12-05 11:08:21 UTC