Batch building includes unchanged targets

[Sean Houghton]

I'm trying to batch build with a custom builder but my action is always
passed the entire set of source/target pairs - even if the source and
target files haven't changed. I've verified (with --tree=status) that
scons considers them "E C" (they exist and are current) but the --tree
output looks like it doesn't work properly with batch building.

My test action looks like this:

Action.Action('echo $CHANGED_TARGETS', batch_key=True,
targets='$CHANGED_TARGETS')


Any ideas?
-Sean

[Dirk Bächle]

Hi Sean,

Post by Sean Houghton
I'm trying to batch build with a custom builder but my action is
always passed the entire set of source/target pairs - even if the
source and target files haven't changed. I've verified (with
--tree=status) that scons considers them "E C" (they exist and are
current) but the --tree output looks like it doesn't work properly
with batch building.
Action.Action('echo $CHANGED_TARGETS', batch_key=True,
targets='$CHANGED_TARGETS')
Any ideas?

nope, not yet. But it would help if you could give a bit more info about
how exactly you call the Builder, if possible as a short and
self-contained example. Then I can try to reproduce the error on my side...

Best regards,

Dirk

[Sean Houghton]

I discovered a bug in my batch builder caused by a missing file flush which
resulted in constant rebuilds. However, the problem is now that any time a
single input changes the entire set of sources is passed to the batch
builder, regardless of change state.

I've created a small repro and example output at
https://gist.github.com/seanhoughton/7148635

Post by Dirk Bächle
Hi Sean,

Post by Sean Houghton
I'm trying to batch build with a custom builder but my action is always
passed the entire set of source/target pairs - even if the source and
target files haven't changed. I've verified (with --tree=status) that
scons considers them "E C" (they exist and are current) but the --tree
output looks like it doesn't work properly with batch building.
Action.Action('echo $CHANGED_TARGETS', batch_key=True,
targets='$CHANGED_TARGETS')
Any ideas?

nope, not yet. But it would help if you could give a bit more info about
how exactly you call the Builder, if possible as a short and self-contained
example. Then I can try to reproduce the error on my side...
Best regards,
Dirk
______________________________**_________________
Scons-users mailing list
http://four.pairlist.net/**mailman/listinfo/scons-users<http://four.pairlist.net/mailman/listinfo/scons-users>

[Dirk Bächle]

Post by Sean Houghton
I discovered a bug in my batch builder caused by a missing file flush
which resulted in constant rebuilds. However, the problem is now that
any time a single input changes the entire set of sources is passed to
the batch builder, regardless of change state.

Thanks for the example...

You're using the batch functionality, which basically means that you
give SCons a black box and say: "Here is my batch builder. It gets N
input files, and has M outputs (=targets)."

With this, SCons will register an NxM dependency for the DAG where each
of the M targets depends on all N sources. This is what you're currently
seeing, and I don't think it's a bug...it's what batch building should do.

If you don't want this behaviour, don't use the batch build but process
your files in single calls (single_source=True). This will also give you
less implicit dependencies overall, so your build usually performs
better regarding speed and memory.

Regards,

Dirk

[Sean Houghton]

I guess "batch building" is an overloaded phrase. In my experience batch
building means processing a set of independent items that all share a
common processing pipeline. In my case I'm submitting the items to be
built to a distributed build system that does a better job scheduling if it
knows about all of the items up front rather than one at a time.

What's the purpose of the targets="$CHANGED_TARGETS" parameter? There's no
documentation on the batch building so I'm trying to learn all I can by
following the MSVC builder in the source.

I'm looking at possibly using Node.changed() to skip processing
source/target pairs that haven't changed since the last build. This looks
promising but I don't like bypassing the built in behavior if I don't have
to.

Post by Sean Houghton
I discovered a bug in my batch builder caused by a missing file flush
which resulted in constant rebuilds. However, the problem is now that any
time a single input changes the entire set of sources is passed to the
batch builder, regardless of change state.
Thanks for the example...

You're using the batch functionality, which basically means that you give
SCons a black box and say: "Here is my batch builder. It gets N input
files, and has M outputs (=targets)."

With this, SCons will register an NxM dependency for the DAG where each of
the M targets depends on all N sources. This is what you're currently
seeing, and I don't think it's a bug...it's what batch building should do.

If you don't want this behaviour, don't use the batch build but process
your files in single calls (single_source=True). This will also give you
less implicit dependencies overall, so your build usually performs better
regarding speed and memory.

Regards,

Dirk

______________________________**_________________
Scons-users mailing list
Scons-***@scons.org
http://four.pairlist.net/**mailman/listinfo/scons-users<http://four.pairlist.net/mailman/listinfo/scons-users>

[Dirk Bächle]

Sean,

Post by Sean Houghton
I guess "batch building" is an overloaded phrase. In my experience
batch building means processing a set of independent items that all
share a common processing pipeline. In my case I'm submitting the
items to be built to a distributed build system that does a better job
scheduling if it knows about all of the items up front rather than one
at a time.

this sounds very interesting. Which distributed build system are we
talking about? And if you already seem to have a good way of building
things, which role do you want SCons to play?

Post by Sean Houghton
What's the purpose of the targets="$CHANGED_TARGETS" parameter?
There's no documentation on the batch building so I'm trying to learn
all I can by following the MSVC builder in the source.

Yeah, documentation about this is scarce and I'd have to dig into the
source code now to give you a correct answer. The MAN page lists all the
reserved variables (like CHANGED_TARGETS) with a short explanation of
each...

Post by Sean Houghton
I'm looking at possibly using Node.changed() to skip processing
source/target pairs that haven't changed since the last build. This
looks promising but I don't like bypassing the built in behavior if I
don't have to.

I don't think you'll have to hack around in the core like that. From
what I understood so far, this is exactly what "$CHANGED_TARGETS" is
for...picking and processing only the targets that aren't up-to-date and
need a rebuild.

Regards,

Dirk

[Bill Deegan]

Dirk,

If I remember correctly the batch compilation logic was only implemented
(By Steven Knight (?)) for the visual C compiler.
With the intent that at some point it would be generally implemented. I'm
not sure how much plumbing is in place.

-Bill

Post by Dirk Bächle
Sean,

Post by Sean Houghton
I guess "batch building" is an overloaded phrase. In my experience batch
building means processing a set of independent items that all share a
common processing pipeline. In my case I'm submitting the items to be
built to a distributed build system that does a better job scheduling if it
knows about all of the items up front rather than one at a time.

this sounds very interesting. Which distributed build system are we
talking about? And if you already seem to have a good way of building
things, which role do you want SCons to play?
What's the purpose of the targets="$CHANGED_TARGETS" parameter? There's

Post by Sean Houghton
no documentation on the batch building so I'm trying to learn all I can by
following the MSVC builder in the source.
Yeah, documentation about this is scarce and I'd have to dig into the

source code now to give you a correct answer. The MAN page lists all the
reserved variables (like CHANGED_TARGETS) with a short explanation of
each...
I'm looking at possibly using Node.changed() to skip processing

Post by Sean Houghton
source/target pairs that haven't changed since the last build. This looks
promising but I don't like bypassing the built in behavior if I don't have
to.

I don't think you'll have to hack around in the core like that. From what
I understood so far, this is exactly what "$CHANGED_TARGETS" is
for...picking and processing only the targets that aren't up-to-date and
need a rebuild.
Regards,
Dirk
______________________________**_________________
Scons-users mailing list
http://four.pairlist.net/**mailman/listinfo/scons-users<http://four.pairlist.net/mailman/listinfo/scons-users>

[Dirk Bächle]

Post by Bill Deegan
Dirk,
If I remember correctly the batch compilation logic was only
implemented (By Steven Knight (?)) for the visual C compiler.
With the intent that at some point it would be generally implemented.
I'm not sure how much plumbing is in place.

Yes, I've been wading through the current sources a lot recently, and
found several places with comments like "TODO: batch...". So it's
probably not the safest feature to use... ;)

Dirk