[Scons-users] problem with new Jar builder-wrapper in 3.0.1

[Mats Wichmann]

Per the instructions still on the scons.org page, I'm emailing this here
prior to filing a ticket.

When I tried our build with 3.0.1., the CI builders for the java
versions tipped over.

The error message looks like this:

TypeError: coercing to Unicode: need string or buffer, list found:

the relevant bit of the backtrace seems to be:

15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_with_java/java/examples-java/simpleclient/SConscript",
line 25:
15:34:53 example_jar = jdk_env.Jar(target='simpleclient.jar',
source=[simpleclient_classes, 'MANIFEST.MF'])
15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_with_java/scons-local/scons-local-3.0.1/SCons/Environment.py",
line 224:
15:34:53 return self.method(*nargs, **kwargs)
15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_with_java/scons-local/scons-local-3.0.1/SCons/Tool/jar.py",
line 174:
15:34:53 if os.path.isfile(s):
15:34:53 File "/usr/lib/python2.7/genericpath.py", line 29:
15:34:53 st = os.stat(path)


The code in question in scons in Tool/jar.py is brand new:

"""
A pseudo-Builder wrapper around the separate Jar sources{File,Dir}
Builders.
"""

The invocation that triggered the failure looked like this:

simpleclient_classes = jdk_env.Java(target='classes',
source=['src/main/java'])
example_jar = jdk_env.Jar(target='simpleclient.jar',
source=[simpleclient_classes, 'MANIFEST.MF'])

(the second line is the one from the traceback)

so the return from the Java builder is passed to Jar as part of the
source list, but the new code does not appear to expect the type it is
passed as a result. It's not clear if our call to Jar is illegal, but
it worked fine through 3.0.0. It *looks* like the intent is not to do
it the way we do, since the preceding call to the Java builder seems
unnecessary per this doc snippet:

"Any .java files in the source list will be compiled to .class files by
calling the Java Builder. "

So it would be good to figure out if this is a new bug in jar.py, or if
the documentation should be explicit about disallowing the way we are
calling it, which at least to my reading it is not explicit about.

thanks,

-- mats

[Bill Deegan]

Mats,

All return values from builders are NodeLists, (list of nodes)

Can you try:
source=simpleclient_classes+['MANIFEST.MF'])

I've also added a link to this thread in the pull request. Hopefully Daniel
Moody will respond here.

-Bill

Post by Mats Wichmann
Per the instructions still on the scons.org page, I'm emailing this here
prior to filing a ticket.
When I tried our build with 3.0.1., the CI builders for the java
versions tipped over.
15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_
with_java/java/examples-java/simpleclient/SConscript",
15:34:53 example_jar = jdk_env.Jar(target='simpleclient.jar',
source=[simpleclient_classes, 'MANIFEST.MF'])
15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_
with_java/scons-local/scons-local-3.0.1/SCons/Environment.py",
15:34:53 return self.method(*nargs, **kwargs)
15:34:53 File
"/home/jenkins-ci/workspace/iotivity-verify-linux_secured_
with_java/scons-local/scons-local-3.0.1/SCons/Tool/jar.py",
15:34:53 st = os.stat(path)
"""
A pseudo-Builder wrapper around the separate Jar sources{File,Dir}
Builders.
"""
simpleclient_classes = jdk_env.Java(target='classes',
source=['src/main/java'])
example_jar = jdk_env.Jar(target='simpleclient.jar',
source=[simpleclient_classes, 'MANIFEST.MF'])
(the second line is the one from the traceback)
so the return from the Java builder is passed to Jar as part of the
source list, but the new code does not appear to expect the type it is
passed as a result. It's not clear if our call to Jar is illegal, but
it worked fine through 3.0.0. It *looks* like the intent is not to do
it the way we do, since the preceding call to the Java builder seems
"Any .java files in the source list will be compiled to .class files by
calling the Java Builder. "
So it would be good to figure out if this is a new bug in jar.py, or if
the documentation should be explicit about disallowing the way we are
calling it, which at least to my reading it is not explicit about.
thanks,
-- mats
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Post by Bill Deegan
Mats,
All return values from builders are NodeLists, (list of nodes)
source=simpleclient_classes+['MANIFEST.MF'])

indeed, this is the conclusion I was arriving at just now, as the way it
is called looked like:

source=[[classA, classB, classC], 'MANIFEST.MF']

and the embedded list is what causes the problem. there's only one
instance of it in our code, the others used another technique, each with
a comment "scons computes this wrong so work around".

The suggestion you give doesn't work, for reasons I'm not yet sure of,
the workaround from the other examples does. I can hopefully try to
trace that a little deeper so I understand what is happening.

Post by Bill Deegan
I've also added a link to this thread in the pull request. Hopefully Daniel
Moody will respond here.
-Bill

Post by Mats Wichmann
Per the instructions still on the scons.org page, I'm emailing this here
prior to filing a ticket.
When I tried our build with 3.0.1., the CI builders for the java
versions tipped over.

[Mats Wichmann]

Post by Mats Wichmann

Post by Bill Deegan
Mats,
All return values from builders are NodeLists, (list of nodes)
source=simpleclient_classes+['MANIFEST.MF'])

indeed, this is the conclusion I was arriving at just now, as the way it
source=[[classA, classB, classC], 'MANIFEST.MF']
and the embedded list is what causes the problem. there's only one
instance of it in our code, the others used another technique, each with
a comment "scons computes this wrong so work around".
The suggestion you give doesn't work, for reasons I'm not yet sure of,

sorry, "doesn't work" as in:

Preparing target
out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar...
jar cfm
out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar
java/examples-java/simpleclient/MANIFEST.MF <function jarSources at
0x7ff8f7800aa0>
sh: -c: line 0: syntax error near unexpected token `newline'
sh: -c: line 0: `jar cfm
"out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar"
"java/examples-java/simpleclient/MANIFEST.MF" <function jarSources at
0x7ff8f7800aa0>'
scons: ***
[out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar]
Error 1


the "function jarSources" is confusing and presumably the problem, it
certainly shouldn't be part of the shell command passed to invoke jar...

[Bill Deegan]

Can you make a simple testcase to reproduce?

Then we'll be able to add to the test suite and work on resolving it more
easily.

-Bill

Post by Mats Wichmann

Post by Mats Wichmann

Post by Bill Deegan
Mats,
All return values from builders are NodeLists, (list of nodes)
source=simpleclient_classes+['MANIFEST.MF'])

indeed, this is the conclusion I was arriving at just now, as the way it
source=[[classA, classB, classC], 'MANIFEST.MF']
and the embedded list is what causes the problem. there's only one
instance of it in our code, the others used another technique, each with
a comment "scons computes this wrong so work around".
The suggestion you give doesn't work, for reasons I'm not yet sure of,

Preparing target
out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar...
jar cfm
out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar
java/examples-java/simpleclient/MANIFEST.MF <function jarSources at
0x7ff8f7800aa0>
sh: -c: line 0: syntax error near unexpected token `newline'
sh: -c: line 0: `jar cfm
"out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar"
"java/examples-java/simpleclient/MANIFEST.MF" <function jarSources at
0x7ff8f7800aa0>'
scons: ***
[out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar]
Error 1
the "function jarSources" is confusing and presumably the problem, it
certainly shouldn't be part of the shell command passed to invoke jar...
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Bill Deegan]

Also what is the other technique used elsewhere in your codebase to resolve
the issue?

Post by Bill Deegan
Can you make a simple testcase to reproduce?
Then we'll be able to add to the test suite and work on resolving it more
easily.
-Bill

Post by Mats Wichmann

Post by Mats Wichmann

Post by Bill Deegan
Mats,
All return values from builders are NodeLists, (list of nodes)
source=simpleclient_classes+['MANIFEST.MF'])

indeed, this is the conclusion I was arriving at just now, as the way it
source=[[classA, classB, classC], 'MANIFEST.MF']
and the embedded list is what causes the problem. there's only one
instance of it in our code, the others used another technique, each with
a comment "scons computes this wrong so work around".
The suggestion you give doesn't work, for reasons I'm not yet sure of,

Preparing target
out/linux/x86_64/debug/java/examples-java/simpleclient/simpl
eclient.jar...
jar cfm
out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar
java/examples-java/simpleclient/MANIFEST.MF <function jarSources at
0x7ff8f7800aa0>
sh: -c: line 0: syntax error near unexpected token `newline'
sh: -c: line 0: `jar cfm
"out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar"
"java/examples-java/simpleclient/MANIFEST.MF" <function jarSources at
0x7ff8f7800aa0>'
scons: ***
[out/linux/x86_64/debug/java/examples-java/simpleclient/simpleclient.jar]
Error 1
the "function jarSources" is confusing and presumably the problem, it
certainly shouldn't be part of the shell command passed to invoke jar...
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Post by Bill Deegan
Also what is the other technique used elsewhere in your codebase to resolve
the issue?

not using the return from the Java builder.

A working (but as noted not ideal) example does this:

simpleclientserver_classes = jdk_env.Java(target='classes',
source=['src/main/java'])
# jdk_env.Jar(target='simpleclientserver.jar',
# source=[simpleclientserver_classes, 'MANIFEST.MF'])
# SCons is incorrectly calculating the output classes for the
simpleclientserver sample.
# The following is a work around for the build command above. The
unfortunate
# side effect of this work around results in a larger output.
example_jar =jdk_env.Jar(target='simpleclientserver.jar',
source=['classes','MANIFEST.MF'],
JARCHDIR='$SOURCE')


that is, "classes' string (whatever that means) instead of using
simpleclientserver_classes as part of the source description.

[Bill Deegan]

Ahh gotcha.
Yes if you have inner classes,etc Java creates files that sometimes SCons
will miss.
(different java versions were (and maybe still do) generate different file
names for inner classes,etc..

So specifying the directory is safer, but you may end up with dangling
files if the non-one-to-one .java -> .class files change.
(Assuming you don't delete the classes directory on clean).

Re the manifest file. I'll have to dig a bit deeper to figure that one out.
Likely not until next week and the weekend is filled with a wedding and
travel. :)


-Bill

Post by Bill Deegan

Post by Bill Deegan
Also what is the other technique used elsewhere in your codebase to

resolve

Post by Bill Deegan
the issue?

not using the return from the Java builder.
simpleclientserver_classes = jdk_env.Java(target='classes',
source=['src/main/java'])
# jdk_env.Jar(target='simpleclientserver.jar',
# source=[simpleclientserver_classes, 'MANIFEST.MF'])
# SCons is incorrectly calculating the output classes for the
simpleclientserver sample.
# The following is a work around for the build command above. The
unfortunate
# side effect of this work around results in a larger output.
example_jar =jdk_env.Jar(target='simpleclientserver.jar',
source=['classes','MANIFEST.MF'],
JARCHDIR='$SOURCE')
that is, "classes' string (whatever that means) instead of using
simpleclientserver_classes as part of the source description.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Okay, this has turned into a complete rathole of a day, sigh. There's a
reason I hate Java, it always wastes my time :)

One thing that's going wrong is that we don't get our manifest file
actually included in the jarfile, we get a default one instead. This
seems to be due to an unaware check in Tool/jar.py, namely from this
sequence:

for s in source:
s = env.subst(s)
if isinstance(s, SCons.Node.FS.Base):
...
else:
if os.path.isfile(s):

if the file is the manifest file, it should be picked by this "if", but
it is not if I pass "MANIFEST.MF" in the list to Jar. However if I pass
it as a file object using File("MANIFEST.MF") it is picked up. In other
words,following this call:

simpleserver_classes = jdk_env.Java(target='classes',
source=['src/main/java'])

this call fails to pick up the manifest:

example_jar = jdk_env.Jar(target='simpleserver.jar',
source=simpleserver_classes + ['MANIFEST.MF'])

but this call successfully includes it:

example_jar = jdk_env.Jar(target='simpleserver.jar',
source=simpleserver_classes + [File('MANIFEST.MF')])

Is this expected behavior?

(adding a bunch of debugs to Tool/jar.py shows that in the former case
it has failed the first three checks under the else: clause above which
are os.path.isfile, os.path.isdir and s[-len(java_suffix):], and fallen
through to the else clause at the end which makes the wrong assumption -
it's either a swig file or an uncreated directory).

Is this due to variantdir stuff?

[Daniel Moody]

Hey Mats,

I just joined the users mailing list, and recreated the email from the link
(https://pairlist4.pair.net/pipermail/scons-users/2017-November/006469.html),
so sorry if the formatting gets messed up.

Regarding this issue, the Jar builder does not handle embedded lists. I saw
you created a flat list and still the manifest did not work correctly.

I tried to recreate your issue with the flat source list and manifest
source but I could not. I created a test which I thought was similar to
what you are doing in your build here:
https://github.com/SConsProject/scons/pull/18/commits/e474eb4e85d18fe56df4bbbdc180df951fbb20f0

And seems to be working with changes I submitted to the jar.py tool.

Let me know if you have any ideas on how I can write a test that can
reproduce your issue, or if these changes fix it. :)

[Mats Wichmann]

Post by Daniel Moody
Hey Mats,
I just joined the users mailing list, and recreated the email from the link
(https://pairlist4.pair.net/pipermail/scons-users/2017-November/006469.html),
so sorry if the formatting gets messed up.
Regarding this issue, the Jar builder does not handle embedded lists. I saw
you created a flat list and still the manifest did not work correctly.

non-flat lists were our problem. there was a prospect of having three
instances, but two had already been changed, leaving the older code
commented out, probably due to problems with including the manifest. I
actually toyed with using Flatten() in our code to make the fix explicit
for future developers. I may go back to that, though if scons is going
to accept it, maybe I don't need to - needing a flat list is fine as
long as it's documented :)

Post by Daniel Moody
I tried to recreate your issue with the flat source list and manifest
source but I could not. I created a test which I thought was similar to
https://github.com/SConsProject/scons/pull/18/commits/e474eb4e85d18fe56df4bbbdc180df951fbb20f0
And seems to be working with changes I submitted to the jar.py tool.
Let me know if you have any ideas on how I can write a test that can
reproduce your issue, or if these changes fix it. :)

So to be clear, if a manifest is not specified, jar creates one. So our
builds do end up with a manifest file, just not the one we wanted to put
there. From just glancing at it, your test looks like it will check for
that case, since it looks for specific contents. In our instance, we
tried to supply:

===
Manifest-Version: 1.0
Class-Path: iotivity.jar
Created-By: 1.8.0_151 (Oracle Corporation)
Main-Class: org.iotivity.base.examples.SimpleServer
===

but the "default" one that ends up in the jarfile looks like:

===
Manifest-Version: 1.0
Created-By: 1.8.0_151 (Oracle Corporation)
===

It was only adding debug prints to jar.py that showed me it wasn't
actually recognizing a manifest file in the source list and to try
something different - if I turn it into a node via File('MANIFEST.MF')
it's picked up fine by that part of the code, but otherwise it fails all
of the checks and is discarded.

If I'd been a little more awake I would have just grokked it from the
output line from building, which of course now is quite clear:

jar cf
out/linux/x86_64/debug/java/examples-java/simpleserver/simpleserver.jar
-C out/linux/x86_64/debug/java/examples-java/simpleserver/classes ...

as opposed to a working:

jar cfm
out/linux/x86_64/debug/java/examples-java/simpleserver/simpleserver.jar
java/examples-java/simpleserver/MANIFEST.MF -C
out/linux/x86_64/debug/java/examples-java/simpleserver/classes ...

no 'm' flag, no manifest file.

I'm still not 100% clear why it is not being grokked as a manifest but
as I said, I think it's variant-dir stuff. With debug on a failing case:

XXX found non-node path: MANIFEST.MF
XXX we are in:
/home/mats/iotivity.work/out/linux/x86_64/debug/java/examples-java/simpleclientserver

Namely it's going to do os.path.isfile() in a place where Python doesn't
see any files, only SCons sees them there through mirroring.

[Bill Deegan]

Is your MANIFEST.MF a generated file (created by some other SCons logic),
or a regular file that exists before SCons is run?

Post by Daniel Moody

Post by Daniel Moody
Hey Mats,
I just joined the users mailing list, and recreated the email from the

link

Post by Daniel Moody
(https://pairlist4.pair.net/pipermail/scons-users/2017-

November/006469.html),

Post by Daniel Moody
so sorry if the formatting gets messed up.
Regarding this issue, the Jar builder does not handle embedded lists. I

saw

Post by Daniel Moody
you created a flat list and still the manifest did not work correctly.

non-flat lists were our problem. there was a prospect of having three
instances, but two had already been changed, leaving the older code
commented out, probably due to problems with including the manifest. I
actually toyed with using Flatten() in our code to make the fix explicit
for future developers. I may go back to that, though if scons is going
to accept it, maybe I don't need to - needing a flat list is fine as
long as it's documented :)

Post by Daniel Moody
I tried to recreate your issue with the flat source list and manifest
source but I could not. I created a test which I thought was similar to
https://github.com/SConsProject/scons/pull/18/commits/

e474eb4e85d18fe56df4bbbdc180df951fbb20f0

Post by Daniel Moody
And seems to be working with changes I submitted to the jar.py tool.
Let me know if you have any ideas on how I can write a test that can
reproduce your issue, or if these changes fix it. :)

So to be clear, if a manifest is not specified, jar creates one. So our
builds do end up with a manifest file, just not the one we wanted to put
there. From just glancing at it, your test looks like it will check for
that case, since it looks for specific contents. In our instance, we
===
Manifest-Version: 1.0
Class-Path: iotivity.jar
Created-By: 1.8.0_151 (Oracle Corporation)
Main-Class: org.iotivity.base.examples.SimpleServer
===
===
Manifest-Version: 1.0
Created-By: 1.8.0_151 (Oracle Corporation)
===
It was only adding debug prints to jar.py that showed me it wasn't
actually recognizing a manifest file in the source list and to try
something different - if I turn it into a node via File('MANIFEST.MF')
it's picked up fine by that part of the code, but otherwise it fails all
of the checks and is discarded.
If I'd been a little more awake I would have just grokked it from the
jar cf
out/linux/x86_64/debug/java/examples-java/simpleserver/simpleserver.jar
-C out/linux/x86_64/debug/java/examples-java/simpleserver/classes ...
jar cfm
out/linux/x86_64/debug/java/examples-java/simpleserver/simpleserver.jar
java/examples-java/simpleserver/MANIFEST.MF -C
out/linux/x86_64/debug/java/examples-java/simpleserver/classes ...
no 'm' flag, no manifest file.
I'm still not 100% clear why it is not being grokked as a manifest but
XXX found non-node path: MANIFEST.MF
/home/mats/iotivity.work/out/linux/x86_64/debug/java/examples-java/
simpleclientserver
Namely it's going to do os.path.isfile() in a place where Python doesn't
see any files, only SCons sees them there through mirroring.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Mats Wichmann]

Post by Bill Deegan
Is your MANIFEST.MF a generated file (created by some other SCons logic),
or a regular file that exists before SCons is run?

regular file.