Discussion:
[Scons-users] SCons interfers with the constructor of io.TextIOWrapper when pywin32 is installed
François Beaune
2018-08-23 17:12:35 UTC
Permalink
Hi,

I'm using scons v3.0.1 under Windows 10:

SCons by Steven Knight et al.:
script: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14
13:16:53, by bdbaddog on hpmicrodog
engine: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14
13:16:53, by bdbaddog on hpmicrodog
engine path:
['c:\\python36\\lib\\site-packages\\scons-3.0.1\\SCons']
Copyright (c) 2001 - 2017 The SCons Foundation

Under (and only under) Python 3.x, when the pywin32 Python module is
installed, scons wraps a number of classes from the io module
(io.BufferedReader, io.BufferedWriter, io.BufferedRWPair,
io.BufferedRandom, io.TextIOWrapper) in order to run some custom code when
they are constructed. This is done in SCons\Platform\win32.py lines 83-92.

I don't know the exact purpose of this, however it appears that SCons makes
the incorrect assumption that all classes it wraps have a fileno() method,
which is not the case for io.TextIOWrapper() when it wraps streams such as
io.StringIO or io.BytesIO.

Here is a minimal repro case, tested with scons v3.0.1, Python 3.6.5 and
pywin32 223:

import sys
import xml.etree.ElementTree as ET

def bob(env, target, source):
element = ET.Element("bob")
try:
ET.tostring(element, encoding="utf-8", method="xml")
except:
print(sys.exc_info())

env = Environment(
BUILDERS={
'bob': Builder(action=Action(bob))
})

env.bob("output", [])

Output:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
bob(["output"], [])
(<class 'io.UnsupportedOperation'>, UnsupportedOperation('fileno',),
<traceback object at 0x00000297BAF30B08>)
scons: done building targets.

Best,
Franz
Mats Wichmann
2018-08-23 19:29:45 UTC
Permalink
Post by François Beaune
Hi,
script: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14
13:16:53, by bdbaddog on hpmicrodog
engine: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14
13:16:53, by bdbaddog on hpmicrodog
['c:\\python36\\lib\\site-packages\\scons-3.0.1\\SCons']
Copyright (c) 2001 - 2017 The SCons Foundation
Under (and only under) Python 3.x, when the pywin32 Python module is
installed, scons wraps a number of classes from the io module
(io.BufferedReader, io.BufferedWriter, io.BufferedRWPair,
io.BufferedRandom, io.TextIOWrapper) in order to run some custom code when
they are constructed. This is done in SCons\Platform\win32.py lines 83-92.
I've run into a couple of problems with this wrapping as well. One
actually made it into an issue:

https://github.com/SCons/scons/issues/3136

which is clearly not the same, but adds to wondering about this wrapping.
François Beaune
2018-08-23 19:48:59 UTC
Permalink
Interesting. I wonder if io.TextIOWrapper shouldn't simply be excluded from
the wrapping since it's itself a wrapper?

Franz
Post by François Beaune
Post by François Beaune
Hi,
script: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668,
2017/11/14
Post by François Beaune
13:16:53, by bdbaddog on hpmicrodog
engine: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668,
2017/11/14
Post by François Beaune
13:16:53, by bdbaddog on hpmicrodog
['c:\\python36\\lib\\site-packages\\scons-3.0.1\\SCons']
Copyright (c) 2001 - 2017 The SCons Foundation
Under (and only under) Python 3.x, when the pywin32 Python module is
installed, scons wraps a number of classes from the io module
(io.BufferedReader, io.BufferedWriter, io.BufferedRWPair,
io.BufferedRandom, io.TextIOWrapper) in order to run some custom code
when
Post by François Beaune
they are constructed. This is done in SCons\Platform\win32.py lines
83-92.
I've run into a couple of problems with this wrapping as well. One
https://github.com/SCons/scons/issues/3136
which is clearly not the same, but adds to wondering about this wrapping.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Bill Deegan
2018-09-06 04:01:09 UTC
Permalink
In a lot of cases non-trivial python builders should be moved to a script
which runs outside SCons.
This way you can avoid the GIL.
Additionally there have been some race conditions between file closing and
reference on windows which seem to show up more frequently for non-trivial
python builder logic inside scons.
(Think parsing and writing a complex file such as XML)..

BTW. the io wrapping is intended to ensure that file handles aren't shared
between threads for such which can cause some race conditions on Win32..

-Bill
Post by François Beaune
Interesting. I wonder if io.TextIOWrapper shouldn't simply be excluded
from the wrapping since it's itself a wrapper?
Franz
Post by François Beaune
Post by François Beaune
Hi,
script: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668,
2017/11/14
Post by François Beaune
13:16:53, by bdbaddog on hpmicrodog
engine: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668,
2017/11/14
Post by François Beaune
13:16:53, by bdbaddog on hpmicrodog
['c:\\python36\\lib\\site-packages\\scons-3.0.1\\SCons']
Copyright (c) 2001 - 2017 The SCons Foundation
Under (and only under) Python 3.x, when the pywin32 Python module is
installed, scons wraps a number of classes from the io module
(io.BufferedReader, io.BufferedWriter, io.BufferedRWPair,
io.BufferedRandom, io.TextIOWrapper) in order to run some custom code
when
Post by François Beaune
they are constructed. This is done in SCons\Platform\win32.py lines
83-92.
I've run into a couple of problems with this wrapping as well. One
https://github.com/SCons/scons/issues/3136
which is clearly not the same, but adds to wondering about this wrapping.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users
Loading...