[Scons-users] Problem installing scons via pip

Discussion:

Martin Ritter

2018-09-21 09:16:03 UTC

Hi all,

I have a very weird problem installing scons via pip. It works fine if I
install just scons alone. But I have a large requirements.txt and would
like to install all my packages at once checking their integrity so I'm
running

pip install --require-hashes -r requirements.txt

and I get the `--single-version-externally-managed` error even though
wheel is installed (python 3.6, pip 18.0, setuptools 40.4, wheel 0.3.1)

If I run `pip wheel` first it seems to work fine as the wheel is picked
up and installed instead

Now the easiest solution for me would be if someone could just upload a
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?

Cheers,

Martin

Martin Ritter

2018-09-21 12:08:32 UTC

Permalink

Hi again,

more investigation and quite some time later I found out that my problem
is due to --no-cache-dir which causes pip not to create wheels but
install directly ... sorry for the noise.

Still, I would be happy if scons would provide wheels directly on pypi :D

Cheers,

Martin

Post by Martin Ritter
Hi all,
I have a very weird problem installing scons via pip. It works fine if I
install just scons alone. But I have a large requirements.txt and would
like to install all my packages at once checking their integrity so I'm
running
pip install --require-hashes -r requirements.txt
and I get the `--single-version-externally-managed` error even though
wheel is installed (python 3.6, pip 18.0, setuptools 40.4, wheel 0.3.1)
If I run `pip wheel` first it seems to work fine as the wheel is picked
up and installed instead
Now the easiest solution for me would be if someone could just upload a
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?
Cheers,
Martin

--
Dr. Martin Ritter

LMU München, Excellence Cluster Universe
Boltzmannstrasse 2, 85748 Garching

Tel: (+49) 89 35831-7152
Fax: (+49) 89 3299-4002

Bill Deegan

2018-09-21 12:21:49 UTC

Permalink

I'll see if we can make that happen on next release..

Post by Martin Ritter
Hi again,
more investigation and quite some time later I found out that my problem
is due to --no-cache-dir which causes pip not to create wheels but
install directly ... sorry for the noise.
Still, I would be happy if scons would provide wheels directly on pypi :D
Cheers,
Martin

--
Dr. Martin Ritter
LMU MÃŒnchen, Excellence Cluster Universe
Boltzmannstrasse 2, 85748 Garching
Tel: (+49) 89 35831-7152
Fax: (+49) 89 3299-4002
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Eric Fahlgren

2018-09-21 15:28:35 UTC

Permalink

+1

I have exactly the same situation as Martin (big requirements.txt, forward
dependency on 'wheel' always borks a new install, the --single... flag
causes a crash later), so would be very happy to see a scons.whl as the
default PyPi package. Any pointers into the source where I might be able
to help make this happen?

Eric

Post by Bill Deegan
I'll see if we can make that happen on next release..

Post by Martin Ritter
Hi all,
I have a very weird problem installing scons via pip. It works fine if

Post by Martin Ritter
install just scons alone. But I have a large requirements.txt and would
like to install all my packages at once checking their integrity so I'm
running
pip install --require-hashes -r requirements.txt
and I get the `--single-version-externally-managed` error even though
wheel is installed (python 3.6, pip 18.0, setuptools 40.4, wheel 0.3.1)
If I run `pip wheel` first it seems to work fine as the wheel is picked
up and installed instead
Now the easiest solution for me would be if someone could just upload a
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?
Cheers,
Martin

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Mats Wichmann

2018-09-21 16:20:09 UTC

Permalink

Post by Eric Fahlgren
+1
I have exactly the same situation as Martin (big requirements.txt, forward
dependency on 'wheel' always borks a new install, the --single... flag
causes a crash later), so would be very happy to see a scons.whl as the
default PyPi package. Any pointers into the source where I might be able
to help make this happen?
Eric

it's a "known issue" that the packaging of scons isn't very modern. and
could really benefit from someone who has knowledge of how Python
packaging happens now, since it's evolved quite a lot over the years.

look at these files to start:

bin/install_scons.py
src/setup.py
src/test_setup.py

my understanding is once packaging is done "the right way", then
uploading wheels is trivial - at least this one makes it sound so:

https://pythonwheels.com/

Bill Deegan

2018-09-21 17:05:27 UTC

Permalink

src/setup.py and setup.cfg likely what you want to look at.
Any help there is welcome.

Post by Bill Deegan
I'll see if we can make that happen on next release..

Post by Martin Ritter
Hi all,
I have a very weird problem installing scons via pip. It works fine if

Post by Martin Ritter
install just scons alone. But I have a large requirements.txt and

would

Post by Martin Ritter
like to install all my packages at once checking their integrity so

I'm

Post by Martin Ritter
running
pip install --require-hashes -r requirements.txt
and I get the `--single-version-externally-managed` error even though
wheel is installed (python 3.6, pip 18.0, setuptools 40.4, wheel 0.3.1)
If I run `pip wheel` first it seems to work fine as the wheel is

picked

Post by Martin Ritter
up and installed instead
Now the easiest solution for me would be if someone could just upload

Post by Martin Ritter
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?
Cheers,
Martin

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Eric Fahlgren

2018-09-21 18:51:46 UTC

Permalink

Just to confirm: the current src/setup.py is used /only/ for
packaging/product installation? (So, if I totally replace it to build a
wheel and do nothing else, that would be ok?)

I am very far from being an expert on packaging, but I just created
whl-building setup files for a couple of packages in the past month, so
it's fresh in my mind and thus it's possible that I could actually do
something here before I forget everything I learned.

Eric

Post by Bill Deegan
src/setup.py and setup.cfg likely what you want to look at.
Any help there is welcome.

Post by Eric Fahlgren
+1
I have exactly the same situation as Martin (big requirements.txt,
forward dependency on 'wheel' always borks a new install, the --single...
flag causes a crash later), so would be very happy to see a scons.whl as
the default PyPi package. Any pointers into the source where I might be
able to help make this happen?
Eric

Post by Bill Deegan
I'll see if we can make that happen on next release..

Post by Martin Ritter
Hi all,
I have a very weird problem installing scons via pip. It works fine

if I

Post by Martin Ritter
install just scons alone. But I have a large requirements.txt and

would

Post by Martin Ritter
like to install all my packages at once checking their integrity so

I'm

0.3.1)

Post by Martin Ritter
If I run `pip wheel` first it seems to work fine as the wheel is

picked

Post by Martin Ritter
up and installed instead
Now the easiest solution for me would be if someone could just upload

Post by Martin Ritter
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?
Cheers,
Martin

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Bill Deegan

2018-09-21 19:19:10 UTC

Permalink

Eric,

I'd say give it a shot and we'll take a look at what you create.
Note there are a bunch of xml files and other data files which should be
installed.
There are also an handful of scripts scons, sconsign, etc. Ideally the
setup would handle creating these and create scons.exe on windows.

I toyed around with replacing the existing setup.py a while back, but ran
out of time to done.

-Bill
SCons Project Co-Manager

Post by Eric Fahlgren
Just to confirm: the current src/setup.py is used /only/ for
packaging/product installation? (So, if I totally replace it to build a
wheel and do nothing else, that would be ok?)
I am very far from being an expert on packaging, but I just created
whl-building setup files for a couple of packages in the past month, so
it's fresh in my mind and thus it's possible that I could actually do
something here before I forget everything I learned.
Eric

Post by Bill Deegan
src/setup.py and setup.cfg likely what you want to look at.
Any help there is welcome.

Post by Eric Fahlgren
+1
I have exactly the same situation as Martin (big requirements.txt,
forward dependency on 'wheel' always borks a new install, the --single...
flag causes a crash later), so would be very happy to see a scons.whl as
the default PyPi package. Any pointers into the source where I might be
able to help make this happen?
Eric

Post by Bill Deegan
I'll see if we can make that happen on next release..

Post by Martin Ritter
Hi all,
I have a very weird problem installing scons via pip. It works fine

if I

Post by Martin Ritter
install just scons alone. But I have a large requirements.txt and

would

Post by Martin Ritter
like to install all my packages at once checking their integrity so

I'm

Post by Martin Ritter
running
pip install --require-hashes -r requirements.txt
and I get the `--single-version-externally-managed` error even

though

Post by Martin Ritter
wheel is installed (python 3.6, pip 18.0, setuptools 40.4, wheel

0.3.1)

Post by Martin Ritter
If I run `pip wheel` first it seems to work fine as the wheel is

picked

Post by Martin Ritter
up and installed instead
Now the easiest solution for me would be if someone could just

upload a

Post by Martin Ritter
wheel of scons to pypi which would allow me to directly install that
wheel instead. Would that be possible?
Cheers,
Martin

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Eric Fahlgren

2018-09-21 21:26:02 UTC

Permalink

It's a lot simpler than I thought it would be. First "*pip install wheel*"
(already had it in my environment). Next, change setup.py:509 to use
setuptools instead of distutils for the setup:

#distutils.core.setup(**arguments)
import setuptools
*setuptools*.setup(**arguments)

You're pretty much done. I'm on Windows 10 x64, using Windows CPython
3.6.6, setuptools 40.0.0, wheel 0.31.1, and had to do a bit of hacking to
fix what appear to be platform-specific things, but since I don't have a
Linux or Mac box available, I really don't want to make any presumptions:

1) At setup.py:44, I manually replaced __VERSION__ with "3.0.1" so I could
run the setup script directly.
2) In setup.py:397 I had to add ".py" extensions to the four entries in the
"scripts" array. This is because setup was complaining "can't find
'script/scons'", so apparently distutils knows to do this and setuptools
doesn't or something.
scripts = [
'script/scons*.py*',
'script/sconsign*.py*',
'script/scons-time*.py*',
'script/scons-configure-cache*.py*',
3) Corresponding to the above, I had to turn off adding the ".py" later at
line 327-329; I just commented out that whole if block:
# if is_win32:
# scons += '.py'
# scons_ver += '.py'

Built it, ran pip, ran scons, it all worked.

cd src
python setup.py *bdist_wheel*

... blah blah blah for thousands of lines ...

ll dist

-rwx------+ 1 efahlgren Domain Users 4181330 2018-09-21 14:03
scons-3.0.1-py2.py3-none-any.whl*

pip install dist/scons-3.0.1-py2.py3-none-any.whl

Processing o:\scons\src\dist\scons-3.0.1-py2.py3-none-any.whl
Installing collected packages: scons
Successfully installed scons-3.0.1

t:/Python36/Scripts/scons.py

scons: Reading SConscript files ...
Windows 10 build...

Bill Deegan

2018-09-21 21:49:36 UTC

Permalink

Take a look at
https://github.com/SCons/scons/wiki/SimplifiedReleaseProcedure

In that process bootstrap.py eventually copies the files for packaging into
build/scons where it runs setup.py
I think it copies the scons.py to scons

I'd love to get setup.py using "console_scripts" to create the bin/scons..
etc

Post by Eric Fahlgren
It's a lot simpler than I thought it would be. First "*pip install wheel*"
(already had it in my environment). Next, change setup.py:509 to use
#distutils.core.setup(**arguments)
import setuptools
*setuptools*.setup(**arguments)
You're pretty much done. I'm on Windows 10 x64, using Windows CPython
3.6.6, setuptools 40.0.0, wheel 0.31.1, and had to do a bit of hacking to
fix what appear to be platform-specific things, but since I don't have a
1) At setup.py:44, I manually replaced __VERSION__ with "3.0.1" so I could
run the setup script directly.
2) In setup.py:397 I had to add ".py" extensions to the four entries in
the "scripts" array. This is because setup was complaining "can't find
'script/scons'", so apparently distutils knows to do this and setuptools
doesn't or something.
scripts = [
'script/scons*.py*',
'script/sconsign*.py*',
'script/scons-time*.py*',
'script/scons-configure-cache*.py*',
3) Corresponding to the above, I had to turn off adding the ".py" later at
# scons += '.py'
# scons_ver += '.py'
Built it, ran pip, ran scons, it all worked.

cd src
python setup.py *bdist_wheel*

... blah blah blah for thousands of lines ...

ll dist

-rwx------+ 1 efahlgren Domain Users 4181330 2018-09-21 14:03
scons-3.0.1-py2.py3-none-any.whl*

pip install dist/scons-3.0.1-py2.py3-none-any.whl

Processing o:\scons\src\dist\scons-3.0.1-py2.py3-none-any.whl
Installing collected packages: scons
Successfully installed scons-3.0.1

t:/Python36/Scripts/scons.py

scons: Reading SConscript files ...
Windows 10 build...
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Eric Fahlgren

2018-09-22 02:03:42 UTC

Permalink

Thanks, Bill, I did a mini-build based on that wiki page and got this down
to two small changes:

In SConstruct at line 744 add
* commands.append("$PYTHON $PYTHONFLAGS $SETUP_PY bdist_wheel")*

* whl = os.path.join(build,*
* 'dist',*
* "%s-py2.py3-none-any.whl" % pkg_version) #
There must be a better way?*
* distutils_targets.append(whl)*

In src/setup.py at 509:
#distutils.core.setup(**arguments)
*import setuptools*
*setuptools.setup(**arguments)*

Here's that mini-build that produces the wheel, which I installed and used
to build our product code.

cd scons
rm -fr build/
t:/Python36/python bin/update-release-info.py release

Updating src\CHANGES.txt...
Updating src\Announce.txt...
Updating SConstruct...
Updating README.rst...
Updating testing\framework\TestSCons.py...
Updating doc\user\main.xml...

t:/Python36/python bootstrap.py > & ! build.log
ll build/scons/dist/

total 3932
drwxr-x---+ 1 efahlgren Domain Users 0 2018-09-21 18:14 ./
drwxr-x---+ 1 efahlgren Domain Users 0 2018-09-21 18:14 ../
-rwxr-x---+ 1 efahlgren Domain Users 443123 2018-09-21 18:14
scons-3.1.0a20180921.tar.gz*
-rwxr-x---+ 1 efahlgren Domain Users 927976 2018-09-21 18:14
scons-3.1.0a20180921.win-amd64.tar.gz*
-rwxr-x---+ 1 efahlgren Domain Users 1284801 2018-09-21 18:14
scons-3.1.0a20180921.win-amd64.zip*
-rwxr-x---+ 1 efahlgren Domain Users 671140 2018-09-21 18:14
scons-3.1.0a20180921.zip*
-rwxr-x---+ 1 efahlgren Domain Users 683514 2018-09-21 18:14
scons-3.1.0a20180921-py2.py3-none-any.whl*

pip install build/scons/dist/scons-3.1.0a20180921-py2.py3-none-any.whl

Processing
o:\scons\build\scons\dist\scons-3.1.0a20180921-py2.py3-none-any.whl
Installing collected packages: scons
Successfully installed scons-3.1.0a20180921

---
I get a bunch of warnings in the build.log, I think the more modern
setuptools is normalizing the names per PEP 508.

T:\Python36\lib\site-packages\setuptools\dist.py:398: UserWarning:
Normalizing '3.1.0.alpha.20180921' to '3.1.0a20180921'
normalized_version,

Then the "Install" commands (copying build/scons/dist/* to build/dist)
fail, since the generated PyPi-compatible file names don't match those
known to SCons...

Install file: "build\scons\dist\scons-3.1.0.alpha.20180921.tar.gz" as
"build\dist\scons-3.1.0.alpha.20180921.tar.gz"
scons: *** [build\dist\scons-3.1.0.alpha.20180921.tar.gz]
build\scons\dist\scons-3.1.0.alpha.20180921.tar.gz: No such file or
directory
scons: building terminated because of errors.

Take a look at
https://github.com/SCons/scons/wiki/SimplifiedReleaseProcedure
In that process bootstrap.py eventually copies the files for packaging
into build/scons where it runs setup.py
I think it copies the scons.py to scons
I'd love to get setup.py using "console_scripts" to create the
bin/scons.. etc

Post by Eric Fahlgren
It's a lot simpler than I thought it would be. First "*pip install
wheel*" (already had it in my environment). Next, change setup.py:509
#distutils.core.setup(**arguments)
import setuptools
*setuptools*.setup(**arguments)
You're pretty much done. I'm on Windows 10 x64, using Windows CPython
3.6.6, setuptools 40.0.0, wheel 0.31.1, and had to do a bit of hacking to
fix what appear to be platform-specific things, but since I don't have a
1) At setup.py:44, I manually replaced __VERSION__ with "3.0.1" so I
could run the setup script directly.
2) In setup.py:397 I had to add ".py" extensions to the four entries in
the "scripts" array. This is because setup was complaining "can't find
'script/scons'", so apparently distutils knows to do this and setuptools
doesn't or something.
scripts = [
'script/scons*.py*',
'script/sconsign*.py*',
'script/scons-time*.py*',
'script/scons-configure-cache*.py*',
3) Corresponding to the above, I had to turn off adding the ".py" later
# scons += '.py'
# scons_ver += '.py'
Built it, ran pip, ran scons, it all worked.

cd src
python setup.py *bdist_wheel*

... blah blah blah for thousands of lines ...

ll dist

-rwx------+ 1 efahlgren Domain Users 4181330 2018-09-21 14:03
scons-3.0.1-py2.py3-none-any.whl*

pip install dist/scons-3.0.1-py2.py3-none-any.whl

Processing o:\scons\src\dist\scons-3.0.1-py2.py3-none-any.whl
Installing collected packages: scons
Successfully installed scons-3.0.1

t:/Python36/Scripts/scons.py

scons: Reading SConscript files ...
Windows 10 build...
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users