Some packages just install some Python binaries, that may need their
shebang fixed.
This change adds some utilities to help with that and try to centralize the
sed rules a bit.
It also removes the logic from the `python-package-install.sh` into the
`python-package[3].mk` files. This does 2 things:
1. It minimizes the need for the shell script to know the Python
version 2/3
2. Makes the logic re-usable in packages; especially if the install rules
differ a bit
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
As I remember this worked.
But since `set -e` is set, I am a bit paranoid about it. In the sense that
it may fail if `ver` != 3.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
"python -m compileall" has a default maximum recursion level of 10, i.e.
it will descend up to 10 levels of subdirectories when looking for
source files to compile. This is usually sufficient but there are
packages that include more than 10 levels (botocore,
https://github.com/openwrt/packages/pull/8214#discussion_r270056741).
This adds the "-r" command line option to the call to compileall to
increase the max recursion level (currently set to 20).
This also patches Python 2's compileall.py to add this max recursion
level option. (Python 3's compileall.py already supports this option.)
This also applies some related changes to python-package-install.sh:
* Use the "-delete" option with find instead of exec'ing rm / rmdir. For
the case of removing empty directories (in delete_empty_dirs()), this
has the added benefit of simplifying the code, as the "-delete" option
implies "-depth", and thus find "does the right thing" (removing empty
directories depth-first).
* Remove the backslash in "-name" patterns (for find), as they are not
regular expression but glob patterns.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Currently, all files in usr/bin (presumably all Python scripts) are run
through sed to replace the shebang; sed will overwrite the file whether
or not a match is found. This causes symlinks to be overridden and made
into copies of their targets. python[3]-base and python[3]-dev are
affected by this.
This adds the --follow-symlinks flag to sed, in addition to using
$(SED), so that symlinks are not overridden.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This extends the Python[3] shebang fixup to all packages.
Only Python scripts in `/usr/bin` will be handled at the moment. Later it
may make sense to also cover executables in `/bin`, though typically Python
executables shouldn't be placed there.
Previously the shebang handling was only done for python[3]-pip &
python[3]-setuptools.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Piping to xargs does not handle spaces in paths too well, because it splits
up the paths.
For deleting empty dirs, we also need to do several retries, otherwise
`find` will try to go through the directories after they're deleted.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
It's a common operation for both Python & Python3, so move it to the
script `python-package-install.sh` script.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Following a discussion on bugs.python.org:
* https://bugs.python.org/issue29708
* https://bugs.python.org/msg313384
It seems that setting a fixed value to PYTHONHASHSEED guarantees that
the bytecodes are generated consistently/in a reproducible manner.
Hopefully, this is the last bit to make Python3 build reproducible.
Tested this locally on a few files [that were not reproducible without
this change].
The PYTHONHASHSEED is only assigned to the host Python/Python3 during
compilation of byte-codes [from python source].
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
The only difference just a parameter for Python3
[ -b to compile bytecodes in legacy mode ].
No need to keep 2 almost identical files now
that they're exported.
I'm a bit scared of that param, since it may get
removed at some point.
But let's see until then.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Since `lang/python` is it's own folder of Python packages
(for both Python 2 & 3), and these build rules are needed
in a lot of packages [especially Python packages],
putting them here makes sense architecturally,
to be shared.
This also helps get rid of the `include_mk` construct
which relies on OpenWrt core to provide, and seems
like a broken design idea that has persisted for a while.
Reason is: it requires that Python 2/3 be built to provide
these mk files for other Python packages,
which seems like a bad idea.
Long-term, there could be an issue where some other feeds
would require these mk files [e.g. telephony] for
some Python packages.
We'll see how we handle this a bit later.
For now we limit this to this feed.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This reverts commits 4333d1dcbf and
074d2863be, making Python packages
discoverable again by pkg_resources.
Fixes#5361.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
That way some python packages can choose
to keep their egg-info dirs, if they want to, or they're needed.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Depending on execution order the `python-package-install.sh`
script would return a non-zero err code.
So, this enforces that all commands in the script
don't fail (via the `set -e` directive).
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
I admit this may be be a bit aggressive, but the lang
folder is getting cluttered/filled up with Python, PHP, Perl,
Ruby, etc. packages.
Makes sense to try to group them into per-lang folders.
I took the Pythons.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Well, they're not yet compiled, but in the next commit
they should be.
People have been complaining [citation needed] to me
via email or via Github that Python's performance is crap
because it packages sources directly and they're not compiled.
And Python has to compile the sources on each run, and
on-the-fly.
Allowing compilation caching is also a no-no, because
I'll get complaints that the flash storage fills up
whenever a Python app runs.
So, to give the user a choice, the new de-facto packaging
for Python packages will be:
* ship compiled + [ preferably ] optimized files
* package sources separately
The problem is that this doubles the number of packages
in LEDE/OpenWrt, but build-times should not suffer a big
hit, since the compilation is done once, and the
install phase should not be too intensive.
Oh, and people don't need ship source packages if
they don't want to.
To do that, a packager needs to just call
`$(eval $(call BuildPackage,python-<package>-src))`
The `python-` prefix is important.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Alexandru Ardelean
Jeffery To