Refreshed patches.
And apply hack for line-endings in pep517 (from pip).
Hack comment:
# FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
# I actually tried this in a number of ways and the only way to fix this is to implement
# a poor-man's dos2unix using sed.
# The issue is with the pip package; it seems that it throws in some Windows line-endings
# and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
# We can get rid of this once this is solved upstream and in pip:
# https://github.com/pypa/pep517/pull/130
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
I tried to install matplotlib by using pip and it failed with the following output:
2021-08-18T11:52:26,171 Collecting matplotlib
2021-08-18T11:52:26,173 Created temporary directory: /tmp/pip-unpack-wuth2u0e
2021-08-18T11:52:26,565 Using cached matplotlib-3.4.3.tar.gz (37.9 MB)
2021-08-18T11:52:38,659 Added matplotlib from https://files.pythonhosted.org/packages/21/37/ 197e68df384ff694f78d687a49ad39f96c67b8d75718bc61503e1676b617/matplotlib-3.4.3.tar. gz#sha256=fc4f526dfdb31c9bd6b8ca06bf9fab663ca12f3ec9cdf4496fb44bc680140318 to build tracker '/tmp/pip- req-tracker-u30x8pht'
2021-08-18T11:52:38,660 Running setup.py (path:/tmp/pip-install-afiyiers/ matplotlib_8e717e38862f4976a3d6cb1832ba3261/setup.py) egg_info for package matplotlib
2021-08-18T11:52:38,661 Created temporary directory: /tmp/pip-pip-egg-info-kbtiezxq
2021-08-18T11:52:38,662 Running command python setup.py egg_info
2021-08-18T11:52:38,831 Traceback (most recent call last):
2021-08-18T11:52:38,832 File "<string>", line 1, in <module>
2021-08-18T11:52:38,832 File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 8, in <module>
2021-08-18T11:52:38,833 ModuleNotFoundError: No module named '_distutils_hack'
More details about it:
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968410
Once I applied the patch from the bug tracker, I got further to install
it.
Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
Python works with GNU DBM and with Berkley DBM.
Berkley DBM has been under Oracle for some time.
And it's not clear how many Python users actually use DBM.
In the packages feed, we have both libdb47 (which is now under Oracle) and
GNU DBM. The GNU DBM has a compatibility layer for Berkley DBM.
There are newer versions than libdb47, but it's probably not worth having
them yet. The libbd47 tarball is ~40+ MB. Odds are newer versions will be
bigger and more bloated.
This change merges the old `python3-gdbm` package into the `python3-dbm`
package, since they are effectively using the same underlying library now,
i.e. gdbm.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
I seem to forget to check/select setuptools and pip (that come bundled with
Python).
This change will do a simple 'ls' on the 2 wheel files, so that the build
fails even if just building Python.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Python3 comes with a built-in readline module. It wasn't included up until
now; mostly because it wasn't considered.
This change introduces it as a sub-package of the main Python3 package.
readline support is included in Python.
libreadline pulls libncursesw as a package, so python3-ncurses was
updated to pull libncursesw as well.
It should be the same package; mostly done for consistency.
Resolves the issue reported here:
https://forum.openwrt.org/t/python3-repl-missing-readline/90039
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Refreshed Python patches.
Updated pip & setuptools version.
For pip, patch '001-pep517-pyc-fix.patch' was reworked.
Also, the current version of the bundled pip (21.1.1) no longer supports
Python2, so the 'py2.py3' suffix gets replaced with just py3.
For setuptools, there is no longer a script/module:
https://github.com/pypa/setuptools/pull/2544
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Related to discussion:
https://github.com/openwrt/packages/pull/14060
Every once in a while a version bump will occur that requires an ABI
change. Example: Python 3.8 to 3.9. When this happens some Python packages
would need to be rebuilt. In setups where everything gets rebuilt, this
isn't a problem.
It's usually a bigger problem when needing to upgrade something via
opkg.
To accommodate for this, we add a libpython with it's own ABI_VERSION
flag. If this ABI_VERSION changes, then this should propagate forward.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Includes fixes for:
* CVE-2021-3177 - ctypes: Buffer overflow in PyCArg_repr
* CVE-2021-23336 - urllib parse_qsl(): Web cache poisoning - semicolon
as a query args separator
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
The crude loop I wrote to come up with this changeset:
find -L package/feeds/packages/ -name patches | \
sed 's/patches$/refresh/' | sort | xargs make
Signed-off-by: Ilya Lipnitskiy <ilya.lipnitskiy@gmail.com>
The Python version of python-config is installed when building on macOS,
rather than the shell script version when building on Linux. When run on
macOS, the Python version of python-config will return values with
Mac-specific customizations.
This patches the python-config install recipe so that which version is
installed can be controlled by the package makefile. When building on
macOS, this installs the Python version for host Python and the shell
script version for target Python.
This also updates Host/Compile and Host/Install to use the default host
build recipes, so that the various HOST_* variables are taken into
account automatically.
Fixes https://github.com/openwrt/packages/issues/14652
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Refreshed patches.
Dropped 'patches-setuptools/004-site-patch.patch'
Does not apply anymore. Setuptools has removed site.py support:
https://github.com/pypa/setuptools/issues/2165
If this is still needed, we may need to re-think it's implementation.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This adds a config option PYTHON3_HOST_PIP_CACHE_WORLD_READABLE; if
enabled, chmod will be run after pip install to make all
files/directories in the host pip cache world-readable.
Supersedes https://github.com/openwrt/packages/pull/13012.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This version includes fixes for:
* CVE-2020-15801 - Fixes python3x._pth being ignored on Windows
* CVE-2019-20907 - Avoid infinite loop when reading specially crafted
TAR files using the tarfile module
This also:
* Remove patches that are included in the update
* Add a dependency in python3-distutils for python3-email[1]
[1]: https://github.com/python/cpython/blob/v3.8.5/Lib/distutils/dist.py#L10
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This version includes fixes for:
* CVE-2020-14422: Hash collisions in IPv4Interface and IPv6Interface
* CVE-2020-15523: Python uses invalid DLL path after calling Py_SetPath
on Windows
This version also includes support for OpenSSL 1.1.x builds that use
'no-deprecated' and '--api=1.1.0'[1], and so this removes the previous
OpenSSL-related patches.
This also backports fixes for security issues, including:
* CVE-2019-20907: Infinite loop in the tarfile module
This also updates the setuptools and pip packages to 47.1.0 and 20.1.1,
respectively.
[1]: https://github.com/python/cpython/pull/20566
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This lets the Python build process set _PYTHON_HOST_PLATFORM instead of
forcing an explicit value.
Also:
* Save the target _PYTHON_HOST_PLATFORM value during Build/InstallDev
for use when building target Python packages (in python3-package.mk).
* Use the (mostly) default PYTHON_FOR_BUILD value, instead patch
configure to remove the platform triplet from the sysconfigdata file
name.
* Remove the "CROSS_COMPILE=yes" make variable (there is no indication
that this variable is necessary).
* Force host pip to build packages from source instead of downloading
binary wheels.
Previously, host pip can download universal (platform-independent)
wheels but not platform-specific wheels, because of the custom
_PYTHON_HOST_PLATFORM value. (Packages that do not have universal
wheels would be compiled from source.)
With a correct _PYTHON_HOST_PLATFORM, host pip can install
platform-specific wheels as well. However, the pre-built shared object
(.so) files in these wheels will have the host's platform triplet in
their file names. When target Python packages are built (using the
target's _PYTHON_HOST_PLATFORM), Python will not use these shared
object files.
By forcing host pip to build packages from source, the built shared
object files will not have the platform triplet in their file names.
(Host Python has been patched to remove the platform triplet from file
names.) This allows these packages to be used when building target
Python packages.
(The net effect of this complete change is that platform-dependent
packages will continue to be compiled from source, while
platform-independent packages will now also be compiled from source.)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Previously, binaries installed by Python packages will have a
non-suffixed Python 2 version and a suffixed Python 3 version, e.g. pip
and pip3. With the removal of Python 2, the non-suffixed names are no
longer taken.
This adds symlinks for the non-suffixed names linking to the suffixed
scripts (or in the case of pip, easy_install, and python-config, to the
fully-versioned scripts).
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
When a Python package is installed from source (i.e. using setup.py)
into a custom location (with --home), setuptools may want to create a
site.py file in the custom location. This file is created based on the
source code of site-patch.py, a file bundled with setuptools.
Because the normal OpenWrt setuptools package does not contain Python
source code, this file is missing and the installation will end with an
error.
This copies site-patch.py to site-patch.py.txt so that it will be
included in python3-setuptools, and patches setuptools to look for this
file.
See https://github.com/openwrt/packages/issues/12223
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
The ssl module assumes OpenSSL can load the default trust anchors (root
CA certificates).
From https://github.com/openwrt/packages/issues/12209
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds the --enable-optimizations configure option (for
profile-guided optimization) for both host and target Python, and the
--with-lto configure option (for link-time optimization) for target
Python (for non-MIPS platforms).
Currently, compiling Python with LTO leads to link errors on mips and
mipsel. (Compiling with LTO appears to succeed on mips64 but there is
only one mips64 target available for convenient testing.)
This also cleans up the host and target configure options:
* Sort options/variables
- Alphabetically
- Flags/options before child-process environment variables
- Group options by type (enable/disable/with/without)
- Static options/variables before conditional ones
* Remove the prefix/dir options, as they are the same as the defaults
set by the build system
* Remove --with-threads, as it is no longer a valid option (threads are
always enabled)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Python will record the values of CC, CXX, AR, and READELF (and other
configure options) used during compilation. pip will use these programs
when asked to compile extension modules on the target device.
* If ccache is used during build, CC and CXX will be ccache_cc and
ccache_cxx, respectively, which are not available on-device (#11912).
* If an external toolchain is used during build, the values of these
variables will contain the external toolchain prefix, which may not be
available on target.
* If the normal toolchain is used during build, AR and READELF will
contain the toolchain prefix, but the names of ar and readelf
on-device do not contain the prefix; they are named "ar" and
"readelf".
This changes the values of these variables in Python's files to match
the names available on-device, and without any toolchain prefix.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
PYTHON_VERSION is a holdover from Python 2; all Python 3 variables are
prefixed with PYTHON3 (or some variation with "3").
This updates all uses of PYTHON_VERSION to PYTHON3_VERSION.
This also sets PYTHON3_PKG_BUILD:=0 before python3-package.mk is
included in the python3 Makefile.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Time to make it more official.
Running 'python' should get you an interpreter running, and that one is now
Python3.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Currently, python3-pip installs the same script as pip3 and pip3.8 to
usr/bin. This changes pip3 to be a symlink to pip3.8.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This issue was introduced after upgrading to 3.8.
Patch `006-remove-multi-arch-and-local-paths.patch` was dropped. It was
tested on x86, but when using the x86_64 compiler in OpenWrt, the issue is
present.
The issue seems to be about Ubuntu/Debian's multi-arch support in
Python/Python3 setup [which I forgot about]. The code runs it regardless of
whether it cross-compiles or not, and for OpenWrt, this causes issues, as
it introduces absolute include paths from the host system.
Fixes https://github.com/openwrt/packages/issues/10452
Fixes: 48277ec915 ("python3: bump to version 3.8")
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This required a bit work to get working, compared to other versions. So,
some things have changed a bit more significantly.
Some highlights:
* there is no longer a pgen executable, seems this is now part of
libpython; let's see what this means for us in the future
* blake2 hash (from OpenSSL) detection needs some fixing; will upstream
added patch 002-fix-blake2-detection.patch
* removed all bpo patches; those should be fixed in upstream
* some needed to be manually re-applied as stuff changed:
- 001-enable-zlib.patch - file changed
- 004-do-not-write-bytes-codes.patch - file changed
- 015-abort-on-failed-modules.patch - variable was renamed
cross_compiling -> CROSS_COMPILING
* 017_lib2to3_fix_pyc_search.patch - the code changed, it does not seem to
have the original problem with respect to file-extension, as there
does not seem to be any special extension logic anymore there
* 006-remove-multi-arch-and-local-paths.patch - dropped patch; I can't
remember the full-details of this issue; it was something with
Debian/Ubuntu's multi-arch stuff; it was probably added maybe due to
some overzealous (on my part) thingy caused by some weird reports,
that I could never solve; let's have this patch dropped and see
* make package/python3/refresh to reduce fuzz for the rest
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
These patches address issues:
CVE-2019-16056: email.utils.parseaddr mistakenly parse an email
CVE-2019-16935: A reflected XSS in python/Lib/DocXMLRPCServer.py (for
Python 2.7)
CVE-2019-16935 was fixed for python3 in #10109
Links to Python issues:
https://bugs.python.org/issue34155https://bugs.python.org/issue38243
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This package is required by other packages to run some binaries via
`load_entry_point`.
So, this splits this package away from setuptools.
setuptools is pretty big, akd pkg-resources is also big, but not as big.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
* Remove $$$$(pkg-config --static --libs libcrypto libssl) from
HOST_LDFLAGS
Having this leads to an "unknown type name 'u_int'" error on Mac.
Removing it doesn't appear to affect Python's ability to find
buildroot LibreSSL.
* Change -Wl,-rpath=... to -Wl,-rpath,... in HOST_LDFLAGS
The equals sign version is not supported by the Mac linker (appears to
be an GNU extension). The comma version is supported; -rpath and its
argument will be separated by a space when passed to the linker.
* Add ac_cv_header_libintl_h=no to HOST_CONFIGURE_VARS for Mac
Python on Mac doesn't expect to use libintl, but if gettext-full is
compiled for host, it will try, leading to undefined symbol errors
during compilation. This prevents configure from finding libintl.h.
Fixes#7171.
Fixes#9621.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
- Update setuptools to 40.8.0
- Update pip to 19.0.3
- Refreshed patches
- Removed 4 patches (2 of them was included in 3.7.3 and other two are
included in this release)
Makefile python3:
- Move PKG_MAINTAINER above PKG_LICENSE
Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
This patch, taken from buildroot, avoids the use of host paths when
compiling third-party extensions.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Build/InstallDev is passed a second argument, a path where host binaries
should be placed (ultimately $(STAGING_DIR)/host).
This change moves python[3]-config to that directory.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
These patches address issue:
CVE-2019-9948: Unnecessary URL scheme exists to allow local_file://
reading file in urllib
Link to Python issue:
https://bugs.python.org/issue35907
Issue 35907 is still currently open, waiting for a decision for
Python 3.5; these patches for Python 2.7 and 3.7 have been merged.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
These patches address issues:
CVE-2019-9740: Python urllib CRLF injection vulnerability
CVE-2019-9947: Header Injection in urllib
Links to Python issues:
https://bugs.python.org/issue36276 (resolved duplicated of 30458)
https://bugs.python.org/issue35906 (resolved duplicated of 30458)
https://bugs.python.org/issue30458
Issue 30458 is still currently open, waiting for a decision for
Python 3.5; these patches for Python 2.7 and 3.7 have been merged.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This changes the "patched" indicator files for host setuptools and pip
to include their PKG_RELEASE values. This also removes host setuptools
and/or pip before host install, if the installed copy does not match the
version (and PKG_RELEASE) of the copy to be installed.
This will allow added or removed patches to affect host setuptools /
pip, since these changes will cause PKG_RELEASE to be incremented.
This also fixes the host install error, when the install tries to patch
an already patched copy of setuptools. (This error occurs because the
existing indicator files do not have version numbers in their file
names, whereas host install expected version numbers to be present.)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds the current setuptools/pip version numbers to the indicator
files' names, which should allow upgraded versions to be patched.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This uses two find commands to delete __pycache__ contents then the
__pycache__ directories, rather than a for loop.
The second command omits a -empty test, so that if the first command
doesn't remove all directory contents for some reason, the second
command will return an error (find will not delete a non-empty
directory).
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Alexandru Ardelean
Josef Schlehofer
Jeffery To
Ilya Lipnitskiy
Rosen Penev
Yorkie Liu
Eneas U de Queiroz