This adds a recipe, Py3Build/InstallBuildDepends, that installs the
requirements listed in HOST_PYTHON3_PACKAGE_BUILD_DEPENDS. This allows
other (non-Python) packages to install host Python packages by calling
this recipe, without having to know the internals of python3-package.mk.
This also updates apparmor to call this recipe.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
When running FindStdlib and running DependsCheckHostPipVersionMatch at
the same time, both commands were joined together resulting in a syntax
error.
Signed-off-by: Michal Vasilek <michal.vasilek@nic.cz>
It often happens that we update a package to a new version (e.g. cffi) to a
newer version, but we forget to update the version for cffi in the
`lang/python/host-pip-requirements/cffi.txt` file.
This check adds a minimal check, so that when a build occurs for a Python
package, if there is a mention/listing of this package in
`lang/python/host-pip-requirements/` it will check that the versions match.
This way, when we update a package, we get a build failure and update the
host version as well.
This will omit packages (like Cython) that are not packaged for OpenWrt,
but are host-side dependencies only.
But until we find some mechanism to check for those, we will probably only
notice to update them when another build occurs (at the very least).
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
In hash-checking mode[1], pip will verify downloaded package archives
(source tarballs in our case) against known SHA256 hashes before
installing the packages.
As a consequence, this requires the use of requirements files[2] and
pinning packages to known versions.
The syntax for package Makefiles has changed slightly;
HOST_PYTHON3_PACKAGE_BUILD_DEPENDS no longer accepts requirement
specifiers like "foo>=1.0", only requirements file names (which are the
same as package names in the most common case).
This also updates affected packages, in particular:
* python-zipp: "setuptools_scm[toml]" has been split into
"setuptools-scm toml" to reuse the requirements file for
setuptools-scm (the extra depends installed by "setuptools_scm[toml]"
is toml).
* python-pycparser: This previously used ply 3.10, whereas the
requirements file will now install 3.11.
[1]: https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
[2]: https://pip.pypa.io/en/stable/user_guide/#requirements-files
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This lets the Python build process set _PYTHON_HOST_PLATFORM instead of
forcing an explicit value.
Also:
* Save the target _PYTHON_HOST_PLATFORM value during Build/InstallDev
for use when building target Python packages (in python3-package.mk).
* Use the (mostly) default PYTHON_FOR_BUILD value, instead patch
configure to remove the platform triplet from the sysconfigdata file
name.
* Remove the "CROSS_COMPILE=yes" make variable (there is no indication
that this variable is necessary).
* Force host pip to build packages from source instead of downloading
binary wheels.
Previously, host pip can download universal (platform-independent)
wheels but not platform-specific wheels, because of the custom
_PYTHON_HOST_PLATFORM value. (Packages that do not have universal
wheels would be compiled from source.)
With a correct _PYTHON_HOST_PLATFORM, host pip can install
platform-specific wheels as well. However, the pre-built shared object
(.so) files in these wheels will have the host's platform triplet in
their file names. When target Python packages are built (using the
target's _PYTHON_HOST_PLATFORM), Python will not use these shared
object files.
By forcing host pip to build packages from source, the built shared
object files will not have the platform triplet in their file names.
(Host Python has been patched to remove the platform triplet from file
names.) This allows these packages to be used when building target
Python packages.
(The net effect of this complete change is that platform-dependent
packages will continue to be compiled from source, while
platform-independent packages will now also be compiled from source.)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds a script that searches a Python package's source code to find
imports for separately-packaged standard library modules.
The script can be run by calling make with the configure target and
"PY3=stdlib V=s" arguments, e.g.
make package/python3-lxml/configure PY3=stdlib V=s
This also updates the readme on how to call this script, as well as more
information on Python package dependencies in general.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This replaces the use of BUILD_VARIANT with PYTHON3_PKG_BUILD to opt
in/out of the default Python package build recipe (Py3Build/Compile).
PYTHON3_PKG_BUILD defaults to true (1), i.e. if a package includes
python3-package.mk, then by default it will set the package's
Build/Compile to Py3Build/Compile.
If PYTHON3_PKG_BUILD is set to 0 before python3-package.mk is included,
then Build/Compile will not be modified.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
There are no bug reports or other evidence to suggest Python is not
compatible with MIPS16 compilation.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
* Remove PYTHON3_BIN_DIR, it isn't used anywhere in the repo
* Rephrase *-src package description
* Reduce Py3Package/$(1)/install indentation
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This extracts filespec export and processing into
Py3Package/ProcessFilespec.
This also allows the filespec variable to be explicitly set to an empty
value, to bypass filespec processing. (The default filespec is also
available as Py3Package/filespec/Default to be explicitly assigned to
the filespec variable.)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This moves functionality from python-package-install.sh into
python3-package.mk, so that they can be reused separate from filespec
processing.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Group Python3/* recipes together, group Py3Package and Py3Build
together.
This also adds headings and whitespace to separate major sections.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This renames "internal" recipes to use the Python3/ prefix and clarifies
the names (RunTarget to Run, Mod to ModSetup, Shebang to FixShebang).
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This changes the recipe name prefix from Build/Compile/HostPy3 to
HostPython3, and clarifies some of the names (RunHost to Run, Mod to
ModSetup).
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
HostPython3 only adds a few environment variables before running host
Python. It has only two users, Build/Compile/HostPy3RunHost and
Build/Compile/HostPy3RunTarget.
HostPython3 also accesses $(PYTHON3PATH), even though python3-host.mk
does not include python3-package.mk, where the variable is defined.
This removes HostPython3 and has its two users run host Python directly.
This also combines the environment variables of HostPython3 and the two
users into HOST_PYTHON3_VARS and PYTHON3_VARS.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds a 'Package/<pkg>-src/config' definition with a 'depends on
<pkg>' line, which will hide <pkg>-src unless <pkg> is selected. This
makes the long list of python packages a bit shorter, and also indents
the src package:
<M> python3-base................................ Python 3.8 interpreter
< > python3-base-src................. Python 3.8 interpreter (sources)
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
This fixes a typo with the default PYTHON3_PKG_SETUP_GLOBAL_ARGS.
Since in make context non-defined variables are empty anyway, this doesn't
produce any issues. The fix is more semantic in nature.
Fixes https://github.com/openwrt/packages/issues/11790
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Some packages just install some Python binaries, that may need their
shebang fixed.
This change adds some utilities to help with that and try to centralize the
sed rules a bit.
It also removes the logic from the `python-package-install.sh` into the
`python-package[3].mk` files. This does 2 things:
1. It minimizes the need for the shell script to know the Python
version 2/3
2. Makes the logic re-usable in packages; especially if the install rules
differ a bit
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This changes the python[3]-django dependencies in packages to be
non-selecting, and adds an MDEPENDS line so that the *-src packages get
placed inside the django menu as well.
Added MENU:= to the src-package definitions in python[3]-package.mk,
so it does not import that setting from the binary package.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
* Add --cache-dir option to set the pip cache to a directory in
$(DL_DIR), instead of pip's default (build user's ~/.cache/pip),
fixes#9066
* Add --disable-pip-version-check option, since the version check only
prints a message saying a new version is available
* Combine host_python_pip_install and host_python_pip_install_host into
Build/Compile/HostPy[3]PipInstall
* Remove --root and --prefix options, since this function is only used
to install packages to host Python's default site-packages directory
(setting these may serve to confuse pip)
* Pass all of $(HOST_PYTHON[3]_PACKAGE_BUILD_DEPENDS) to the function,
since pip can handle multiple arguments/packages
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Some packages (PyYAML,
https://github.com/openwrt/packages/pull/8482#discussion_r270692276)
recognize "global" options to setup.py; these must appear before the
"install" command on the command line.
This adds PYTHON[3]_PKG_SETUP_GLOBAL_ARGS, which let packages set these
global options.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Currently, all files in usr/bin (presumably all Python scripts) are run
through sed to replace the shebang; sed will overwrite the file whether
or not a match is found. This causes symlinks to be overridden and made
into copies of their targets. python[3]-base and python[3]-dev are
affected by this.
This adds the --follow-symlinks flag to sed, in addition to using
$(SED), so that symlinks are not overridden.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
If users want to define these before including python[3]-package.mk, these
vars will be overridden during the include.
So, override these vars if they haven't been defined.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This adds a variable (PYTHON_PKG_SETUP_DIR / PYTHON3_PKG_SETUP_DIR) that
allows a Python package Makefile to control the directory where setup.py
is called (as part of PyBuild/Compile/Default /
Py3Build/Compile/Default).
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This fixes Package/*/install to call PyPackage/*/install correctly.
Previously, if a package used the default PyPackage/*/install, then it
would not called. (A custom-defined PyPackage/*/install would be called
with no issue.)
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
It's a common operation for both Python & Python3, so move it to the
script `python-package-install.sh` script.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
For python `src` packages we should clear out the DEPENDS
to prevent recursive deps from happening.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
The only difference just a parameter for Python3
[ -b to compile bytecodes in legacy mode ].
No need to keep 2 almost identical files now
that they're exported.
I'm a bit scared of that param, since it may get
removed at some point.
But let's see until then.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Since `lang/python` is it's own folder of Python packages
(for both Python 2 & 3), and these build rules are needed
in a lot of packages [especially Python packages],
putting them here makes sense architecturally,
to be shared.
This also helps get rid of the `include_mk` construct
which relies on OpenWrt core to provide, and seems
like a broken design idea that has persisted for a while.
Reason is: it requires that Python 2/3 be built to provide
these mk files for other Python packages,
which seems like a bad idea.
Long-term, there could be an issue where some other feeds
would require these mk files [e.g. telephony] for
some Python packages.
We'll see how we handle this a bit later.
For now we limit this to this feed.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This reverts commits 4333d1dcbf and
074d2863be, making Python packages
discoverable again by pkg_resources.
Fixes#5361.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
That way some python packages can choose
to keep their egg-info dirs, if they want to, or they're needed.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
To install Python packages host side, that
may be needed for a build.
The intent, is to try to reduce host-side Python
packages being installed via LEDE/OpenWrt build system.
Because those seem like a pain to maintain.
The idea is adapted from Yousong's `python-packages`
package.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Similar to LEDE/OpenWrt's Build/Compile/Default rule,
and other similarities like this.
This should allow Python packages to define
PyBuild/Compile rules to do specific stuff per
package.
The advantage of using these (over just overriding
Build/Compile) is the VARIANT mechanism that is
in place to support packaging both for Python & Python3.
So, PyBuild/Compile will get picked up for the Python
variant build, and Py3Build/Compile will get picked
up for the Python3 variant build.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
I admit this may be be a bit aggressive, but the lang
folder is getting cluttered/filled up with Python, PHP, Perl,
Ruby, etc. packages.
Makes sense to try to group them into per-lang folders.
I took the Pythons.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Technically, this just updates build details.
No functionality change to package itself.
So, no need to increase PKG_RELEASE on this change.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
And move \`python3-package.mk\` include, same as in python package.
This is mostly to have consistency/similarity between python & python3
packages.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
--sysconfigdir and --sbindir are obtained from --prefix
and --exec-prefix, so no need to have them explicitly
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Currently, the python3-package.mk framework relies on HOST_BUILD_PREFIX to
refer to the $(STAGING_DIR)/host directory but using the HOST_BUILD_PREFIX
variable requires the use of include/host-build.mk which in turn includes
python3-package.mk, leading to target redefinition errors.
In order to provide a global, uniform way to refer to the host staging
directory, LEDE introduced a new variable STAGING_DIR_HOSTPKG which points
to $(STAGING_DIR)/host for now with the purpose of eventually being able to
relocate that directory in the future.
This commit changes python3-package.mk to ...
- stop including include/host-build.mk (revert of #3423)
- replace usages of $(HOST_BUILD_PREFIX) with $(STAGING_DIR_HOSTPKG)
- warn and fallback to $(STAGING_DIR)/host if STAGING_DIR_HOSTPKG is
unavailable
The fallback code will ensure that the python host build infrastructure
continues to work properly on older OpenWrt and LEDE versions until the
STAGING_DIR_HOSTPKG is fully settled in and can be removed some time
in the future.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jeffery To
Michal Vasilek
Alexandru Ardelean
Eneas U de Queiroz
Jo-Philipp Wich
Matthias Schiffer