Hannu Nyman c13b537885 | 6 years ago | |
---|---|---|
.. | ||
Flask | 6 years ago | |
Jinja2 | 6 years ago | |
MarkupSafe | 6 years ago | |
Werkzeug | 6 years ago | |
bcrypt | 6 years ago | |
chardet | 7 years ago | |
click | 7 years ago | |
django | 6 years ago | |
django-appconf | 7 years ago | |
django-compressor | 7 years ago | |
django-constance | 6 years ago | |
django-formtools | 6 years ago | |
django-jsonfield | 7 years ago | |
django-picklefield | 6 years ago | |
django-postoffice | 6 years ago | |
django-ranged-response | 6 years ago | |
django-restframework | 6 years ago | |
django-simple-captcha | 6 years ago | |
django-statici18n | 6 years ago | |
django-webpack-loader | 6 years ago | |
et_xmlfile | 7 years ago | |
flup | 6 years ago | |
gunicorn | 6 years ago | |
itsdangerous | 6 years ago | |
jdcal | 6 years ago | |
micropython | 6 years ago | |
micropython-lib | 6 years ago | |
openpyxl | 6 years ago | |
passlib | 6 years ago | |
pillow | 6 years ago | |
pyjwt | 6 years ago | |
pyodbc | 6 years ago | |
python | 6 years ago | |
python-asn1crypto | 7 years ago | |
python-attrs | 6 years ago | |
python-automat | 6 years ago | |
python-certifi | 6 years ago | |
python-cffi | 6 years ago | |
python-constantly | 6 years ago | |
python-crcmod | 7 years ago | |
python-crypto | 6 years ago | |
python-cryptography | 6 years ago | |
python-curl | 7 years ago | |
python-dateutil | 6 years ago | |
python-dns | 6 years ago | |
python-dpkt | 6 years ago | |
python-egenix-mx-base | 6 years ago | |
python-enum34 | 6 years ago | |
python-evdev | 6 years ago | |
python-gmpy2 | 6 years ago | |
python-gnupg | 6 years ago | |
python-hyperlink | 6 years ago | |
python-idna | 6 years ago | |
python-incremental | 6 years ago | |
python-ipaddress | 6 years ago | |
python-ldap | 6 years ago | |
python-lxml | 6 years ago | |
python-mysql | 6 years ago | |
python-oauthlib | 6 years ago | |
python-packages | 7 years ago | |
python-parsley | 6 years ago | |
python-pcapy | 6 years ago | |
python-pip-conf | 7 years ago | |
python-ply | 6 years ago | |
python-psycopg2 | 6 years ago | |
python-pyasn1 | 6 years ago | |
python-pyasn1-modules | 6 years ago | |
python-pycparser | 6 years ago | |
python-pyopenssl | 6 years ago | |
python-pyptlib | 6 years ago | |
python-pyserial | 7 years ago | |
python-qrcode | 7 years ago | |
python-requests | 6 years ago | |
python-requests-oauthlib | 6 years ago | |
python-service-identity | 6 years ago | |
python-six | 6 years ago | |
python-twisted | 6 years ago | |
python-txsocksx | 6 years ago | |
python-urllib3 | 6 years ago | |
python-yaml | 6 years ago | |
python-zope-interface | 6 years ago | |
python3 | 6 years ago | |
python3-bottle | 6 years ago | |
pytz | 6 years ago | |
rcssmin | 6 years ago | |
simplejson | 6 years ago | |
vobject | 6 years ago | |
README.md | 6 years ago | |
python-host.mk | 7 years ago | |
python-package-install.sh | 6 years ago | |
python-package.mk | 6 years ago | |
python-version.mk | 6 years ago | |
python3-host.mk | 7 years ago | |
python3-package.mk | 6 years ago | |
python3-version.mk | 6 years ago |
This section describes specifics for the Python packages that are present in this repo, and how things are structured.
In terms of license, contributing guide, etc, all of that information is described in the top README.md file, and it applies here as well. This document attempts to cover only technical aspects of Python/Python3 packages, and maybe some explanations about how things are (and why they are as they are).
This sub-tree came to exist after a number of contributions (Python packages) were made to this repo, and the lang subtree grew to a point where a decision was made to move all Python packages under lang/python.
It contains the 2 Python interpreters (Python & Python3) and Python packages. Most of the Python packages are downloaded from pypi.org. Python packages from pypi.org are typically preferred when adding new packages.
If more packages (than the ones packaged here) are needed, they can be downloaded via pip or pip3. Note that the versions of pip
& setuptools
[available in this repo] are the ones that are packaged inside the Python & Python3 packages (yes, Python & Python3 come packaged with pip
& setuptools
).
In order to build the Python[3] interpreters, a host Python/Python3 interpreter needs to be built, in order to process some of the build for the target Python/Python3 build. The host Python[3] interpreters are also needed so that Python bytecodes are generated, so the host interpreters need to be the exact versions as on the target. And finally, the host Python[3] interpreters also provide pip & pip3, so that they may be used to install some Python[3] packages that are required to build other Python[3] packages. That's why you'll also see a Python/Python3 build & staging directories.
As you're probably thinking, this sounds [and is] somewhat too much complication [just for packaging], but the status of things is-as-it-is, and it's probably much worse than what's currently visible on the surface [with respect to packaging Python[3] & packages].
As mentioned earlier, Python[3] packages are shipped with bytecodes, and the reason for this is simply performance & size. The thought/discussion matrix derives a bit like this:
python-src
& python3-src
can be provided for people that want the source codeBy default, automatic Python[3] byte-code generation is disabled when running a Python script, in order to prevent a disk from accidentally filling up. Since some disks reside in RAM, this also means not filling up the RAM. If someone wants to convert Python source to byte-code then he/she is free to compile it [directly on the device] manually via the Python interpreter & library.
The basis of all these packages are:
These 2 are normal OpenWrt packages, which will build the Python interpreters. They also provide python[3]-pip
& python[3]-setuptools
. Each Python or Python3 package is actually split into multiple sub-packages [e.g. python-email, python-sqlite3, etc]. This can be viewed inside lang/python/python/files & lang/python/python3/files.
The reason for this splitting, is purely to offer a way for some people to package Python/Python3 in as-minimal-as-possible-and-still-runable way, and also to be somewhat maintainable when packaging. A standard Python[3] installation can take ~20-30 MBs of disk, which can be somewhat big for some people, so there are the python[3]-base
packages which bring that down to ~5 MBs. This seems to be good enough (and interesting) for a number of people.
The Python[3] interpreters are structured like this:
python-base
(and python3-base
), which is just the minimal package to startup Python[3] and run basic commandspython
(and python3
) are meta-packages, which install almost everything (python[3]-base [plus] Python[3] library [minus] some unit-tests & some windows-y things)python-light
(and python3-light
) are python
(and python3
) [minus] packages that are in lang/python/python/files or lang/python/python3/files ; the size of these 2 packages may be sensible (and interesting) to another group of peopleAll other Python & Python3 packages (aside from the 2 intepreters) typically use these files:
Note that Python/Python3 packages don't need to use these files (i.e. python[3]-package.mk
& python[3]-host.mk
), but they do provide some ease-of-use & reduction of duplicate code, especially when packaging for both Python & Python3. And they do contain some learned-lessons about packaging Python/Python3 packages, so it's a good idea to use them.
A Python package can be packaged for either Python or Python3 or both.
This section will describe both, and then it can be inferred which is for which.
Packaging for both Python & Python3 uses the VARIANT
mechanism for packaging inside OpenWrt. (#### FIXME: find a link for this later if it exists)
It's important when packaging for both Python & Python3 to override this variable, so that the build directory differs for each variant.
Typically it's just something like:
PKG_BUILD_DIR:=$(BUILD_DIR)/$(BUILD_VARIANT)-pyasn1-$(PKG_VERSION)
Where pyasn1
should be some other name, or maybe PKG_NAME
This should be added before this include:
include $(INCLUDE_DIR)/package.mk
In many cases, this needs to be overriden. This is usually because the way Python packages are archived, don't follow the convention of other tar.gz
packages.
So, something like:
PKG_UNPACK=$(HOST_TAR) -C $(PKG_BUILD_DIR) --strip-components=1 -xzf $(DL_DIR)/$(PKG_SOURCE)
should be added.
It's not important whether this is after or before include $(INCLUDE_DIR)/package.mk
If packaging for Python, add this after include $(INCLUDE_DIR)/package.mk
include ../python-package.mk
If packaging for Python3, add this after include $(INCLUDE_DIR)/package.mk
include ../python3-package.mk
Order doesn't matter between python-package.mk
& python3-package.mk
.
These will make sure that build rules for Python or Python3 can be specified and picked up for build.
This part is similar to default OpenWrt packages.
It's usually recommended to have a Package/<PKG_NAME>/Default
section that's common for both Python & Python3.
Example:
define Package/python-lxml/Default
SECTION:=lang
CATEGORY:=Languages
SUBMENU:=Python
URL:=https://lxml.de
DEPENDS:=+libxml2 +libxslt +libexslt
endef
Then for each variant do something like:
define Package/python-lxml
$(call Package/python-lxml/Default)
TITLE:=python-lxml
DEPENDS+=+PACKAGE_python-lxml:python-light +PACKAGE_python-lxml:python-codecs
VARIANT:=python
endef
define Package/python3-lxml
$(call Package/python-lxml/Default)
TITLE:=python3-lxml
DEPENDS+=+PACKAGE_python3-lxml:python3-light
VARIANT:=python3
endef
Some considerations here (based on the example above):
DEPENDS
are correct for both variants; as seen in the example above, python-codecs
is needed only for python-lxml
; that's because python3-codecs
doesn't exist and is included in python3-base
; most of the times they are similar, sometimes they are not+PACKAGE_python-lxml:<dep>
as seen in the above example ; the reason for this is build-time reduction ; if you want to build Python3 only packages, this won't build Python & Python packages + dependencies ; this is a known functionality of OpenWrt build deps
+PACKAGE_python-lxml
conditional deps creates circular dependencies [for some weird reason], then this can be omittedVARIANT=python
or VARIANT=python3
must be addedPackage/python3-<something>
& Package/python3-<something>
; this convention makes things easier to follow, though it could work without naming things this thisTITLE
can be something a bit more verbose/neat ; typically the name is short as seen aboveFollowing these, 2 more definitions are required:
define Package/python-lxml/description
The lxml XML toolkit is a Pythonic binding
for the C libraries libxml2 and libxslt.
endef
define Package/python3-lxml/description
$(call Package/python-lxml/description)
.
(Variant for Python3)
endef
Typically, the description is the same for both, so just mentioning that one is a variant of the other is sufficient.
If all the above prerequisites have been met, all that's left is:
$(eval $(call PyPackage,python-lxml))
$(eval $(call BuildPackage,python-lxml))
$(eval $(call Py3Package,python3-lxml))
$(eval $(call BuildPackage,python3-lxml))
The $(eval $(call PyPackage,python-lxml))
part will instantiate all the default Python build rules so that the final Python package is packaged into an OpenWrt.
And $(eval $(call BuildPackage,python-lxml))
will bind all the rules generated with $(eval $(call PyPackage,python-lxml))
into the OpenWrt build system.
These packages will contain byte-codes and binaries (shared libs & other stuff).
If a user wishes to ship source code, adding 2 more lines creates 2 more packages that ship Python source code:
$(eval $(call PyPackage,python-lxml))
$(eval $(call PyPackage,python-lxml-src))
$(eval $(call BuildPackage,python-lxml))
$(eval $(call Py3Package,python3-lxml))
$(eval $(call Py3Package,python3-lxml-src))
$(eval $(call BuildPackage,python3-lxml))
The name *-src
must be the Python package name; so for python-lxml-src
a equivalent python-lxml
name must exist.
Some packages need custom build rules (because they do).
For building, if a user specifies a PyBuild/Compile
& Py3Build/Compile
rule, this will be used to build/compile the package, instead of the default one defined in python[3]-package.mk
.
For installing files on the target, 2 build rules are used:
PyPackage/$(1)/filespec
& Py3Package/$(1)/filespec
which are Python library files relative to /usr/lib/pythonX.Y
; by default this is /usr/lib/python$(PYTHON[3]_VERSION)/site-packages
; most Python[3] packages generate files that get installed in this sub-folderPyPackage/$(1)/install
& Py3Package/$(1)/install
is similar to Package/$(1)/install
; these allow binary (or other files) to be installed on the targetBoth the 2 above rules generate a Package/$(1)/install
build rule, which gets picked up by the build system. Both can be used together (they are not mutually exclusive), and provide a good enough flexibility for specifying Python[3] packages.
These can be installed via pip and ideally they should only be installed like this, because it's a bit simpler than running them through the OpenWrt build system. Build variants on the host-side build are more complicated (and nearly impossible to do sanely) in the current OpenWrt build system.
Which is why [for example] if you need python cffi on the host build, it's easier to just add it via:
HOST_PYTHON_PACKAGE_BUILD_DEPENDS:="cffi==$(PKG_VERSION)"
HOST_PYTHON3_PACKAGE_BUILD_DEPENDS:="cffi==$(PKG_VERSION)"
[cffi is one of those packages that needs a host-side package installed for both Python & Python3].
This works reasonably well in the current OpenWrt build system, as binaries get built for this package and get installed in the staging-dir $(STAGING_DIR)/usr/lib/pythonX.Y/site-packages
.