faster to compile.
A small selection of packages was tested going from:
Executed in 696.30 secs fish external
usr time 82.98 mins 395.00 micros 82.98 mins
sys time 9.02 mins 0.00 micros 9.02 mins
to:
Executed in 592.20 secs fish external
usr time 84.84 mins 361.00 micros 84.84 mins
sys time 8.85 mins 57.00 micros 8.85 mins
Tested by running make -j 12 and wiping staging/build_dir/target_x
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Contains following list of changes:
ab4c3471b261 tests: add cram based unit tests
7b4e3241e1bd tests: add cgi-io built with clang sanitizers
21831f45d16d Disable session ACLs during unit testing
2f525417b5df Add initial GitLab CI support
57f1c4f18cb6 Add .gitignore
09f9ac5066ee Fix off-by-one in postdecode_fields
ed8ce0d5d28b Add fuzzing of utility functions
a61581819800 Add fuzzing of multipart_parser
6b0615b728ed Refactor utility functions into static library
a0ed2c9a7a72 Fix clang compiler errors
232659da19a4 Fix possible NULL dereference
8e5719b37a67 Fix warnings reported by clang-10 static analyzer
b99aa8a64cca Remove Makefile
Signed-off-by: Petr Štetiar <ynezz@true.cz>
No functional changes, just moved the sources into out of tree
project[1] so it's going to be easier to do CI with unit testing,
fuzzing etc.
1. https://git.openwrt.org/?p=project/cgi-io.git;a=shortlog
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Create an anonymous inode in /tmp using O_TMPFILE and attempt to link the
file in place using linkat(). Only fall back to the old file copy when
linking the tempfile fails.
Avoids double memory use if both the temporary upload file and the
destination file are located in /tmp.
Ref: https://github.com/openwrt/luci/issues/3654
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Allocate dynamic buffer memory for decoding post data and allow post
requsts up to 128KB compared to the previos 1KB limit.
Also support downloading /proc and /sys files by falling back to
chunked transfer encoding when the file size cannot be determined.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
In the command read side, close the superfluous write end of the pipe
early to ensure that EOF is reliably detected. Without that change, splice
calls to read from the pipe will occasionally hang until the CGI process
is eventually killed due to timeout.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Implement a new "cgi-exec" applet which allows to invoke remote commands
and stream their stdandard output back to the client via HTTP. This is
needed in cases where large amounts of data or binary encoded contents
such as tar archives need to be transferred, which are unsuitable to be
transported via ubus directly.
The exec call is guarded by the same ACL semantics as rpcd's file plugin,
means in order to be able to execute a command remotely, the ubus session
identified by the given session ID must have read access to the "exec"
function of the "cgi-io" scope and an explicit "exec" permission rule for
the invoked command in the "file" scope.
In order to initiate a transfer, a POST request in x-www-form-urlencoded
format must be sent to the applet, with one field "sessionid" holding
the login session and another field "command" specifiying the commandline
to invoke.
Further optional fields are "filename" which - if present - will cause
the download applet to set a Content-Dispostition header and "mimetype"
which allows to let the applet respond with a specific type instead of
the default "application/octet-stream".
Below is an example for the required ACL rules to grant exec access to
both the "date" and "iptables" commands. The "date" rule specifies the
base name of the executable and thus allows invocation with arbitrary
parameters while the latter "iptables" rule merely allows one specific
set of arguments which must appear exactly in the given order.
ubus call session grant '{
"ubus_rpc_session": "...",
"scope": "cgi-io",
"objects": [
[ "exec", "read" ]
]
}'
ubus call session grant '{
"ubus_rpc_session": "...",
"scope": "file",
"objects": [
[ "/bin/date", "exec" ],
[ "/usr/sbin/iptables -n -v -L", "exec" ]
]
}'
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Fixes following errors:
main.c:458:37: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
main.c:463:17: error: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Werror=sign-compare]
main.c:518:35: error: comparison of integer expressions of different signedness: ‘ssize_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
main.c:157:3: error: ignoring return value of ‘read’, declared with attribute warn_unused_result [-Werror=unused-result]
main.c:763:3: error: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Werror=unused-result]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Currently cgi-io try to read data after the data ended.
- Adds "-" to whitelist char
- In main_upload is tried to consume the buffer while it's already readed by the while loop before
Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
Instead of always replying with a generic 500 internal server error code,
use more appropriate codes such as 403 to indicate denied permissions.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Add a new `cgi-download` applet which allows to retrieve the contents
of regular files or block devices.
In order to initiate a transfer, a POST request in x-www-form-urlencoded
format must be sent to the applet, with one field "sessionid" holding
the login session and another field "path" containing the file path to
download.
Further optional fields are "filename" which - if present - will cause
the download applet to set a Content-Dispostition header and "mimetype"
which allows to let the applet respond with a specific type instead of
the default "application/octet-stream".
Below is an example for the required acl rules to grant download access
to files or block devices:
ubus call session grant '{
"ubus_rpc_session": "...",
"scope": "cgi-io",
"objects": [
[ "download", "read" ]
]
}'
ubus call session grant '{
"ubus_rpc_session": "...",
"scope": "file",
"objects": [
[ "/etc/config/*", "read" ],
[ "/dev/mtdblock*", "read" ]
]
}'
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Use the `cgi-io` scope to check for permission to execute the requested
command (`upload`, `backup`) and the `file` scope to check path
permissions.
The reasoning of this change is that `cgi-io` is usually used in
conjunction with `rpcd-mod-file` to transfer large file data out
of band and `rpcd-mod-file` already uses the `file` scope to manage
file path access permissions. After this change, both `rpc-mod-file`
and `cgi-io` can share the same path acl rules.
Write access to a path can be granted by using an ubus call in the
following form:
ubus call session grant '{
"ubus_rpc_session": "...",
"scope": "file",
"objects": [
[ "/var/lib/uploads/*", "write" ]
]
}'
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
The CONTRIBUTING.md requests an (or multiple) SPDX identifier for GPL
licenses. But a lot of packages did use a different, non-SPDX style with a
"+" at the end instead of "-or-later".
Signed-off-by: Sven Eckelmann <sven@narfation.org>
Introduce further ACL checks to verify that the request-supplied
upload location may be written to. This prevents overwriting things
like /bin/busybox and allows to confine uploads to specific directories.
To setup the required ACLs, the following ubus command may be used
on the command line:
ubus call session grant '{
"ubus_rpc_session": "d41d8cd98f00b204e9800998ecf8427e",
"scope": "cgi-io",
"objects": [
[ "/etc/certificates/*", "write" ],
[ "/var/uploads/*", "write" ]
]
}'
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Report SHA256 checksums in addition to the MD5 ones to make cgi-io suitable
for sysupgrade image verification.
Also allow stat(), md5sum and/or sha256sum to fail and respond with a JSON
null value instead, leaving it to the frontend to handle errors as needed.
Fixes#4790.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Remove an improperly placed semicolon in order to solve the following
compiler error:
.../main.c:144:3: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
if (execl("/bin/busybox", "/bin/busybox", "md5sum", file, NULL));
^~
.../main.c:145:4: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
return NULL;
^~~~~~
cc1: all warnings being treated as errors
Fixes#4723.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
luci2-io-helper: bugfix buckup script read timeout
Reading files from stdin will block for ever. The uhttpd is killing the
backup process after script_timeout.
Switching read to non blocking mode and add a waitpid for the slave
process does not end in a script_timeout anymore.
Signed-off-by: Florian Eckert <Eckert.Florian@googlemail.com>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Fixes build errors with external toolchains:
[ 33%] Building C object CMakeFiles/cgi-io.dir/main.c.o
/home/florian/dev/openwrt/trunk/build_dir/target-mipsel-unknown-linux-gnu_glibc/cgi-io/main.c:30:21:
fatal error: libubus.h: No such file or directory
#include <libubus.h>
^
compilation terminated.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Jo-Philipp Wich
Rosen Penev
Petr Štetiar
Ansuel Smith
Sven Eckelmann
Jianhui Zhao
Daniel Golle
Florian Fainelli
John Crispin