Commit Graph

124 Commits

Author SHA1 Message Date
Sebastian Kemper
f58d323534 build: allow file modes per binary package
Currently the global variable PKG_FILE_MODES is used for all ipkg
creations. This works for Makefiles which output a single package, or
variants of a single package.

But if a Makefile outputs multiple packages that each contain different
files, setting PKG_FILE_MODES causes build failure when any of the files
in the variable do not exist in the folder that is currently being
packaged.

Example:

/openwrt/staging_dir/host/bin/fakeroot -l /openwrt/staging_dir/host/lib/libfakeroot.so -f /openwrt/staging_dir/host/bin/faked /openwrt/scripts/ipkg-build -m "/usr/lib/mariadb/plugin/auth_pam_tool_dir:root:376:0750" /openwrt/build_dir/target-mips_24kc_musl/mariadb-10.4.13/ipkg-mips_24kc/mariadb-server-plugin-disks /openwrt/bin/packages/mips_24kc/packages
+chown: cannot access '/openwrt/build_dir/target-mips_24kc_musl/mariadb-10.4.13/ipkg-mips_24kc/mariadb-server-plugin-disks//usr/lib/mariadb/plugin/auth_pam_tool_dir': No such file or directory

This commit changes the file mode handling a bit. The file mode can now
be set either globally via PKG_FILE_MODES (no behavior change) or on a
per-package basis via FILE_MODES. This way specific file modes can be
used for any particular package.

This behavior is already used for other OpenWrt variables, hence it is
familiar:

PKG_MAINTAINER vs MAINTAINER
PKG_SOURCE_SUBDIR vs SUBDIR
PKG_LICENSE vs LICENSE
...

Signed-off-by: Sebastian Kemper <sebastian_ml@gmx.net>
2020-09-09 14:13:59 +01:00
Felix Fietkau
7716a43092 build: fix extreme build system slowdown caused by SOURCE_DATE_EPOCH changes
Adding inline shell invocations in per-target variables causes them to be
executed over and over again, which causes a significant slowdown.
Fix this by evaluating it only once per package directory

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2020-09-05 11:26:57 +02:00
Paul Spooren
62d5ec7306 build: store SourceDateEpoch in manifest
The usage of granular `SOURCE_DATE_EPOCH` for packages is an
incrementing integer which could be useful for downstream tooling,
therefore add it to the packages manifest.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-31 11:18:06 +01:00
Paul Spooren
fd29f2bcd9 build: store granular timestamps in packages
With the new `SOURCE` argument of `get_source_date_epoch` it is possible
to set package timestamps based on actual package changes rather thane
$TOPDIR changes.

This commit adds a new variable PKG_SOURCE_DATE_EPOCH which is used by
the `ipkg` build script. As a fallback the existing SOURCE_DATE_EPOCH is
used or as last resort the current time.

The redundant checks for `.git/` and `.svn/` are removed.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-31 11:18:06 +01:00
Paul Spooren
353ce2e521 build: ipkg-build use fakeroot with PKG_FILE_MODES
The `ipkg-build` script converts a folder into a `opkg` installable
package. Until now it would use root:root for all packages and try to
preserve file modes.

This has the two drawbacks of packages want to add non-root files or add
SUID files, like the `sudo` package does.

To give more flexibility regarding file modes and avoid init script
hacks, a new variable called `PKG_FILE_MODES`. The variable contains a
list of files modes in the format `path:owner:group:mode`.

An example for the `sudo` package below:

```
PKG_FILE_MODES:=\
        /usr/bin/sudo:root:root:4755 \
        /etc/sudoers:root:root:0440
```

The `ipkg-build` now runs within a fakeroot environment to set any mode
and directly store it in the resulting `ipk` package archive.

Both options `-o` and `-g` are no longer required due to the introduction
of the more flexible `-m` options, which takes the `PKG_FILE_MODES` as
input.

Lastly the option `-c` is removed as it's unused within the script.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-31 11:13:12 +01:00
Eneas U de Queiroz
11192484fa build: reduce number of files passed to ipk-remove
Instead of using xargs to pass a huge number of files to
script/ipkg-remove, which will usually pick only one, use a more
restrictive wildcard so that, currently, at the most 325 files are
examined, instead of up to over 2,300.  The 325-file package is python,
which is picking up python3* ipks.  It is about to be removed.
Runner-up is ddns-scripts with 7 files.

This makes a second run of make package/luci/compile go from
real 16.40s; user 17.42s; sys 2.73s
to
real 10.71s; user 9.51s; sys 1.27s

There is a caveat though: if one were to remove the ABI_VERSION of a
package that ends in a digit [0-9], then the old package ipk will not be
removed from the bin directory by make package/abc2/clean.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-07-11 13:33:28 +02:00
Eneas U de Queiroz
78d1f3ac61 build: call ipkg-remove using xargs if #args>=512
The wildcard call to clean up luci package (luci*) can pick up over
2,300 files when the full tree is built. Running make package/luci/clean
or a second run of make package/luci/compile would fail with an
'Argument list too long' error.

To avoid that, a maybe_use_xargs function was created that runs the
command straight as usual if the number of arguments is < 512, or saves
the list in a temporary file and feeds it to xargs otherwise.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-07-11 13:33:28 +02:00
Eneas U de Queiroz
40acc13ca7 build: package-ipkg: avoid calling wildcard twice
Instead of calling $(wildcard) to check if the removal list is empty,
then calling it again to actually remove the files, define a function so
that the arguments are expanded only once when it gets called.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-07-11 13:33:28 +02:00
Florian Eckert
4d86e2ce06 buildsystem: add CPE information to ipkg packages and manifest files
Common Platform Enumeration (CPE) is a structured naming scheme for
information technology systems, software, and packages.

This information already exists in some makefiles. In order for the
information to be processed further, it should also be added to the
manifest file and the control file of ipkg packages.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
2020-05-05 22:07:59 +02:00
Xu Wang
44304c1d67 base-files: fix build for /sbin/pkg_check
Setting CONFIG_IPK_FILES_CHECKSUMS=y causes sha256 checksum files to be
included with the packages to check for corruption. This commit fixes two
issues:
- /sbin/pkg_check was being removed incorrectly if IPK_FILES_CHECKSUMS=y
- checksums were being saved in the wrong file

Signed-off-by: Xu Wang <xwang1498@gmx.com>
2020-01-14 17:52:34 +01:00
Jonas Gorski
5ede3fcdf7 build: warn when packages have no associated install section
Declaring a package without the appropriate install section is an easy
mistake to make, especially when renaming packages. Since this is also
easy to detect, warn about it when it happens.

Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
No-objections-at-all-by: Jo-Philipp Wich <jo@mein.io>
2019-06-25 14:42:07 +02:00
Eneas U de Queiroz
a41f474d17 build: add support to && in DEPENDS
Adds support to && operand in DEPENDS.  Also, fixes generation of ||
dependencies by scripts/package-metadata.pl.

The precedence order from higher to lower is && then ||.  Use of
parentheses to change the order is not supported. As before, they are
silently ignored.  Use them for readability only.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com> [DMARC removal]
2019-05-31 11:21:22 +02:00
Jo-Philipp Wich
790bce92ad build: formatting fixes for per-provide ABI_VERSION suffixes
- Filter out potential duplicates with the package name
   (e.g. when renaming libfoo1 w/ ABI_VERSION:=1 to libfoo)
 - Use the GetABISuffix macro to properly separate the suffix
   with a dash in case the basename ends with a number

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2019-01-24 12:51:39 +01:00
Jo-Philipp Wich
60558790a2 build: extend ABI_VERSION suffixing to provides
When a library package specifies additional provides, e.g. libncurses
which provides libncursesw, we should also append the abi version
suffix to each provide, since there may be more than one package
providing the virtual library.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2019-01-24 10:39:29 +01:00
Michal Hrusecky
74450124f6 build: Optionally provide file checksums in package metadata
This may be useful if you don't entirely trust your flash and want to be able
to check for corruptions.

Signed-off-by: Michal Hrusecky <Michal@Hrusecky.net>
2019-01-22 09:22:25 +01:00
Jo-Philipp Wich
e6bcf1e4ac build: add ABI_VERSION to binary package names
Add the ABI_VERSION source makefile variable to the binary package basename
and resolve source dependencies on packages with ABI_VERSION set to such
expanded names.

If for example a package specifies DEPENDS:=libopenssl while the OpenSSL
Makefile specifies ABI_VERSION:=1.0.0, the resulting ipk control data
dependency will be "Depends: libopenssl1.0.0" and the libopenssl ipk file
will be called "libopenssl1.0.0_<version>_<arch>.ipk".

The next time a library such as OpenSSL is updated to an incompatible
version, the ABI_VERSION shall be changed accordingly to prevent opkg from
simply upgrading to an incompatible library without considering the
dependencies of already installed packages.

Also introduce another "SourceName" control field which is required by
the newly introduced "scritps/ipkg-remove" to determine the proper related
.ipk files to delete upon buildroot package clean operations.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2019-01-19 14:32:12 +01:00
Felix Fietkau
7c306ae640 build: fix compile error when a package includes itself in PROVIDES
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2018-07-20 10:40:58 +02:00
Michael Heimpold
2cb75cd8b9 build: allow defining license information per binary package
At the moment, license information can only be specified on a
"per source package" level while other metadata fields (e.g. maintainer)
can be given for each binary package. Apply the same logic for license
fields as well. This can be used e.g. in cases where a library is
distributed under some license while related tools are distributed
under a different one.

Signed-off-by: Michael Heimpold <mhei@heimpold.de>
2017-11-25 20:02:04 +01:00
Yousong Zhou
73bc636aba build: ipkg: new field Alternatives
It's a list of specs of the following form seprated by commas to
describe alternatives provided by this package

    <prio>:<path>:<altpath>

<path> will be a symbolic link to <altpath> of the highest <prio>

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2017-05-02 22:10:51 +08:00
Felix Fietkau
1f22957247 build: clean up redundant touching of the package install info file
Instead of using a separate .clean stamp file, remove the install info
file on compile, then append the install package list afterwards

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-02-26 13:31:44 +01:00
Felix Fietkau
3cd7f43017 build: fix package build after deleting staging dir
Ensure that the <package>.provides files are always created

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-02-09 13:51:40 +01:00
Florian Fainelli
25e2cf392a package-ipkg: Do not fail build without base-files
If the base-files package is not selected, we will fail executing the
very first postinst script:

make[3]: Leaving directory `/local/users/fainelli/openwrt/trunk'
cp -fpR
/local/users/fainelli/openwrt/trunk/build_dir/target-arm_xscale_musl-1.1.15_eabi/root-orion
/local/users/fainelli/openwrt/trunk/build_dir/target-arm_xscale_musl-1.1.15_eabi/root.orig-orion
./usr/lib/opkg/info/busybox.postinst: line 3:
/local/users/fainelli/openwrt/trunk/build_dir/target-arm_xscale_musl-1.1.15_eabi/root-orion/lib/functions.sh:
No such file or directory
./usr/lib/opkg/info/busybox.postinst: line 4: default_postinst: command
not found
postinst script ./usr/lib/opkg/info/busybox.postinst has failed with
exit code 127
make[2]: *** [package/install] Error 1

Check for the existence of lib/functions.sh, and if it does not exist,
just bail out gracefully.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
2017-01-24 11:07:45 -08:00
Felix Fietkau
f784b3e195 build: fix regression in handling ABI_VERSION
The code was updated without changing $$@ into the proper target
filename

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-21 18:30:51 +01:00
Felix Fietkau
5aa03e16a7 build: disable CONFIG_AUTOREMOVE for packages/tools where QUILT is used
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-19 13:30:22 +01:00
Felix Fietkau
87d9e3462d build: stage unstripped packages inside PKG_BUILD_DIR
This makes the build slightly more efficient by avoiding the need to
re-stage packages on every full build run.

It is also necessary for the upcoming CONFIG_AUTOREMOVE feature

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-18 23:57:08 +01:00
Felix Fietkau
8aa93dafc8 build: write ABI_VERSION file when staging the package
The make target does not rewrite the file if the contents didn't
change. This causes make to always consider the target old.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-18 23:57:08 +01:00
Felix Fietkau
9228d1c066 build: introduce extra targets that contain only proper dependencies
This can be used to check if targets like prepare or compile are up to date

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-18 23:57:08 +01:00
Rafał Miłecki
b522292405 base-files: add support for overlaying rootfs content
This adds support for install-overlay define. When used in package it
allows installing files to a special directory that gets copied to the
root when installing it.
It allows overwriting files provided by other packages.

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
2016-12-20 09:35:36 +01:00
Felix Fietkau
64c386c566 build: remove stale .ipk files if package dir changes
If a package nonshared status is changed, a stale .ipk file might still
be present in the old package directory. Remove the .ipk file from all
package directories when building a new one (or explicitly running
clean)

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-11-08 10:49:51 +01:00
Jo-Philipp Wich
317b3556a4 include: properly update .install stamp files
Right now the $(PKG_INSTALL_STAMP) files are only written if a package is
selected as <*> but never deleted or emptied if the corresponding package
is getting deselected.

For ordinary packages this usually is no problem as the package/install
recipe performs its own check for enabled packages when assembling the
list of install stamp files to consider, but this logic might fail under
certain circumstances for packages providing multiple build variants.

In case of a multi-variant package, the buildroot first checks if any
of the variants is enabled, then resolves all variants of the common
source package and finally processes the corresponding .install stamp
files of all variants, relying on the assumption that only the selected
.install stamp file exists.

When an initially selected variant is getting deselected or changed from
<*> to <m> and another variant is marked as <*> instead, the .install
stamp file of the deselected variant remains unchanged and a second
.install stamp file for the newly selected variant is getting created,
causing the package/install recipe to pick up two .install stamps with
conflicting variants, leading to opkg file clashes.

This issue happens for example if package "ip" is set to <m> and package
"ip-full" to <*> -  the install command will eventually fail with:

     * check_conflicts_for: The following packages conflict with ip:
     * check_conflicts_for: 	ip-full *
     * opkg_install_cmd: Cannot install package ip.

In order to fix the problem, always process the removal requests or the
.install stamp files, even for deselected packages but only write the
package base name into the stamp file if the corresponding package is
marked as builtin.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-11-02 01:01:35 +01:00
Matthias Schiffer
ec041920b7
include/package-ipkg.mk: use TARGET_PATH_PKG in Package/*/install steps
Package/*/install was using a different PATH than all other steps like
Build/Install, which was confusing and easily led to mistakes.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2016-10-06 01:57:39 +02:00
Felix Fietkau
71d533eea9 package-ipkg: do not include feeds.mk at metadata dump time to speed up scanning
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-06-16 11:56:24 +02:00
Matthias Schiffer
3ee6c17cd1
package-ipkg.mk: fix Provides for packages with multiple PROVIDES entries
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2016-06-16 04:15:02 +02:00
Jo-Philipp Wich
d9ad55a609 include: remove unused FeedPackageDir macro arguments
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-04-22 12:02:53 +02:00
Jo-Philipp Wich
5170393f8c include: choose package output directory based on repository info
Use the new repository metadata field to choose the output directory of the
final package archives.

Non-sharable packages will be placed in the per-target package directory
while the rest will be placed in a per-repository sub directory within the
$OUTPUT_DIR/packages/$CPU_TYPE/ prefix.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-04-06 21:55:44 +02:00
Felix Fietkau
e597a4190e package-ipkg.mk: fix typo (patch from #19540)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45568
2015-04-23 08:27:22 +00:00
Felix Fietkau
d600f9e957 package-ipkg.mk: fix typo
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45194
2015-03-31 18:31:08 +00:00
Felix Fietkau
765c73f309 package-ipkg.mk: add missing semicolon (#19381)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45193
2015-03-31 18:29:30 +00:00
Felix Fietkau
640318e620 build: ensure that preinst and postrm are executable (fixes #19372)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45187
2015-03-31 15:13:27 +00:00
Felix Fietkau
55c5d10ca9 tools: replace ipkg-utils with a reduced ipkg-build variant in scripts/
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45129
2015-03-29 07:35:26 +00:00
Felix Fietkau
61dd3c8013 build: make postinst/prerm scripts executable
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45128
2015-03-29 07:35:19 +00:00
John Crispin
4a1d29c5fc Fix possible fail to check for dependencies
Few characters in filenames (a plus sign, a dot) can be interpreted
specially by grep. This can lead to the omission of missing package
dependency. For example if we would have "some.file.so" then it
matches also "some2file.so". -F switch off special meaning
of any character and -x match against whole line.

Signed-off-by: Sławomir Demeszko <s.demeszko@wireless-instruments.com>

SVN-Revision: 44247
2015-02-03 10:11:06 +00:00
Nicolas Thill
d8263c3025 include: don't use extended regex for deps checking
Closes: #18851

Signed-off-by: Nicolas Thill <nico@openwrt.org>

SVN-Revision: 44224
2015-01-31 08:15:46 +00:00
Jo-Philipp Wich
c3787d4c0c include: use anchored patterns for dependency checking
Anchor search pattern when testing dependencies, otherwise the check may
succeed in cases where it shouldn't, e.g. when matching "udp_tunnel.ko"
against "ip6_udp_tunnel.ko".

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 44207
2015-01-29 21:59:17 +00:00
Jo-Philipp Wich
4de4827ddc include: unbreak conffiles, postinst & prerm exports
Changeset r43017 reworked the ipkg control metadata generation but broke
the export of conffiles, postinst and prerm defines.

Change the code back to rely on shvar and shexport, this is required to
properly output multiline contents.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 43041
2014-10-24 09:24:51 +00:00
Felix Fietkau
949ab1487f build: only print skipped packages that were selected
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43023
2014-10-22 08:57:50 +00:00
Felix Fietkau
d8278dee4d kernel: use /bin/sh instead of bash to speed up processing of many kmod-* packages
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43022
2014-10-22 08:57:45 +00:00
Felix Fietkau
d081edf7eb build: clean up and optimize ipkg control generator code
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43017
2014-10-22 08:57:16 +00:00
Felix Fietkau
edc8e2022a build: optimize printing of disabled packages
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43015
2014-10-22 08:57:07 +00:00
Felix Fietkau
c6ebce701e build: propagate package conflicts to opkg metadata
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 42771
2014-10-05 16:41:38 +00:00