Commit Graph

168 Commits

Author SHA1 Message Date
Paul Spooren
2496f436a8
build: use zstd for IB, toolchain, SDK and LLVM compression
Use similar efficient but faster compression algorithm.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2024-04-13 08:05:04 +02:00
Paul Spooren
d997477775 treewide: remove implicit SUBTARGET
Historically it's possible to leave the `SUBTARGETS` undefined and
automatically fallback to a "generic" subtarget. This however breaks
various downstream scripts which may have expectations around filenames:

While some targets with an explicit generic subtarget contain `generic`
in the filenames of artifacts, implicit "subtargets" don't.

Right now this breaks the CI[1], possibly also scripts using the ImageBuilders.

This commit removes all code that support implicit handling of
subtargets and instead requires every target to define "SUBTARGETS".

[1]: https://github.com/openwrt/openwrt/actions/runs/8592821105/job/23548273630

Signed-off-by: Paul Spooren <mail@aparcar.org>
2024-04-08 21:53:05 +02:00
Florian Eckert
a998a12a2f imagebuilder: add check if target is sourced from feed
The image generation would fail, if the target is included from a feed.
To fix this, check if targets is found in the feed directory.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
Tested-by: Thomas Richard <thomas.richard@bootlin.com>
2024-03-29 20:00:27 +01:00
Florian Eckert
27d227b682 imagebuilder: check if BOARD is located in the feeds sub directory
Fixes the regression so that targets that were installed via a feed can
also be build again with the Image Builder.

Fixes: 84ec8c4 ("imagebuilder: copy from buildroot only target/linux")
Signed-off-by: Florian Eckert <fe@dev.tdt.de>
Tested-by: Thomas Richard <thomas.richard@bootlin.com>
2024-03-29 20:00:27 +01:00
Daniel Golle
3a67c5b662 Revert "build: align SOURCE path for build system and SDK"
This reverts commit 131e41614d.
Sadly it makes menuconfig fail with
tmp/.config-package.in:171: glob failed: No files found "feeds/base/utils/busybox/Config.in"
make: *** [/usr/src/openwrt/include/toplevel.mk:136: menuconfig] Error 1

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2024-02-12 19:03:17 +00:00
Paul Spooren
131e41614d build: align SOURCE path for build system and SDK
Building a package in the build system or the SDK results in different
values for the `SOURCE` property, it's either `packages/<package name>`
or `feeds/base/<package name>`. The reason is that the SDK handles
`openwrt.git` as an external feed called while the build system contains
the *base* packages directly.

Since packages created with either method are (ideally) the same (bit
for bit), align the content of SOURCE. To do so this commit creates a
symlink from `feeds/base` to `$(TOPDIR)/package` and adopts the SOURCE
when building from inside the build system.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2024-02-12 17:06:41 +01:00
a-gave
84ec8c41e7
imagebuilder: copy from buildroot only target/linux/generic and target/linux/<target> to reduce the size
This reduces the size of a single imagebuilder by about 40MB
In example for the target ath79 it would be the sum of generic and <target> directories, so about 16MB,
instead of the whole size of the target directory, about 53MB:
11M	target/linux/generic/
3.9M	target/linux/ath79/

Signed-off-by: a-gave <agave@dracaena.it>
2024-01-15 16:49:12 +01:00
Tomasz Maciej Nowak
e40b9a7fa0 ib: split out processing user provided packages
Some device recipes remove default target packages. If user tries to add
them back they will be ignored, since packages list is processed in one
go. Process the device recipe packages first and do user ones later, so
additions won't get filtered out.

Signed-off-by: Tomasz Maciej Nowak <tmn505@gmail.com>
2023-07-15 17:02:42 +02:00
Michael Pratt
d87a8aa148
treewide: add ORIG_PATH variable
Add a variable that stores the original value of $PATH
in the host system's shell, before Make alters it.

This can be useful for when it is necessary
to ignore symlinks and programs made by the build system.

Define this new variable before all instances of
'export PATH:=' or similar.

Signed-off-by: Michael Pratt <mcpratt@pm.me>
2023-06-05 08:31:47 +02:00
Paul Spooren
7b7edd25a5 imagebuilder: allow to specific ROOTFS_PARTSIZE
Setting this options modifies the rootfs size of created images. When
installing a large number of packages it may become necessary to
increase the size to have enough storage.

This option is only useful for supported devices, i.e. with an attached
SD Card or installed on a hard drive.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2023-03-14 19:00:47 +01:00
Florian Eckert
744a5ab9ce target/imagebuilder: add help text and rename whatdepends to package_whatdepens
Add the command `package_whatdepends` to show all package that have a
dependency to this packages.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
2023-01-06 18:44:45 +01:00
Florian Eckert
c75a565d2c target/imagebuilder: update help text for manifest target
- Add a help text for the manifest command.
- Unify command detail help text to use `manifest`.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
2023-01-06 18:44:45 +01:00
Florian Eckert
e1cf4688ec target/imagebuilder: change help text for image build
Using command `image` to unify help text.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
2023-01-06 18:44:44 +01:00
Florian Eckert
7b60dba954 target/imagebuilder: add depends target to show all package that gets installed
This commits adds the makefile targets `depends` this wrapper is a call
to `opkg depends`. This command shows which runtime dependencies exist
if this package is installed into the image.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>
2023-01-06 18:44:44 +01:00
Daniel Golle
5cf5dce05a
imagebuilder: export SOURCE_DATE_EPOCH to environment
Export SOURCE_DATE_EPOCH to environment so filesystem and image
creation tools will make use of it.
Fixes reproducibility of images generated with the ImageBuilder.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2022-04-10 00:56:38 +01:00
Petr Štetiar
621f39d1f4 imagebuilder: fix broken image generation with external targets
When using external targets there is a symlink being created for the
target under target/linux which then becomes dangling under Image
Builder. Fix it by dereferencing the possible symlink.

Tested on IB with external target, ipq40xx and mvebu.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2022-03-25 14:28:00 +01:00
Paul Spooren
15e55a2190 imagebuilder: fix local packages/ folder
This commit fixes commit "2999f810ff: build,IB: include kmods only in
local builds" which cause the local packages/ folder only to be added
for local builds but no longer for ImageBuilder created by the Buildbot.

The commits intention was to use remote kmods repositories rather than
storing them locally. Accidentally the entire handling of the local
`packages/` was removed.

Re-add the folder and include a README describing what it can be used
for.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-11-05 19:18:48 -10:00
Paul Spooren
a7fdd4de59 imagebuilder: show architecture in make info output
Using `make info` show the current target, revision, default packages
and available profiles. This commits adds the used architecture.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-09-23 21:37:19 -10:00
Sven Roederer
6967903b01 imagebuilder: unset BINARY_FOLDER and DOWNLOAD_FOLDER in final archive
Using these config-options to customize the folders used at build-time
makes these folder settings appear in generated archive. This causes the
imagebuilder to be not portable, as it's going to use the build-time folders
on the new systems. Errors look like:

  mkdir: cannot create directory '/mnt/build': Permission denied
  Makefile:116: recipe for target '_call_image' failed
  make[2]: *** [_call_image] Error 1
  Makefile:241: recipe for target 'image' failed
  make[1]: *** [image] Error 2

The build-time settings of these folders are passed into the archives via
.config file.
The expected behavior is that after unpacking the imagebuilder acts like
these settings have their defaults, using intree folders. So unset the
build-time settings.

Signed-off-by: Sven Roederer <devel-sven@geroedel.de>
2021-05-08 12:14:00 +02:00
Paul Spooren
0f7cd97f81 build,ib: add STRIP_ABI option for manifest
The ImageBuilder `make manifest` prints all installed packages. This
function can be used to create a list of package and corresponding
package versions before attempting image creation.

When called with `--strip-abi` OPKG can automatically strip attached
ABIVersions from package names. Make this function accessible for the
ImageBuilder by adding a `STRIP_ABI` variable.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-16 11:00:08 -10:00
Jo-Philipp Wich
6a46615f37 imagebuilder: invoke bundle-libraries.sh w/o buildroot dirs in $PATH
Invoke bundle-libraries.sh with any buildroot related directory entries
removed from $PATH to avoid picking up cross versions of utilities like
ldd which will not properly work when used against host executables.

This should fix executable bundling for glibc-target imagebuilders.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2020-12-29 01:07:42 +01:00
Matthew Gyurgyik
aab36200e7 imagebuilder: fix partition signature
When building images with the imagebuilder, the partition signature
never changes. The signature is generated by hashing SOURCE_DATE_EPOCH
and LINUX_VERMAGIC which are undefined. Prepopulate these variables, as
done by the SDK.

Signed-off-by: Matthew Gyurgyik <matthew@gyurgyik.io>
2020-11-24 14:46:05 -10:00
Paulo Machado
b19a684f46 imagebuilder: fix main entry makefile
Remove a syntax error from ImageBuider Makefile

Acked-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Paulo Machado <pffmachado@yahoo.com>
2020-11-23 03:13:46 +00:00
Paul Spooren
418362b1cc imagebuilder: add package signature verification
The ImageBuilder downloads pre-built packages and adds them to images.
This process uses `opkg` which has the capability to verify package list
signatures via `usign`, as enabled per default on running OpenWrt
devices.

Until now this was disabled for ImageBuilders because neither the `opkg`
keys nor the `opkg-add` script was present during first packagelist
update.

To harden the ImageBuilder against *drive-by-download-attacks* both keys
and verification script are added to the ImageBuilder allowing `opkg` to
verify downloaded package indices.

This commit adds `opkg-add` to the ImageBuilder scripts folder. The keys
folder is added to ImageBuilder $TOPDIR to have an obvious place for users to
store their own keys. The `option check_signature` is appended to the
repositories.conf file. All of the above only happens if the Buildbot
runs with the SIGNATURE_CHECK option.

The keys stored in the ImageBuilder keys/ are the same as included in
the openwrt-keyring package. To avoid the chicken-egg problem of
downloading and verifying a package, containing signing keys, the keys
are added during the ImageBuilder generation. They are same as in
shipped images (stored at `/etc/opkg/keys/`).

To allow a local package feed in which the user can add additional
packages, a local set of `usign` and `ucert` keys is generated, same as
building OpenWrt from source. The private key signs the local repository
inside the packages/ folder. The local public key is added to the keys/
folder to be considered by `opkg` when updating repositories. This way a
local package feed can be modified while requiring `opkg` to check
signatures for remote feed, making HTTPS optional.

The new option `ADD_LOCAL_KEY` allows to add the local key inside the
created images, adding the advantage that sysupgrades can validate the
ImageBuilders local key.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-19 22:15:00 +00:00
Paul Spooren
2e282537d0 imagebuilder: fix sstrip
Without an absolute path to staging_dir/host/bin/sstrip the Makefile
tries to run a host installed version of sstrip, which is likely not
available.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-09 10:54:30 +00:00
Paul Spooren
04757f964b build,IB: reload packages/ only if existing
With the fix of external kmod feeds it is possible to ship the
ImageBuilder without any packages except the pseudo packages kernel and
libc. Therefore the local package feeds becomes optional.

This commit adds a check to the package_reload function to only run if
the local feed is existing.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-10-30 00:39:09 +00:00
Paul Spooren
2999f810ff build,IB: include kmods only in local builds
The buildbots generate a kmod archive which should be used instead of a
local copy. This is possible due to the introduction of a kernelversion
specific feed.

This commit adds the ability of using only signed package feeds.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-10-30 00:39:09 +00:00
Paul Spooren
29fd93da14 imagebuilder: add missing libfakeroot files
The `libfakeroot` files are currently missing in the ImageBuilder. As
`fakeroot` is always built, copy those files unconditionally.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-09-29 00:06:32 +01:00
Paul Spooren
84024e245f build: add whatdepends target to imagebuilder
The package manager `opkg` offers the function `whatdepends` to print
packages that depend on a specific package.

This feature is useful when used in a CI to not only build an upgraded
package but all packages with a dependency.

Usage:
    make whatdepends PACKAGE=libipset

The resulting list can be fed into a SDK building all packages and warn
if anything fails.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-31 11:18:24 +01:00
Paul Spooren
8c9a788172
build: SDK/IB reproducible and faster compression
Both IB and SDK now use the same logic for packing.

This commit add reproducible multithread compression to the SDK and
corrects the file mtime for both. Previously all files where just copied
over from the build system, generating random mtimes.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-23 18:55:49 +02:00
Paul Spooren
941ec28b35 imagebuilder: Remove json_info_files/ before build
The folder `json_info_files` contains multiple JSON files which describe
created firmware images. The folder is not removed between builds as the
ImageBuilder does not use `image.mk`.

Not removing the JSON files result in a merged `profiles.json` file
containing entries for outdated or non-existing images.

This commit adds the `json_info_files/` cleanup step to the ImageBuilder
Makefile.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-11 11:12:56 +01:00
Paul Spooren
4f38063640 imagebuilder: pass IB=1 on checking requirements
The patch 4a1a58a3  build, imagebuilder: Do not require libncurses-dev
was supposed to remove libncurses as a requirement for the ImageBuilder.
However as the IB=1 is only exported during building, not for checking
requirements, it did never actually work.

This commit export IB=1 to the requirement check.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-11 10:49:18 +01:00
Paul Spooren
07449f692c build: refactor JSON info files to profiles.json
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").

They are useful for firmware wizards and script checking for
reproducibility.

Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.

This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.

Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.

Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.

As before, this creation is enabled by default only if `BUILDBOT` is set.

Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-04-03 12:17:45 +02:00
Martin Schiller
b204fcdb07 target/imagebuilder: use multi-thread support for xz compression
This speeds up the packing of the imagebuilder a lot:

 imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
 imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
 imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
 imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
 imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
 imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
 imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
 imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
 imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
 imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s

To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.

Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-10-09 09:13:44 +02:00
Paul Spooren
07926d7def imagebuilder: fix make info for empty SUPPORTED_DEVICES
For x86/64 (maybe more) target the SUPPORTED_DEVICES variable is empty
which causes the `&&` junction to fail, producing a non zero exit code.

Tested-by: Paul Spooren <mail@aparcar.org>
Fixed-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Paul Spooren <mail@aparcar.org>
2019-08-14 08:14:52 +02:00
Richard Musil
71ab2c9d17 imagebuilder: new DISABLED_SERVICES make variable
Adds a new variable DISABLED_SERVICES to ImageBuilder Makefile, which
defines a list of services (installed as /etc/init.d/*) to be disabled
during the build of a custom image (normally all are enabled).

It comes handy when a particular service should not be run under normal
circumstances, but should be ready in the image for situations when it
might be needed.

Signed-off-by: Richard Musil <risa2000x@gmail.com>
2019-05-15 13:34:24 +02:00
Daniel Golle
d6fa04a437 IB: include SUPPORTED_DEVICES in 'make info' output
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2019-03-11 05:32:29 +01:00
Paul Spooren
483498808e ib: show current revision based on $(REVISION)
This is useful in for the attendedsyupsgrade server (asu) to
distinguish between snapshot version. Currently asu can't tell devices
requesting a snapshot build if the same build is already installed.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2019-03-08 01:18:12 +01:00
Paul Spooren
f37afec866 ib: show unified target based on $(TARGETID)
Instead of showing a slightly more readable target like
"ar71xx (Generic)" print the more generic format "ar71xx/genric"

Signed-off-by: Paul Spooren <mail@aparcar.org>
2019-03-08 01:18:12 +01:00
Daniel Golle
13c379e5c6 ib: display whether profile comes with image metadata
Having image metadata (and signature) appended is a condition for
semi-automated sysupgrade, hence IB needs to be able to tell which
images will end up with metadata.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2019-03-06 14:50:42 +01:00
Paul Spooren
ad5c2897ec imagebuilder: manifest function show stderr
This really simplifies debugging, if a package is not found or a feed is
not reachable, a proper stderr is printed. Currently it would only say
`_call_manifest` failed.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2018-08-06 06:54:42 +02:00
Paul Spooren
869b0d11db imagebuilder: add function to show manifest
Tested with 18.06.0-rc2/ar71xx/generic/tl-wdr4300-v1, image & list

This PR is based on the work of @fewckert[1] with slight improvements.

Add function `manifest` to show the manifest of the produced image,
before actually building it. The manifest contains an orderd list of
package name and version.

This is usefull to check package dependencies but also determine a
unique and reproducible image name before building the package. The
sysupgrade server[2] builds images on request with individual package
selection. To distignish between created images which contain differnt
packages, the EXTRA_IMAGE_NAME is set to a shortend hash of the
manifest's content. So far the image was renamed afterwards as the
manifests content was unknown, however this corrupts the signed
sha256sums. This patch allows a clean solution as to dtermine the
manifest in advance and set the EXTRA_IMAGE_NAME accordingly.

[1]: https://github.com/lede-project/source/pull/1591
[2]: https://github.com/aparcar/attendedsysupgrade-server

Signed-off-by: Paul Spooren <mail@aparcar.org>
2018-07-30 15:55:21 +02:00
Matthias Schiffer
a02a69d5f8
imagebuilder: remove split patch dirs from imagebuilder archive
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2018-03-07 09:59:08 +01:00
Matthias Schiffer
2fbf669730
imagebuilder: reuse rootfs preparation from rootfs.mk
In addition to removing redundant code, this fixes various issues in
IB-generated images that have been fixed in prepare_rootfs before,
including better handling of CONFIG_CLEAN_IPKG and enabling of initscripts
from FILES.

We also reuse the opkg macro and remove --force-... flags that have been
removed from rootfs.mk as well.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2018-03-07 09:59:08 +01:00
Jo-Philipp Wich
fd30187c87 imagebuilder: fix reference to removed VERSION_SED variable
Fixes: ff8e9a4ecb ("treewide: combine VERSION_SED and VERSION_SED_SCRIPT")
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2018-02-02 14:41:35 +01:00
Zoltan HERPAI
d2c06eb075 merge: etc: update remaining files
Signed-off-by: Zoltan HERPAI <wigyori@uid0.hu>
2017-12-08 19:41:18 +01:00
Zoltan HERPAI
7b5c989ab9 merge: targets: update image generation and targets
Signed-off-by: Zoltan HERPAI <wigyori@uid0.hu>
2017-12-08 19:41:18 +01:00
Daniel Golle
d80d1b6c42 imagebuilder: don't rewrite package list output
No longer rewrite opkg list output in package_list function, remove
the awk call in the pipe (which was intended for a single specific
use-case).

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2017-06-17 01:24:43 +02:00
Daniel Golle
1b555e1d2b imagebuilder: clean package_list
commit 19ac879954 (imagebuilder: add package_list function) introduced
a new function 'package_list' to the imagebuilder Makefile.
Unfortunately the package list was poluted by stdout noise of the
Makefile itself as well as opkg. Redirect those outputs to stderr to
make sure that the package_list returned doesn't contain progress
info output but really only packages.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2017-06-17 00:54:46 +02:00
Paul Spooren
19ac879954 imagebuilder: add package_list function
The imagebuilder can now list all available packages by using make
package_list. This is usefull for scripts to retrieve a list of all
packages with versions (and size)

Signed-off-by: Paul Spooren <paul@spooren.de>
[daniel@makrotopia.org: fixed commit message]
2017-06-15 00:36:08 +02:00