Commit Graph

60 Commits

Author SHA1 Message Date
Paul Spooren
8c9a788172
build: SDK/IB reproducible and faster compression
Both IB and SDK now use the same logic for packing.

This commit add reproducible multithread compression to the SDK and
corrects the file mtime for both. Previously all files where just copied
over from the build system, generating random mtimes.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-08-23 18:55:49 +02:00
Martin Schiller
b204fcdb07 target/imagebuilder: use multi-thread support for xz compression
This speeds up the packing of the imagebuilder a lot:

 imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
 imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
 imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
 imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
 imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
 imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
 imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
 imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
 imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
 imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s

To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.

Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-10-09 09:13:44 +02:00
Matthias Schiffer
a02a69d5f8
imagebuilder: remove split patch dirs from imagebuilder archive
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2018-03-07 09:59:08 +01:00
Jo-Philipp Wich
fd30187c87 imagebuilder: fix reference to removed VERSION_SED variable
Fixes: ff8e9a4ecb ("treewide: combine VERSION_SED and VERSION_SED_SCRIPT")
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2018-02-02 14:41:35 +01:00
Jo-Philipp Wich
37e7a1734f imagebuilder: fix bundling of DTS sources
Refer to LINUX_KARCH instead of ARCH when bundling DTS files in the image
builder tarball.

While we're at it, also dereference symbolic links when copying as some
kernel architectures contain symbolic links in their DTS directories.

This fixes aarch64 imagebuilders such as brcm2708/bcm2710 ones in particular
as the kernel refers to "aarch64" as "arm64" internally.

Ref: https://forum.lede-project.org/t/lede-image-builder-problem/3680

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2017-05-14 21:43:06 +02:00
Felix Fietkau
9467ce42da build: get rid of host.mk
Defined required host related variables in toplevel.mk instead

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-02-26 13:31:44 +01:00
Jo-Philipp Wich
7d57db4d9b build: introduce STAGING_DIR_IMAGE
Introduce a new location STAGING_DIR_IMAGE which is intended to be used by
bootloader iamges and similar image-related artifacts.

This directory is guaranteed to be persistent across kernel upgrades which
might involve a removal of KERNEL_BUILD_DIR and is guranteed to be bundled
with the image builder.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2017-01-27 16:53:31 +01:00
Felix Fietkau
619c8fa922 imagebuilder: remove existing debug kernel image
Reduces tarball size and improves build time

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-13 19:46:45 +01:00
Felix Fietkau
d1514e8f84 imagebuilder: remove existing root filesystem images
Reduces tarball size and improves build time

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-13 19:46:45 +01:00
Jo-Philipp Wich
72d751cba9 build: rework library bundling
Rework the bundle-libraries.sh implementation to use a more robust approach
for executing host binaries through the shipped ELF loader and libraries.

The previous approach relied on symlinks pointing to a wrapper script which
caused various issues, especially with multicall binaries as the original
argv[0] name was not preserved through the ld.so invocation. Another down-
side was the fact that the actual binaries got moved into another directory
which caused executables to fail looking up resources with paths relative
to the executable location.

The new library wrapper implements the following improvements:

 - Instead of symlinks pointing to a common wrapper, each ELF executable
   is now replaced by a unqiue shell script which retains the original
   program name getting called

 - Instead of letting ld.so invoke the ELF executable directly, launch
   the final ELF binary through a helper program which fixes up the argv[0]
   argument for the target program

 - Support sharing a common location for the bundled libraries instead of
   having one copy in each directory containing wrapped binaries

Finally modify the SDK build to wrap the staging_dir and toolchain binaries
which allows to use the SDK on systems with a different glibc version.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2017-01-10 12:27:28 +01:00
Daniel Engberg
636a069c42 target/imagebuilder: Switch to xz compression instead of bz2
Switch to xz compression instead of using bz2.
Saves about 20% of total size (ar71xx)

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
2016-10-15 11:36:51 +02:00
Daniel Dickinson
71370d2c55 target/{sdk,imagebuild}: Fix for symlink-tree
With symlink tree some directories are just symlinked which
means IB and SDK end up with a symlink instead of an actual
directory; this fixes the missing files by dereferencesing
the directories instead of copying the symlinks.

Signed-off-by: Daniel Dickinson <lede@cshore.thecshore.com>
2016-09-30 00:39:25 +02:00
Alexander Couzens
2e980479c1 IB/SDK/toolchain: use lower cases filenames
Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
2016-06-01 17:54:36 +02:00
Alexander Couzens
4a7c653400 IB/SDK/toolchain: use VERSION_DIST_SANITIZED instead of VERSION_DIST
VERSION_DIST can contains spaces which produces problems when used as file name

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
2016-06-01 17:54:36 +02:00
Jo-Philipp Wich
b9466382b5 imagebuilder: use correct package directory when bundling kmods and libc
The libc and kernel package files moved since the introduction of shared
packages and the changed output directory layout. This causes the generated
ImageBuilder archive to lack the necessary "libc" and "kernel" meta packages,
leading to opkg install errors later on.

Use the FeedPackageDir macro to figure out the proper source directory to use.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-04-22 12:22:57 +02:00
Jo-Philipp Wich
da46d2b228 imagebuilder: fix standalone operation
Fix standalone ImageBuilders after the package layout rework.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-04-13 01:46:09 +02:00
Felix Fietkau
d69efa4e15 targets: Use configured distribution name for SDK, IB, and Toolchain tarball names
We allow to configure the version distribution name;
let's also use it for the tarballs (SDK, ImageBuilder,
and SDK).

Signed-off-by: Daniel Dickinson <openwrt@daniel.thecshore.com>

SVN-Revision: 48081
2016-01-03 14:44:42 +00:00
Jo-Philipp Wich
3c5357d6ef feeds: use common macro "FeedSourcesAppend" to populate opkg configurations
This introduces a common macro to assemble the correct url templates to
avoid code duplication and have the feed config handling in a central place.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 45799
2015-05-28 09:47:37 +00:00
Jo-Philipp Wich
6c21bcaa65 IB: use online repositories
Change the IB packaging to only embed libc, kernel and kmod packages by default
and generate repositories.conf to refer to the remote package repositories.

Introduce a new config option CONFIG_IB_STANDALONE which restores the old
behaviour of building self contained IB archives.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 45772
2015-05-27 11:22:20 +00:00
Jo-Philipp Wich
65ec08d1b0 imagebuilder: align filename with SDK
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 44313
2015-02-07 21:01:48 +00:00
Jo-Philipp Wich
188032c8e3 imagebuilder: don't pregenerate package index, strip host binaries
The package index is generated on first use anyway, therefore it makes no
sense to continue shipping it.

Also sstrip the bundled host binaries when packing the IB to save some
additional space.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 44293
2015-02-06 12:31:31 +00:00
Jo-Philipp Wich
2adab34c20 imagebuilder: use FreeBSD compatible tar invocation
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 44288
2015-02-06 00:27:06 +00:00
Jo-Philipp Wich
15d8db1f8c build: add version number to filenames
This commit introduces a new option CONFIG_VERSION_FILENAMES which causes
OpenWrt to embed the version number in generated image files, SDK- and
ImageBuilder archives.

The option is enabled by default if CONFIG_VERSIONOPT is set.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 43869
2015-01-08 11:02:00 +00:00
John Crispin
caa671d873 imagebuilder: fix typo that copied grub to the wrong folder
Signed-off-by: John Crispin <blogic@openwrt.org>

SVN-Revision: 41824
2014-07-24 15:46:12 +00:00
John Crispin
941dab02d5 imagebuilder: copy dts folder
Signed-off-by: John Crispin <blogic@openwrt.org>

SVN-Revision: 41790
2014-07-21 18:42:00 +00:00
John Crispin
68e8614009 imagebuilder: x86 fails to build inside the imagebuilder
Signed-off-by: John Crispin <blogic@openwrt.org>

SVN-Revision: 41580
2014-07-10 22:02:31 +00:00
Felix Fietkau
897b2e4b98 target/imagebuilder: add missing package index (fixes #16675)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 40968
2014-06-02 13:27:31 +00:00
Jo-Philipp Wich
69a6fd0190 imagebuilder: bundle dtc (#12953, #14741)
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 40767
2014-05-14 12:29:36 +00:00
John Crispin
3686b8d279 derive system from hostcc to allow building inside a chroot
SVN-Revision: 33228
2012-08-22 15:07:32 +00:00
Jo-Philipp Wich
a062e08bb4 use bundle-libraries.sh to install required host utilities into imagebuilder staging tree
SVN-Revision: 33196
2012-08-15 13:28:26 +00:00
John Crispin
76e4cd2a24 add $(SUBTARGET) to the filename if available
SVN-Revision: 32007
2012-05-30 18:04:15 +00:00
Jo-Philipp Wich
cfe30549a1 don't overwrite version.mk, substitute REVISION instead
SVN-Revision: 31327
2012-04-17 09:31:16 +00:00
Jo-Philipp Wich
3cb72f38fd populate repositories.conf from version.mk
SVN-Revision: 31268
2012-04-12 23:08:13 +00:00
Jo-Philipp Wich
cdbeb0e80b copy kernel .config, needed by mkfs.jffs2 to detect compression capabilities
SVN-Revision: 23639
2010-10-26 13:34:21 +00:00
Jo-Philipp Wich
ea5904ff93 - sync with host opkg changes - get rid of autogenerated opkg.conf - utilize dl/ as package cache, useful for remote downloads
SVN-Revision: 23039
2010-09-12 22:04:09 +00:00
Jo-Philipp Wich
699876fa6b - add a helper script to generate opkg.conf, attempt to detect package architecture from packages/ - fix package defaults when no package override is given, IB previously aggregated the defautls of all profiles - introduce a repositories.conf, allows using remote opkg repositories in imagebuilder
SVN-Revision: 22978
2010-09-08 00:36:35 +00:00
Jo-Philipp Wich
cdd44e7984 - correctly copy .config when scripts/env is used in the buildroot - prevent user provides PACKAGES from overriding per-profile defaults
SVN-Revision: 21213
2010-04-27 22:16:40 +00:00
Nicolas Thill
8c2b753329 imagebuilder: various fixes * only copy host tools from $(STAGING_DIR_HOST)/bin * remove more target patches, and also target files * combine the 3 find commands into a single one
SVN-Revision: 20597
2010-03-30 11:41:51 +00:00
Gabor Juhos
3ba43e6528 imagebuilder: add REVISION variable (closes #6795)
SVN-Revision: 20064
2010-03-08 17:00:46 +00:00
Felix Fietkau
34939cad39 get rid of $Id$ - it has never helped us and it has broken too many patches ;)
SVN-Revision: 15242
2009-04-17 14:09:46 +00:00
Nicolas Thill
be1a9ff41d fix ImageBuilder (closes: #3863)
SVN-Revision: 12370
2008-08-22 13:45:54 +00:00
Felix Fietkau
bdb9c7351c some more build system cleanup
SVN-Revision: 12329
2008-08-17 11:14:13 +00:00
Felix Fietkau
b81933c951 strip -$(KERNEL) from sdk/imagebuilder names
SVN-Revision: 9742
2007-12-13 22:31:56 +00:00
Felix Fietkau
fa9710c7ee fix imagebuilder compile
SVN-Revision: 9736
2007-12-13 21:04:21 +00:00
Felix Fietkau
93575b53a7 Use $(CP) instead of $(INSTALL_BIN) for binaries.
Signed-off-by: Andy Boyett <agb-openwrt@padded-cell.net>

SVN-Revision: 9694
2007-12-09 18:53:06 +00:00
Felix Fietkau
dfa7618bd6 fix the image builder
SVN-Revision: 9549
2007-11-14 23:11:07 +00:00
Florian Fainelli
43dc71fb0d Fix the imagebuilder generation (does not mean it works with the recent changes)
SVN-Revision: 8710
2007-09-09 15:09:10 +00:00
Florian Fainelli
c54708272f Fix image builder generation (#2213)
SVN-Revision: 8412
2007-08-14 09:47:29 +00:00
Florian Fainelli
163dab630d Fix the imagebuilder compilation (#2213)
SVN-Revision: 8410
2007-08-13 16:30:54 +00:00
Florian Fainelli
e7113073f3 Suppress spaces in uname -m
SVN-Revision: 7594
2007-06-12 16:48:10 +00:00