Commit Graph

73 Commits

Author SHA1 Message Date
David Bauer
9a9cf40dd9 download: add mirror alias for Debian
Add an alias for Debian packages and download them from the Debian
mirror redirector.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-26 20:41:00 +01:00
Paul Spooren
8286f3a3d3 treewide: unify OpenWrt hosted source via @OPENWRT
Multiple sources are hosted on OpenWrts source server only. The source
URLs to point to the server vary based on different epochs in OpenWrts
history.

Replace all by @OPENWRT which is an "empty" mirror, therefore using the
fallback servers sources.cdn.openwrt.org and sources.openwrt.org.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-02-05 12:00:24 -10:00
Paul Spooren
a17b8eaa2e build: use SPDX license tags
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2021-02-05 14:54:47 +01:00
Felix Fietkau
14bad2001f build: prevent excessive re-evaluation of PKG_VERSION
version_abbrev uses $(shell) and the ?= is causing make to run the command
over and over again, causing a significant build slowdown

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2020-10-16 21:39:10 +02:00
Matthias Schiffer
4696112ea2
build: add zstd support to pack/unpack functions
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2020-05-31 11:03:30 +02:00
Kamil Wcislo
09c428ec6b
build: add PKG_SOURCE_URL_FILE support
It seems that there is a missing PKG_SOURCE_URL_FILE support.
This little fix adds the support for packages to change the name of the
downloaded file.

Sometimes it is desirable to change the downloaded archive file name, like
for mitigating name conflicts for different packages (some files on the server
could be named like, e.g. 2018-01-01.tar.gz) or for the cases that there is
no name for the file in the URL (e.g. http://someserver.com/download).

Signed-off-by: Kamil Wcislo <kamil.wcislo@lpnplant.io>
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2019-11-23 13:35:43 +01:00
Felix Fietkau
7ec092e641 Revert faulty tree push
Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85)
Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e)
Revert "ramips: add preliminary support for WIO ONE" (085141dc5b)
Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539)
Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d)
Revert "toolchain: add autoconf-lean" (fdb30eed03)
Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758)

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2019-08-12 12:27:06 +02:00
Felix Fietkau
6fa0e07758 build: allow overriding the filename on the remote server when downloading
Github releases usually don't contain the project name in the release
filename, which makes them very inconvenient to use from the build
system. Add support for naming the local file differently.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2019-08-12 11:43:38 +02:00
Eneas U de Queiroz
8252511dc0 build: remove sgid permission from tar
Otherwise tar will keep the sgid bit when running
from a sgid-set directory, resulting in a different
file being generated.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
[reworked commit message, removed DMARC]
2019-04-06 19:14:06 +02:00
Yousong Zhou
9009efa18b download.mk: enable DownloadMethod/github_archive
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
04b9f85873 scripts/dl_github_archive.py: rename from download.py
- Make the code more GitHub-specific
 - Requires mirror hash to work with .gitattributes
 - Use different API depending on whether PKG_SOURCE_VERSION is a
   complete commit id or other ref types like tags
 - Fix removing symbolic link
 - pre-clean dir_untar for possible leftovers from previous run

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
e48ea13b3b download.mk: add more comments
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
e15565a01c download.mk: restore the old dl_method implementation
Seems like the python download.py dl_method call causes serious
performance regression for fresh "make defconfig" as reported in
FS#1621.  GitHub tarball download will also be disabled with this

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-06-29 13:54:17 +08:00
Yousong Zhou
75ab064d2b build: download code from github using archive API
A new python script scripts/download.py is added to fetch tarballs using
GitHub archive API [1], then repack in a reproducible way same as the
current DownloadMethod/git

GitHub imposes a 60 reqs/hour rate limit on unauthenticated API
access[2].  This affects fetching commit date for feeding tar --mtime=
argument.  However, observation indicates that archive download is NOT
subject to this limit at the moment.  In the rare cases where download
fails because of this, we will falback to using DownloadMethod/git

The missing piece in the GitHub API is that it cannot provide in the
tarball dependent submodules's source code.  In that case, the
implementation will also fallback to using DownloadMethod/git

 [1] Get archive link, https://developer.github.com/v3/repos/contents/#get-archive-link
 [2] Rate limiting, https://developer.github.com/v3/#rate-limiting

v2 <- v1:

 - allow passing multiple urls with --urls argument
 - add commit ts cache.  can be helpful on retry

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-06-27 10:51:27 +08:00
Jo-Philipp Wich
4700544e40 downloads.mk: introduce name-agnostic PROJECT_GIT variable
Introduce a name-agnostic PROJECT_GIT variable poiting to
https://git.openwrt.org/ and declare LEDE_GIT and OPENWRT_GIT
as aliases to it.

After some transition time we can drop this alias variables.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2018-01-10 21:27:31 +01:00
Philip Prindeville
76ba01a392 build: remove @ as it's causing an error
Since $(DownloadMethod/unknown) is being invoked in the expansion of
$(call locked ...) anyway, you can't have an @ because the shell
doesn't know what to do with it.

Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
2017-09-20 08:50:49 +02:00
Felix Fietkau
5919cc2dc4 build: let make check warn about use of legacy PKG_MD5SUM variable in feeds
The variable rename change has been merged in OpenWrt now, so it's
possible to convert the feeds as well.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-13 10:23:43 +01:00
Felix Fietkau
84bd74057f build: use mkhash to replace various quirky md5sum/openssl calls
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-05 11:09:12 +01:00
Felix Fietkau
ffb0181a87 build: add defaults for PKG_SOURCE, PKG_SOURCE_SUBDIR, PKG_VERSION
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-22 16:42:20 +01:00
Felix Fietkau
d49059693b build: add FIXUP option for make check
This will attempt to automatically fix common mistakes like using MD5
instead of SHA256, using the MD5SUM variable instead of HASH, or even a
missing mirror file hash.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-17 10:36:45 +01:00
Felix Fietkau
7a315b0b5d build: implement make check and make package/X/check
This is intended to be used for a wide array of package sanity checks.

The first check that is implemented is for the hash of downloaded files.
It checks:
  - Missing hash
  - Use of SHA256 instead of MD5
  - dl/<file> hash not matching hash in makefile
  - deprecated MD5SUM variable

The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-17 10:36:25 +01:00
Felix Fietkau
6a5cc2d085 include/package.mk: sync default value for hash fallback with mirror hash
Simplifies dealing with PKG_HASH vs PKG_MD5SUM

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 15:40:07 +01:00
Felix Fietkau
881c5b47ec build: remove duplicate Download/default definition from include/host-build.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 15:35:21 +01:00
Felix Fietkau
7416d2e046 build: replace MD5SUM variables with HASH
Since we've switched to preferring SHA256 over MD5, the old variable
name is misleading. Packages using the old name remain compatible.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 13:49:24 +01:00
Felix Fietkau
1b9a39c528 download.mk: improve download tarball reproducibility
Store only numeric user/group id. Group 0 is 'root' on linux and 'wheel' on
BSD and Mac OS X.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:26 +01:00
Felix Fietkau
19d3b78304 download.mk: remove code duplication in $(TAR) call
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:26 +01:00
Felix Fietkau
dbbfd41118 download.mk: use $(error) instead of a regular shell error
Useful for further rework

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:25 +01:00
Florian Fainelli
b9aab34eb4 include/download.mk: Allow specify DownloadMethod specific options
This is going to be used to migrate the hand rolled git clone for the kernel
into using the git download method. The kernel uses custom options that we may
have to pass down.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
2016-12-04 11:41:51 +01:00
Daniel Engberg
cbc80805bd include/download.mk: Use -7e compression instead of -6 by default
Adds a slightly higher compression level to xz by default which roughly raises memory usage from 100MiB to about 200MiB during compression, about 10MiB for decompression. (Source: xz manpage)

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
2016-10-05 15:15:19 +02:00
Jo-Philipp Wich
5d86dc791e include/download.mk: generate reproducable SCM tarballs
Apply a number of changes to the tarball generation in order to produce
identical files on different systems:

1) Use an explicit `gzip -cn` to avoid storing file mtime in the gzip header
2) Instruct `tar` to unconditionally use uid and gid 0 for archive members
3) Instruct `tar` to sort archive members by file name
4) For SCMs that do not preserve file modification times like Git or Mercurial,
   use the date of the last commit to the repository and pass it as `--mtime`
   value to `tar`

After these changes, locally produced tarballs generated from SCM checkouts
should be identical on any system, simplifying the mirroring of cache archives.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-10-05 00:45:59 +02:00
Jo-Philipp Wich
3a085dcbe3 download.mk: use HTTPS for git.lede-project.org
Some Git versions have issues following the HTTP->HTTPS redirect and since
the keyring package is fetched from this host, switching to HTTPS is a
sensible choice anyway.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-06-01 15:18:42 +02:00
John Crispin
f5c4d963ff include: add lede git server url
Signed-off-by: John Crispin <john@phrozen.org>
2016-04-29 04:15:24 +02:00
Álvaro Fernández Rojas
694f060550 download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw files from github repos.

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2016-04-03 22:02:46 +02:00
John Crispin
5c408ede6c download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw github repository files

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>

SVN-Revision: 49138
2016-04-09 10:25:34 +00:00
Felix Fietkau
59121fdd69 buildroot: improve git submodule handling for packages
Move the `--recursive` switch from `git clone` to `git submodule`
so that submodules are cloned for upstream branches where the
PKG_SOURCE_VERSION commit-ish has a different .gitmodules
configuration than the repository default.

This is, for example, required when the master branch for a source
package does not use submodules, but its topic branch for OpenWRT
does.

This changes the buildroot dependency from git-1.6.2 to git 1.7.12.2,
which was released September 2012.

Signed-off-by: Darik Horn <dajhorn@vanadac.com>
Signed-off-by: Karl Palsson <karlp@tweak.net.au>

SVN-Revision: 48830
2016-02-29 20:12:25 +00:00
Jo-Philipp Wich
ac5b226735 download.mk: fix packed checkout mirroring support
Changeset r48416  broke the downloading of mirrored, packed scm checkouts.

Fix this by removing the "@" sign in front of the download command which is
now executed as part of a larger shell command under flock.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 48733
2016-02-17 08:44:37 +00:00
Felix Fietkau
c5db626ac5 build: download.mk: bugfix download.pl argument call order
Change wrong download.pl argument call order.
See download.pl argument list.

Signed-off-by: Florian Eckert <Eckert.Florian@googlemail.com>

SVN-Revision: 48427
2016-01-21 13:28:18 +00:00
Felix Fietkau
a81ad971df build: add locking for downloads (fixes race conditions with multiple variants)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48416
2016-01-20 21:31:50 +00:00
Felix Fietkau
883b5b8191 build: add @APACHE download facility
The Apache Software Foundation offers diverse download mirros.

For packaging Apache software a new alias @APACHE is defined.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>

SVN-Revision: 48270
2016-01-17 10:47:32 +00:00
Felix Fietkau
11b7bc593c build: add a variable pointing to the main openwrt git repositories (useful if we want to support using a mirror later)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48117
2016-01-04 15:11:43 +00:00
Felix Fietkau
9b68fe961b build: add support for choosing a different url filename part than the output file
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 47591
2015-11-22 19:06:33 +00:00
John Crispin
b0a1c7dc0d include/download.mk: Add download mirrors for tools from GNU Savannah (bug #15184)
I defined a new download method @SAVANNAH in include/download.mk and scripts/download.pl,
and converted quilt and qemu to use that method.

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

SVN-Revision: 42840
2014-10-08 08:01:39 +00:00
Felix Fietkau
d1969d761f build: add support for packing version control downloads with xz
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 40446
2014-04-10 14:43:26 +00:00
Jo-Philipp Wich
97fc11af79 Incorrect checkout of git submodules
When checking out git packages, buildroot doesn't seem to track the revisions
correctly of any submodules referenced by that project. As a result, the
submodule stays at whatever revision was referenced by the head of the master
branch. Running a 'git submodule update' after the checkout fixes this problem.

Signed-off-by: Owen Kirby <osk@exegin.com>

SVN-Revision: 38359
2013-10-10 15:01:50 +00:00
Jo-Philipp Wich
5d253542d7 include: download.mk: correct tar argument order when packing checkouts
GNU TAR v1.26 will fail with "no such file or directory" errors when the
output file does not immediately follow the -f switch.

SVN-Revision: 34460
2012-12-02 18:35:47 +00:00
Felix Fietkau
e8ff034c8b download.mk: refuse to download mirror tarballs from packages using version control if no mirror md5sum was provided
SVN-Revision: 32079
2012-06-06 13:45:53 +00:00
Jens Muecke
6204553450 Enable recursive download of git sources.
SVN-Revision: 30967
2012-03-17 21:33:13 +00:00
Felix Fietkau
b568a64f8c add support for md5sum checks for mirrored tarballs of packages with version control source urls
SVN-Revision: 26260
2011-03-21 23:02:06 +00:00
Felix Fietkau
fd0deca65d download.mk: properly quote download urls
SVN-Revision: 26129
2011-03-13 18:38:43 +00:00
Jo-Philipp Wich
8fae8c0a6f download.mk: support https:// downloads
SVN-Revision: 24688
2010-12-18 21:43:10 +00:00