Commit Graph

83 Commits

Author SHA1 Message Date
Christian Marangi
261925a869
scripts/download.pl: generilize and simplify download tool check
Generilize download tool check and skip other check if a download tool
has been found.
While at it also reintroduce c836ca84e8
that was previously dropped with aria2c support.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-28 16:55:10 +02:00
Christian Marangi
f1b3958d02
scripts/download.pl: fix support for aria2c download tool on macos
Currently we use /dev/shm to place aria2c tmp file. This is not present
on macos. Use the openwrt tmp directory instead of the linux-only
/dev/shm to save compatibility with more os.

Fixes: d391236269 ("download.pl: add aria2c support")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-26 20:50:10 +02:00
Christian Marangi
5df60f5c24
scripts/download.pl: fix mirrors regression for curl and wget
With the introduction of aria2c support, curl and wget no longer try to
download the file from mirrors. Fix this regression by emptying the
remaining mirrors list only when aria2c is used.

Fixes: d391236269 ("download.pl: add aria2c support")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-26 20:50:09 +02:00
Christian Marangi
295e0ed7a9
scripts/download.pl: fix whitespace in mirror urls and drop for
Fix whitespace in mirror urls and replace for loop with join+map logic.

Fixes: d391236269 ("download.pl: add aria2c support")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-25 15:54:12 +02:00
Bradford Zhang
d391236269
download.pl: add aria2c support
Use aria2c download tool by default on package download if available in
the system.
aria2c permits to use multiple mirrors and may improve download speed on
special context where servers are hard to reach.

Co-authored-by: Christian Marangi <ansuelsmth@gmail.com>
Signed-off-by: Bradford Zhang <zyc@zyc.name>
[ fix wrong var in the script and improve commit description ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-25 14:56:45 +02:00
Petr Štetiar
50a48faa1b scripts/download.pl: fix downloads with wget
Several users of wget for downloads (curl is not available in the
system) have reported broken download functionality:

 wget --tries=5 --timeout=20 --output-document=-  https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.142.tar.xz
 http://: Invalid host name.

Thats all happening due to '' was passed as an argument, which got later
expanded to http://.

In the context of a list constructor '' is not nothing, it is an empty
string element.  So fix it by using () as it will yield "nothing" and
thus not introduce an empty string element.

Fixes: #10692
Fixes: 90c6e3aedf ("scripts: always check certificates")
Signed-off-by: Jo-Philipp Wich <jo@mein.io> [shellwords() -> ()]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2022-09-13 09:14:09 +02:00
Petr Štetiar
c836ca84e8 scripts/download.pl: silence can't exec curl warning
When running build in verbose mode `make V=s` we can see a lot of
following warnings when curl is not available in the system:

 Can't exec "curl": No such file or directory at scripts/download.pl line 77.

So lets fix it by redirecting of the stderr to null hole.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2022-09-13 09:14:09 +02:00
Josh Roys
90c6e3aedf
scripts: always check certificates
Remove flags from wget and curl instructing them to ignore bad server
certificates. Although other mechanisms can protect against malicious
modifications of downloads, other vectors of attack may be available
to an adversary.

TLS certificate verification can be disabled by turning oof the
"Enable TLS certificate verification during package download" option
enabled by default in the "Global build settings" in "make menuconfig"

Signed-off-by: Josh Roys <roysjosh@gmail.com>
[ add additional info on how to disable this option ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2022-09-10 15:24:22 +02:00
Rui Salvaterra
6b822189b1 toolchain: binutils: add support for version 2.38
While at it, also fix the download.xs4all.nl and add ftp.gnu.org to the GNU
mirrors, as it seems to be updated faster.

Deleted (upstreamed):
600-Close_the_file_descriptor.patch [1]

[1] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=5a98fb7513b559e20dfebdbaa2a471afda3b4742

Signed-off-by: Rui Salvaterra <rsalvaterra@gmail.com>
2022-03-09 10:23:25 +00:00
Ansuel Smith
88204bfa82 treewide: drop use of which
Ubuntu started to flag which as deprecated and it
seems which is not really standard and may vary
across Distro.
Drop the use of which and use the standard 'command -v'
for this simple task.
Which is still present in the prereq if some package/script
still use which.
A utility script called command_all.sh is implemented that
will just mimic the output of which -a.

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
2022-01-17 09:14:26 +01:00
Leonardo Mörlein
b993b68b6c build: introduce $(MKHASH)
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if

    make TOPDIR="$(pwd)" -C "$pkgdir" compile

was called manually. In most of the cases, I just saw warnings like this:

    make: Entering directory '/home/.../package/gluon-status-page'
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    [...]

While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.

After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.

Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
2021-05-13 15:13:15 +02:00
David Bauer
9a9cf40dd9 download: add mirror alias for Debian
Add an alias for Debian packages and download them from the Debian
mirror redirector.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-26 20:41:00 +01:00
David Bauer
01c01d9861 download: use mirror redirector for GNOME downloads
Use the GNOME mirror redirector as the primary download source for GNOME
packages.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-26 20:41:00 +01:00
David Bauer
f04e007491 download: remove broken mirrors
These mirrors don't exist anymore. Remove them.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-22 00:57:04 +01:00
Paul Spooren
8286f3a3d3 treewide: unify OpenWrt hosted source via @OPENWRT
Multiple sources are hosted on OpenWrts source server only. The source
URLs to point to the server vary based on different epochs in OpenWrts
history.

Replace all by @OPENWRT which is an "empty" mirror, therefore using the
fallback servers sources.cdn.openwrt.org and sources.openwrt.org.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-02-05 12:00:24 -10:00
Paul Spooren
7521aedff4 scripts: sources CDN as fallback in download.pl
In case the default sources for a package fail use the CDN rather than
our own mirror. In case the CDN fails, fallback to our mirror.

Also remove mirror1 which isn't available anymore.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-01-27 22:46:08 -10:00
Petr Štetiar
52a5d0d27f download.pl: properly cleanup intermediate .hash file
It seems like after a build the /dl dir seems to now contain a .hash
file for each source file due to inproper cleanup so fix it by removing
those intermediate files before leaving the download action.

Fixes: 4e19cbc553 ("download: handle possibly invalid local tarballs")
Reported-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-27 22:27:08 +01:00
Petr Štetiar
4e19cbc553 download: handle possibly invalid local tarballs
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.

From now on, we're going to always check the downloaded tarballs before
considering them valid.

Steps to reproduce:

 1. Remove cached tarball

   rm dl/libubox-2020-08-06-9e52171d.tar.xz

 2. Download valid tarball again

   make package/libubox/download

 3. Invalidate the tarball

   sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile

 4. Now compile with corrupt tarball source

   make package/libubox/{clean,compile}

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-27 14:46:13 +01:00
David Bauer
d369993898 scripts: download.pl: retry download using filename
With this commit, the download script will try downloading source files
using the filename instead of the url-filename in case the previous
download attempt using the url-filename failed.

This is required, as the OpenWrt sources mirrors serve files using the
filename files might be renamed to after downloading. If the original
mirror for a file where url-filename and filename do not match goes
down, the download failed prior to this patch.

Further improvement can be done by performing this only for the
OpenWrt sources mirrors.

Signed-off-by: David Bauer <mail@david-bauer.net>
2020-11-20 02:35:43 +01:00
David Bauer
fe82ea049e scripts: download.pl: fix indentation
Signed-off-by: David Bauer <mail@david-bauer.net>
2020-09-11 17:35:19 +02:00
Jo-Philipp Wich
bf96eb55c8 Revert "scripts/download: add sources CDN as first mirror"
This reverts commit c737a9ee6a.

The source CDN has been discontinued in its current form and will take a
while to be reestablished. Even then it makes little sense to put a CDN
before other CDNs such as kernel.org, apache.org, sourceforge etc.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2020-08-24 15:55:14 +02:00
Paul Spooren
c737a9ee6a scripts/download: add sources CDN as first mirror
OpenWrt now has a CDN for sources at sources.cdn.openwrt.org which
mirrors sources.openwrt.org.

Downloading sources outside Europe or US (mainland) could
result in low throughput, extremely slowing down the first compilation of
the build system.

This patch adds sources.cdn.openwrt.org as the first mirror to offer
worldwide fast download speeds by default. If the CDN goes down for
whatever reason, the script jumps to the next available mirror and
downloads requested files as before (in regional varying speed).

Signed-off-by: Paul Spooren <mail@aparcar.org>
Acked-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-04-14 23:16:55 +01:00
Jiri Kastner
dc34c695c4 scripts/dowload.pl: add archive.apache.org to apache mirror list
apache mirrors holds only latest releases, to download
older releases, one must use archive.apache.org to get
them.

Signed-off-by: Jiri Kastner <cz172638@gmail.com>
2019-12-23 00:21:46 +01:00
Yousong Zhou
18dc533172 download.dl: fix @KERNEL mirror urls
- Prepend "/pub" for mirror.rackspace.com
 - Use https for download.xs4all.nl and mirrors.mit.edu

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-09-16 08:11:59 +00:00
Yousong Zhou
0f3667864d download.pl: use https://source.openwrt.org
https://sources.lede-openwrt.org now redirects to there

https://downloads.openwrt.org/sources returns 404, so remove it here

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-08-18 15:19:22 +00:00
Rosen Penev
1ce846580a download.pl: Change OpenWrt mirrors to HTTPS.
These have supported HTTPS for quite a while. I have not seen any obvious breakage.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2018-05-01 11:12:15 +02:00
Rosen Penev
9ea90209f9 download.pl: Change SourceForge address to HTTPS.
SourceForge has supported HTTPS for its downloads for a long time now.

I have not been able to see any failures resulting from this change.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2018-05-01 11:12:15 +02:00
Jo-Philipp Wich
f4b9d9d6f1 base-files: allow skipping of hash verification
When calling a download target, hash verification is now completely
skipped if we set PKG_HASH=skip.

This allows to easily bump package version:

$ make package/<mypackage>/download PKG_HASH=skip V=s
$ make package/<mypackage>/check FIXUP=1 V=s

This will download the new version of the package, and then automatically
update PKG_HASH with the hash of the new version.  Of course, it is still
the responsibility of the packager to ensure that the new tarball is
legitimate, because it is downloaded from a possibly untrusted source.

Fixes: b30ba14e ("scripts/download.pl: fail loudly if provided hash is unsupported")
Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Acked-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: John Crispin <john@phrozen.org>
2017-12-14 09:29:31 +01:00
Baptiste Jonglez
b30ba14e2a scripts/download.pl: fail loudly if provided hash is unsupported
Currently, if the provided hash is unsupported (length different from 32
or 64 bytes), we happily download the requested file without any kind of
checksum verification.

This is quite dangerous and may provide a false sense of security, because
a single typo in the hash (e.g. one character deleted by mistake) may skip
checksum verification entirely.

Instead, fail immediately if we don't support the provided hash.
In particular, if an external package repository decides to change the
hash algorithm one day, we will now fail loudly instead of skipping
checksum verification without complaints.

Note: if some users of scripts/download.pl knowingly provide an empty hash
because they don't need checksum verification, this change will break
them.  This does not seem to be the case currently, but if this feature is
ever needed, an option should be added to download.pl instead of relying
on the hash being empty.

Fixes: eaa4eba10a ("scripts/download.pl: add SHA-256 support")

Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
2017-09-11 17:13:02 +02:00
Zoltan Gyarmati
0a761fe8ef scripts/dowload.pl: use glob to expand target dir
If CONFIG_DOWNLOAD_FOLDER is set to for example "~/dl", the download
script fails to create the .hash and .dl files with the following
errors:
 Cannot create file ~/dl/dropbear-2017.75.tar.bz2.dl: No such file or directory
 sh: 1: cannot create ~/dl/dropbear-2017.75.tar.bz2.hash: Directory nonexistent

If the tarball already exists in the ~/dl dir, it's properly found and
used, so this issue only affects the download.pl script.
 This patch calls glob() on the target dir parameter, which will expand `~`.

Signed-off-by: Zoltan Gyarmati <mr.zoltan.gyarmati@gmail.com>
2017-08-30 18:12:36 +02:00
Daniel Engberg
99c4295126 scripts/download.pl: Adjust URLs
Internet2 isn't considered a trusted issuer meaning that https links to
rit.edu will fail.

The host mirror.csclub.uwaterloo.ca has a trusted SSL cert and peering
is good so it can replace rit.edu without performance issues.

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
[Jo-Philipp Wich: rewrapped commit message]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2017-07-21 08:09:45 +02:00
Felix Fietkau
cbe0a7ecc0 scripts/download.pl: print the command used to download files
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-05-25 19:01:07 +02:00
Felix Fietkau
2fd1503378 scripts/download.pl: fix error message on hash mismatch
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-04-04 12:33:37 +02:00
Felix Fietkau
74c09e714e download.pl: fix detecting download errors with curl
Avoid treating error pages as successfully downloaded content

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-31 11:24:19 +01:00
Felix Fietkau
84bd74057f build: use mkhash to replace various quirky md5sum/openssl calls
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-05 11:09:12 +01:00
Brian J. Murrell
fc6b6f4583 download.pl: use curl in preference to wget
Because wget doesn't know how to do Negotiate authentication with a proxy
and curl does, use curl if it's present. The user is expected to have a
~/.curlrc that sets the options necessary for any proxy authentication.

A ~/.curlrc is completely optional however and curl will work in exactly
the same manner as wget without one.

Signed-off-by: Brian J. Murrell <brian@interlinx.bc.ca>
[Jo-Philipp Wich: Rework code to detect curl usability by checking --version,
                  Use vararg style open() to bypass the shell when downloading,
                  Use Text::ParseWords to decompose env vars into arguments]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2017-01-03 14:26:41 +01:00
Felix Fietkau
651bc94df4 download.pl: check for existing file before the first download attempt
In the build system, flock will prevent multiple concurrent downloads
for the same file. However, if one download request for the same file is
waiting for another one to finish, it will result in downloading the
same file twice consecutively.

Prevent this issue by exiting immediately if the file has already been
downloaded

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 15:16:47 +01:00
Daniel Engberg
8372a7f922 download.pl: Rework URLs
Provide HTTPS URL when possible, try to keep 8 mirrors per entry and spread
over several locations of the world. Since most active contributors are in
US/CA and/or EU prioritize mirrors that are within those regions if possible.

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
2016-05-28 09:14:55 +02:00
Petr Štetiar
e0b241bb48 scripts/download.pl: Use CDN for kernel downloads
More info at https://www.kernel.org/introducing-fastly-cdn.html

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2016-05-23 13:39:50 +02:00
Felix Fietkau
111285e742 download.pl: use http://sources.lede-project.org as download mirror
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-05-13 16:32:35 +02:00
Álvaro Fernández Rojas
694f060550 download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw files from github repos.

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2016-04-03 22:02:46 +02:00
John Crispin
5c408ede6c download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw github repository files

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>

SVN-Revision: 49138
2016-04-09 10:25:34 +00:00
Felix Fietkau
9a94538272 scripts/download.pl: fix sha256 hash command (#21931)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48809
2016-02-27 16:20:06 +00:00
Felix Fietkau
883b5b8191 build: add @APACHE download facility
The Apache Software Foundation offers diverse download mirros.

For packaging Apache software a new alias @APACHE is defined.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>

SVN-Revision: 48270
2016-01-17 10:47:32 +00:00
Felix Fietkau
526f470c5f download.pl: fix openssl sha256 digest output
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48259
2016-01-16 10:24:15 +00:00
Felix Fietkau
eaa4eba10a scripts/download.pl: add SHA-256 support
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48253
2016-01-16 00:19:41 +00:00
Felix Fietkau
4eaeee7151 scripts/download.pl: cleanup GNOME source location definitions
Cleanup the @GNOME source download location definitions:
* remove dead and stale mirrors
* adjust to changes at directory structure
* add one new working mirror

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

SVN-Revision: 47825
2015-12-10 12:40:08 +00:00
Felix Fietkau
9b68fe961b build: add support for choosing a different url filename part than the output file
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 47591
2015-11-22 19:06:33 +00:00
Hauke Mehrtens
accc5e3693 script: downlaod: change mirror for kernel.org
kernel.org now suggests a different mirror address. this one also
support IPv6 connections and was faster for me.

Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>

SVN-Revision: 46875
2015-09-11 17:59:12 +00:00
Felix Fietkau
3d640eccc1 scripts/download.pl: add support for adding mirrors through an environment variable
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43654
2014-12-12 12:35:23 +00:00