How do I use PREMIRRORS
in Bitbake local configurations or recipes?
I want to provide my own download locations for some slow or inaccessible third-party URLs, but the official PREMIRRORS
documentation is vague and lacks examples.
BitBake recipes specify how a particular package is built. Recipes consist of the source URL (http, https, ftp, cvs, svn, git, local file system) of the package, dependencies and compile or install options. They also store the metadata for the package in standard variables.
Bitbake is a task scheduler that parses Python and Shell script mixed code, which we called Recipes. The code parsed generates and runs tasks. They are a set of steps orders according to the code's dependencies.
The openembedded-core layer provides a utility tool called bitbake-layers, which performs actions associated with layers. It allows you to perform actions including the following: see the list of actual layers taken into account in your configuration. add existing layers to your configuration.
Note: These results are based on experimentation with Yocto 2.3, but probably apply to 2.5 as well.
Suppose that your recipe file contains this target URL:
SRC_URI = "http://download.example.com:8080/foo/bar/baz-1.0.tar.gz"
Then in your local.conf
, you can define your custom download location as:
PREMIRRORS_prepend = "http://download\.example\.com:8080/.* http://my-mirror.example.com/copies/\n"
In this default case (with no special placeholders) Bitbake does not include the additional /foo/bar
path elements, and instead tries to download just the filename from http://my-mirror.example.com/copies/baz-1.0.tar.gz
These samples use special predefined placeholders, which are detailed in the next section.
SRC_URI = "https://example.com:1234/foo/bar.zip"
PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/PATH\n"
http://mirror.local/foo/bar.zip
SRC_URI = "https://example.com:1234/foo/bar.zip"
PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/MIRRORNAME\n"
http://mirror.local/example.com.1234.foo.bar.zip
SRC_URI = "ftp://example.com:1234/foo/bar.zip"
PREMIRRORS_prepend = "(\w+)://example\.com(:\d+)?/.* TYPE://mirror.local/PATH\n"
ftp://mirrors.local/foo/bar.zip
PREMIRRORS
parses all matched URIs and provides five special placeholder values in the target URI. Supposing the matched URI is http://host.example.com:1234/foo/bar/baz.txt
:
TYPE https
HOST host.example.com%3A1234
PATH foo/bar/baz.txt
BASENAME baz.txt
MIRRORNAME host.example.com.1234.foo.bar.baz.txt
PREMIRRORS
variableThe PREMIRRORS
variable consists of series of lines (separated by \n
) each with a regular expression to match a URI, and then a replacement string, with both portions separated by a space.
Bitbake tries them in order of appearance, and you generally want your private mirrors to take priority, so prepend onto PREMIRRORS
, ex:
PREMIRRORS_prepend = "http://original/location/.* http://alternate/location/\n"
You can add entries to PREMIRRORS
inside your bitbake recipes, but it is not recommended, since a major use of PREMIRRORS
is for people reusing your recipe in some other context or location.
Instead, you can put it inside your local.conf
file in an existing build directory. Alternately, edit the source template which Poky script use when creating a new local.conf
in a fresh build-directory.
SOURCE_MIRROR_URL
?The SOURCE_MIRROR_URL
is a quick way to add a series of PREMIRROR
entries for all supported protocols. For example, this setting:
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "TYPE://mirror.local/PATH"
is the same as writing:
PREMIRRORS_prepend = "\
cvs://.*/.* TYPE://mirror.local/PATH \
svn://.*/.* TYPE://mirror.local/PATH \
git://.*/.* TYPE://mirror.local/PATH \
gitsm://.*/.* TYPE://mirror.local/PATH \
hg://.*/.* TYPE://mirror.local/PATH \
bzr://.*/.* TYPE://mirror.local/PATH \
p4://.*/.* TYPE://mirror.local/PATH \
osc://.*/.* TYPE://mirror.local/PATH \
https?$://.*/.* TYPE://mirror.local/PATH \
ftp://.*/.* TYPE://mirror.local/PATH \
npm://.*/?.* TYPE://mirror.local/PATH \
"
It seems the INHERIT
+SOURCE_MIRROR_URL
directives will still work if used in your local.conf
(as opposed to a particular recipe.) However, Bitbake will emit warnings, so it may not be the intended use-case. Ex:
WARNING: Invalid protocol in PREMIRRORS: ('cvs://.*/.*', 'TYPE://mirror.local/PATH')
The -D
debug flag will cause bitbake
to emit information about what URLs it attempts to download from. You can also use -C do_fetch
, which will force it to try the fetch step and re-download anything needed for the given recipe.
bitbake -D -C do_fetch software-recipe-name-here
Here's some example debug output, showing the PREMIRROR
URL it attempts to access:
DEBUG: some-software-1.0 do_fetch: Trying PREMIRRORS
DEBUG: some-software-1.0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /home/user/build_foo/DL_DIR 'http://mirror.local/path/to/the/filename.ext
If you need to experiment and run bitbake
many times, it will be faster to temporarily put your new PREMIRRORS_prepend
directive into a particular test-recipe, as opposed to modifying the local.conf
. This is because Bitbake won't need to re-parse all the other recipes whenever you change it.
http://host:123/foo
?Apparently there's no easy way to get the 123
on its own. While PREMIRRORS
allows you to match with regular expressions, it does not seem to support using captured text from the match inside the replacement URI.
The port number is present inside HOST
and MIRRORNAME
, but there's no standard mechanism to split those values apart.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With