Difference between revisions of "Heat/LocalPackageCacheMirror"

Latest revision as of 04:38, 15 April 2013

For frequently repeated stack creation, or scenarios where your internet connection is either slow or bandwidth limited, it can be desirable to locally cache package repository data and packages, such that the instance creation (which typically includes installing additional packages from the remote repositories) is quicker.

The heat developers have been using the following approaches to speed up stack launches:

Squid Cache

Steve Baker's blog post about setting up a local squid proxy:

A similar approach can be used to install a squid proxy on a local development machine (asalkeld had notes on setting this up on Fedora in the old GitHub wiki, perhaps he still has notes which can be pasted in here?)

Local Repository Mirror

An alternative approach is to locally mirror the entire repository and updates - this is useful if you have multiple machines running the same OS and you want to do network installs etc, basically you need a local server box running an http server, and a script which runs periodically (e.g nightly via a cron job) to sync with a Fedora mirror which supports rsync. You can then build a JEOS image (or modify an existing one) to point yum at the local mirror.

Note these instructions are basic, see Fedora Notes on setting up public/private mirrors for more comprehensive mirroring instructions/options.

The following gives an overview of steps which can be used to create and use a simple local mirror:

Clone this github gist
Edit the fedora_rsync.sh to match the required version of Fedora, and point at a mirror geographically close to your location
Run an initial sync (will take a long time and a lot of space, allow at least 60G of disk space per Fedora version/arch)
Setup a cron-job to periodically sync updates, e.g nightly
Make the mirror accessible over the network via a method supported by yum (e.g via http as in the example repo files in the gist)
Edit the two .repo files to point to your local mirror location

Now you need to modify your JEOS image to point to the local mirror instead of the internet mirrorlist/repos. This can be done either via rebuilding with a modified TDL, or more quickly by modifying a copy of an existing, known-good JEOS image using libguestfs tools, e.g:

cd /var/lib/libvirt/images/

cp F17-x86_64-cfntools.qcow2 F17-x86_64-cfntools-localmirror.qcow2

virt-edit -a F17-x86_64-cfntools-localmirror.qcow2 /etc/yum.repos.d/fedora.repo -e 's/enabled=1/enabled=0/'

virt-edit -a F17-x86_64-cfntools-localmirror.qcow2 /etc/yum.repos.d/fedora-updates.repo -e 's/enabled=1/enabled=0/'

virt-copy-in -a F17-x86_64-cfntools-localmirror.qcow2 ~/git/gists/5345217/fedora-localmirror.repo /etc/yum.repos.d

virt-copy-in -a F17-x86_64-cfntools-localmirror.qcow2 ~/git/gists/5345217/fedora-updates-localmirror.repo /etc/yum.repos.d

You can use virt-ls and/or virt-cat to verify the modifications made, then update the image in glance:

glance delete <old image id>

glance image-create --name=F17-x86_64-cfntools --disk-format=qcow2 --container-format=bare < /var/lib/libvirt/images/F17-x86_64-cfntools-localmirror.qcow2

@@ Line 9: / Line 9: @@
 === Local Repository Mirror ===
-An alternative approach is to locally mirror the entire repository and updates - this is useful if you have multiple machines running the same OS and you want to do network installs etc, basically you need a local server box running an http server, and a script which runs periodically (e.g nightly via a cron job) to sync with a local mirror which supports rsync.  You can then build a JEOS image (or modify an existing one) to point yum at the local mirror.
+An alternative approach is to locally mirror the entire repository and updates - this is useful if you have multiple machines running the same OS and you want to do network installs etc, basically you need a local server box running an http server, and a script which runs periodically (e.g nightly via a cron job) to sync with a Fedora mirror which supports rsync.  You can then build a JEOS image (or modify an existing one) to point yum at the local mirror.
 Note these instructions are basic, see [http://fedoraproject.org/wiki/Infrastructure/Mirroring Fedora Notes on setting up public/private mirrors] for more comprehensive mirroring instructions/options.