Squid Proxying Package Manager

Introduction

I use a squid proxy at home to reduce the load on my broadband connection. This has come in useful for when i want to download updates to my Linux boxes. I use a mixture of Debian and Ubuntu boxes which can be updated via apt-get. I found that it was easy and useful to point apt at my squid proxy server so that packages already downloaded by one computer could be quickly downloaded by others.

Pre-requisites

This procedure only really makes sense if you have more than one computer running the same distro which use the same source. It may be useful for example if you have multiple Fedora boxes, multiple Ubuntu boxes or multiple Debian boxes. But it probably wouldn’t make sense if you had one of each of those.

A squid proxy server. I have mine running on a box which has a few gig of disk space. Nothing special, by no means would it mirror the whole of a repository, just cache the most recent requests.

A distro with a package management system and tools that you can point at a proxy, or a package management system that uses the http get method to obtain new packages. Debian based systems such as Ubuntu and Knoppix, and to the best of my knowledge Redhat based systems such as Fedora, fall into this category.

Squid config

Mine is a real basic install of squid. The only thing I changed was the maximum file system cached by squid. If I had left this at the default then large software packages such as openoffice.org and firefox might not be cached. This would defeat the whole point of caching – to reduce load on the connection. I therefore changed the squid config on the server from the default 4MB to a larger value. To do this after installing squid edit the file /etc/squid/squid.conf and find an existing line like below, or add this line.

maximum_object_size 102400 KB

That’s 100MiB. Which is quite a lot. Feel free to adjust to taste.

The only other settings were standard ones to allow my clients to access the proxy. My network is 192.168.129.0 and this is the config in the /etc/squid/squid.conf which allows those clients to access the proxy.

hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin ? no_cache deny QUERY acl all src 0.0.0.0/0.0.0.0 acl mynet src 192.168.129.0/255.255.255.0 http_access allow mynet http_access deny all

Most of the above should already be in your config file, but may be commented out. Feel free to read the documentation in the config file.

Downloading Updates

There’s a few ways to download updates on various distros. The tool I use on Ubuntu is apt-get or aptitude but things will look very similar for yum or up2date. Here’s what you might normally do to update.

sudo aptitude update sudo aptitude dist-upgrade

Here’s what I do now. (Note: colossus is the hostname of my proxy server)

export http_proxy=http://colossus:3128/ sudo aptitude update sudo aptitude dist-upgrade

That’s it. Requests now go through the proxy. Here’s the proof from /var/log/squid/access.log on the proxy server.

First we have one client downloading openoffice.org2. Note the TCP_MISS indicating this isn’t in the cache.

1141328780.215 255474 192.168.129.208 TCP_MISS/200 29882642 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-core_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328782.178   1962 192.168.129.208 TCP_MISS/200 221616 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/python-uno_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328825.221  43040 192.168.129.208 TCP_MISS/200 4891082 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-writer_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328856.570  31346 192.168.129.208 TCP_MISS/200 3673226 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-calc_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328873.024  16452 192.168.129.208 TCP_MISS/200 1927980 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-draw_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328878.841   5814 192.168.129.208 TCP_MISS/200 650262 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-impress_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328882.960   4115 192.168.129.208 TCP_MISS/200 375876 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-math_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328909.582  26620 192.168.129.208 TCP_MISS/200 2997606 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-base_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328910.383    800 192.168.129.208 TCP_MISS/200 91761 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org_2.0.1oob680m5-0ubuntu2_i386.deb - DIRECT/82.211.81.182 application/x-debian-package 1141328911.152    767 192.168.129.208 TCP_MISS/200 86133 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org2_2.0.1oob680m5-0ubuntu2_all.deb - DIRECT/82.211.81.182 application/x-debian-package

And here comes the second machine getting the same packages. Note the TCP_HIT noting the packages are in the cache and as such are served up by the proxy.

1141329032.977     82 192.168.129.220 TCP_HIT/200 601488 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-l10n-en-us_2.0.1oob680m5-0ubuntu2_all.deb - NONE/- application/x-debian-package 1141329035.530   2546 192.168.129.220 TCP_HIT/200 26147042 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-common_2.0.1oob680m5-0ubuntu2_all.deb - NONE/- application/x-debian-package 1141329039.239   3702 192.168.129.220 TCP_HIT/200 29882650 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-core_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329039.264     21 192.168.129.220 TCP_HIT/200 221624 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/python-uno_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329039.730    462 192.168.129.220 TCP_HIT/200 4891090 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-writer_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.110    356 192.168.129.220 TCP_HIT/200 3673234 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-calc_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.299    181 192.168.129.220 TCP_HIT/200 1927988 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-draw_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.365     59 192.168.129.220 TCP_HIT/200 650270 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-impress_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.402     32 192.168.129.220 TCP_HIT/200 375884 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-math_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.693    283 192.168.129.220 TCP_HIT/200 2997614 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org-base_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.706      8 192.168.129.220 TCP_HIT/200 91769 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org_2.0.1oob680m5-0ubuntu2_i386.deb - NONE/- application/x-debian-package 1141329040.720     10 192.168.129.220 TCP_HIT/200 86141 GET http://gb.archive.ubuntu.com/ubuntu/pool/main/o/openoffice.org/openoffice.org2_2.0.1oob680m5-0ubuntu2_all.deb - NONE/- application/x-debian-package

The following is always nice to see – especially when your broadband connection maximum speed is about 110K (1Mb download).

Fetched 71.5MB in 7s (9137kB/s)

Further Funkyness

I also have a script (which HugoMills wrote and I tweaked) which downloads latest packages overnight but does NOT install them. I have this in my /etc/cron.daily so my boxes always have the latest packages overnight and I can then choose to install them the next day. Very handy for me because I have a metered broadband connection where I have a higher “allowance” overnight than I do during the day.

export http_proxy=http://colossus:3128/ aptitude update >/dev/null aptitude autoclean >/dev/null RESULT=`aptitude -s -y upgrade | grep upgraded, | sed -e "s/[^0-9,]//g" | cut -d, -f1-2`  if [[ :$RESULT|!= :0,0 ]] then echo "Upgrade required for the following packages:" aptitude -s -y -v upgrade | sed -n -e '/Inst/{' -e 's/Inst (.*)((.*) .*/1 2/' -e p -e '}' aptitude -d -y -v upgrade >/dev/null fi

A side effect of running this overnight through cron is that I get an email each day from each box on my LAN telling me what new packages are available, and have been downloaded. It’s very handy indeed – thanks Hugo!