A couple of weeks ago I wrote about backing up Apache and MySQL to Amazon S3, the final piece of the puzzle I was looking for was to automatically purge all backups older than 7 days from S3. It took me a little while to put all the pieces in place, and I wanted to write it all down as much for myself as anyone else since I am sure I won’t remember all this if I need it again.
I found a nice utility names s3cleaner a simple python script that uses age and regex patterns to delete files from S3 buckets.
The first challenge was to run a python script on my server (hosted at Dreamhost) I needed to create a sandboxed Python environment. I found some instructions here. The first step is to download virtual-python.py and run it. From a command line you can simply run
wget http://peak.telecommunity.com/dist/virtual-python.py (to download the script)
Once it is downloaded run the script
Next you have to download ez_setup.py from the command line run
wget http://peak.telecommunity.com/dist/ez_setup.py
Once it is downloaded execute it, the key here is you are now using the sandboxed install
of python, point to python in the path shown in the image above, not simply python like as
in the first step.
You now have a sandboxed installation of python which will let you run s3cleaner but wait not so fast,
s3cleaner relies on boto which provides the python interface to Amazon Web Services. Download boto here,
and copy it up to your server. Once on your server install it, again make sure you use the sandboxed python
not the system wide one
You are now ready to place s3cleaner.py on your server and run it you need to specify the following flags
when running
A word of caution, for bucket name it only accepts a bucket name not a subdirectory, so
everything that matches your deletion criteria (–-maxage and –-regex) anywhere in the
bucket will be deleted.
Download List
virtual-python.py and ez_setup.py
boto
s3cleaner