Using CDN's to cache source downloads?

[Jon South]

On Sun, Jun 26, 2011 at 16:56, Auke Kok <sofar at foo-projects.org> wrote:

>
> I've been messing around with some new plugin ideas, and I came up with the
> idea to write one that makes lunar fallback to using a CDN for failed
> downloads. This would provide a great extra mirror service at almost no cost
> to implement to us (and, coral CDN already does this for free for anyone).
>
> Here's how it works:
>
> 1) we "seed" the CDN.
>
> Basically, we run a script daily on a machine that finds all the compatible
> URL's in moonbase, and goes and tries to "download" these URL's through the
> CDN. After that, the CDN has a cache, and anyone can, at a later stage get a
> copy through the CDN.
>
> We don't need to actually download the entire thing to seed it. Just doing
> a "HEAD $URL" is enough to make the CDN get the file for us, so this is
> extremely efficient.
>
> 2) we add a plugin that downloads the source through the CDN instead of
> directly.
>
>
> I already wrote the plugin, and a basic seeder. I'm wondering if people
> think this would be useful or not. Also, it's entirely dependent on the
> reliability of the CDN itself.
>
> Ideas? feedback welcome. See attached code for details.
>

This looks pretty cool, but is this allowed by coral's usage policies? Funny
enough, their Usage FAQ is currently not responding (is it not on a CDN?).
Also, would the CDN be tried first, or as a last resort? I think speed and
reliability of the CDN could make this flip-flop a bit.

I wonder what it would cost to run a caching mirror via Amazon S3 or some
other comparable CDN? Maybe we could get one going?

-Striker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://foo-projects.org/pipermail/lunar/attachments/20110627/691b47bb/attachment.html>