I'm building something which could uncharitably be described as "an open proxy providing attack amplification as a service", but which I'd rather describe as "a core part of tools like RSS feed readers": ask it to monitor a URL, and it'll check periodically to see if it has changed and tell you so. I'd like to make it a publicly available service so people can more easily build new tools for noticing when web pages change. What are some ways I can encourage good applications and discourage abuse?
@jamey Aside from the obvious "honour response codes" (and probably report those back to the requester?), I think a lot of the issues can be avoided by setting a reasonable minimum interval (5 minutes?) and making sure that you only make one request per URL even if many users of the service are asking for it to be monitored. That would essentially guarantee that you make at most one request per 5 minutes to a given URL.
An addition could be to disallow or automatically delay monitoring of URLs that respond slowly, to account for the *really* intensive processes (eg. tarball generation) where even every 5 minutes would be too much.
@joepie91 Yes, good suggestions! So far I've set a ten minute minimum—we'll see if that's too short or too long—and I do only monitor any given URL once. I think I also need to limit the rate of requests within a single domain, even if they're for different URLs. I'll have to think more about using response time as a hint, that's an interesting point I hadn't really considered!