Speeds and Feeds: RSS feed management, validation, and performance »

We use this blog to talk about issues we see around securing, managing and scaling APIs and web services.  We also see many of these same issues and requirements with feeds.  Arguably, feeds - specifically RSS and Atom feeds - might just be the most common type of XML API.

Feeds are are growing beyond just being a great way to keep people abreast of changes to news or a blog and becoming a great way to aggregate or syndicate content to partners or customers or applications in general.

Our media customers, like MTV Networks, use RSS feeds management as a way to distribute updates about their content to customers and partners. Essentially, the RSS feed becomes a “catalog” of what they’re pushing out to their web site partners right now.   
 
The needs of feeds: customization, validation, and caching

In general, everything we said about API analytics, API traffic management and API scalability can apply to feeds that you offer or consume as well.     Do you know who is looking at your feeds, how the traffic changes over time, and what kind of response time your feeds are delivering? Do you have any way to limit access to your feed if it becomes tremendously popular? And while most feeds associated with blogs, for instance, have no access control so that they are open to the whole Internet, there are also feeds that need authentication and authorization just like any web service.

But aside from these other “web services” types of requirements, feeds have some special requirements that come up all the time – customization, validation, and caching.
 
We see feed validation issues frequently. Feeds are more likely to have risks with broken links, especially if you are aggregating frequently updated content from around the web.  Feeds often don’t match the proper XML schema or aren't valud XML, as often they are generated or manually assembled. Sometimes the feed provider needs to address these issues, and other times the feed consumer has to find a way to deal with the bad input.
 
Customization is another issue. The basic feed on your blog is pretty much the same for everyone – but for others that is not the case. Lots of companies use feeds to push content to their partners, and each partner may demand a different format. Sometimes a partner needs only certain items, or a certain number of items, or they need the feed transformed into different formats like RSS, MRSS, or Atom. It’s not unusual for a feed provider to have to provide 10 or 20 (or more) custom variations of the same feed if content syndication is important to the business.    You may want to drive these with existing profiles or business rules in SQL or LDAP.
 
Caching is the last, but certainly not the least. Sometimes feeds are just static HTML files served by a web server, but sometimes they’re not. And when they’re not, feeds can be slow. We’ve seen some infrastructure that takes over two seconds to produce a feed – that’s a long time. When that happens, a little caching can go a long way.

So think about feed management when considering API management or content syndication.  If you want to experiment with API management on on a feed, Apigee is a good way to get a feel for what stats  on feed usage, latency, data volumes, and error rates might offer.