Mobile Apps 101: Key patterns you need to know (video & slides) »

Thanks to all who participated in the Mobile Apps 101 Webinar last week about "designing apps that people want to use." The video and slides are below.

Check them out for an introduction to some key patterns for mobile app development - repeatable patterns that represent functionality for the front and back-end of mobile apps.  Thanks @edanuff, @gbrail, @timanglade.

We'd also love to hear your thoughts, insights, or questions over on the api-craft forum.

X.commerce: A boldly open platform & OS for commerce in the cloud »

Today we formally announced our collaboration with X.commerce in providing the API infrastructure for X.commerce, an eBay Inc. company.

X.commerce is designed to help merchants reach consumers at the edge of their businesses - where they shop today - online and on millions of mobile devices.

X.commerce is a great example of a platform going boldly open. Many platform businesses focus on apps and business processes on top of the platform layer (apps on top of an OS, system integrators on top of Salesforce, and so on). X.commerce's innovative approach also enables 3rd parties to plug net new capabilities into the X.commerce fabric from below, making them available for use in apps.

These plug-ins, such as tax calculation, customer support services, or rewards management, will extend the eBay capabilities in the ecosystem for thousands of potential customers and collaborators to take advantage of.  This also includes the potential for capabilities offered by 3rd parties that are competitive with those offered by eBay.

It's analogous to Linux's loadable kernel modules or Microsoft's device drivers - not just enabling apps above, but many devices below that can be used by apps; each device added enriches the app ecosystem. The X.commerce platform begins to look like an OS for commerce in the cloud.

eBay understands that more capabilities plugging into the commerce OS means more apps being built on top. This should lead to increased traffic and transactions through the platform. In turn, X.commerce will likely see very rapid ecosystem evolution and growth. It's a bold move and is likely to be rewarded by the ecosystem and the market. Bravo!

Scaling your API: Predict, Prepare for, Overcome Challenges (video & slides) »

Thanks to all who participated in last week's Webinar, Scaling APIs: Predicting, Preparing for & Overcoming Challenges, in which we examined common bottlenecks and hurdles when scaling your API. We discussed some tips and tricks for how to overcome those challenges as your API traffic grows 10x, 100x, 1000x . . .

The video (~50 min.) and slides are below. Thanks @gbrail and @brianpagano.

We'd love to continue the discussion and get more of your thoughts, insights, or questions on the api-craft forum.

Hiring great people to run your API program: Where to start? »

Customers ask us all the time "How do we hire great people to run our API program?"

We tell them the same thing we think about when hiring at Apigee. It's about passion and it's about finding people who can work with context, not control.

We lead with passion, not with skills or profiles. We look for people who get what we're doing at a deep level. We focus on how we function as a company. And great hires for API programs are often found from within, or from the ranks of the folks who are already passionate about the approach/program.

Skills and job profiles will get you only so far - passion and talent will take you all the way.

Just like our customers, we serve new constituents every day. We look for people who can explore new boundaries and solve hard problems; people who can make decisions every day with general direction and in a given context.

This is why we also look to hire people that thrive in an environment of context, not control. People who can build the context and have the judgement to take action based on their knowledge of the business problem at hand versus needing a 'command and control' structure.

Context, not control is a huge shift from the leadership and management styles of the past. Netflix continues to be one of the pioneers of this approach and describes it really well here. More on this soon.

“APIs: A Strategy Guide” O’Reilly author webinar & free chapter »

Last week O'Reilly hosted a webinar by the authors of the new O'Reilly book "APIs: A Strategy Guide."  Video and slides are below. 

The book is an overview of API strategy for business executives and this webinar dives into both public and private API strategies.

Thanks to O'Reilly, @daniel_jacobson, @gbrail and @danwoodscito.

Courtesy of O'Reilly, a free book chapter is posted here.

I need an API: Where do I start? »

One of the most important questions that enterprises ask us has changed from "Why APIs?" to "I know I need an API; where do I start?"

Start where you have the most amount of pain - where the business drivers are. If your pain point is in meeting demand for mobile and social apps, then start there - with an internal API strategy. If you need to innovate with partners to deliver on a backlog of business development opportunities, start with a partner API. If you need to inspire a broad community of app developers to innovate and create growth and new opportunities, start with an open API.

The lines between internal, partner, and open blur a little more every day as companies innovate at the edge of their businesses. And what starts as one flavor of API can become a different opportunity down the road. We've seen it a bunch of times - an API team learns from internal and partner projects, develops the know-how and courage to open an API to the world of innovative developers who can take the API and the business in creative and valuable directions. We've seen companies have huge success in the opposite direction - businesses starting with Open APIs and realizing that the bulk and the most valuable innovation is coming from partner and internal collaboration.

As long as you start where the business pain is -  you're starting in the right place.

Announcement: Free eBook on Web API Design »

We wrote an API design book! In 2009, a couple of my fellow Apigeeks (@earth2marsh & @gbrail) and I were frustrated with how many bad, inconsistent Web APIs we encountered in the world. So, we started cataloging the bad (and good) things we saw people doing to other people through API design. After three years of discussions, presentations and implementations with technologists all over the world (online & in real life), we are happy to announce the beta version of a new eBook.

Web API Design: Crafting Interfaces that Developers Love 

Although I did my best to chase down errors, there are likely mistakes and bugs in the book. Plus, it's shocking how poorly code and eBook publishing get along. Despite some blemishes though I think you'll enjoy kicking back and reading some of the world's best thinking about API design.

Big thanks to all the freelancers, corporate geeks, open source contributors, bloggers and doers who helped craft our thinking about API design. Please enjoy the book and please let us hear any feedback: @landlessness or .(JavaScript must be enabled to view this email address)

API Product Management: Driving Success through the Value Chain (slides & video) »

Thanks to all who attended the API Product Management: Driving Success through the Value Chain Webinar.

Video and slides are here. Thanks to @jenmazzon, @sramji and a big thanks to our special guest host, Michael Hart @michaelhart. As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

OAuth: The Big Picture - Free e-Book »

APIs and the app economy - an explosion of innovation »

Startup Weekend Detroit 4 was hosted the weekend of February 17, 2012, at the M@dison Building, home of Detroit Venture Partners.

Watch this 3 minute video* - you can feel the positive energy that APIs & the app economy unleash. More than 100 entrepreneurs from around North America attended and created real products and startups during a single weekend.

What hit me watching this video is that there is really an explosion of innovation happening in the world today.

When you think about all the events happening... the hackathons, the startup weekends, the maker events... there is a LOT going on.

You can find events for whatever topic you are interested in.... general mobile hackathons, social hackathons, cleanweb, water, heathcare, civic/open government, music, gaming, connected car, more.

Programmable Web is tracking 55 events coming in the next few weeks: http://blog.programmableweb.com/2012/02/23/55-upcoming-hackathons-civic-music-photos-and-robotics/

At the same time, the 'bar' for getting something started has been materially lowered.  With a swipe of your credit card you can spin up server instances in EC2 or spawn an app environment on Heroku or other CaaS/PaaS platforms.  Tools like Sencha Touch, PhoneGap, Titanium and others are available to easily build device applications.

Mobile app developers don’t even have to build their own user management backend because platforms like UserGrid provide full user management instantly.

It all makes for a potent cocktail of innovation.

Companies wanting to be a part of that innovation need to get the core of their business exposed via APIs and get into the game.

(*Video by Hendrickson Video: hendricksonvideo.com/)

User experience, data quality, and dealing with mistakes »

As discussed in previous posts, it's important for app developers and API providers to understand how to protect users, apps, and APIs from abuse and how to deal with malicious attacks when they happen.

It's also important to think about how to design protection systems to optimize impact on abusers while minimizing impact on legitimate users.

Good design for online safety systems requires the ability to dial the aggressiveness of the protection, detection, and remediation mechanisms up and down. This is useful when dealing with a wave of attack that is temporary. That is to say, the system might need to be fine tuned to be aggressive against traffic originating from a suspicious source followed by a reduction in the aggressiveness when the attack has ceded.

The flip side of having the ability to dial up and down the aggressiveness of the system is the potential of denying high quality service to legitimate users. Mistakes in this realm are called “false positives” and happen when good entities (legitimate traffic, users, apps) are labeled as “not good” by classification systems. As a result, they receive low quality of service, or are even denied service. Any and all such mistakes can impose a huge cost in terms of users, usage, and profits.

Any online safety technology requires a system of checks and balances to ensure that the aggressiveness of the system is fine tuned to maximize true positives and minimize false positives. There are various mechanisms to achieve such quality. However they are a topic for a different blog post.

For a number of reasons it is impossible to achieve a 0% false positive rate. First, abusers and attackers do their best to appear as legitimate users to the system. Secondly, there is a “learning latency” between the time it takes for a system to learn new bad behavioral patterns and “publish” them to their detection systems. In addition, systems will never be able to correctly distinguish new bad users/apps from new good users/apps and aggressive systems can bundle new good and new bad users/apps together.

One possible remediation for the inevitable false positive rate is to build a user experience that is simple, intuitive and minimizes the intrusiveness of such technologies in to the user’s workflow. It is critical to realize that an extremely large majority of users, apps, or developers who will find themselves in a remediation flow will be legitimate users (because this flow will break the cost model of the abusers who are interfacing with the service programmatically). Anything that can be done to reduce the cost this flow imposes on legitimate users should be high priority.

One caveat to the above assertion is that it is worth the abuser’s time to experience a remediation flow first-hand, as it allows them to figure out how to (at scale) circumvent the remediation flow. A well designed flow should not assume otherwise.

Dealing with malicious attacks »

In my last post I looked at some of the ways malicious users can attack your apps and APIs and ways to mitigate the risk of attack. This time we'll look at some more ways to push back against attackers.

When a malicious user or app is detected, the user or the app can be blocked, throttled or denied service. The confidence in the "maliciousness" of an app or a user can be used to take service denial actions or even to reduce the QoS available for such users and apps. Similarly, end users should be notified about actions taken against apps that have been authorized by the user to act on their behalf using their credentials. 

Let’s take a look at some of the push-back mechanisms in detail.

Quotas: Quotas can be applied at various levels i.e. at any one of user, app, IP address, or API level. In addition, quotas can be defined over multiple durations of time - that is quotas over windows of seconds, minutes, daily, monthly, and so on. A sensible quota strategy should use the traffic patterns generated by an app or seen by an API to deduce “good”, “bad” and “suspicious” behaviors over different sources and time durations and then enforce the resulting quotas judiciously. Denying legitimate traffic service because of misconfigured or stale quotas can cost usage and annoy legitimate users who can then in turn decide to leave the app or service. Automatic quota management is a good approach for scenarios where quota management is a necessity - that is for apps or APIs that have a high abuse potential.

Throttling: Like quotas, throttling can be applied to users, apps, IP addresses or APIs. Since throttling impacts the rate at which clients can make requests, a good throttling strategy requires integration with an intelligent traffic monitoring solution. Such a solution can employ and disengage various throttling measures based on changes in traffic while ensuring that higher volume traffic sources do not cause the lower volume traffic to starve for service.

Tiered Traffic: It is crucial for any developer or API provider to reserve and deliver the highest quality of service to the set of users who are “desirable”. In other words, to support users who promote the API or app, which in turn increases usage and profitability. Since profitability is a function of cost and revenue, a set of users that offers relatively high value but can be served at a lower cost (through caches etc.) are also good contenders for the higher tier. Users who are suspicious, unknown, or new or exhibiting mixed behaviors can be limited to lower tier traffic.

Blocking: Outright blocking is another option that can be implemented by “black lists” that are built and managed by the provider. Blocking when done extensively requires black list management, remediation, and expiration policies to ensure that entities on the black list are able to escalate and prove their legitimacy and so be removed from the black list.

Proofs: Users, devices, and apps can be asked to prove their legitimacy through “proofs” that force them to verify that they are human (e.g. CAPTCHA) or verify their identity (e.g. SMS or voice mail based code). There are numerous services that offer APIs that enable such functionality.

Regressive Actions: Often historical analysis is required to correctly classify the activity or behavior of an app or user. Offline historical analysis can be used to perform such classifications and “regressive actions” can be taken based on the results. In other words, the abuser is punished after they have carried out an attack. This ensures that no other users or providers can be targeted. In addition, regressive actions can be used to protect end users who have yet to be targeted by such attacks. For example, phishing email can be removed from a user's inbox if they have not yet read the email.

It is important to note that no single technique is a silver bullet and high-value services and users need to be protected through a mixture of these techniques. In addition, if there is enough money to be made by attacking users and providers, one should expect the abusers to push back against these techniques and attempt to game the system to be able to thwart or circumvent these defenses.

Next time, we'll look at designing online safety systems as they require the ability to dial the aggressiveness of the protection, detection, and remediation mechanisms up and down.

Protecting users, apps, and APIs from abuse »

Abusers or spammers are the bad guys looking to make money by getting unsuspecting end users or consumers of online services to interact with malicious content or spam that leads to one or more of the following scenarios:

• Eyeballs on spam content that lead to clicks and purchases.
• Gathering users’ private information through keyloggers (or other spyware) on the user’s machine or device which is then sold to the highest bidder.
• Phishing for users’ private information such as SSN, credit card #, or passwords and selling those to the highest bidder.
• Installing malicious software on users’ machines or devices, which in turn steals more of their information or uses their bandwidth or storage for carrying our further attacks.

Any workflow that creates or consumes content, shares or reshares content, sends or receives communications can be vulnerable to attack.

Online attacks almost always contain a "payload" that either delivers the attack or leads the user to another location where the attack is completed. Any time a piece of content is created or consumed or a communication is sent or received, there is an inherent "payload" that is delivered. In the wrong hands, this payload can be malicious.

Blogs can be spammed with comments which contain spam or malware, or which employ phishing techniques that redirect users to other malicious locations on the Internet.

In the same way, users can receive emails that contain spam or malware or other phishing techniques. Users can be redirected to malicious sites in their browser. Users' machines can be attacked and malware (such as adware, spyware, key loggers and viruses) installed which can scrape users’ private information and send it to the abusers.

Thinking about apps and APIs as assets

Apps like APIs can be considered "assets" that can be used to carry out attacks against end users.

APIs (and especially free APIs) provide services that eventually reach other users. Even if an API has a usage cost associated with it, if the return on investment for an abuser for planning and carrying out an attack is higher than the cost of using the API (at scale), even paid APIs can be abused.

Both developers and enterprises have apps that are highly monetizable if abused. App-based attacks fall into the following general categories:

Malicious apps: These apps are written with the sole goal of abusing unsuspecting users who download and install them. With the increasing use of single sign on services and integration of social networks with apps through social network platforms, a user's social network can be used to propagate these attacks.

Well-intentioned apps with vulnerabilities: These apps are not written to be malicious but have vulnerabilities either in their code (e.g. the apps's functionality can be scripted or controlled through hooks provided by the app) or in their business logic (e.g. no throttling of calls from a single user to the back-end system). An abuser can exploit these vulnerabilities to carry out attacks on enterprise APIs or on the end users of the apps.

Well intentioned apps exploited by a malicious user: In this case, an abuser can use some legitimate capability offered by the app as a foundation to carry out attacks that propagate through the end user's social network and spread through further usage.

Protecting your assets

You don't want your APIs or apps used for abuse. It is crucial for both developers and enterprises to protect their “assets”. If an end user develops a perception that a certain app or service is "dangerous", usage declines, growth can be reversed, and revenue and profit suffer.

Remember that abusers are most often out there to make money and if the cost to carry out an attack is less than the value derived from the attack, it will continue to be a sound business investment for the abusers.

Some things that developers can do to protect their apps and end users (and their friends)

• Monitor usage of your app and its impact on the users. Build and maintain a proxy for user reputation and models of good and bad behavior in the content of the app.
• Enable end users to report suspicious behavior of the app or of other users of the app.
• Work with the enterprise or API provider to ensure that your app is not creating suspicious loads on the API.

Some things that enterprises can do to protect their APIs

• Build traffic monitoring solutions and models to rate traffic as safe or abusive.
• Ensure that any content or communication created or shared through the API is free of malicious payloads.
• Invest in mechanisms to report and notify suspicious user and app behavior to the app developers.
• Build reputation systems for users, content and IP addresses to be able to quickly classify traffic, users and apps as good or bad OR desirable or undesirable.

Next time we'll look at some of the ways to push back against attackers, including use of quotas, throttling, and so on.

OAuth: Implementing OAuth 2.0 »

In a recent OAuth post, we recommended that if your API can require HTTPS, use OAuth 2.0.  Otherwise, use OAuth 1.0a.

How should you use OAuth 2.0?

There are three types of credentials in OAuth 2.0.

BEARER TOKEN: This type requires SSL. A bearer token is a big random number. Bearer tokens are easier for developers than OAuth 1.0a because they don’t have to write the signature. This is the most straightforward of the credential types to implement.  Use it!

MAC TOKEN: Like OAuth 1.0a, uses signature instead of SSL. The spec is still changing so I recommend that you wait.

SAML: Makes sense if the API Team and the developers really want SAML. The OAuth standards team is working on ways to use OAuth with SAML but that option is still changing. I recommend that you wait to implement SAML.

In short, I recommend that you use bearer tokens, as it is the simplest to implement.

How many legs?

I often get this question – how many legs does OAuth have and how many legs should I use?

Let’s look at the 3-legged and 2-legged terminology that’s developed around OAuth.

3-legged OAuth refers to the fact that there are three entities - a user, a client (application), and a server (service) involved. Both the user and the app have credentials.

2-legged OAuth refers to the case where the OAuth signature method is being used to identify just the application – that is, to identify just the client and not the user. This is often done using SSL. But the OAuth 1.0 signature can be used.

In my opinion, 2-legged OAuth uses only a part of OAuth and 3-legged OAuth is really OAuth.

If your API needs to authenticate users, you should use 3-legged OAuth. It has benefits for the API Providers and above all, it will mean improved security and better end user and consumer experiences with web and mobile apps.

Would love to hear your thoughts or questions over on the API Craft Google groups.

OAuth: Is it worth the effort? »

In the previous few blog posts, I’ve talked about what OAuth is and when and why I recommend it.

This time, let's explore some of the reasons why OAuth is more cumbersome and complicated for developers than plain passwords as well as some recommendations to help you make the decision between OAuth 2.0 or 1.0a.

Because OAuth eliminates password sharing between web apps, and password storage on mobile devices, and importantly for the sake of your end-users’ experience and security, I believe it is worth the effort.

How many versions of OAuth and which one should you use?

OAuth 1.0 The first "production" version of OAuth 1.0 didn't actually do authentication delegation correctly and as a result, the spec itself had to be patched. No one should be using OAuth 1.0 now.

OAuth 1.0a Not an IETF standard but stable and well understood.
Uses a signature to exchange credentials between client and server. This means that you get API security without SSL (not only is the password never exchanged but the OAuth token is never exchanged – it’s encrypted inside the signature). But coding the signature right is tricky.

OAuth 2.0 Actively under development in the IETF.
It is getting more stable and changes less with each draft – core flows are unlikely to change. Supports a ‘bearer token’, which is easier to code than the signature but requires SSL. However, most APIs should be using SSL by default anyway. Optional specs support signatures, SAML, and so on, but those specs are not yet stable.

The authentication dance is painful

There are a lot of great sites where you can get the details of the OAuth 1.0a authentication flow chart, so I'm not going to tackle that here. It’s not simple. It gets a little simpler with bearer tokens in OAuth 2.0 but because of the security requirements and the fact that you have to exchange identity without a password, it's always going to be a little complex.

Signatures are painful

The good news is that in OAuth 2.0 signatures are optional. You can do SSL and use a bearer token.

As I mentioned above, getting the OAuth 1.0a signature algorithm right is difficult. If you’re going to use OAuth 1.0a, I recommend you use one of the many libraries that are available.

Where do you store the credentials on the client?

There is no magic. You have an OAuth token and secret that needs to be on your mobile device and accessible.

You could encrypt it, but then you need access to the key that decrypts it.  You could also break them into smaller pieces.

What should you do? 

Be patient. If your API can require HTTPS, use OAuth 2.0 with bearer tokens. (Use the latest draft of 2.0.) Otherwise, use OAuth 1.0a.

It's possible to build an effective API right now using either a current draft of OAuth 2.0 (see Facebook) or OAuth 1.0a (see Twitter).

Bottom line, with your end-users’ experience and security top-of-mind, I believe OAuth is worth the effort.

Next time: Implementing OAuth 2.0

OAuth: For server-to-server APIs? »

In my last post, OAuth: Why it's good for API providers, I talked about why I recommend OAuth for API providers when they are exposing APIs for web or mobile apps.

This time, a short follow up to look at a couple of scenarios in which OAuth might not be the best solution.

APIs designed to be used only by servers.

When the only clients of your API are servers, OAuth is not the best solution. Having a separate set of authentication credentials for each app is a nice feature of OAuth, but for server-to-server use, the need to log in securely using a browser gets in the way.

Simply assigning a unique password to each app is probably sufficient. Two-way SSL is another good, albeit cumbersome approach.

I believe that OAuth is unnecessary for APIs that are used by a small number of internal or partner systems, and which are used by servers and not by end users who have passwords.

However, think ahead! If you discover that those APIs that are only accessed by servers today are useful by other types of clients (like web or mobile) tomorrow, then you'll need to support OAuth.

Are there other scenarios for which OAuth is unnecessary or even a bad idea?

Just like the server-to-server scenario, I think that OAuth doesn’t make sense in the following scenarios:

-       Anything that requires commercial levels of trust. For example, when your security model requires the capabilities of a PKI infrastructure.

-       One-time tokens. OAuth is a lot of complexity and machinery to make one API call.

Bad ideas include creating your own ‘version’ of OAuth or creating something that’s like OAuth but different. Sticking with standards, and focusing your development efforts on creating great apps seems like a better idea than rolling your own security scheme.

Next time: We'll talk about some OAuth complexities, and why it's worth the effort.

OAuth: Why it’s good for API providers »

In OAuth: The valey key metaphor and OAuth: Flow for mobile apps, we talked about why OAuth is good for users - how it allows users to grant third-parties access to their web services or mobile apps without sharing their passwords.

This time, why OAuth is good for API providers whether they are exposing APIs for web apps or APIs designed for mobile apps.

OAuth means that Web apps that expose APIs don’t have to share passwords.

There are two alternatives

-       The security risk of typing your password into every single web app you use – an unacceptable risk!

-       Some sort of universal single sign-on mechanism (run by a third party that everyone agrees upon and trusts) – it doesn’t exist!

Therefore, for all web apps that expose APIs to other web apps - we recommend OAuth.

OAuth eliminates the need to store a password on a mobile device adding a layer of security when an API is used by mobile apps built by untrusted developers for a public API.

An OAuth token is hard to guess. It is tied to a particular app and device. It can be revoked without affecting other devices and apps.

OAuth allows the API provider to revoke tokens for an individual user, for an entire app, without requiring the user to change their original password. This is critical if a mobile device is compromised or if a rogue app is discovered.

OAuth future proofs the authentication process. Because users’ browsers are redirected to a place that’s under the control of the API team to get the OAuth token, the API team can control what to do at that point in the workflow. 

In other words, OAuth is flexible enough to support any of your company’s specific workflows (think SSL, multi-factor authentication, terms of service, changes to terms of service, and so on…) without requiring a change to the client or server.

Therefore, for APIs designed for mobile and native apps – we recommend OAuth

Next time: OAuth in a server-to-server API scenario.

OAuth: Flow for mobile apps »

In my previous post, I talked about how OAuth allows users to grant third-parties access to their web services without sharing their passwords.

In that previous example, our user (Bob) accessed his Twitter account through the bit.ly web site.

This time, let's look at what happens when Bob is using a mobile app instead of a web app.

It’s pretty much the same for a mobile app as a web app. When you fire up your mobile app and want to access Twitter, you don’t want to have to give the mobile app your Twitter password and store it on your phone.

In the same way as happens for the web app,  the phone app opens a browser window and directs Bob to login to Twitter (and authorize bit.ly to use his Twitter account) - using OAuth.

The phone app never sees Bob’s Twitter password.

The phone app uses its OAuth credentials to retrieve credentials for Bob. It stores these locally. In this way,

OAuth allows this one phone app access to the Twitter service on behalf of one user (Bob).

This all happens through your browser. There are both advantages and disadvantages to this:

- The advantage of this is that the app never sees your password.

You don’t have to trust the person that built the app or worry about losing your phone (as much).

If Bob loses his phone, he can log into Twitter and revoke the credentials that Twitter gave the phone app. (A thief cannot uncover the password, only the token.)

- The disadvantage is that the app has to open up a browser window. This possibly breaks the flow of a nicely designed UI for a mobile app.

There are ways in OAuth around this but you eliminate many of the security reasons for OAuth in doing so. Twitter and Facebook for example, have made the choice to have everybody log in through the browser.

Next time: Why OAuth is good for API providers.

Armrest & REST API Design »

Last week I flew from Doha, Qatar to Brussels, Belgium. The fella in the aisle seat fell asleep before takeoff. The road warrior code of chivalry indicates one should never wake a fellow traveler.

So I stayed in my window seat for 8 hours. After a few minutes it became obvious that my armrest (singular, my neighbor had usurped the other one) was uncomfortable.

Here's a picture of the armrest.



The armrest has many problems. The biggest: you can't rest your arm on it.

If you try to relax, your elbow gets sucked by gravity into the cutout and pinched unpleasantly by hard plastic. If you deal with the pain and fall asleep you will wake to find your ears exploding because you inadvertently cranked up the volume, the flight attendant will be tapping your shoulder asking why you paged him and your reading light will be shining in your dilated eyes.

The Airbus A330-200 in which I was traveling holds 250 people. That's 2,000 man hours of discomfort per flight injected into the universe because of bad design.

So, why does such an obviously bad, pain-inflicting design make it into the world?

Because it cost-effectively solves other, non-customer-related problems! In this case, it is a clever way to store the remote controller while still keeping it accessible to passengers. It’s also done in a way that doesn't require an expensive reworking of the interior: cut off part of the armrest, put a hole in it and reattach on a hinge. Done! In-air multimedia problem solved!

What does this have to do with REST APIs?

Even with the best of intentions, API teams often stop short of doing the right things for their customers. Customers of APIs are application developers. Application developers are smart, busy, and born to suffer (especially Objective-C developers, NSThatsADifferentTopic). But they are also often musicians, gamers and aesthetes. They are attracted to things like truth and beauty. And they are attracted to good, self-consistent API design.

So, why do API teams often stop short of delighting developers? Because the time and cost of learning to do the right things and getting them done often seems prohibitive! At Apigee, we share everything we learn on our blog, twitter @Apigee, YouTube, API Craft on Google Group, and in our webinars.

There's no reason to walk the road to creating a fantastic API alone. If you know of an API or an API team that's not doing things right or is trying to go it alone, please send them to us. We'll get them straightened out.

OAuth: The valet key metaphor »

OAuth has taken off as a standard way for apps and websites to handle authentication. But OAuth is a confusing spec that can be hard to pin down.

I wanted to talk a little about what is OAuth and when you should use it for your API – hopefully pin it down a little in a few blog posts. I covered a lot of this in OAuth: The Big Picture. Check out the video and slides! 

Let’s start with what is OAuth and why it came about.

The developers of OAuth set out to solve the problem that services and passwords don’t mix well as you start to combine apps and mash them up.

Imagine, in today’s environment of web APIs and mobile apps, if every web site that used an API from another web site had to share and store that web site password. Soon you’d have the proliferation of your passwords all over the Internet with every service you’ve used from Facebook, to Twitter, to Skype, to your bank account. Do you trust them all with your password? 

What is OAuth?

It is another way to authenticate to a service - a security protocol that allows users to grant third-party access to their web resources without sharing their passwords.

OAuth supports this “delegated authentication” between web apps using a security token called an "access token." Rather than relying on a single password as the master key for every app that accesses an API, OAuth uses this token.  

An OAuth token gives one app access to one API on behalf of one user.

Eran Hammer-Lahav, a spec author for OAuth, compares the token to a valet key.  This is an apt metaphor.

For most people, their car is one of their most valuable possessions, valued in tens of thousands of dollars.  They are convenient places to leave our other valuables like computers and clothing.  Yet we are sometimes required to give them to a parking attendant or valet whom we’ve not met before. A valet key solves the problem – it’s an access token with limited rights that can operate the vehicle but not grant access to the trunk or glove box.

For great information about the history of OAuth, see Eran Hammer-Lahav’s guide. 

How does it work?

Simply, there are three entities (legs) to consider for an OAuth scenario:

• The user of a service – let’s call him Bob
• A client (a Web app, a mobile app) – let’s take bit.ly as an example client
• The server (where the service runs) – let’s take twitter as an example

How does our user Bob interact with Twitter through his bit.ly account?

1 - Bob visits bit.ly on the web, which uses a service provided by Twitter. Bob already has logins for bit.ly and Twitter.

2 - Behind the scenes, bit.ly uses it’s OAuth credentials to begin the authentication process for Bob.

Bit.ly redirects Bob temporarily to twitter.com to log in. (bit.ly never sees Bob’s Twitter password).

(The page hosted by twitter that asks if an application can access Twitter on your behalf is probably familiar to most of us by now.)

Note that twitter shows Bob what rights the app is asking for, and importantly what the app will not be able to do – in this case, see that bit.ly cannot access Bob’s direct messages or see his twitter password. 

3 - If this sign in is successful, bit.ly uses its own OAuth credentials (token) to retrieve credentials for Bob (that’s the valet key that allows bit.ly to use Twitter on Bob’s behalf).

4 - Bit.ly stores Bob’s credentials along with Bob’s account. They allow him to use bit.ly and only bit.ly to access Twitter.

Why is it important?

What if bit.ly is hacked and someone steals all the passwords. Bob will have lost his bit.ly password but the thieves don’t have his Twitter password.

The Twitter API team, knowing that bit.ly has been hacked, can decide to revoke bit.ly OAuth credentials. Now everyone that uses bit.ly can’t use twitter. This is extreme but can be important when an app is severely comprised so you don’t have to change your passwords.

On the other hand, if Bob decides to change his Twitter password, the token that bit.ly has for Bob that allows him to access Twitter is unaffected.

Bit.ly never had Bob’s twitter password so he doesn’t have to do anything at bit.ly – and he will be able to access twitter through bit.ly next time he tries.

Next time: The OAuth flow for mobile apps

HUGE: Running an API at Scale »

Thanks to all who participated in last week's strategy webinar - HUGE: Running an API at Scale.

And thanks to our speakers @sramji, @brianpagano, and @edanuff.

Here are the slides and video. We'd love more of your thoughts, insights, or questions on the api-craft forum.  

API Trends: What to expect in 2012 »

Thanks to all who participated in last week's webinar:  "API Trends: What to expect in 2012" with Sam @sramji, Anant @jhingran, and Brian @brianpagno.

Here are the video and slides. Thanks for a great interactive session.

We'd love to hear more of your thoughts and questions; please join the conversation on the api-craft forum.

RESTful API Design: API Virtualization »

Last time, we looked at why you might consider complementing your API with an SDK or code libraries. In the series so far, we've covered a lot of tips and tricks for designing pragmatic RESTful APIs. 

You may be asking -  

How do I follow all these best practice guidelines and still maintain and iterate my APIs? 

What should I be thinking from an architectural perspective in terms of implementing these best practices?

Add an API virtualization layer 

I recommend you give yourself a buffer or virtual layer between the interface on top and the API implementation on the bottom.

The app that consumes the API is on top. The API virtualization layer isolates the application and the API. Make a clean design for the app on top and turn the problem into an integration problem.

 In other words, work at integrating your design on top of the key APIs through the virtualization layer.

Don't start in code, and try to build up from your business logic to a clean API interface.

RESTful API Design: complement with an SDK »

So far in the API design series, I've looked at best practices for designing pragmatic RESTful APIs.

This time, I'll talk about complementing APIs with code libraries and SDKs.

What to do when building a UI requires a lot of domain knowledge?

This image is an example of a twitter app displayed in a newsroom.

A developer created this cool visualization - a tweet referencing another twitter user.

The Twitter API is one of the simplest app domains imaginable: the primary object is 140 characters of text. Yet, the developer who created this needed quite a bit of domain knowledge - they needed to parse the tweet, color code it, and display it in a certain way.

The API team shouldn't try to solve presentation problems with the API. That's what code libraries do. Instead

Complement your API with code libraries and a software development kit (SDK)

That is, don't overburden your API design. A lot of what's needed in scenarios like this is on the client side and you can push that burden to an SDK.

The SDK can provide the platform-specific code that developers use in their apps to invoke API operations - meaning you keep your API clean.

Other reasons you might consider complementing your API with an SDK

• Speed adoption on a specific platform - for example Objective C SDK for iPhone. Lots of experienced developers are just starting off with objective C+ so an SDK might be helpful.
• Simplify integration effort required to work with your API - If key use cases are complex or need to be complemented by standard on-client processing.
• An SDK can help reduce bad or inefficient code that might slow down service for everyone.
• As a developer resource - Good SDKs are a forcing function to create good source code examples and documentation. Both Yahoo! and Paypal have good examples.
• To market your API to a specific community - you upload the SDK to a samples or plug-in page on the platform's existing developer community.

Next time: API Virtualization

RESTful API Design: chatty APIs »

In this post in the series, let's think about how app developers use that API you're designing and dealing with chatty APIs.

Imagine how developers will use your API

When designing your API and resources, try to imagine how  developers will use it to say construct a user interface, an iPhone app, or many other apps.

Some API designs become very chatty - meaning just to build a simple UI or app, you have dozens or hundreds of API calls back to the server.

The API team can sometimes decide not to deal with creating a nice, resource-oriented RESTful APIs, and just fall back to a mode where they create the 3 or 4 Java-style getter and setter methods they know they need to power a particular user interface.

I don't recommend this. You can design a RESTful API and still mitigate the chattiness.

Be complete and RESTful and provide shortcuts

First design your API and its resources according to pragmatic RESTful design principles and then provide shortcuts.

What kind of shortcut?

Say you know that 80% of all your apps are going to need some sort of composite response, then build the kind of request that gives them what they need.

Just don't do the latter instead of the former. First design using good pragmatic RESTful principles!

Take advantage of the partial response syntax

The partial response syntax I discussed in a previous post can help.

To avoid creating one-off base URLs, you can use the partial response syntax to drill down to dependent and associated resources.

In the case of our dogs API, the dogs have association with owners, who in turn have associations with veterinarians, and so on. Keep nesting the partial response syntax to get back just the information you need.

    /owners/5678?fields=name, dogs(name)

Next, SDKs and code libraries.

RESTful API Design: making requests »

We've covered singular vs. plural nouns to label your resources, tips for search, handling errors, and more.

Now lets take a look at what some API requests and responses look like for our dogs API.

Create a brown dog named Al

POST /dogs
name=Al&furColor=brown
Response
200 OK

{
"dog":{
"id:"1234",
"name": "Al",
"furColor": "brown"
}
} 

Rename Al to Rover - Update

PUT /dogs/1234
name=Rover
Response
200 OK

{
"dog":{
"id:"1234",
"name": "Rover",
"furColor": "brown"

}
} 

Tell me about a particular dog

GET /dogs/1234

Response
200 OK

{
"dog":{
"id:"1234",
"name": "Rover",
"furColor": "brown"
}
}


Tell me about all the dogs

GET /dogs
Response
200 OK

{
"dogs":
[{"dog:{
"id:"1233",
"name": "Fido",
"furColor": "white"}},
{"dog:{
"id:"1234",
"name": "Rover",
"furColor": "brown"}}]
}

 Delete Rover :-(

DELETE /dogs/1234
Response
200 OK

Next time: chatty APIs.

RESTful API Design: how many versions? »

Engage customers where they are - at the edge of your enterprise »

Wayne Gretzky has this great quote -
A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be.

A few days ago I had a great conversation with a big retailer (and with many other customers since then) about engaging customers on the edge of the enterprise and think it's worth sharing.

Today our systems of record in enterprises are ERP and billing systems. That will continue to be the case but the problem is that all these systems of record represent where the puck has already been - what has already happened.

We'd like to predict where the puck is going to be. Today, lots of information can be gathered and analyzed quickly, and in real-time. So you can know what your customer is doing right now. Based on what they are doing now, or have just done, you can predict what they will do, or want to do next.

We're talking about figuring out how to play where the puck is going.
Imagine, you are a leading retailer. A customer walks into your retail location. If they check in (Foursquare, Facebook, Google+, . .  on their mobile device), and you have their credentials, you can know their profile and recent history.  

Where have they been? Did they just leave Walmart or Tiffany? Their spending patterns will obviously be different.

Can you know from their Twitter stream whether they were grocery shopping or anniversary shopping? Knowing this, and correlating it with what they've already bought from you, gives you a perspective and clues to their intent.

Imagine if you can make this information available to a sales person on the spot? Now you've provided that sales person a context in which to interact with the customer. They're equipped to provide a great and customized experience.

The only way to make this happen is if your enterprise is interacting with all of the different streams of data that are available and correlating this data with what you already have inside the enterprise.

Use it all - data in the core enterprise, from outside the enterprise and most importantly from the edge of the enterprise (mostly mobile apps) - to make context-sensitive interactions for customers.

That's what engaging customers on the edge is.

A look at the BlueVia Developer Platform »

BlueVia, a global developer platform initiative of Telefonica, has partnered with Apigee to provide the BlueVia API console for developers to explore and test their APIs.

So we asked them to sit down with us to tell us a little about BlueVia. In this video, BlueVia's Dan Appelquist and Jose Valles discuss the BlueVia Platform with Jeremy Perlman of Apigee.

Tune in to learn what BlueVia is, how the platform provides access to Telefonica's assets and helps developers take apps, web services, and ideas to market.

And it's BlueVia's first birthday and they're celebrating by giving developers a chance to win a MacBook Air and Samsung Galaxy Nexus. Check the BlueVia blog for details.

BlueVia Developer Platform

RESTful API Design: authentication »

This time, in this series about pragmatic RESTful API Design, I'll discuss authentication.

There are many schools of thought - my colleagues at Apigee and I don't always agree on how to handle authentication - but overall here's my take.

Let's look at these three top services. See how each of these services handles things differently:

PayPal

Permissions Service API

Facebook

OAuth 2.0

Twitter

OAuth 1.0a

Note that PayPal's proprietary three-legged permissions API  was in place long before OAuth was conceived.

What should you do?

Use the latest and greatest OAuth - OAuth 2.0 (as of this writing)

Don't do something *like* OAuth, but different
It will be frustrating to app developers if they can't use an OAuth library in their language because of your variation.

Next time: Versions - how many? Meanwhile, I'd love to hear from you over on the API Craft Google Group.

RESTful API Design: tips for handling exceptional behavior »

In the last post in this series, I talked about the top level domain (the stuff on the other side of the URL) and in this RESTful API design series so far, I've dealt with baseline, standard behaviors.

This time I'll explore some of the exceptions that can happen - when clients of Web APIs can't handle all the things we've discussed.

For example, sometimes clients intercept HTTP error codes, or support limited HTTP methods.

What are ways to handle these situations and work within the limitations of a specific client?

Client intercepts HTTP error code

One common thing in some versions of Adobe Flash - if you send an HTTP response that is anything other than HTTP 200 OK, the Flash container intercepts that response and puts the error code in front of the end user of the app.

Therefore, the app developer doesn't have an opportunity to intercept the error code. App developers need the API to support this in some way.

Twitter does an excellent job of handling this.

They have an optional  parameter suppress_response_codes. If set to true, the HTTP response is always 200.

/public_timelines.json?
suppress_response_codes=true
HTTP status code: 200{"error" : "Could not authenticate you."}

Notice that this parameter is a big verbose response code. (They could have used something like src, but they opted to be verbose.)

This is important because when you look at the URL, you need to see that the response codes are being suppressed as it has huge implications about how apps are going to respond to it.

Overall recommendations:

1 - Use suppress_response_codes = true

2 - The HTTP code is no longer just for the code

Our rules from earlier (what about errors?) change. In this context, the HTTP code is no longer just for the code - the program - it's now to be ignored. Client apps are never going to be checking the HTTP status code as it is always the same.

3 - Push any response code that we would have put in the HTTP response down into the response message

In my example below, the response code is 401 - see it in the response message.
Also include additional error codes and verbose information in that message.

Always return OK
/dogs?suppress_response_codes = true 

Code for  ignoring
200 - OK 

Message for people & code
{response_code" : "401", "message" : "Verbose, plain language description of the problem with hints about how to fix it."
"more_info" : "http://dev.tecachdogrest.com/errors/12345", "code" : 12345}

Client supports limited HTTP methods

It is common to see support for GET and POST and not PUT and DELETE.

To maintain the integrity of the four HTTP methods, I suggest you use the following methodology commonly used by Ruby on Rails developers:

Make the method an optional parameter in the URL.

Then the HTTP verb is always a GET but the developer can express rich HTTP verbs and still maintain a RESTful clean API.

Create
/dogs?method=post

Read
/dogs

Update
/dogs/1234?method=put&location=park

Delete
/dogs/1234?method=delete

WARNING:

It can be very dangerous to provide post or delete capabailities using a GET method because if the URL is in a web page then a web crawler like the Googlebot can create or destroy lots of content inadvertently. Be sure you understand the implications of supporting this approach for your applications' context.

Next time: authentication

RESTful API Design: consolidate API requests in one subdomain »

In the series so far, we've talked about everything that comes after the top level domain. This time, let's explore stuff on the other side of the URL.

Here's how Facebook, Foursquare, and Twitter handle this: 

Facebook provides two APIs.

They started with api.facebook.com, then modified it to orient around the social graph graph.facebook.com  

Foursquare has one API. 

Twitter has three APIs; two of them focused on search and streaming.

It's easy to understand how Facebook and Twitter ended up with more than one API.

It has a lot to do with timing and acquisition, and it's easy to reconfigure a CName entry in your DNS to point requests to different clusters.

But if we're making design decisions about what's in the best interest of app developer, I recommend following Foursquare's lead:

Consolidate all API requests under one API subdomain.

It's cleaner, easier and more intuitive for developers who you want to build cool apps using your API.

Facebook, Foursquare, and Twitter also all have dedicated developer portals.

developers.facebook.com
developers.foursquare.com
dev.twitter.com

How to organize all of this? 

Your API gateway should be the top-level domain

api.teachdogrest.com 

Your developer portal should be (in keeping with spirit of REST) developers.yourtopleveldomain.

developers.teachdogrest.com 

Do Web redirects

Then optionally, if you can sense from requests coming in from browser where the developer really needs to go, you can redirect.

Say a developer types api.teachdogrest.com in the browser but there's no other information for the GET request, you can probably safely redirect to your developer portal and help get the developer where he really needs to be.

• api -> developers (if from browser)
• dev -> developers
• developer -> developers

Next time: Exceptional behavior!

RESTful API Design: what about counts? »

What's the right way to do counts?

This question comes up a lot. This post talks about just one interesting way to approach it.

When I first kicked off the conversation in a recent RESTful API Design webinar, I asked developers to chime in with ideas about how best to design for counts, and the conversation over on our API Craft Google group is underway.

Check it out!

So here's where we started.

Say, you want a count of all of some resource (scoped based on whatever the requester has access to).

/dogs/count

In this way, you think about "count" as a reserved word.

This means that you can never have "count" as an identifier on a resource - doing so would return that specific resource.  Otherwise, this would get a count of the resource (dogs) available in the database.

But this is only one way to think about getting counts. I'd love to hear more of your ideas and approaches over on API Craft.

Next: your URL and the top level domain

RESTful API Design: tips for search »

In the most post in this series about Pragmatic REST API design, I talked about handling responses that don't involve resources. This time, a somewhat related topic - search. 

What about searching?

While a simple search could be modeled as a resourceful API (for example, dogs/?q=red), a more complex search across multiple resources requires a different design.

This will sound familiar if you've read the aforementioned API design tip about using verbs not nouns when results don't return a resource from the database - rather the result is some action or calculation. what about responses that don’t involve resources?

If you want to do a global search across resources, I suggest you follow the Google model

Global search

/search?q=fluffy+fur

Here, search is the verb; ?q represents the query.

Scoped search

To add scope to your search, you can prepend with the scope of the search. For example, search in dogs owned by resource ID 5678

/owners/5678/dogs/search?q=fluffy+fur

Formatted results

For search or for any of the action oriented (non-resource) responses, you can prepend with the format as follows:

/search.xml?q=fluffy+fur

Next: what about counts?

Check out the full series on Pragmatic REST API Design. Also we'd love to hear more of your comments and questions over on API Craft.

RESTful API Design: what about responses that don’t involve resources? »

In a related post in this series about Pragmatic REST API design, I talked about partial responses and pagination. Check out the full series. This time:

What's an API response that doesn't involve resources?

API calls that send a response that's not a resource per se are not uncommon depending on the domain. I've seen it in financial services, Telco, and the automotive domain to some extent.

Actions like the following are your clue that you might not be dealing with a "resource" response.

• Calculate
• Translate
• Convert

For example, you want to make a simple algorithmic calculation like how much tax someone should pay, or do a natural language translation (one language in request; another in response), or convert one currency to another. None involve resources returned from a database.

In these cases:

Use verbs not nouns

For example, an API to convert 100 euros to Chinese Yen

/convert?from=EUR&to=CNY&amount=100

Make it clear in your API documentation that these are different.

Simply separate out a section of documentation that makes it clear that you use verbs in cases like this because these calls are not resource related.

Next time: what about searching?

RESTful API Design: what about attribute names? »

In a recent post in this series about Pragmatic REST API design, I talked about formats - supporting multiple formats and working with JSON as the default. Check out the full series.

This time, let's talk about what happens when a response comes back.

You have an object with data attributes on it. How should you name the attributes?

Here are API responses from a few leading APIs:

Twitter

"created_at": Thu Nov 03 05:19;38 +0000 2011"

Bing

"DateTime": "2011-10-29T09:35:00Z"

Foursquare

"createdAt": 1320296464

They each use a different code convention. Although the Twitter approach is familiar to me as a Ruby on Rails developer, I think that Foursquare has the best approach.

How does the API response get back in the code? You parse the response (JSON parser); what comes back populates the Object. It looks like this

var myObject = JSON.parse(response);

If you chose the Twitter or Bing approach, your code looks like this. It's not JavaScript convention and looks weird - looks like the name of another object or class in the system, which is not correct.

timing = myObject.created_at;

timing - myObject.DateTime;

Recommendations

Use JSON as default

Follow JavaScript conventions for naming attributes

- Use medial capitalization (aka CamelCase)

- Use uppercase or lowercase depending on type of object

This results in code that looks like the following, allowing the JavaScript developer to write it in a way that makes sense for JavaScript.

"createdAt": 1320296464

timing = myObject.createdAt;

Next: What to do when your API returns something other than a resource.

RESTful API Design: support multiple formats »

In the most recent post in this series about Pragmatic REST API design, I talked about pagination and partial response. Check out the full series. This time:

What about formats? Should you support only one format or multiple formats?
I recommend that you support more than one format - that you push things out in one format and accept as many formats as necessary. You can usually automate the mapping from format to format.

Here's what the syntax looks like for a few key APIs

Google Data

?alt=json

Foursquare

/venue.json

Digg*

Accept: application/json
?type=json

* the type argument, if present, overrides the Accept header

Digg allows you to specify in two ways: in a pure RESTful way in the Accept header or in the type parameter in the URL. This can be confusing - at the very least you need to document what to do if there are conflicts.

I recommend the Foursquare approach

To get the JSON format from a collection or specific element.

dogs.json

/dogs/1234.json

Developers and even casual users of any file system are familiar to this dot notation. It also requires just one additional character (the period) to get the point across.

What about defaults?

In my opinion, JSON is winning out as the default format. JSON is the closest thing we have to universal language. Even if the back end is built in Ruby on Rails, PHP, Java, Python etc., most projects probably touch JavaScript for the front-end. It also has the advantage of being terse - less verbose than XML.

Next time: what about attribute names?

RESTful API Design: can your API give developers just the information they need? »

In a recent post in this series about Pragmatic REST API design, I talked about tips for versioning your APIs. Check out the full series.

This time - pagination and partial response - how do you give developers exactly the information they need?

Partial response

Take for example the Twitter API - a request for a tweet. You'll get much more than a typical twitter app often needs - including the name of person, the text of the tweet, a timestamp, how often the message was retweeted, and a lot of metadata.

Let's look at how several leading APIs handle giving developers just what they need in responses, including Google who pioneered the idea of partial response.

LinkedIn

/people:(id,first-name,last-name,industry)

This request on a person returns ID, first name, last name, and industry.

LinkedIn uses this terse :(...) syntax which isn't self-evident. Plus it's difficult for a developer to reverse engineer the meaning using a search engine. But this is the way that LinkedIn does partial selection.

Facebook

/joe.smith/friends?fields=id,name,picture

Google

?fields=title,media:group(media:thumbnail)

Google and Facebook have a similar approach, which works well.

They each have an optional parameter called fields after which you put the names of fields you want to be returned.

As you see in this example, you can also put sub-objects in responses to pull in other information from additional resources.

Add optional fields in a comma delimited list

The Google approach works extremely well.

Here's how to get just the information we need from our dogs API using this approach:

/dogs?fields=name,color,location

It's simple to read; a developer can select just the information an app needs at a given time; it cuts down on bandwidth issues, which is important for mobile apps.

The partial selection syntax can also be used to include associated resources cutting down on the number of requests needed to get the required information.

Make it easy for developers to paginate objects in a database

It's almost always a bad idea to return every resource in a database.

Let's look at how Facebook, Twitter, and LinkedIn handle pagination.

• Facebook uses offset and limit
• Twitter uses page and rpp (records per page)
• LinkedIn uses start and count

Semantically, Facebook and LinkedIn do the same thing. That is, the LinkedIn start & count is used in the same way as the Facebook offset & limit.

Say you want to get records 50 through 75 from each system. You would use:

• Facebook -  offset 50 and limit 25
  
• Twitter - page 3 and rpp 25 (records per page)
• LinkedIn - start 50 and count 25
  

Use limit and offset

I recommend limit and offset. It is more common, well understood in leading databases, and easy for developers.

/dogs?limit=25&offset=50

What about defaults?

My loose rule of thumb for default pagination is limit=10 with offset=0.

limit=10&offset=0

The pagination defaults are of course dependent on your data size. If your resources are large, you probably want to limit it to fewer than 10; if resources are small, it can make sense to choose a larger limit.

In summary:

Support partial response by adding optional fields in a comma delimited list

Use limit and offset to make it easy for developers to paginate objects.

Next time: dealing with multiple formats.

RESTful API Design: version & format in URLs or headers? »

In a recent tips for versioning post and on the RESTful API Design webinar, I described some best practices for versioning your APIs - move the v as far left in the URL as possible; place it at the highest scope (e.g. /v1/resources); and use a simple ordinal number, no dot notation.

I wanted to follow up with a little more on versioning, inspired by some great discussion on API craft (keep it coming!).

As I mentioned on the video, there is a strong school of thought about putting format and version in the header.

I think that sometimes people are forced to put the version in the header because they have multiple inter-dependent APIs. That is often a symptom of a bigger problem, namely, they are usually exposing their internal mess
instead of creating a usable API facade on top.

That's very different than saying putting the version in the header is a symptom of a problematic API design. It's not!

In fact, using headers is more correct for many reasons: it leverages existing HTTP standards, it's intellectually consistent with Fielding's vision, it solves some hard real-world problems related to inter-dependent APIs, and more.

However, I think the reason most of the popular APIs do not use it is because it's less fun to hack in a browser.

Simple rules I follow:

If it changes the logic I write to handle the response, put it in the URL so I can see it easily.

If it doesn't change the logic for each response, like OAuth stuff, put it in the header.

These for example, all represent the same resource:

dogs/1
Content-Type: application/json

dogs/1
Content-Type: application/xml

dogs/1
Content-Type: application/png

The code I would write to handle the responses would be very different.

There's no question the header is more correct and it is still a very strong API design.

Next in the series on RESTful API Design: pagination and partial response - how to return just what developers need.

Meanwhile, I'd love to hear more of your thoughts in the API Craft Google group.

Video demo: New updates to the API Console »

This week we released some updates to our Apigee API Console. Here is a short demo of:

• a new API provider page that lists all the API consoles
• a new API Resources page that lists all of the API operations 
• UI changes to the Console itself (split screen, awesome bar, and more)

For now these updates are limited to just the Twitter console, but we'll soon roll them out more broadly.

We'd love your feedback on these changes - you can drop us a line at .(JavaScript must be enabled to view this email address).  Thanks!

'

RESTful API Design: tips for versioning »

In the last post in this series about Pragmatic REST API design, I talked about designing error conditions in your APIs. Check out the full series here.

This time - versioning - one of the most important considerations when designing your pragmatic API.

Never release an API without a version and make the version mandatory.

Let's see how three top API providers handle versioning.

Twilio uses a timestamp in the URL (note the European format).

At compilation time, the developer includes the timestamp of the application when the code was compiled. That timestamp goes in all the HTTP requests.

When the request comes into Twilio, they do a look up. Based on the timestamp they identify the API that was valid when this code was created and route accordingly.

It's a very clever and interesting approach, although I think it is a bit complex. It can be confusing to understand whether the timestamp is compilation time or the timestamp when the API was released, for example.

Salesforce.com uses v20.0, placed somewhere in the middle of the URL.

I like the use of the v. notation. However, I don't like using the .0 because it implies that the interface might be changing more frequently than it should. The logic behind an interface can change rapidly but the interface itself shouldn't change frequently.

Facebook also uses the v. notation but makes the version an optional parameter.

This is problematic because as soon as Facebook forced the API up to the next version, all the apps that didn't include the version number broke and had to be pulled back and version number added.

How to think about version numbers in a pragmatic way with REST?

Specify the version with a 'v' prefix. Move it all the way to the left in the URL so that it has the highest scope (e.g. /v1/dogs).

Use a simple ordinal number - v1, v2, and so on. Don't use the dot notation like v1.2 because it implies a granularity of versioning that doesn't work well with APIs--it's an interface not an implementation.

Make the version mandatory.

How many versions to maintain?

Next time -Giving developers just the information they need -  pagination and partial response.

Please join the API Craft conversation on Google groups.

RESTful API Design: what about errors? »

In the previous posts in this series about Pragmatic REST API design, I talked about simplyfing associations, using the HTTP ? to hide complexities and optional parameters, choosing plural nouns and concrete names, and more. See the series here.

What about errors in the context of RESTful API best practices? Many software developers, including myself, don't always like to think about exceptions and error handling but it is a very important piece of the puzzle for any software developer, and especially for API designers.  

Why is good error design especially important for API designers?

Bottom line, it's about making your APIs intuitive and making developers successful.

First, developers learn to write code through errors.  The "test-first" concepts of the extreme programming model and the more recent "test driven development" models represent a body of best practices that have evolved because this is such an important and natural way for developers to work.

From the perspective of the developer consuming your Web API, everything at the other side of that interface is a black box. Errors therefore become a key tool providing context and visibility into how to use an API.

Secondly, in addition to when they're developing their applications, developers depend on well-designed errors at the critical times when they are troubleshooting and resolving issues after the applications they've built using your APIs are in the hands of their users.

How to think about errors in a pragmatic way with REST?

Let's take a look at how three top APIs approach it.

Facebook
No matter what happens on a Facebook request, you get back the 200 status code - everything is OK. Many error messages also push down into the HTTP response. Here they also throw an #803 error but with no information about what #803 is or how to react to it.

Twilio
Twilio does a great job aligning errors with HTTP status codes. Like Facebook, they provide a more granular error message but with a link that takes you to the documentation. Community commenting  and discussion on the documentation helps to build a body of information and adds context for developers experiencing these errors.

SimpleGeo
Provides error codes but with no additional value in the payload. 

A couple of best practices

Use HTTP status codes
Use HTTP status codes and try to map them cleanly to relevant standard-based codes.

There are over 70 HTTP status codes. However, most developers don't have all 70 memorized. So if you choose status codes that are not very common you will force application developers away from building their apps and over to wikipedia to figure out what you're trying to tell them. 

Therefore, most API providers use a small subset. For example, the Google GData API uses only 10 status codes, Netflix uses 9, and Digg, only 8.

How many status codes should you use for your API?

When you boil it down, there are really only 3 outcomes in the interaction between an app and an API:

• Everything worked
• The application did something wrong
• The API did something wrong

Start by using the following 3 codes. If you need more, add them. But you shouldn't go beyond 8.

• 200 - OK
• 404 - Not Found
• 500 - Internal Server Error

If you're not comfortable reducing all your error conditions to these 3, try picking among these additional 5:

• 201 - Created
• 304 - Not Modified
• 400 - Bad Request
• 401 - Unauthorized
• 403 - Forbidden

(Check out this good Wikipedia entry for all HTTP Status codes.)

It is important that the code that is returned can be consumed and acted upon by the application's business logic - for example, in an if-then-else, or a case statement. 

Make messages returned in the payload as verbose as possible

RESTful API Design: simplify associations, sweep complexities under the HTTP ? »

In the previous post in this series about Pragmatic REST API design, I talked about choosing plural versus singular nouns and concrete names over abstract. See RESTful API Design: plural nouns and concrete names.

This time we'll explore API design considerations when handling associations between resources and parameters like states and attributes.

Associations
Resources almost always have relationship to other resources. What's a simple way to express these relationships in a RESTful world?

Let's look again at the API we modeled in nouns are good, verbs are bad - the API that interacts with our dogs resource. Remember, we had two base URLs:

/dogs

/dogs/1234

We're using HTTP verbs to operate on the resources and collections. Our dogs belong to owners. To get all the dogs belonging to a specific owner, or to create a new dog for that owner, do a GET or a POST:

GET owners/5678/dogs

POST owners/5678/dogs

Now, the relationships can be complex. Owners have relationships with vetinerians, who have relationships with dogs, who have relationships with food, and so on. It's not uncommon to see people string these together making a URL 5 or 6 levels deep. Remember that once you have the primary key for one level, you don't need to include the levels above that because you've already got your specific object. In other words, you shouldn't need too many cases where a URL is deeper than what we have above - resource name/identifier/resource name.

Sweep complexity behind the HTTP ?
Most APIs have intricacies beyond the base level of a resource. Complexities can include many states that can be updated, changed, queried, as well as the attributes associated with a resource.

Make it simple for developers to use the base URL by putting optional states and attributes behind the HTTP question mark. To get all red dogs running in the park:

GET /dogs?color=red&state=running&location=park

In summary, keep your API intuitive by simplifying the associations between resources, and sweeping parameters and other complexities under the rug of the HTTP question mark.

Next time, what about errors?

RESTful API Design: plural nouns and concrete names »

In the first post in this series, Are you a Pragmatist or a RESTafarian?, I proposed that "pragmatic REST" is a design issue.

In the second post, RESTful API Design: nouns are good, verbs are bad, I outlined some of  the API design practices that work well: Nouns in URLs are good. Verbs are bad. Try to limit your API to 2 base URLs per resource.

This time, we'll explore how to pick the nouns for your URLs.

Plural nouns are better than singular
Should you choose singular or plural nouns? You'll see popular APIs use both. Let's look at a few examples  - key resources for these businesses are expressed as either singular or plural:

Foursquare
/checkins

GroupOn
/deals

Zappos
/Product

Given that the first thing most people probably do with a RESTful API is a GET, I think it reads more easily and is more intuitive to use plural nouns. But above all, avoid a mixed model in which you use singular for some resources, plural for others. Being consistent allows developers to predict and guess the method calls as they learn to work with your API.

Concrete names are better than abstract
Achieving pure abstraction is sometimes a goal of API architects. However, that abstraction is not always meaningful for developers.

Take for example an API that accesses content in various forms  - blogs, videos, news articles, and so on.

An API that models everything at the highest level of abstraction -  as /items or /assets in our example - loses the opportunity to paint a tangible picture for a developer to know what they can do with this API. It is more compelling and useful to see the resources listed as blogs, videos, and news articles.

The level of abstraction depends on your scenario. You also want to expose a manageable number of resources. Aim for concrete naming and to keep the number of resources between 12 and 24.

In summary, an intuitive API uses plural nouns and concrete names rather than abstract.

Next time, simplifying associations and hiding complexities using the HTTP ?.

RESTful API Design: nouns are good, verbs are bad »

In the first post in this series about pragmatic REST API design, I defined 'pragmatic REST' as looking at API design from the developer point of view. Now let's get into specific design practices we've seen work well.

The #1 principle in pragmatic RESTful design is: keep simple things simple.

Keep your base URL simple and intuitive

The base URL is the most important design affordance of your API.  A simple and intuitive base URL design will make using your API easy.

A design affordance is a design element that communicates how something should be used without requiring documentation.  A door handle's design should communicate whether you pull or push. But here's an example of a conflict between design affordance and documentation - not an intuitive interface!

My key litmus test for simple API design and pragmatic REST is:  only 2 base URLs per resource

Let's model an API around a simple object or resource (dogs) and create a RESTful API that interacts with dogs.

The first base URL is for collections; the second is for a specific element in the collection.

Boiling it down to this level will also force the verbs out of your base URLs.   

Keep verbs out of your base URLs

Many RESTful APIs start by using a method-driven approach to URL design. These method-based URLs sometimes contain verbs -  sometimes at the beginning, sometimes at the end.

For any resource that you model, like our dog, you can never consider one object in isolation. Rather, there are always related and interacting resources to account for - like owners, veterinarians, leashes, food, squirrels, and so on.

Think about the method calls required to address all the objects in the dogs world. The URLs for our resource might end up looking something like this:



It's a slippery slope - soon you have a huge list of URLs and no consistent pattern making it difficult for developers to learn how to use your APIs.
 
Use HTTP verbs to operate on the collections and elements

For our dog resources, we have two base URLs that use nouns as labels, and we can operate on them with HTTP verbs. Our HTTP verbs are POST, GET, PUT, and DELETE. (I think of them as mapping to the old metaphor of CRUD (Create-Read-Update-Delete).)  

With our two resources (/dogs and /dogs/1234) and the four HTTP verbs, we have a rich set of capability that's intuitive to the developer. Here is a chart that shows what I mean for our dogs.



The point is that a developer probably doesn't need the chart to grok how the API behaves. They can experiment with and learn the API without having to dig into the documentation.  

In summary: Use 2 base URLs per resource.  In your URLs - nouns are good; verbs are bad.

Next time:  how do you pick your nouns?

Please join the API Craft conversation on Google groups.

“Bigger, Better Business with OAuth” API strategy webinar - video & slides »

Thanks to all that attended yesterday's webinar "Bigger, Better Business with OAuth."  

Unfortunately, we had a Webex outage which cut the webinar audio off after a few minutes, but thanks to @sramji and @landlessness for finishing out the webinar for the video (and slides below) after the outage.

OAuth: The next Big Thing in Security »

Today at #defragcon our @sramji gave this talk making the case for OAuth as a business imperative.

(As Sam mentions on the last slide, no developers were harmed in the production of this presentation.) 

API Design: Are you a Pragmatist or a RESTafarian? »

I wanted to write some posts on RESTful API design best practices we've observed over the last year.   I covered a lot of these recently in a API design presentation but want to write more detail out.

First - let's start with our overall point of view on API design.

We advocate pragmatic, not dogmatic REST. What do I mean by dogmatic?   

You might have seen discussion threads on true REST - some of them can get pretty strict and wonky. 

 @mikeschinkel sums it up well - defining the RESTafarian. 

Our view: approach API design from the 'outside-in' perspective.

This means we start by asking - what are we trying to achieve with an API?

The API's job is to make the developer as successful as possible.   

Why? Look at the value chain below - the developer is the lynchpin of our entire API strategy.

So design the API to maximize developer productivity and success as the primary design principle.  

This is what we call pragmatic REST.

Pragmatic REST is a design problem.

You have to get the design right, because design communicates how it will be used.

So now the question is  - what is the design with optimal benefit for the app developer?

The design asethetic for APIs is to think about design choices from the developer point of view.

This is pretty much the guiding principle for all the specific tips that we'll dig into next.

Next time:   considerations in designing your base URLs.

“RESTful API Design: Second Edition”  Webinar - Video & slides »

Here are the slides and video for last week's RESTful API Design webinar, thanks again to everyone for joining.

Check out some additional tips and advice for designing Pragmatic RESTful APIs on the API Best Practices blog.

We'd love more thoughts or questions - there is a thread on the api-craft forum.  

Video and Slides - “Developer segmentation for your API”  strategy webinar »

Thanks to all that attended last week's API strategy webinar "Developer Segmentation for your API" on developer community and market size. (And thanks to our speakers @sramji  and @landlessness).

Here are the video and slides (and as always, you are free to use under Creative Commons).

I have an SDK - do I need an API? »

A frequent question from people who have an existing SDK is “Do I need an API?”  

First, what do we mean by “SDK” and “API”?

An SDK, software development kit, is usually compiled code that is distributed to developers to embed on local devices.

An API, application programming interface, is a set of operations exposed for invocation by a remote system.  (For the purposes of this discussion, we will keep the definition of an API at this high level and not delve into REST vs SOAP, good API design, etc.).

These days, SDKs are generally used when specialized processing will need to be done on the remote device.  Some examples of this might be speech recognition or 3D graphics.  Making these calls remotely could be expensive or impractical.  Some people distribute SDKs when there doesn’t seem to be a compelling reason (specialized processing).  In these situations, all would be better served by the existence of an exposed API.

An API provides a stable interface that, like any good software architecture, hides the ever changing back-end implementation from front-end developers.   API providers generally provide a dedicated web site containing the details of the API and all documentation.  Since nothing needs to be distributed, an API is logistically easier and also eases the burden of compilation against multiple OS and hardware platforms.

Even if you provide an SDK, you probably still need an API. 

(Remember, in this context, an API is a remotely exposed interface, not the set of functions in the SDK).

In today’s world, almost every application will need to communicate with remote servers, mash together other data, connect with social networks, etc…. 

While the SDK can be effective for building the application and performing specialized processing on the device, that application still needs to work with these remote services. 

Having a well managed API exposed between the applications and your back-end provides many benefits (logging, security, policy enforcement, caching, pagination, transformation,…).

There is no reason that an SDK and API need to be mutually exclusive.  While you should probably default to the use of an API whenever possible, SDKs and APIs can be friends when the situation calls for it.

If you're interested in this topic, we wrote more about this earlier in "Do you need an SDK for your API?" and we also just recently released Apigee SDKs for our own OAuth API, supporting Java, Objective C, and Javascript

“Your API is not a website!” - webinar video and slides »

Thanks to those that joined us for last week's API how to webinar presentation "Your API is not a website".   (and thanks to our speakers @gbrail and @brianpagano)

Video and slides are below!

Video and Slides - “Boss, we need an API!”  strategy webinar »

Thanks to all that attended last week's API strategy webinar "Boss, we need an API". (And thanks to our speakers @sramji, @brianpagano, and @landlessness).

Here are the video and slides (and as always, you are free to use under Creative Commons).

Join us this Thursday for our next API webinar "Your API is not a website!" by @gbrail - you can sign up here!

Trust, Safety, and the OAuth API »

At Apigee, we've been developing a secure, robust API management platform since 2005 and have been running it in the cloud since 2008.  We're proud that some of the most demanding enterprise customers like Netflix, Comcast, and GameSpy have made Apigee technology part of their API platform.

We believe that it should be easy to start working with APIs, so last week we launched a new service called the OAuth API, which takes the pain out of getting up and running with APIs that use OAuth.

The story of the OAuth API is the story of yet another function that we used to think of as occurring on a device but that is now moving into the cloud.

It marks the beginning of a trend of APIs being manipulated before they get to the client, since some things are better handled as a service. Authentication mediation is one of those things.

Apigee has built a business of using our API Gateway to publicly expose backend APIs in a safe way, such as by adding OAuth as the mechanism by which applications can be granted permissions by end-users of applications.

The OAuth API takes that model and reverses it, taking something that is complicated for developers to implement (especially across multiple APIs) and simplifying it.

But is it secure?

Should you trust your OAuth keys to a cloud provider, and to Apigee in particular?

In many ways, the cloud might be more secure, since providers do things you might never do on your own. Far too many applications never bother with encrypting users' secrets in their own databases. When you use the OAuth API, you get that for free, in addition to the API normalization and long-term maintenance wins. 

If you do trust the cloud, should you trust Apigee? We respectfully submit that it comes down to three things: technology, processes, and people. 

Technology

Our technology processes billions of OAuth transactions a day. Our Apigee Free tools—including the OAuth API—are built upon this same platform that powers 250 enterprise customers.  It is a core requirement that our servers be both security hardened and highly available.

Processes

We encrypt the secrets entrusted to our database. We run security audits. We involve our security team members throughout the design and architecture process.We use the same software, made by the same people, working under the same processes behind the products that power the APIs of the most demanding Fortune 500 companies. We use industry standard security practices.

And in the end, it's still an OAuth token that is being stored, not a user's password. It can be revoked by a user at any time, and developers can still invalidate their consumer key and secret at any time.

People

Here at Apigee we have lots of experience in API security and operations in the cloud, again as a result of working with demanding enterprise customers for years, many of whom entrust us to protect their API traffic, their organizations, and their customers.

Q&A for OAuth and API Best Practices Webinar »

Thanks to all that attended last week's webinar:  OAuth: The Big Picture (slides and video here).

There were great questions  - here are thoughts on those we didn't have time to discuss.

Q: Would you recommend only using oauth for passwords and usernames?  What about more confidential items like card numbers? (John G)

 I'm not sure I completely understand - but are you asking - if a user can authenticate to OAuth using a credit card number rather than a username/password?

That's theoretically possible but I'm not sure it's a good idea. Also, keep in mind that most credit-card authorization systems use a unique ID to map your transaction to an actual credit card, and for PCI compliance the actual number itself is stored in a PCI-compliant card system. In other words,  I'm not sure that OAuth would be solving a new problem here.

Q: Can you talk about the practices around validity periods of tokens, given that their decoupled from password validity? (Eve Maler)

 Sure. First, why do OAuth tokens need to be refreshed at all?

For one, when an OAuth 1.0a or 2.0 "mac" signature is used without the benefit of SSL, it could theoretically be cracked given enough time and computing resources. However when SSL is in use, you can't do this by just sniffing the network, although if you can intercept a request some other way then you could theoretically do this. So, one reason to refresh the token is so that it can't be cracked.

Another reason is just because as users pass their computers around, phones, etc, it's possible that they won't realize that they contain OAuth tokens. This is a much more subtle and undefined reason than the first.

Some applications give out tokens that never expire, which leads to the theoretical "cracking" example if you are not using SSL. On the other hand, I've seen applications with tokens that expire in 15 minutes, which is not very user-friendly. I don't have a specific recommendation right now and I'm not sure if I have seen another as well.

I actually think that this is a good area for some new research if there's someone out there looking for a project -- I don't think that we've quantified this well.

Q: If the user is entering the user/pass on the browser, even though it may be a trusted website URL, it is still vulnerable for some client to sniff the credentials as it is on the machine itself!! (Kaushik)

 Sure -- nothing on a computer can be completely trusted, and the person operating the machinery can be trusted least of all!

What OAuth does for a web app, is keep the passwords from the server that you are using and prevents your password from being spread around the Internet from server to server. Given the number of security breaches in which servers are compromised and user accounts exposed,  this fixes a big problem.

What OAuth does for a mobile app is keep the password from the application that you are running. I think you can trust the browser embedded in your OS more than you can trust the application built by a developer that you don't know, but I know that others here disagree. (And yes, I realize that an app can put up a fake browser screen. I wonder if such an app can get through the Apple App Store? Does anyone have knowledge about how real a threat this is?)

Q: What options does Apigee support for server-to-server API calls where we want to verify the caller is a valid caller?  i.e. request signature? (Mark Visosky)

 We can support two-way SSL, WS-Security with UserNameToken profile or X.509 profile, SAML, XML encryption / digital signatures, and of course HTTP basic authentication.

Q: if i change my password on twitter, then try to use a previously working app, it makes me reauth (Daniel)

 I didn't know that about Twitter and should have tried before the call.

Q: Thanks for the webinar, you've clearly thought a lot about authentication and security issues.  However, I cannot grant you the point that opening up a web browser where you enter your password to the service provider keeps the user's password secure.  A malicious desktop or mobile application will be able to trick the user into divulging their password by spoofing the oAuth page in some way. oAuth sign in page, I mean (Kevin)

Thanks for the questions. There are certainly other ways to trick a user into divulging their password even beyond what you suggest... That doesn't mean that we have yet abandoned passwords completely.

However, an API provider that DOES want to abandon passwords and replace them with some sort of multi-factor authentication, VPN, SAML, hardware dongle, whatever, can do that by replacing the authentication screen in OAuth with one that requires whatever that API provider wants WITHOUT requiring any change at all on the client. This is a lot harder to do once there are clients out there that have hard-coded username/password authentication.

Q: if it again asks for reauth then does that mean it is using the changed password somehow..? (Kaushik)

An API has the option of revoking the OAuth tokens at any time, and some may choose to revoke them whenever the password is changed. However, OAuth doesn't require or specify that.

Q: Isn't the full OAuth dance only necessary if a user needs to authorize another application to perform actions on their behalf?  i.e. giving authorization to bit.ly to submit tweets through my twitter account.  What about API's where we just want to validate callers are who they say they are but no end-user is involved. (Mark V)

 For server-to-server communications, two-way SSL, or "regular" SSL with HTTP Basic and a long, random password are fine ways to solve this IMHO.

For applications that need to authenticate themselves as "the app" rather than as a particular user,  you can assign each app a unique and random "application key" and include that in each request. Use SSL as well if keeping the app key a secret is important to your business.

Q: Do services need to store/control which third-parties can retrieve your OAuth credentials? Or is it generally the rule that if you expose OAuth, any app can use it? (Kenneth)

 In order to build an application that uses an API via OAuth, the application needs a set of application credentials -- that's the case for OAuth 1.0 and 2.0.

Q: i want to write an OAuth2 *provider* - what libraries are available to me? (Kevin H)

There is a long list of open-source projects at oauth.net under "Code." It's not something that can usually be just "plugged in" to a server, however, because it depends on how you want to authenticate users, where you want to store credentials, etc. At this point in time I should mention that an API gateway product such as Apigee Enterprise is one great way to get it done quickly.

Q: If oAuth requires SSL in 2.0, the request signing step becomes useless, no? (Kevin W)

When OAuth 2.0 is used with a "bearer token," then there is no secret and no request signing -- you use SSL instead.  When OAuth 2.0 is used with a "Mac token" then it is just like 1.0a.

Q: Are the specs for SSL ,SAML assertion available to be used or implemented for my server..? (Kaushik R)

Absolutely. Check ietf.orf and search for "OAuth." There are active groups working on all these specs.

Q: If you're requiring SSL, why not just trade un/pw for a token set and be done with it?  This assumes that the oAuth sign in page is security theater, which I maintain that it is. (Kevin)

@Eve but you can use any RSA spec, as long as the client and server know what to do. (Dan)

un/pw/api key or whatever? (Kevin)

Even the father of OAuth, Eran Hammer-Lahav, says using SSL as a means of authentication is broken (ie, OAuth 2.0 is broken) - (Dan)

 Yes, he is wary of SSL.  That's why he personally ensured that OAuth 2.0 also includes the "mac token" option, which is a way to use OAuth 2.0 with a signature and without SSL, just like 1.0a. This particular spec is newer and changing more quickly but it'll be an important option soon. Given the amount of work he is doing on the spec I don't think he thinks it's broken.

Q: How complex it is to implement OAuth in our own servers..? (Kaushik)

 The hard part is integrating the OAuth flow with your existing authentication mechanisms, so that the browser is redirected and all that. There is also a bunch of work involved in implementing the various authentication flows, and there are quite a few of them. I think there are much more complex things to implement but OAuth does have its challenges.

Q: Where is the OAuth token stored? is it Server side in your example Twitter, or client side BIT.LY or BOTH? I think you mentioned in case of mobile apps it is stored on mobile. (Arpit)

 It depends who is accessing the API on your behalf. The bit.ly web site, for instance,  must store them on the server since it's a web site. The Twitter mobile apps, however,  store it on the mobile device itself.

Q: So are they saying that banks shouldn't use OAuth for banking transactions? (Kenneth K)

 If a bank has a fiduciary responsibility, for example, to sign every transaction with an RSA key that is stored on a hardware key store and managed by a PKI with tight policies and procedures around key distribution, then OAuth isn't going to be used in such an environment any more than plain old username / password combinations. That's the case for financial applications that deal with individual transactions worth millions or billions of dollars.

Seeing that many of us bank on our mobile phones or web browsers using just a username and password for authentication, I think that OAuth is just fine for a wide variety of other financial apps.

Q: The oAuth login screen is security theater - I can spoof it or otherwise trick the user if I am a malicious mobile or desktop app developer.  If you can't force me (the third party app developer) to NOT store the un/pw, why make me goof around with an embedded browser page in my desktop/mobile app?  Thoughts? (Kevin)

 I think I tried to answer this elsewhere, but in general if you really want to do this then OAuth 2.0 actually includes an authentication flow that lets you get a token by simply sending the username / password to the server.

Q: oAuth gives a false sense that your transactions is secure, but the spec really does not mandate any message level security?? (Praveen)

 Well the spec strongly recommends it but the spec can't force it -- it is up to each individual API to require message-level security in the right places.

Q: Advanced talk should have a topic on usage of the "SCOPE" parameter (Vladimir)

I think that the whole "scope" thing would be an interesting seminar,  yes. Let's consider it,  but we'll need a lot of good examples!

Thanks again to everyone and hope to see you on our next webinar:   "Boss, we need an API!"

API Adoption Tip: One simple, open method »

Whether you are creating an API for internal use, partners or the wide open world of application developers -  here is a tip that could help with initial adoption of your API.

Create at least one simple, open API method.

What I mean by simple and open is an API satisfying these criteria:

• no signup required
• no authentication required
• uses HTTP GET
• returns JSON

A good example - is the the Twitter public timeline API method - no signup or authentication required.

Many API teams we work with are creating sophisticated, highly secure APIs with complex authentication and authorization requirements. The API's primary purpose is to access private, secure data.

Why would an API team working on such private stuff take the time to create at least one simple, open API method? Adoption.

If a developer can copy and paste a simple URL into his browser and see JSON coming back then he's already used your API. If he's already a user then getting him to sign up and figure out authentication will be easier for him to stomach.

How do you find an appropriate API method to create?

One place to start is your company's website. Maybe it has a listing of the products and services you sell? Or the people on the management team?

Find some information that is already publicly available and wrap an API around it. Your developers will thank you by using it, signing up and creating compelling apps.

For more on developer adoption - check out our new webinar video on Developer marketing and adoption How-Tos and best practices

Announcing the BlueVia API Console »

We're excited to announce that we've partnered with BlueVia, the new developer platform from Telefonica, to offer developers an Apigee API Console!

BlueVia's APIs gives developers access to Telefonica's network assets and data, including APIs for SMS, MSS, user contextual information, advertising and location.

The BlueVia API Console is built with Apigee To-Go, our free product to let you build, skin and embed your own API Console. With the BlueVia API console, developers will now be able to learn, explore and test the APIs more easily, and start building apps faster. Check out this video demo: 

One thing we love about the BlueVia platform is that it's committed to giving developers between 10-50% of the revenue generated by API transactions.

Additionally, it gives developers 70% of application sales and recurring subscription revenue. Helping developers make money is really important for platform providers, and Telefonica's BlueVia is also a strong example of telco innovation with open data, services and APIs. You can read more about telco APIs, the implications on network operations and frameworks for thinking about tools and technology in our whitepaper on Telco API Management. 

PCI and your API, part 3:  Know your Data »

(This is part 3 of our series on implications of PCI compliance for your API, including previous entries on Cloud computing and PCI and Pragmatic PCI. )

You’ve hardened your processes, separated environments, encrypted tables in your DBs, trained your developers and IT staff.  Then comes your audit and your auditors jump in and run a script against your DB.  Ding, ding, ding....  Left and right you start seeing things that look like Primary Account Numbers!  What? Where? How did this happen?

One of the challenges of PCI is that you’ll be focused on your cardholder environment and your payment processing tools and applications.  Meanwhile, you’ve got API’s, web forms, chat windows, log files, support tickets and any number of other places for data to hide.  Your customers, developers and employees will likely have innocently created a PCI Compliance risk. 

What other streams does data enter your system via?  Do you have a customer support or CRM tool?  Often customers, not so PCI saavy, will send you info they shouldn’t.  An email with a Credit Card number (aka PAN) in it asking for a refund, or a chat window or a comment in an API call.

You didn’t expect this channel of communication to be used to send credit card data, yet sure enough there are a magic 16 digits uncovered at the wrong time, during your assessment.

The antidote?  Know your data.  Know what information is flowing through your system, what information is stored and what might be masked on collection or display.  If you’re an API provider, this can mean watching your APIs for sensitive information passing through.  Not only PAN data, but other data can be useful to avoid storing unencrypted or storing at all.  Social Security Numbers are another sticky piece of data you’d like to avoid if possible.

Leveraging tools to help you discover your hidden vulnerabilities is one tactic as is encrypting vulnerable tables.  Eliminating those vulnerabilities is a better route.  As we shift towards an API Economy, knowing the data passing through your APIs grows ever important in achieving and maintaining PCI compliance.

Shameless Plug#1:  Ask us about Apigee PCI Gateway Policies and how we can help you know your data.

Shameless Plug #2:  For more on this topic, tomorrow we're hosting a live webinar on "Does your API need to be PCI Compliant"?   We'll send you the video if you can't make the live webinar.

PCI and APIs in the Cloud »

Previously we talked about Pragmatic PCI and applying just enough process to ensure you understand and execute your processes in a PCI compliant manner.  

What about when you inject “The Cloud” into the picture?  

PCI Compliance isn’t something that someone can sell you and even a PCI compliant environment can be misused - creating a hole in your assessment.

What is special about the cloud from a PCI perspective?  

First off, you don’t control the physical environment and therefore you are dependent upon your provider’s physical security measures to maintain compliance.  This doesn’t need to be a problem as there are numerous providers available now that can provide “bare metal” that is certified compliant for you to work from. 

You still are likely to have the responsibility of maintaining the virtual machine environment, updating operating systems, app servers, frameworks, applications and databases.   How do you offload that responsibility even further?  PCI is all about the cardholder environment and the protection of Primary Account Numbers (PANs).  

Cloud or on-premise, the guidelines are the same and keeping your exposure minimized is key to simplifying your PCI compliance.  

Isolating your cardholder environment and ensuring servers have a single purpose is key area of compliance and if you can leverage a cloud environment and a providers physical security to meet this goal, so much the better.

For more on PCI and how it might impact your API strategy, check out our live webinar this week - "Does your API need to be PCI compliant?"  If you can't catch the webinar we'll send you the video..

(And a shameless Plug: Apigee also offers a full managed PCI-compliant Cloud API management service. )

Next up, Knowing Your Data before the Auditors Do.

Does your API need PCI compliance? »

Many APIs start out serving content.   But if you eventually want to transact with your API and take credit card orders - you need to understand the implications of PCI DSS compliance.

PCI is a set of requirements that protects your customers and your business from the release of sensitive credit card information.   You don't buy technology to that makes you PCI compliant.  Instead, PCI is a process and checklist of standards that those accepting credit card data must adhere to (more on this here).   But it's important that the technology you use support and maintain your PCI compliance process.

Earlier this week we announced our Apigee Enterprise Cloud PCI - for companies that want to transact with their API, yet take advantage of the cloud for their API management and infrastructure.

On July 16 2011, we'll host a live webinar on the topic of PCI and considerations foryour API.   We'll also be writing a bit more on this topic over the next few days.

Have a great holiday.

Video and Slides: Is your API naked?  API Platform and Ops Considerations »

Thanks to all that attended last week's API Best Practices Webinar #5 "Is your API Naked? API Platform and Operations Considerations" (and thanks to our presenters @gbrail and @landlessness). Video and slides are below.

Our next API webinar, "Your API Sucks!  Why developers hang up and how to stop that" with @landlessness and @earth2marsh, is June 14th at 11am PST (sign up here!)

(And you can see all our API best practices webinars to date here)

Video and Slides: API Metrics - What to Measure »

Thanks to all that attended last week's API Best Practices Webinar #4, API Metrics - What to Measure (and thanks to our presenters @brianpagano and @landlessness). Video and slides are below.

Our next API webinar, "Is your API Naked? API Technology and Ops Considerations" with @landlessness and @gbrail, is June 14th at 11am PST (sign up here!)

Video and Slides: 10 Patterns in Successful API Programs »

Thanks to all that attended last week's API Best Practices webinar #3, 10 Patterns in Successful API Programs (and thanks to our presenters @gbrail and @landlessness). The video, slides, and Q&A is below.

Our next API webinar, "API Metrics: What to Measure" with @landlessness and @brianpagano, is June 2nd at 11am PST (sign up here!)

Questions and Answers from the Live Webinar 
 

Q: What is the best way to throttle/rate limit, software vs hardware?

A: If you’re talking about rate limiting at the API level, based on IP address, OAuth token, username, etc., then there are a bunch of good software solutions, including the Apigee Gateway. Some companies, like us, offer their software packaged inside a hardware appliance as well, but there’s nothing special that hardware can offer when it comes to this kind of rate limiting.

Q: Which OAuth do you recommend - 1.1, 2.0 or wait?

A: It’s hard to answer generally for all APIs, but the latest drafts of OAuth 2.0 are pretty mature, and include options that support a lot of use cases. A big advantage of OAuth 2.0 is that you can secure an API using just a security “bearer token” and SSL, or you can use a signature like OAuth 1.0 supported.

Q: What if I am in a big enterprise that has adopted SOAP as a standard for APIs, but I want to expose as REST.  How deadly, weird, etc... is protocol translation?

A: It’s not deadly or weird at all - lots of our customers are doing it. It’s very common to have internal services based on SOAP that meet the needs of the internal needs, but which don’t necessarily make sense to expose to the Internet in that form.  Putting a translation layer in front that can make use of protocol mediation and data transformation to make the SOAP API look like a REST API that supports JSON makes a lot of sense. The performance impact will be minimal unless you are talking about large data volumes (in the 10,000s of requests/second and above) and in that case you can horizontally scale if you need to throw CPU at the problem.

Q: Does it make sense to provide some kind of "notification" to users, which would  inform that certain part of the data is changing?

A I think it depends on the API! Twitter is a good example (I realize that we keep using them) -- they started with an API that requires you to poll, and they evolved to also offer a streaming API that pushes data out with less latency. But keep in mind that any kind of streaming or push API, including a full streaming API like Twitter, Webhooks (a little less complex), or Web Sockets, is going to be more work for a developer to adopt -- if you choose to offer this it should be in addition to something simpler to use that might offer slower performance.

Q: Does Apigee have insight to Cloud Computing Providers exposing API and the adoption of that by users (both individual consumers and businesses)?

A:  In general we’re seeing that every cloud computing provider is offering an API at some point, and that they’re getting used. I’m not sure that we have any more specifics...

Q: Do you have good example of content API ?

A:  We know of some great APIs in the works that are coming out from some leading media providers. In the UK, the Guardian newspaper is a good example of a content API.

Q: Do you recommend any framework (Python or PHP) for a content API ? 

A: There are many and we should do another webinar on this topic. I (Brian) personally like Ruby on Rails because you will get an excellent RESTful API design for free. There are frameworks for Java like JAX-RS. And for PHP there is FRAPI.

Q: Our API is a paid product. Does it make sense to make it free for developers?

A: You should at least expose some of the API methods for free - also known as a freemium model. It’s a really good way to promote adoption. Related to that it’s also good to have API methods that do not require any authentication - usually a simple read-only method.

Q: Is anyone familiar with open source rate limiting software?

A I know that some of the big guys like Twitter build frameworks on top of memcached -- it’s horizontally scalable and it has an atomic integer increment operation that works well.

Q: Do you recommend any developer community for mobile telecom services, like SMS, billing, etc?

A: Twilio has a good developer community with meetups. Also check out what AT&T is doing. They are starting to put together a solid developer community.

Video: RESTful API Design (Pragmatic, not Dogmatic) »

Recently Brian Mulloy (@landlessness) and Marsh Gardiner (@earth2marsh) hosted a webinar on API design and Pragmatic REST. The video of the recording and the slides are below.

Our next API webinar - 10 Patterns in Successful API Programs - is this Thursday, May 19th at 11am PST, with Brian and Greg Brail @gbrail. Interested in the topic? Then you should sign up now!

RESTful API Design:  Webinar slides and recording »

Thanks to everyone that attended yesterday's RESTful API Design Webinar (and thanks to our presenters @earth2marsh and @landlessness).

The slides are below and here is the full recorded webinar.   (We'll also post a video next week.)

Our next API webinar is "10 Patterns in Successful API Programs" with @landlessness and @gbrail, May 19th at 11am PST (sign up here!) 

API RESTful Design question:  Versioning number in the URL? »

We're starting to get some great questions from signups for next week's API design webinar:  "Pragmatic REST - API Design Fu". 

Here's one we'll comment on now and then talk about more next week.

Q: "For API Versioning.  Doesn't seem very RESTful to version a resource.  Can it be avoided by using implementation and specification versions described in the Sun API Cloud Documentation?"

A: Your question is an insightful one. Versioning is an open issue for REST API design in practice.

Clearly, an API will need to change it's behavior over time. As it changes application developers will need to change their code accordingly. So, versioning is inevitable.  

(caveat: Dan Jacobson of Netflix and NPR's API program has great points on what he calls a 'versionless API' in this presentation around slide 202.")

Our point of view here at Apigee is API design approach we call pragmatic REST.

The driving philosophy of our approach is: what's good for the application developer is good for the API. The Sun Cloud approach to versioning is interesting. The upside is it's powerful and will handle cases of inter-dependent API calls well.

The downside is it's complicated and as a developer starts using it, there is a lot to learn. Plus, because the versions are specified in the header, he won't be able to copy & paste it into the browser to learn it.

Our suggestion - even though it clearly violates pure REST theory - is to include the API version in the URL as a single digit number like /v1, /v2, etc. Whatever complexity is involved in making it "work" is on the shoulders of the API team.

More on this topic in our webinar - sign up here and hope to see you there.

Race conditions: Does your API TOCK TOO much? »

We've written on the topic of API threat detection before, and also outlined a "top 10 API threats" to guard against, but race conditions are another area in which APIs are also vulnerable.  

A race condition is a bug where the output is dependent on a sequence of timing of other events.   APIs are vulnerable to a type of race condition called TOCTTOU (pronounced "TOCK TOO") .   

During this crack in time, malicious users are using race conditions that have been exploited as security vulnerabilities in systems for almost 4 decades.

Historically TOCTTOU attacks have been used to take advantage of a weak filesystem API and limited security controls to OPEN, RENAME, CHANGE OWNER(chown), CHANGE MODE from READ ONLY to READ/WRITE (chmod). Applications such as vi, gedit, rpm, emacs, gdm, and many other GNU programs have been exploited to take advantage of TOCTTOU weaknesses. (See this even longer list at the NIST CVE CCE Vulnerability Database.)

What can you do to reduce race conditions in APIs?

Mediate the request and response with a single atomic event that collapses the check (authenticate) operation with the use (open) events.  If you are building atomic transactions as the design principle, then buffer the system with a checkpoint between the user zone and the service zone — aka, the DMZ.

The use of a mediation proxy in the DMZ to perform the authentication and authorization checks adds latency indeed, but it improves the quality of the system by adding load balancing, packet and message body introspection, event logging, and possibly a second authorization on the response to verify that the user is authorized to receive the object requested.

A second authorization check can also:
-    prevent accidental data leakage
-    makes it more difficult to create a race condition
-    detect if the user is no longer authorized to read the content, or if the user is attempting to access an object without execution, read or write entitlements.

So if you are concerned that your firewall is not doing enough for you, or you are worried that your DMZ has been compromised by the firewall TCP Split Handshake vulnerabilities, then consider using an extra layer of mediation between client side applications and server side APIs.

A policy enforcement point (like our Apigee API Gateway)  - behind the firewall and in front of the API service - reduces the risk of a man-in-the-middle attack.

Are all OAuth implementations Created Equal? »

We've implemented a couple dozen OAuth implementations in the past months, and no two have been the same.

We've seen Oauth 1.0a (2-legged,3-legged), Oauth 2.0 (username password, user-agent, web-server flow) with custom token types (bearer,mac, saml).  Just to name a few.

We wanted to write some posts that discuss factors we see that drive these variances and their implications.  We also want to compare and contrast each of these variances along with our own lessons learned. 

First, why the variances?  In our experience, the primary cause of the variance is the type of application that the API is targeted for, which tend to break into one or more of 3 types.

Partner Apps - typically a "closed" API available only to a select group of partners with whom you collaborate very closely to build the apps. For instance - an health insurance provider who is exposing an API that a partner can use to build some health related activity tracing applications.

Open - Browser / Mobile Apps - APIs are exposed to a large community of 3rd party developer from whom you are expecting a large scale of applications to drive innovation. Here you do not control who, how many and what kind of applications get build using the APIs.   For example, mobile applications that integrate with the social networking sites.

Enterprise Apps -  when the API is exposed for use primarily by other enterprise customers - much like a B2B scenario - to allow the customers the freedom and flexibility of building in house Applications and services that mashup one of more of the applications from multiple vendors for internal optimizations.  A good example could be an enterprise that wants to use the Sales Force API along with one other internal apps to build an internal collaboration app.

Depending on the application type above, your approach to an OAuth implementation may vary:

An API for Partner Apps will typically follow a “User name / password” flow.  In this case the end user’s  user name and password will be passed from the user to the partner app and the partner app transmits this to the origin / backend API. The key consideration here is that the partner app will have direct access to the end users credentials and is trusted to treat this information sensibly.

In the Open Browser / Mobile apps scenario the app itself is not inherently trusted and is not allowed to handle the end user credentials directly. The onus of authenticating the end user falls squarely on the origin / backend API provider. The provider (after the end user authentication is successful) will simply issue an access token to the app – this token will correspond to the set of capabilities that the end user has authorized the application for. The sequence of steps followed in this case is commonly referred to as the “user-agent flow” model.

The Enterprise Apps scenario is by far the most robust flow (and understandably, the most complex one too). Here the access token is given to an intermediary server instead of the client device and this facilitates a more secure transmission of the access token. In this scenario, it is possible to restrict the source from which the web service is invoked. This sequence of steps is commonly referred to as the “web server flow” model.

Sounds complicated?  It is. We will try to simplify each of these variants and throw some more light in a series of follow-up entries. Stay tuned!

Mapping out your API Strategy: Webinar slides and recording »

Thanks to everyone that attended yesterday's API Strategy Workshop Webinar (and thanks to our presenters @sramji and @landlessness)

The slides are below and here is the full recorded webinar. 

Join our next API webinar, "Pragmatic REST:API Design Fu", May 5th at 11am PST.  You can sign up here. 

Free Webinar: Mapping Out Your API Strategy »

Ever need to explain why APIs are so powerful to someone at your company?  Need an easier way to think about the different API strategy options?

Join us for a Webinar - “Mapping Out Your API Strategy” - this Wednesday, April 20 at 11 am PST / 2 pm EST.

It's free (of course) and you can sign up at this registration page.

Brian Mulloy and Sam Ramji will talk about the API market, and propose some ways of thinking (and cool frameworks) for how you can think about API strategy vs. your objectives.  

Punctuated Equilibrium, Celestial Navigation, and APIs - Web 2.0 2011 API Strategy talk »

Sam Ramji from Apigee, along with Dan Jacobson and Michael Hart from Netflix, recently gave this API strategy talk at the Web 2.0 Expo in San Francisco. 

It includes frameworks, best practices and lessons learned to help in thinking of your API strategy from a business model, architecture, and data perspective.   

Why Punctuated Equilibrium and Celestial Navigation? See the slides below and look out for the full video shortly. 

Shiny Shiny Monday: Two-legged OAuth, Custom Token, and more… »

We roll out big features with lots of fanfare, but the little ones roll out as they are ready. On Mondays we'll highlight recent changes here. For this week we've released:

1. Support for file attachments! Now you can attach a photo to your Twitter profile or Facebook via an API call (screenshot below). Note that you can attach via URL or by uploading from your local machine. We recommend the URL method, as that way it will be available to others if you choose to share a snapshot.
2. Custom Token - after launching Apigee To-Go, people clamored for support for custom tokens, so we added it!
3. Two legged OAuth - we've also added 2-legged OAuth, a very valid use case when people have APIs that aren't based around the concept of users granting 3rd parties permissions.

Why you need OAuth for your API or App »

I've been posting a bit on OAuth best practices recently. But I want to take a step back and talk about why OAuth is important in the first place, why it's difficult, and how to think about it for your API.

Why is OAuth important?
 
First, OAuth supports delegated authentication between web apps. That means that if web app X offers an API, and web site Y wants to use that API, then no one needs to type their "X" password into a screen controlled by "Y." This is a big deal and it's being used by a large and growing number of apps.
 
Second, OAuth does this using a security token called an "access token." An OAuth access token is a unique identifier that allows whoever has the token to get access from one and only one application to one and only one API. (The creators of OAuth describe this access token as a "valet  key," and that's a good description.)
 
An access token is different from a password, which allows whoever has the password to get access from any application to an API, because it is tied to a single application. Plus, access tokens are large and random, so they're not subject to a dictionary attack like a password, and they are hard to share. It's not going to be the case that your access token for Facebook is going to be the same one that you use for Foursquare, whereas if you're like most humans you might use the same password!
 
Third, in order to get access to an access token, an application itself must somehow be authenticated. That means that not only do users need security credentials in an OAuth world, but so do applications. This is a good thing because apps can't always be trusted, and with OAuth each API provider has the ability to revoke access to a rogue application if necessary. (Although there's no way to make it totally impossible for a determined hacker to extract the credentials from a mobile app given enough time and effort, the existence of these credentials adds a layer of security that wouldn't be there otherwise.)
 
Fourth, OAuth makes it possible to separate the assignment of API security credentials from the process of authenticating users to a web site. What does that mean? It means that if you are big company with a web app, and you want to offer an API, it's not necessary to wire the API into the web site infrastructure. Instead, the API can be implemented alongside the web site, and only link to one web page for the actual user authentication. (And even this can be done in other ways.) That means that companies of all sizes can build OAuth-enabled APIs without impacting their existing applications, and that's a good thing.
 
Getting all these things right is hard. For instance, the very first "production" version of OAuth 1.0 didn't actually do the first thing -- authentication delegation -- correctly and as a result the spec itself had to be patched (that's why we talk about OAuth 1.0a now). A lot of people looked at 1.0 without realizing this, which goes to show how tricky it is to create a new security standard from the ground up without making big mistakes.
 
Why has OAuth been difficult?
 
Right now, the OAuth world is sadly full and ambiguity. This is largely because of implementation work on the OAuth 2.0 spec, which offers the promise of an easy-to-use OAuth specification, but at the same time keeps changing. We fully expect the rate of change to slow or even stop in the next few months, but no one knows for sure, especially since it is an IETF process, which takes time.
 
However, it takes time to do a job right, and security standards that impact the whole Internet certainly deserve to be done right.
 
Unfortunately, that puts today's API implementors between a rock and a hard place. Until OAuth 2.0 is a little more stable, you have the choice of implementing the latest draft and being prepared for changes, and implementing OAuth 1.0a, which makes life more difficult for customers.
 
What should you do?
 
Be patient. It's possible to build an effective API right now using either a current draft of OAuth 2.0 (see Facebook) or OAuth 1.0a (see Twitter).
 
However, things are going to change a bit for the near future. Some would take this as a reason to give up, "sunset" OAuth, or declare it "dead" way before its time. I think that building a new security technology from the ground up that has all the advantages I enumerated at the top of this article would be a big mistake.  We have two pretty-good options today in OAuth 1.0a the latest drafts of 2.0 -- stick with them.

Pragmatic PCI Compliance and APIs: Just Enough Process »

“If the minimum wasn’t good enough, it wouldn’t be the minimum.”  - Keith W.

Wise words from one of my developers many years ago.  When it comes to tackling PCI Compliance, it is advice well worth taking.

With leaks of sensitive customer information in the news, there’s an increased focus on compliance as more services shift to cloud computing and APIs.  

If you are a merchant of any kind or deal with customer credit card information then you must be aware of PCI compliance regulations that are designed to protect consumer credit card information from exposure.

PCI compliance gets tricky as apps and services move to cloud services and APIs.   If you’re heading down the path of PCI compliance or just trying to position yourself, your APIs and your internal systems better for the future, keeping it simple will help you be successful.

First, Document your Process

The PCI Data Security Standards (PCI DSS) establish the “what” but not the “how” of achieving compliance. 

The how is up to you. But like most audit and process centric assessments, what is most important is being able to articulate what you do to support a particular DSS item - and then being able to show evidence to support that statement.

Identify all of the process standards that apply to you from the DSS and identify the proper owner of those processes as well.  Put together a simple Process Description Template that everyone uses to document their individual processes and adopt a naming convention that calls out the DSS section.  Centralize the storage of those documents and make sure everyone knows where they are. 

Just focus on capturing the “how” of your processes in as lean a manner as possible.  Your assessment team is not going to evaluate quality of the process or the documentation, only that it meets the requirements of the DSS.

With your processes documented "well enough" and easily mapped to the PCI DSS, you’ll discover gaps, strengths and you’ll make your assessors life easier and that makes your life easier.

Next, we'll talk about the special challenges with PCI in cloud computing and APIs, and practices that you can apply to reduce your risk.

Morgan Hall is a project manager with Apigee. Previously he was Director of Business Architecture at TransUnion.

Universal Design Principles Applied to APIs (Part 2 of 4) »

Following Part 1, here are the slides and a video for Part 2 of our series on applying universal principles of design to APIs.  In this part I cover 3 more universal design principles, including:

• Flexibility-Usability Tradeoff
• Hick’s Law
• 80/20 Rule
• Inverted Pyramid

And for more on API Design check out Teach a Dog to REST and REST for SQL Developers

Best Practices for OAuth 2.0 vs. OAuth 1.0 - One year later »

Since we first wrote about OAuth 1.0 vs. 2.0 a while back, lots has been happening and lots has changed. The OAuth community has made progress and made changes, and an increasing number of API providers have deployed APIs that use OAuth 2.0. (Similarly, the number of new OAuth 1.0-enabled APIs doesn't seem to be growing.)

We've been busy helping our customers implement OAuth-based APIs, and we've also been watching the process develop. Here are some things that we've learned: 

OAuth is a solution -- not technology.

In order for an API provider to support OAuth, a number of things have to happen. There must be a place where the OAuth protocol is actually implemented -- all of the different URLs for requesting tokens, issuing tokens, redirecting browsers, and so on. This part is a matter of implementing the standard and doesn't change from implementation to implementation (so, it can be delegated to a third-party component like Apigee Enterprise).

In addition, there has to be somewhere to store OAuth tokens. There is no standard for this -- some use an RDBMS, some use NoSQL, and others use a cloud-hosted service.

Finally, the OAuth solution has to integrate with a provider's "login page," so that when a user wants to get an OAuth token, they are redirected to a legitimate "login" page with a proper SSL certificate,  look and feel, and so on. All of these kinds of things can be different from one provider to another.

At Apigee, we have been working with many companies to deploy APIs built using OAuth. What we've found is that OAuth is not just a technology that you buy or download -- it's a bit of technology and a bit of integration, and to implement it at all you need a platform that can flexibly integrate with what you already have, while adding what you might be missing. 

Be careful.

Since we last blogged about it, OAuth 2.0 has been improved a bit and has changed some basic things -- for instance the name on the HTTP "Authorization" header has changed from "OAuth" to "Bearer." This is a big change, and obviously if an API provider just switches from one value to another without warning, all the clients will break.

OAuth 2.0 is up to draft 13, and with every draft it gets closer to being "done" -- but it's not done yet and I haven't seen anyone say when it will be. That means that if you implement, say, draft 13, you will likely have to change, and find a way to maintain backward compatibility for existing clients.

If that's too much, then implement OAuth 1.0a -- it's as "done" as it's going to get at this point! 

Stick with "bearer tokens" for now in conjunction with SSL.

The latest draft includes support for various types of security tokens,  and a mechanism to plug in new types. The simplest are "bearer tokens." A bearer token is just a big, random string that a client must present on every API call. Bearer tokens are simple because there's no special signature or validation code required on either end. The client is responsible for storing the token in a safe place and sending it with every request. The server is responsible for looking up the token in a database and making sure it's a valid one -- that's it.

However, bearer tokens provide no security unless used in conjunction with SSL -- otherwise, any eavesdropper in a coffee shop can snatch OAuth tokens from  the air. (Fortunately, OAuth tokens can be individually revoked without requiring a password reset.) 

Get ready to change -- or stick with 1.0a.

We said this before but it bears repeating -- if you implement OAuth 2.0, it is likely that you will have to change your servers to support newer versions, and also maintain backward compatibility.

One way to ease the pain is to put version numbers in the URLs that you hand out for the various steps in the OAuth flow. So, instead of using:

http://api.foo.com/oauth/authorize

use:

http://api.foo.com/oauth/13/authorize

because you know that there will be future versions and that they will be different...

Get ready for more options.

 Part of the OAuth 2.0 work defines some new security token types other than "bearer." Right now only "bearer" seems to be sufficiently mature from a spec standard, but this will change over the next few months. Here are some of the options:

1.       Bearer -- as we mentioned before, this is just a long, random string. The advantage is that it's extremely easy to implement on both client and server. The disadvantage is that it requires SSL for all API calls. Also, if API traffic passes through multiple endpoints such as proxies, then the proxies will be able to see the token as it passes by.

2.       Mac -- this is equivalent to the token scheme in OAuth 1.0, in that it requires both a key and secret, and uses an "Hmac" algorithm to encrypt some data on each request. The result is that the request is only valid if both client and server have the same keys, and there is no way for an intermediary to re-create the original request without a password. In other words, it is secure even if SSL is not used or even if the request is intercepted by a proxy.

3.       SAML -- this uses SAML assertions on each request as a way to establish the client's identity. This is helpful if you already have SAML infrastructure for either the client or server in your environment.

Build & Run a Web API for FREE »

With the proliferation of free, cloud services it's possible to build and run interesting mobile and web projects from end-to-end for free--including an awesome web API.

Here are 10 steps to building and running a web API for free.

1. Prepare

Get up to speed on APIs - from their economic and business model rationale to design best practices and principles to developer community practices.

 2. Design

Prioritize your resources (objects) in a Google spreadsheet and start to craft your URLs and HTTP methods. 
 

3. Build

Build your API quickly using Ruby on Rails scaffolding.  And here are some additional Rails API tips from @StefanSiebel.

Store your source code in GitHub.

 4. Deploy

After you create your Rails-based API, deploy it to Heroku.

5. Proxy

Use Apigee's Free service for API analytics, debugging and rate limiting.

6. Document

Document your API using GitHub pages.

7. Engage

Engage developers by embedding an API Console into your docs using Apigee To-Go.

8. Launch

Grab a domain name for your project from a DNS provider like GoDaddy.

Setup CNAME records for:

• GitHub docs: dev.mydomain.com
• Apigee proxy: api.mydomain.com

9. Promote

Use Twitter, Blogger and Facebook to promote your API.

10. Support

Support your community with GetSatisfaction and manage tickets with GitHub's Issues feature.

A couple of us (@earth2marsh & @landlessness) recently built an API with this set of products and had a lot of fun doing it. We'll talk more about that project in a future post.

A Short History of API Authentication (and where it’s going): from HTTP basic to OAuth 2.0 »

Part 1: The Web

In the beginning -- way back in the beginning -- the web was all about open access. Tim Berners-Lee and his colleagues focused on making information available, not on protecting it from unauthorized users.

But as time went on, and as Al Gore took the initiative in liberating the government-run Internet backbone for commercial use (really), the Web became about "e-commerce,"  and e-commerce required security. SSL matured to ensure that sensitive traffic was encrypted all the way from the client to the server and back, and various schemes emerged to allow user authentication.

The oldest and most common format for web authentication is HTTP Basic authentication. This is what you get when you visit a web site and a little browser window pops up requesting a username and password. Every web browser and every major web server supports some form of this.
 
From a web design perspective, HTTP Basic has a big disadvantage in that it's implemented entirely by the browser, and can't be customized for each site. As the quality of graphic design improved on the web, designers soon realized that they wanted not a generic little grey box on the screen, but a carefully-designed login page, with logos, disclaimers, and the like, or a discreet login button on the corner of a web page. The combination of HTTP Basic and HTML just didn't allow this.

The result was the rise of form-based authentication. This is what the vast majority of secure web sites use today. As a user, you visit a web page that prompts you for a username and password. If authentication is successful, then under the covers you are granted a unique cookie, which your web browser sends with each subsequent request. As the user you never see this -- it just looks like you logged in and now the site works -- but under the covers it is quite different from Basic.

Both Basic and form-based authentication rely on SSL to create an encrypted tunnel between the client and server. Without that encrypted tunnel, anyone snooping the Internet or the open Wifi at Starbucks can see the passwords go by in the clear. Fortunately, SSL protects against this very well, but sometimes developers neglect to use it, users neglect to ask for it (as most of us do with Facebook), and sometimes the traffic travels over unencrypted links behind the firewall of a large network.
 
The web community attempted to counter this using HTTP Digest authentication, which encrypts the password using a one-way hash so it's impossible to reverse-engineer even if SSL is not used -- but it still must be implemented by the browser and can't be designed in to a nice UI. It never took off.
 
For a higher level of security, SSL has long supported two-way authentication, which requires that individual end users request digital certificates for each site they plan to visit and install those signatures on their browsers. The overhead of issuing PKI credentials to end users was enormous and this never took off either.
 
Part 2: Early APIs
 
Some early APIs were built right on top of existing web sites built using form-based authentication. The easiest way to implement them was to use the same authentication mechanism, so API developers would create a method called "login" that returns a security token, another method called "logout," and require the security token on every API call.
 
This approach makes the API easy to tack on top of an existing web app, but it is more work on the client side and hurts API adoption. An administrator can't as easily drive the API from the script without logging in, extracting the security token from the response, making the call, and then logging out.
 
Other early APIs just use HTTP Basic authentication. It's simple, works with every client (and with every shell script based on "cURL"), but it requires SSL to be used, often leaving it up to the client to "remember" to use it. Still, it's effective as long as the user has the password for the API handy.
 
Yet others, especially Amazon, decided they wanted to avoid using SSL for performance reasons, but they also wanted to avoid using the uncommon HTTP Digest authentication. (Amazon's S3 is used to store multi-gigabyte files and SSL does make a difference there.) They chose to create their own access control mechanisms based on secret keys and in some cases, digital signatures. The result was a bit of programming for each developer starting out with AWS, but Amazon's services were so useful and cheap that it didn't matter. By now there are numerous libraries to make this process easier.
 
Part 3: APIs get formal
 
The first real access control mechanism aimed at the needs of APIs and API developers is OAuth. The idea came from a popular API (Twitter) and a defunct web site (Ma.gnolia). The goal was to make it possible for a Ma.gnolia user to access Twitter without requiring that each user give Ma.gnolia their Twitter password.
 
The result, OAuth 1.0, is like a "valet key" for an API -- it is a token that gives a single client or web site access to a particular API on behalf of one user. The client or web site can get an OAuth "access token" without ever seeing the user's password because the two web sites do a sort of "credential dance" to exchange the secret token. Once the token is issued, the user can see it or revoke it, thus taking away access from the client or site without requiring a password change.
 
Since then, OAuth 1.0 has seen a patch (it became 1.0a to fix a security flaw), and an IETF committee is working on OAuth 2.0. This new version includes some important simplifications and a wider range of use cases. OAuth has become the de facto standard for API authentication. (for more see my earlier post on OAuth 1.0 vs. OAuth 2.0)
 
The Future
 
OAuth is likely to dominate the world of APIs for a long time. With OAuth 2.0, soon API providers and users will have the option of using a simple "bearer token" in conjunction with SSL, or a signed token that can remain secure without further encryption. OAuth 2.0 is flexible enough to be used in the original "web site to API" use case that OAuth 1.0 was designed to handle, but it can also be used for access by mobile devices, where it can be important to be able to remotely revoke the access tokens that might be stored on your phone without going around and changing innumerable passwords.
 
Still, there remain problems to be solved. What about access for simple management APIs, for instance? Even OAuth 2.0 is cumbersome and when SSL is always in use sometimes a plain old password is sufficient. API providers can and should consider supporting good old Basic authentication alongside OAuth 2.0 if for no other reason than convenience -- as long as SSL and a strong password are required.
 
What about mobile apps? Is there a way for the server hosting an API for, say, a large bank to ensure that the request is coming from an official application, or from a rogue app that is attempting to "phish" passwords by pretending to be the real thing? Can we do something by combining signed applications with server-side validation, or is a secure app store the only way to protect against mobile malware?
 
The world of APIs is evolving and there's no doubt that security technology will continue to evolve along with it.

(For more on API security issues see our other entries on OAuth and API authorization, identity, and security - or get it all with our "Is your API Naked" whitepaper)

REST API design for SQL programmers »

@gbrail put together this short deck mapping SQL concepts to (pragmatic) RESTful API design.  And for more on API design - see Teach a Dog to REST and Universal Design Principles Applied to APIs

REST API Design for SQL developers

Why XML won’t die: XML vs. JSON for your API »

Last week I wrote that if you're API doesn't support JSON and JSONP - you're doing it wrong.   I don't think that's terribly controversial.

But is JSON (and JSONP) perfect for everything you need to support with your API?  Is XML dead?

JSON is especially good at representing programming-language objects. If you have a JavaScript or Java object, or even a C struct, the structure of the object and all its fields can be easily and quickly converted to JSON, sent over a network, and retrieved on the other end without too much difficulty and (usually) comes out the same on both ends.

But not everything in the world is a programming-language object. Sometimes to describe a complex real-world object we have to combine different descriptions and languages from different places, mash them up, and use them to describe even more complex things. The descriptions of these complex things need to be validated, they need to be commented on, they need to be shared and sometimes annotated with additional data that doesn't affect the original structure.

When the world gets complicated and open-ended like that, what's needed is not a programming-language-format object, but a open-ended, extensible -- umm -- markup language. That's what we have today with XML.

For instance, the travel industry (through the Open Axis Group), the insurance industry (through ACORD) and the financial services industry (through FpML) have all spent many person-years developing standards that describe what they do in XML format. Each standard comes complete with a schema, which means that any client or server can validate a document to ensure it is correct enough before starting to parse it, and which makes it easier to edit the document using one of the many of the mature tools that are available.

Sure, parsing and understanding these documents is not simple, but they do not represent simple things. The ability to represent a complex travel itinerary, a life insurance policy, or an interest-rate swap in a standards-based format is a big deal and a triumph of XML technology.

Similarly, look at HTML. (Most HTML is not XML but both come from SGML and are very similar.) HTML works because it can combine both structured and unstructured content in various ways and accept the ability to mash up different standards into one document.

In my opinion, XML will only be dead when the web has replaced HTML with JSON.

So for our APIs, let's embrace JSON -- it's small, simple, and easy to use. But when we have to collaborate on complex documents, pull information from different places, and define complex schemas to represent complex real-world concepts, let's also not forget about good old XML.

Universal Design Principles Applied to APIs (Part 1 of 4) »

The slides and video for Part 1 follow:

For more on API Design - see another presentation and video "Teach a Dog to REST: RESTful API Design Principles" 

We'd love to hear what you think - reach me at brian at apigee or @landlessness

Crossing the Streams:  Handling cross-site API requests (JSONP, CORS, UMP and more) »

My earlier post "Not using JSON and JSONP? You're doing it wrong!"  generated a lot of questions about the best ways to handle cross-site API requests from JavaScript running in the browser.

(Short intro -- the browser won't let you do it with the standard XmlHttpRequest API because of the "same-origin policy." This policy is in your browser's JavaScript engine so when you hit an application at http://www.foo.com, there's no way for http://www.badguys.com to return bad data to your application.)

This is an important topic, because if you are providing an API at api.yourapi.com, no one can use your API from inside a browser unless you implement one, and preferably more than one, of these techniques. It's also important because the state of the art is changing and changing fast.

Here's a run-down of the various techniques today, and how you might choose,  with the inevitable links to Wikipedia to help guide you: 

JSONP:

What it is: You return JSON from your API, and wrap it with a user-supplied "callback" identifier. This turns your JSON response from a JavaScript object into a function call or an assignment statement. Your user uses the DOM API to dynamically insert "script" tags into the web page to cause your API to be invoked, then executes the result as a JavaScript function call to create the object and start using it. 

Why it's good: Works on every browser, and asynchronously like every JavaScript network resource to boot. If users use a library like jQuery it is very easy to use.

Why it's bad: Only works with JSON,  although if you read my last entry then your API returns JSON anyway! It only works with GET, so you have to pollute your beautiful and philosophically-correct REST API by adding something like an "action" query parameter -- this tells your server that even though it just got a "GET" call, it should behave as if it were a POST, and you can't upload data with GET.

Finally, JSONP is really a hack around the security sandbox in your browser. As a result, it lets your JavaScript code download executable code from  another domain and run it inside the browser. If the user of your API trusts you, then that's great, but otherwise they're placing the security of their web app in your hands.

Aside from offering JSON as a response format with callbacks, you can also specify XML as the response with JSONP-X.

CORS: 

What it is: Cross Origin Resource Sharing is a W3C spec that allows a web service to specify that it's OK for it to be invoked from any domain. The API provider implements it by returning a special set of HTTP headers with every API response, and also by implementing an HTTP OPTIONS "pre-flight" request that lets the browser check first to see if a request is going to be OK. The actual spec is awfully complex but what you actually need to know in most cases isn't all that bad.

Why it's good: Because it lets the API consumer easily say, "it's OK, browser -- go ahead and make that API call." Once it's implemented, browsers that understand the CORS spec will automatically open a hole in the cross-origin policy as part of the standard XmlHttpRequest API -- the programmer of the web app doesn't have to do anything special. Plus, it can return any kind of data, not just JSON, and it works with all HTTP verbs. 

Why it's bad: It's pretty new and not all browsers implement it. Firefox 3.5, Chrome 3, and Safari 4 has it. IE8 does too but using a slightly different API. And support in many smaller mobile browsers (like Opera) is sketchy or nonexistent.

In other words, CORS is super-duper awesome as long as you don't care about people who run IE -- whether that's important to you is up to you

UMP:

What it is: UMP is a re-boot of CORS -- it handles 90 percent (or more) of the reasons an API would want to implement CORS without 90 percent of the complexity. It's also backward compatible in that it uses many of the same HTTP headers as CORS.

Why it's good: Like CORS, it allows an API to easily make itself available to web browser clients with a minimum of fuss. Better yet, all the API has to do is set a few HTTP headers, without also implementing an OPTIONS request.

Why it's bad: It's even newer than CORS. (New enough that it doesn't even have a Wikipedia page.) I have been told that recent versions of Chrome support it but it's not clear where else it can work at this point.

There's more to compare CORS and UMP here

Cross domain files

What they are: Flash and Silverlight also have a same-origin policy, but they do not work the same as the JavaScript engine and the techniques above do apply. What they do support are special files, namely crossdomain.xml (for Flash) and clientaccesspolicy.xml (for Silverlight). These function the same way as CORS and UMP headers in that they tell the client when it's OK to invoke the API across domains.

Why they're good: If you have Flash or Silverlight clients for your API, you need these too so that your API can be invoked from  those environments.

Why they're bad: If you don't want or need to support Flash and Silverlight clients then you don't need them.

So which one should I pick?

 If you want your API to be adopted in as many places as possible then you should implement all of them! Seriously:

1.       Always make it possible for your API to return JSON.

2.       Support JSONP as much as you can (since you can't use it to upload data that doesn't fit in the URL) and leave it up to the users of your API to assess the security risks

3.       Support CORS by returning the Access-Control-Allow-Origin header on all API responses and implementing the OPTIONS verb as specified in the CORS spec.

4.       If you do the above your API will also support UMP.

5.       Return a crossdomain.xml and clientaccesspolicy.xml file from the top level of your API domain for Flash and Silverlight clients.

Whew!

At Apigee, we have done the above for users of our Apigee Enterprise product, and since it's so flexible we can change what we support as the standards change.

Not serving JSON AND JSONP? Then you’re doing it wrong! »

If you've used an API recently, you've probably seen that the popular APIs out there support JSON. JavaScript Object Notation is a standard defined a while back by Douglas Crockford from Yahoo. It uses a subset of the JavaScript syntax to simply and effectively describe an object.

In the last few years, JSON has taken its place alongside XML as the de facto way to describe API data. Today's leading APIs support JSON in addition to XML, and an increasing number support only JSON.

JSON is popular because it's simple. Programming-language objects map to and from JSON in a straightforward way that everyone can understand. You need a "JSON parser" to convert JSON into an object (unless you're working in JavaScript) but you don't need to know much about it other than how to make it go.

If you are thinking of building an API, JSON support is critical. Here's why:

JavaScript. JSON is JavaScript. That is, a "JSON object" is literally a small fragment of JavaScript that represents an object and its sub-objects. That means that creating a real JavaScript object based on some JSON text is simple and fast. Web programmers love JSON.

JSONP. This lets JavaScript running inside the browser invoke APIs that reside on a different host on the Internet. This doesn't sound like a big deal but it is actually huge because all browsers implement a "same-origin policy" that otherwise makes this impossible. JSONP is hard to implement, but libraries like jQuery make it easy for the client if the server already supports it.  If you're not serving JSON AND JSONP you're doing it wrong!

Smaller. Smaller is better, especially in the mobile environment, and since JSON doesn't "say" every field name twice like XML does, JSON output is a lot smaller than XML.

Less complicated. JSON is free of namespaces, attributes, multiple "text" nodes, and other complexities of XML. The result is that JSON parsers exist for every language, they're small, and they're fast. Furthermore, if you need to write your own, it's not complicated. The same goes for security -- all that's necessary to prove that a JSON document is valid JSON is a simple regular expression check, which is easily available in nearly every programming environment.

Tools. An increasing number of tools support JSON. JSON support is not ubiquitous yet, but at the rate JSON is gaining it will be soon.

Many APIs now support XML and JSON - like the Twitter API, where JSON is the default. Some APIs support only JSON (like Foursquare's V2 API).

But JSON isn't  for everything. Next up: Why XML isn't dead yet!

Need API and developer adoption?  Try a Hack Day! »

Want to launch your API with a bang?  Or get more internal adoption?

Hack Days (or Hackathons) give developers a day off to build anything they can dream up. No rules. At the end of the day, developers demo for glory and free beer.

Check out the recent FourSquare Hack Day. LinkedIn had a great one. And there are independent Hack Days.

If your API is only available to developers inside the company - even more reason. Why?

- Get the word out - Hack Day is a high profile way to drive adoption and build grass-roots excitement internally.

- Showcase innovation - with great new Hacks and product ideas you'd never have thought of.

- Get those example apps! -  need example apps and code on your developer portal?  Don't DIY or pay someone -use the best Hacks!

- Eat your own dogfood -  Find bugs before releasing your API to partners.

-Get recruits! -   Need good hires for the API team but don't want to ask?

Here are a few tips:

- Dedicate time  - give developers a full day with no distractions.  

- Team up!   Encourage developers to team up and mix skillsets.  

- Demo for glory.  Get your CEO and execs to judge demos at day's end.   The Hack doesn't have to be finished.  Prizes are nice but recognition might be enough.

- Feed me!  Beer and free food can take a OK Hack Day and make it a *great* hack day.

- Start small, but rinse and repeat.  Your first Hack Day might be small and a little rough.  Buy the 3rd or 4th time - you might have Beck playing on your lawn.

- Productize!  Have a labs page or a way to turn great hacks into real product.  

- Provide tools.  Give your developers an API console to learn your new API and debug API problems.

(Thanks to SimpleGeo for the image above)

Two Ton API client:  creating a generation of car hackers »

Aside from curb weight and operating speed - there's not a lot of difference between an iPhone and a fine automobile.  Both are at their most innovative when people drive them with APIs.

Why? See Brian Mulloy's (@landlessness) O'Reilly Detroit Ignite session - "Two Ton API client."  Both a talk track and the slideshare are below. And drive carefully...

Using your existing systems to create new value with APIs »

Companies need to wring value out of their investments; find new ways to serve customers or create new value with their current systems, products, supply chains and partnerships. 

I keep going back to water analogies.  My house has a plumbing infrastructure that was paid for a long time ago.  I recently found new value in that infrastructure, an automatic sprinkler system.  Again the system reminds me of an API.  Through this one common interface I can connect many endpoints (sprinkler heads) and my yard has never looked better.  I have some endpoints that cover the lawn, some that concentrate on shrubs, others take care of the plant beds.  The point is that I tapped in to my existing water pipes and found new value in a 35 year old system. 

Can you do the same with your enterprise systems?

Let's take a common strategy like a mobile device strategy.  First we have to understand what information customers need when using a mobile device like a smartphone.  Certainly location and product come to mind.  Maybe information about their account or orders.  These little gems of information can be exposed through the API interface and carried to many different endpoints; today a smartphone, tomorrow a tablet PC or connected car.  Let's call these 'functional APIs" since they carry a function to the user through a mobile device. 

The next step to realizing a functional API is to map the data elements and functionality of the new API to their location in the existing systems.  These data elements may map to databases, ERP systems, other internal services, etc.  In fact, a single API may map to one or more internal systems.  Any holes that are discovered will have to be filled. 

For example, if you are designing an API and define features or data that aren't available through existing systems, you will have to provide that new system or new data set.  But, in the interest of embracing the legacy systems, you should be able to reuse the majority of what you have by extending it outward through the new abstract API.

Leveraging what you have through an API can reduce your development cost, give you speed to market and also give you more flexibility in the future to integrate with more devices and partners.  Leveraging my existing residential water system yielded a "yard of the month" award, perhaps this strategy can beautify your business as well.

Next we'll talk about the details of integrating with existing systems.   And ots on this in our recent whitepaper "APIs: A Profit Interface - using APIs to grow your business.

Developers Hate Marketing: Launching and marketing your API »

Once you build your API, will developers come?

We've picked up some good stuff from our customers on this topic, many of which we've posted as developer adoption best practices on this blog.

So today we've rolled these in a new whitepaper with our friends at Evolved Media  - Developers Hate Marketing:  Attracting Developers to your API.  

Topics include some thoughts on:

• what do developers expect?
• do's and don'ts in launching your API
• patterns in successful developer programs

You can find it here, we'd love to hear what you think.

Paypal API sandbox console demo interview with @marshacollier »

 Marsha Collier and Sam discuss the API console at the Paypal innovate show this week.  Check it out!

How to learn and explore PayPal’s Adaptive Payments APIs with Apigee Tools »

At PayPal's Innovate 2010 event yesterday, we launched our API Explorer for the PayPal Adaptive Payments APIs, including their new ones. Our own Sam Ramji gave a great demo on stage, but in case you couldn't be there, here's a quick overview of how to get started:

New API Console for Learning and Debugging the PayPal API »

In time for PayPal's Innovate conference, we've added the PayPal API to Apigee's API console. You can now easily view requests and responses to the PayPal API, test and debug, and share your results.

The PayPal API lets developers easily leverage its payment platform in apps, mobile and mashups. The Apigee PayPal console interacts with the PayPal sandbox environment to let you get up and running without needing to transact payments. 

We've made getting started on the API a single-click task by pre-populating custom tokens and sample values for all request parameters, but when you are ready to get your hands dirty, here's a few things to get you going: 

• The Apigee console supports the PayPal API's custom token authentication scheme. The PayPal API uses multiple tokens, so we let you specify them as a custom token so it is easy to stay authenticated--you only need to authenticate once for the entire session.  
•  You can easily configure the request parameters by using the Apigee parameters pop-up. This is designed to support simple experimentation to figure out what you'll need for your apps and integrations. 

Let us know your feedback and check out our other consoles for the Twitter, Facebook, and SimpleGeo APIs. And if you're hanging out at the Innovate conference, ping us on Twitter! We'll be hanging out at the hackathon.

Do you need an SDK for your API? »

In our last couple API strategy workshops, we've had good discussions on a common question for API providers - do you need to offer an SDK (software dev kit) for your API?

In this context we defined an SDK as going beyond offering an API to include platform-specific code that developers use in their apps to invoke API operations, also including the source code, and documentation that developers might need. Here's what we came up with:

Reasons to consider an SDK:

Speed adoption on a specific platform - for example Objective C SDK for iPhone.  Lots of experienced developers are just starting off with objective C+ so an SDK might be helpful.

Simplify integration effort required to work with your API - If key use cases are complex or need to be complemented by standard on-client processing. 

An SDK can help reduce bad or inefficent code that might slow down service for everyone.

As a developer resource - Good SDKs are a forcing function to create good source code examples and documentation.  (See some good examples here and here. )

To market your API to a specific community - you upload the SDK to a samples or plug-in page on the platform's exisitng developer community.

Reasons you might not need an SDK:

If your API is designed for adoption -  standards-based, well documented, etc.. developers may be able to get rolling without a client SDK.

Resources - it's tough to do a SDK for each platform you target, you'll have to pick carefully. Also, it can be time consuming and expensive to create SDKs (tip: maybe packaging up internal samples or test cycles can kickstart things)

Maintenance and versioning - convincing clients to upgrade to the newest version can be rough. Also, capturing the "local flavor" of each language you support can be a challenge.

Complexity -On each platform there might be use cases you don't expect, like keeping application-level secrets off clients, debugging, etc.

More is not necessarily better - fewer, more well-documented, code-level samples can often be the best resource. Facebook provides Android, C#, iPhone, JavaScript, PHP, and Python libraries (all documented differently), but Twitter supplies none. Facebook has more resources but still struggles a bit at times.

Our view: whenever possible, forget the SDK, and spend the time making your API better and more useable, and give good API resources such as example code.  

Agree or disagree? 

Apigee Enterprise 3.8: OAuth 2.0, streaming API, developer keys, and more. »

We're excited to announce the general availability of Apigee Enterprise 3.8!   Some highlights:

OAuth 2.0 – Apigee Enterprise 3.8 is Oauth 2.0, draft version 10, compliant.  If you are running our OAuth 1.0a solution, you can migrate to OAuth 2.0.

Developer Connect - a Drupal-based portal you brand with your look-and-feel and use for publishing API documentation, content, and community management.  Give your developers their own API testing console and other social publishing features such as blogs and forums.   See more developer portal details here. 

Key Manager Service - handle the complete lifecycle for API keys and security tokens (such as OAuth) -  including creation, validation, revocation and expiry of keys and tokens. This service now includes auditing capability and RBAC model for access control.

Streaming API  connections - proxy HTTP responses that use long standing connections. Enable at endpoint level or on-demand, using the context variable.

Distributed Quotas - enforce API usage limits, or quotas,  across cluster nodes. 

Cache and persist API response data on hard-disk, eliminating the dependency between cached data and available RAM. 

REST API for Apigee Analytics - we've updated this to support custom reports with various filters and date ranges. 

Citrix Xen Server 5.5 support -   if you choose to deploy Apigee on-premise, we've added support for another one of the most popular virtualization platforms.

And more..  enhancements to our API Debugging Console, Logging, Load Balancing, Virtual Host and more. 

Enterprise customers should expect us to be in touch soon about 3.8 and all the details are in the release notes - but email us at .(JavaScript must be enabled to view this email address) with any questions!

Check Out Facebook’s New Places API »

How to explore Facebook's new Places APIs in the Apigee Test Console:

Don't have a minute to watch the video? Start by viewing the checkins at the Cookie Jar.

This Week in APIs- July 24-30 »

We might be a little obsessed with APIs here at Apigee- here's what made API fanaticism great this week. 

The Khronos Group released version 4.1 of its OpenGL graphics API, which promises to better support different hardware and software configurations and provide a better 3D graphics experience. It's increasingly important for APIs to address the proliferation of devices and fragmentation of mobile phone OSes, so we're happy to see an open standard making progress- plus, it's 3D! 

Google announced that they will be working with developers on supporting applications with check-in functionality on its Places API, launched this spring. The check-in app space is increasingly competitive and we're excited to see what devs come up with.

Twitter has begun testing a new streaming API that lets desktop apps automatically update their streams in real-time. The User Streams API is being utilized by TweetDeck and Echofon. By transitioning to streaming, Twitter will cut down on rate limit and latency problems- major progress! We're looking forward to seeing this roll out more widely.  

Hot New APIs- 

• Video search engine blinkx unveiled a new mobile API to give access to its huge collection of MP4 videos- including those from the AP and Lonely Planet.
• TypeKit, a font startup, unwrapped a new API that lets developers generate kits from its library and build them into their applications. Very exciting for all you typography geeks! 
• Gamerang is launching a new site to focus on the social side of gaming. As Chris Crum over at WebProNews says, Gamerang is like Netflix for video games- and its new API makes it easy to incorporate the gaming news and social content into sites and apps. 

Did we miss anything? Shoot us an @ sign on Twitter!

This week in APIs-June 27th-July 2nd »

All the best info on APIs this week.

Mozilla's JetPack API Survey: Mozilla is getting ready to release Mozilla 4 beta, and their most anticipated feature is the JetPack SDK. Mozilla did a survey to ask which APIs developers want most. The winner? Page Mods, which allows developers to alter webpages. More browser innovation with APIs is on the way.

Alcatel-Lucent acquires Programmable web: Alcatel-Lucent acquired API news and directory site ProgrammableWeb this week. A smart move and we're exciting to keep seeing great intelligence on the API economy from one of our fav sites.

E*Trade Open API: Online trading giant E*trade just announced their first Open API for third party developers! E*trade is hoping their API will lead to innovative investing applications- we're eager to see what they come up with.

New APIs - Every week a new API is released! Here are three we <3 :

GroopBuy-GroopBuy is a company that hunts down local deals for people. It was launched this year, and they just released their GroopBuy API. Companies that look for local deals and coupons are becoming hot, and developers using their APIs is an important part of that.

Drumbone-Sunlight Labs has just released their Drumbone API for data about legislators, bills, and roll call votes. Just adds to mounting evidence that the government is moving towards Web 2.0!

NakdReality-NakdReality is a company specializing in instant on-location search and augmented reality service. They also just opened up their API for developers, adding to the Geolocation API craze!

API Tip of the Day: Reduce the Learning Curve »

Be consistent when building your API, and try to use standard conventions when possible. This will reduce the learning curve for 3rd party developers.

Hooks, Sockets and Firehoses: Streaming API Technologies Getting Us to REALLY Real-Time »

If you rely on Twitter to see what your friends are having for lunch, waiting 24 seconds to get an update is fine. If you're a stock trader who could have lost a few thousand dollars or more while reading this post, 24 seconds is a very long time to learn that your favorite company just got acquired.

Real-time runs on APIs as much as it runs inside a web browser. This means that the "real-time web" isn't always "real-time"- a web service API is typically a request-response technology, requiring that the client makes a request of the server, waits for a response, and then processes it. This can add critical seconds or milliseconds.

Earlier this month I attended the ReadWriteWeb Real-Time Summit in New York City. In an "unconference" session on streaming API technologies, we discussed 3 technologies bringing us closer to REALLY real time:

HTTP streaming- This technique uses the HTTP protocol, but instead of responding to a request with a single piece of data, the server responds with many small pieces of data that are published as soon as they are available. It's as if you're using HTTP to retrieve a very, very, large file -- the server just keeps pushing out data, one piece at a time. Twitter uses this technique in its Streaming API.

Since HTTP Streaming doesn't work quite the same way that most HTTP-based services work, the program that reads the stream has to be written to process the incoming data, handle reconnects, and so on. It's also difficult to use in a web browser without doing some JavaScript trickery; HTTP Streaming is mainly suitable for server-to-server communication.

WebHooks is a real-time tech with a simple concept: whenever something happens on server A, it makes an HTTP POST to server B and sends it the update directly. Like HTTP Streaming, it only works for server-to-server communications, but unlike HTTP Streaming the barrier to entry is very low for the programmer - anyone who can write a program or script that can process an HTTP POST can consume a WebHook. Unlike HTTP Streaming, however, a WebHook uses one HTTP request for each event, which means that with today's servers a WebHook is going to top out at several thousand events per second, whereas HTTP Streaming can theoretically process many more events.

WebSockets, unlike HTTP streaming and WebHooks, is designed for real-time streaming to the client- useful for creating real-time apps that run in the browser. WebSockets allows a client and server to use the HTTP protocol for some initial interchange, and then agree to communicate bidirectionally. The result is just like a TCP socket -- the client and server can send whatever data they'd like, in any order, in either direction, for as long as they want. WebSockets also supports JavaScript, ensuring you can easily build a high-fidelity, near-real-time application in a web browser using Internet-standard protocols. WebSockets is part of the HTML 5 family of standards, already implemented in Google Chrome and Apple Safari, and will eventually come to all the browsers.<;It was good to see Kaazing, who is leading the development of WebSockets, hanging out at the RRW Real Time Summit. With some existing technology out there and some new stuff on the way, the stage is set to build a real-time web that's a bit closer to real-time.

With some existing technology out there and some new stuff on the way, the stage is set to build a real time web that's a bit closer to real time.

This Week in APIs- June 7-11, 2010 »

Check out the top news in APIs this week!

S3 Import/Export API- Amazon Web Services added an Import/Export API for its Simple Storage Service (S3), an infrastructure service for storing data. The new API makes it easier to move large amounts of data in and out of Amazon- a great move for data portability.

Twitter API Meetup- Twitter held a meetup in its San Francisco offices this week to talk about the Twitter API, showcase some cool apps and celebrate Apple's WWDC. Twitter's Taylor Singletary presented on XAuth, a new authentication method for APIs. Twitter will no longer allow support for basic authentication at the end of the month so Twitter devs, it's time to brush up on the new method! The meetup made it very clear that the XAuth standard was developed to be used across the API community and promises to make authentication safer and easier, so keep an eye out for adoption.

APIs in the world of PaaS- Over at CloudAve, Krishnan Subramanian has a post and some love for Apigee on why APIs are the oxygen of Platform as a Service offerings- check out the article for some stark examples of how APIs can mean the difference between success and failure.

New APIs! A number of companies debuted new APIs- here are a few we thought were especially cool:

• HubSpot, which launched a Lead API to connect HubSpot's lead capture tools to other CRM software. Excellent example of how APIs can be used to connect tools and unleash new functionality. 
• The New York Times introduced a Most Popular API to track the most viewed and shared content- fodder for mashups here! 
• Microsoft introduced its Trial API that allows developers to include free trial versions of their apps in the Phone 7 App Market- more signs that the mobile API world is maturing. 
• As always, ProgrammableWeb's Newest APIs list is a great way to keep on the top of the all the new delicious APIs out there. 

Anything important we missed this week in APIs? Shoot me an email at .(JavaScript must be enabled to view this email address).

API Virtualization »

We've seen an architectural pattern emerge over the last two years that deals with the expanding variety of computing devices and the exploding number of APIs used in modern applications. This pattern is API Virtualization - applying a virtual layer above the API itself to deal with the different concerns that come into play when delivering an API to different device and machine endpoints as well as across a range of different classes of business relationships.

I gave this presentation at GlueCon as a Prezi. The prezi is available here, and I've put the audio with it in the screencast below.

One of the big shifts that has occurred in application development is the move from libraries to APIs for use of pre-written code. APIs enable access to not just logic but data and the the computational power required to render that data appropriately. The Facebook and Twitter APIs are good examples of this, but there are many more including Paypal's X.com, Ebay, Amazon, Netflix, Sears, Blockbuster, Best Buy, MTV and Google.

While applications like Siri which use 20 or more web APIs are still unusual, most applications use several APIs to deal with access to content, social networks, events, identification, payment, and other concerns. These APIs change frequently, as do the applications that consume them, resulting in a web of dependencies and causing chaos in the development lifecycle.

API Virtualization enables these different concerns and dependencies to be isolated from the application or API provider, resulting in a cleaner development process and higher-performing applications.

Where will your customers be this Christmas? »

Keith Morrow has a great article on TechTarget series on how an API can create a disruptive impact on a retailer's ability to reach customers beyond traditional web browser channels.

One key point is that an API can be driven by a relatively small team. 

Yet an API has a disproportionate impact on the business, and this impact scales with low marginal costs.  

This can be an especially effective strategy for retailers that might live in world of thin margins, limited resources, and the constant pressure of the peak holiday season. 

For more on Keith's retail API experience, we commissioned him to write a Retail 2.0 API Strategy whitepaper discussing Retail API opportunities, challenges of launching with tight resources, and best practices.

You can download the whitepaper here.

WWSJD- What Would Steve Jobs Do? Lessons for the Age of APIs »

API Scalability: Cache large ‘chunks’ in the API response »

Highscalability.com has an interesting piece on caching as "secret sauce" for good web performance.

Facebook and Twitter are cache assembly lines -- every web page and API request is served up by many calls to various caches at different levels - assembling the final result from many different chunks. At this scale there is almost no other way to deliver reasonable performance.
 
For APIs - what's the largest chunk of all? The entire API response.

APIs lend themselves nicely to caching responses because it is often easy to identify the cache key. If you follow the REST pattern each URL maps uniquely to a resource that has a well-defined lifecycle.

If you're working on the next Twitter, you will need many layers of caching to deliver great performance, including the filesystem and database caches, and then you will add additional caching layers using products like memcached or Coherence.  But a final tier of caching for the API responses themselves can only help performance, so consider:

• Can API analytics data show the most common or slowest API calls?
• Are there slow API calls that return the same data over and over?
• Can you design using a REST pattern to make it easy to identify individual cache items?

There are a number of ways to speed up calls that are caching candidates, such as memcached or something similar.

Another way is to use an API proxy. A proxy that is sitting in front of your API servers is an efficient and easy way to add an additional caching layer without making any changes to the server tier. We have helped a number of customers drastically improve the performance of their APIs by inserting such a caching tier without touching the clients or servers. 

And caching isn't just helpful if you are building an API. Applications that consume APIs, whether they are running on another web server or on a smartphone, can set up their own API proxy so that responses from many devices are cached in a central place, even if the APIs that the application is depending on can't be depended on themselves to return good performance.

(thanks to jules:stone for the photo!)

A Look at HipChat’s API »

While Twitter and other web 2.0 applications have changed public communications, companies have been clamoring for similar messaging systems for internal use. Used by over 1,000 companies and teams, HipChat is a great solution with deep features to help you supercharge your company's communications. You can create chat rooms for meetings, use it on your mobile phone, have a one-on-one chat with anybody in your company, and securely share files. HipChat recently released their very own API, which is well-designed and documented. Here's a quick demo of what makes HipChat great:

Since Garret Heaton of HipChat uses Apigee, we asked him to answer 5 questions for us:

What do you wish your API would do that it doesn't do today?

Garret:Obviously it'd be nice to have support for all the features users have access to when using HipChat: changing room topics, sending invites, uploading files, creating rooms, etc. It'd also be awesome to have a streaming API (and open up full access via XMPP) so users could build bots and other real-time services.

If you could send one message about building your API back in time to yourself, what would it be?

Garret:Keep it simple! The first versions of our API had a more complex interface for passing auth information, desired response format, and entity IDs. Our initial testers were often confused. It's important for new users to be able to test the API easily using cURL.

What insights into your app or other benefits have you gotten from working with Apigee?

Garret:The 'Response Time' data is really helpful for identifying slower requests. It's really nice not having to build this ourselves. The new debug console is also incredibly helpful when you need to see what your exact request/response data looks like.

What new Apigee feature would you most like to see from Apigee?

Garret:I know you're already working on it, but HTTPS support for mapped domains so our users can use HTTPS [NB: It's coming very soon, we promise!]. Also, a way to create a test console for our API. I think our users would find that very helpful.

How would you like to see developers make use of your API once it is exposed?

Garret:It'd be awesome to have plugins for sending HipChat messages available for lots of other software: Capistrano, Nagios, Subversion, Git, Perforce, Trac, Get Satisfaction, CoTweet, and many other cloud-based tools companies are using these days. We're also looking forward to seeing the creative things people will build that we haven't even thought of yet!

If you're looking for a great collaboration tool to improve your internal communications, be sure to check out HipChat. And keep telling us what Apigee features you'd like to see next so we can provide a better service for you!

API Versioning Hell? Think API Virtualization »

We've seen an architectural pattern emerge over the last two years that deals with the expanding variety of computing devices and the exploding number of APIs used in modern applications.  This pattern is API Virtualization - applying a virtual layer above the API itself to deal with the different concerns that come into play when delivering an API to different device and machine endpoints as well as across a range of different classes of business relationships.

Sam Ramji gave this presentation at GlueCon as a Prezi.  The prezi is available here, and we've put the audio with it in the screencast below.

One of the big shifts that has occurred in application development is the move from libraries to APIs for use of pre-written code.   APIs enable access to not just logic but data and the the computational power required to render that data appropriately.  The Facebook and Twitter APIs are good examples of this, but there are many more including Paypal's X.com, Ebay, Amazon, Netflix, Sears, Blockbuster, Best Buy, MTV and Google. 

While applications like Siri which use 20 or more web APIs are still unusual, most applications use several APIs to deal with access to content, social networks, events, identification, payment, and other concerns.  These APIs change frequently, as do the applications that consume them, resulting in a web of dependencies and causing chaos in the development lifecycle.

API Virtualization enables these different concerns and dependencies to be isolated from the application or API provider, resulting in a cleaner development process and higher-performing applications.

Vzaar: A Platform for Merchandising Video »

We all know YouTube is a famous platform for sharing videos. But the advertising that supports YouTube isn't always wanted when you, yourself, are a business who wants to be taken seriously. This is a problem that vzaar solves—hosting professional content and videos for businesses. They provide a professional video platform that allows you to serve high quality videos for your web applications, and many other businesses. Creating videos for your business is a great way to engage with customers and can be an important revenue generating tool. The best part about vzaar is the control they provide, like how your video gets presented and where. They also have a well-documented API that you can use to plug videos right into your application.

Since the CTO of Vzaar, Adrian Sevitz ,uses Apigee, we asked him to share some of his experiences.

 What insights into your app or other benefits have you gotten from working with Apigee?

 Adrian: Both insight into which calls are being accessed, and what the popular API calls are as well as how many hack attempts we get looking for open urls and holes.

 What new Apigee feature would you most like to see from Apigee?

 Adrian: Two things. The first is the ability to limit users by number of calls so we can sell different access limits. The second is to white or blacklist ulrs being access to provide a layer of protection at the api level.

 What mashup, app, or API do you admire?

 Adrian: App wise we love postmark for our emails and spreedly for our billing. Both of them make our life easier. And what could be better than that?

 When your trying to give your business the right impression on a video platform, vzaar is a great way to push your product or service out! Also we want to hear your ideas on www.apigee.com/support so we can provide a better service for all of our users.

Darwin’s Finches, 20th Century Business, and APIs: Evolve Your Business Model »

What do APIs have in comon with Darwin's observations on evolution, the 20th century garment district, and the Kobayashi Maru?

Sam Ramji makes the case for APIs in his much written about web 2.0 talk  - watch and listen to the full talk or just flip through the slides - both below.

Retailer Markets Moving to Open APIs »

When one of the largest, most successful retailers on the planet makes a move, you can definitely consider it a confirmation of a trend.

Last week, the WSJ reported that Wal-Mart is opening smaller-format stores in new urban locations.  Wal-Mart CEO Mike Duke stated in their annual report that their growth will be driven by “innovated new formats”, which includes the smaller stores and stores with drive-throughs for picking up internet purchases.

Essentially, Wal-Mart is going to where their customers are…. taking their brands to different chnnels where customers spend their time, to be there when those customers want to buy.

What does that have to do with APIs?  Everything.

Retailers as diverse as Sears, Netflix, and BestBuy are opening APIs to take their brands and their services to where their customers spend time.  Those customers are on Facebook, on Twitter, on enthusiast sites, on their iPhones, iPads, Xboxes, etc. 

As the image below from Sam's Web 2.0 talk depicts - you may have to connect to customers across many diverse channels, from smartphones to Xboxes to your car and kiosks.)

Getting your capabilities into these environments where customers are spending time requires that you fit into new formats… like an iPhone or Android app, or an iPad or Xbox app.  You can’t possibly be an expert in the dozens (hundreds?) of platforms, sites, and devices out there.  Developers can help you do that, if you have your catalogs and services exposed via web APIs.

By the way, we have a new whitepaper out on Retail 2.0 today, you can get a sneak preview here.

Working over vendors:  A buyers guide for API management »

The explosion of APIs is driving companies large and small to launch APIs for their customers and partners.  But both public and private APIs require management features well beyond their internal functions.    How should you approach the 'build vs. buy' analysis and evaluating API management solutions and vendors?

As a former CTO that has recently gone through this process, here are some lessons learned on how to scope out requirements, evaluate and select a solution.

Know the problem (and requirements) you are solving for

Many API product managers are initially tasked to spec out only a small slice of the problem (such as "get me a developer portal"  or "we need developer analytics."  ) Your requirements landscape is likely much wider than this.  Make sure your PM thinks about business requirements that have technology implications in areas such as:

1.    Security, Access Control - Authentication methods
2.    Reporting, Analytics -Usage by developer, organization, API, location
3.    Threat Protection - Denial of service attacks
4.    Traffic Management - Routing needs for versioning, operations management
5.    Mediation, translation - Protocol bridge between SOAP, REST, JMS etc
6.    Performance, Scale -     Clustering, caching
7.    Developer support-     Access key management, FAQs, wiki
8.    Operational support - Root cause analysis, logging

(For more on these topics, see this blog series)

Get everyone involved, early.

Because APIs touch partner relationships, they are critical to the business, and there are multiple stakeholders in your organization.  Walk through the lifecycle of an API, what roles support the release of an API, external access, operational support and management. These stakeholders should be represented in the overall product management strategy and accounted for in your requirements for API management.

Considering a vendor? Demand a POC!

The requirements, operational view and deployment strategy will provide an outline of scope for build vs. buy considerations. All of these functions should be outside of the core elements of the APIs themselves to ensure consistency without duplicating efforts for key policy areas that are needed for your APIs.

If you engage an API management technology vendor, these requirements can be prioritized and used to drive a demo of a solution to ensure the capabilities meet your needs. Insist on a POC (proof of concept) for a deeper dive and validation of more technical requirements.  This exercise will quickly allow you to determine commercial solutions that have the potential to meet your needs and provide a concrete plan for demonstrating capabilities that are of material value to your organization.

A POC will tell you a lot about a vendor -  both what they would be like to work with on a day by day basis and how they meet their commitments.  A POC will also tell you how well your own team has thought out what they need and how prepared they are to execute it.    Eliminate vendors that will not step up to a POC.

Also,  make sure your own team goes back and can repro the POC -  without the vendor.     

This diagram below (and a larger version here)  shows how we thought about our POC process.

Revisit the priorities and rollout strategy for API management

This evaluation process should expedite the short list of vendor options that will satisfy your needs and as well as gain internal consensus on what API management means to your organization. You may find the depth and breadth of your requirements will motivate you to find a commercial solution rather than build your own system that will have to be maintained in addition to your core APIs.  

A key consideration for the solution you select is handing ongoing changes after this system is deployed. Once your APIs are operational you’ll need a way to manage updates to numerous policy areas on a regular basis and most likely you will want to avoid any downtime to implement those changes.   For example, my previous role, we did an cost analysis of how many independent API codebase versions we could support at one time.  Our answer?   Two.   This drove us to select an API management technology to  provide a versioning and mediation layer above the API layer.

It is important to review the flexibility of the system to avoid limiting your API management capabilities and practices to a particular solution.

Operational awareness

Policy enforcement is typically the initial area focus for API management but operational visibility is also a key area to focus to include as well. How will you report on usage and do you have a consistent way to collect data to be aggregated? Do you have a common root cause analysis process to enable message level tracing and logging regardless of the API?

Conclusion

The topics around API management are numerous and not unlike the questions that were contemplated during web browser adoption. With a new delivery model of information and services there are multiple stakeholders to consider during a ongoing lifecycle of development and operations.  Developing an understanding of your needs up front will greatly streamline your selection process as well as get a common understanding of what API management entails within your organization.

Darwin’s Finches, 20th Century Business, and Open APIs: Evolve your Business Model »

Today @sramji was invited to speak on API business models and strategy at the Web 2.0 Expo in San Francisco.  

Thanks to our customers and many others for these API best practices and ideas.

Don’t try to market to developers.  Instead, solve their problems. »

Not long ago you could count the number of 'developer marketing' programs on one hand.  Now there are hundreds of programs as Web companies and enterprises open APIs.   These companies know that developer adoption will make their API strategy succeed or fail.

But Developer Marketing is an oxymoron.  Developers hate marketing.   

You cannot drive adoption by 'marketing to developers.'  Sure, you can send offers to your developers but your mileage may vary.

A better formula - understand what's important to developers and give them what they need to reach these goals. Developers want to:

•  build new skills that lead to the best projects and jobs.  This is why new or proprietary tools and programming models are tough to get off the ground - it's a small market of new projects for the developer.
•  increase their productivity.  With good tools and by connecting developers with decent resources and each other for help.  This is why sites like StackOverflow take off. 
•  be recognized for good work and see their products used.  Focus on showcasing their work, not your product.  It's not about you.
•  get paid.  Think App Store model, or affilate marketing networks.

Talk to the folks that made the big developer networks sucessful and you'll hear these points over and over. Some others:

• Developers are not buyers, but are very strong influencers.  There are superstars in the developer world - make them fans and that is the best marketing you'll ever get.
• You can't 'own' or 'use' developers because they have an account on your service.  Developers have lots of options and switching costs might be low from your API.
• Act on their feedback.  Developers are smart and listening and acting on their complaints and ideas is critical to your credibility.
• Developer communities are fragmented.  For example, there is no such thing as an "API developer', but instead there are Twitter or Facebook or Salesforce developers.

Once you have attracted a developer to use your service - they are like gold.   So treat them with respect - don't try to 'use' developers or you might lose them!

Apigee API Contest Winner at iPad Developers Camp - Netflix Actors »

The winners of the Apigee sponsored prize at the iPad developers camp was the "Netflix Actors" application. Check out a quick demo of the application below.

Sam Ramji discusses Apigee’s new Twitter API Tools with Robert Scoble. »

Robert Scoble of Rackspace caught up with Sam Ramji to discuss the latest release of Apigee and all the new Twitter centric features. Check out the full interview below!

OAuth is improving, but still moving »

We've been following the fast-moving debate in the IETF regarding OAuth 2.0.  OAuth, for those of you who have not encountered it already, is a set of authentication technologies for the Internet designed around the concept of an access token.

Access tokens, in the words of Eran Hammer-Lahav, are like valet keys -- they give the holder access to a specific function, for a specific amount of time. For instance, you might use OAuth to give another web site the ability to read photos from your Flickr profile, but not to modify them. OAuth lets you do this, it lets you go back to Flickr and revoke the web site's permissions at any time, and it does it without requiring that you give the site your Flickr password.
 
The current spec, OAuth 1.0a, is implemented in lots of places, and it solves a lot of problems. However, implementing it is no picnic for either the API provider (the server) or for the developer who builds the client. (There are libraries, of course, not to mention products such as our own that simplify this process.)
 
OAuth 2.0 introduces many changes. The most important is that a client may now use a "bearer token." That's a fancy IETF way of saying that an access token can just be a string that the server gives you. On every request, the client passes that token back to the server, the server checks to see if the token is valid, and you're done. This is much simpler to implement than OAuth 1.0a, but it is only secure if you use SSL for every request. Applications that won't or can't use SSL may still use the old way of transmitting each token, which encrypts the token so that it is safe even if SSL is not used or even if it is intercepted by a proxy like Apigee.
 
However, OAuth 2.0 is far from complete. It is currently undergoing lots of discussion on the IETF mailing list, and the spec draft changes daily.
 
That's why I was surprised to read today that Facebook is using OAuth 2.0 to authenticate its own API. Now, some of the key players in OAuth work at  Facebook, and they have chosen to use only a part of the spec, and the part that's arguably the least complicated. I'm sure that they feel that taking this calculated risk now is in the best interest of Facebook and its developer community, but the possibility remains that the spec will change and Facebook will have to change its implementation to match.

(In fact, at the moment I write this, they do not -- the name of the query parameter that holds the token is "access_token" in the Facebook documentation and "oauth_token" in the latest version of the spec repository.)
 
In the meantime, developers building on top of these APIs may have to contend with OAuth 1.0a (the current spec), OAuth 1.0 (an older version that some sites may still use), the draft form of OAuth 2.0 as implemented by Facebook, and even "WRAP," which introduced some of the concepts used in OAuth 2.0.
 
So the good news is that there are lot of good standards being written that can make it easier to produce and consume powerful and secure APIs. The bad news is that those standards are still changing. So stay tuned, and be careful!

Right product at the right time:  API Product Management »

Recently we were asked by a SaaS company exec "can't we just hire someone to come in here and build our API for us?"

Danger, Will Robinson.  Just like any other product in your stable, your API needs to go through your product management practice.  Successful APIs usually have a dedicated API product manager that creates the 'right product at the right time" by continually focusing helping the team stay on target by driving:

1. What is the vision for the API?      How do you go from an idea to great product?  Start by asking what is your vision?  if you were sitting around with your top 5 execs..would they agree?   One good PM framework we've seen really focus an effort is explicitly defining the "VMSO" or "Vision, Mission, Strategy, Objectives" before every major release.  For example:

• Vision - what is "the dream" (example: be the most widely used widget catalog on the planet)
• MIssion - what do we do every day to achieve the dream? (example: have the easiest catalog API to learn and use)
• Strategy - what is our unique approach for achieving our mission? (example: have the smoothest sign up, clearest REST API and best community support)
• Objectives - what are our 1-3 key API metrics to determine if the strategy is working? (example: developer apps, API transactions, API revenue)

2. What is the target customer segment for the API?  Mobile developers?  Your top 10 partners? Affliate marketers? Each segments may need different features, policies, or marketing approaches. Do a customer or developer segmentation analysis and force rank priority segments. 

And if you ask your API team who their target segment is and the answer is 'everybody' - get worried.

3. Develop use cases.  Ask how little, not how much, you can launch with your API.  Taking back functionality is difficult once it's out there.  Identify and prioritize the minimum set of use cases (or user scenarios - such as 'browse catalog information')  and consider throwing out anything outside what's needed for each use case.

4.  Iterate quickly.  It's rare to find a successful API program where the PM doesn't say something like "and after we launched our customers took us in a completely different direction." Consider agile development techniques to help your team iterate quickly. 

5.  Differentiate your API. How is your API or content different than competing APIs in your vertical?  Why should I drop what I'm doing now and use your API?  Using a well-worn PM 'positioning framework' can help the team agree on this beforehand. For example:

What other PM processes do you recommend?

(and thanks to respres for the great photo.)

@sramji demos @apigee @twitterapi features at #chirp »

Sam demoed Apigee's new Twitter API test and debug console at the Twitter Chirp conference yesterday.

There's both a video and the screenr screencast for a clearer view of the screen below. Enjoy!

Introducing new Apigee features for API testing and debugging »

Our Apigee team is delighted to announce powerful new features for API developers for the Twitter's Chirp conference.

Today, Twitter API developers have a new API Testing console that provides a way to explore and share Twitter API testing results via URL with other developers (see and example here).

And the new API Debugger gives developers API message capture, record, and playback for any API that is proxied with Apigee.  You can share your debug snapshots with other developers via URL.

A couple other changes:  we've removed our 'private preview' invitation requirement - anybody can sign up and get started now.   And we've increased our free traffic limit from 10K to 50K per hour.

Finally, if you've used Apigee before, check out our new proxy creation flow.

For a detailed perspective, check out the Apigee blog for more on the evolution of Apigee.

Beyond the Browser: The API web »

Maybe it was because I was reading it on my new iPad, but really enjoyed this post by John Doerr and partners on TechCrunch last week.

Was struck by the phraseology 'beyond the browser' and that we're quickly moving " beyond web sites limited by browsers… to interactive, connected applications with incredible simplicity, speed, and fluidity.... that will transform everything." 

Couldn't agree more, especially when we consider the next 5-10 billion devices sold (via Mary Meeker's amazing Web 2.0 deck below, slide 32) will not be PCs with browsers but connected devices - from the iPad to your Xbox or Tivo that don't use browsers but instead connect to content providers with APIs.

We are moving beyond the browser to the "API web", where much of your company’s web traffic will come from a non-browser source.  Enabling the API web is our mission.

 

Moving the needle: Example API metrics »

It's an old cliche, but it's been said that you can't move the needle if you can't *see* the needle.  So frequently we're asked "what are good metrics to measure an API program?" 

While individual metrics are important - it might be as much about the 'process around metrics.'  Or..how metrics are evangelized and used to drive specific parts of the API product development pipeline.  Specifically:

1. Get early buy-in on the 'top 3'  -  strong API product managers often focus in on 1-3 top level 'strategic' metrics and get early, wide agreement from all parts of the extended team - the sponsoring exec, PM, engineering, BD, and operations. If different stakeholders are measuring success with different metrics (say number of developer sign-ups vs. API traffic vs. revenue) this can pull resources in different directions.  

2. Track against realistic projections.   Set expectations early by modeling anticipated results and then track actuals against this estimate. For example, pick a 'comparable' or competitor's API to guess developer portal traffic, then model the expected developer sign-ups and conversions  (for example, 10% of visitors might ask for a key, 20% of them might built an app, 10% of those apps might drive ongoing traffic, each of those apps might drive a certain volume of traffic, and so on... )  

3. Publish a weekly dashboard, religiously.  Proactively call out how product updates and community activities do or don't move the needle so you can quickly adjust tactics and think of new ideas that might move the needle.

4. Create a metrics 'pipeline' -  How do different metrics diagnose how each stage of the customer conversion process is working?  For example, developer portal traffic might be a good metric to measure the marketing guys. (that is, they might be responsible for getting developers *to* the portal.)  But whether or not a developer converts to ask for a key and then converts into an active API user might be a measure of how effective the PM process is working to create a product that developers want to use.  User experienced bugs can measure development and product QA effectiveness, and so on..

Here is an example of a metrics pipeline that we recently discussed with a customer.

(Thanks to seenoevil for the photo)

Considerations for ‘open data’ API programs »

In my last post I wrote about how the book digitizing effort is trying to monetize underutilized books online.

For anyone contemplating exposing data or capabilities via APIs to create new revenue streams, there are some important implementation lessons that can be learned.

Have control

In Norway’s Bookshelf project, their free online books can only be read online and only in Norway, and cannot be downloaded or printed out. Similarly, with APIs you need to have a way to control who can access your data and from where. API Identity, API authorization and other API security considerations are a must.

Make it (at least something) free

Google must operate under copyright laws, but it has found a way to freely expose extracts or snippets under ‘fair use’. Similarly, you cannot expect uptake of your content or capabilities if some of it cannot be consumed easily and en gratis. Give at least some of what you offer away for free to spur adoption. Technically, this means being able to control what can be consumed at a granular level.

Get started!

Google, Bookshelf, and Europeana have ideas about how they’ll monetize books they put online, but they are still experimenting with the right business models. Executives shouldn’t wait for an iron-clad business model to present itself… you won’t figure out the right model unless you are ‘in the game’.

MotorMouths: An API Provider that Reviews Cars. »

Buying a car is stressful. Getting access to useful, trustworthy information helps. MotorMouths.com pulls together the best reviews for all new cars in the market today. Mashable described them as, "The RottenTomatoes of car reviews." To do this, MotorMouths gathers thousands of reviews from leading critics and then calculates an overall score for every car, based on the average of individual critics' scores. So, for example notice the critic's versus the average score for the Hyundai Genesis:

MotorMouths uses Apigee to gather stats for their own API, so we asked them to share their experience and insight:

What insights into your app or other benefits have you gotten from working with Apigee?

Jeff Smick: We get a much better sense of when and how our affiliates are consuming our API. We can gather a lot of data with Google Analytics on the frontend. Without Apigee gathering data about API usage would have required us to implement our own stats system. As a very small, fast moving, bootstrapped company we wouldn't have had the time to do so.

What new Apigee feature would you most like to see from Apigee?

Jeff Smick: I'd like to see the error code in the error reports. Apigee provides a good overview of which urls are producing errors, but no breakdown as to what those errors are. We use error codes to let API consumers know that they're unauthorized or doing something bad. So it'd be nice to separate those from the true errors.

What mashup or app do you admire that uses the same API(s)?

Jeff Smick: I've always been a big fan of Twitter's API. They've been doing a great job from day one. We also consume the grader.com APIs (twitter grader and blog grader) which are easy to use.

The next time your shopping for a car and want to do some consumer research, do it the fast and simple way by going to MotorMouths.

How do content and transactional APIs differ? »

Recently, during one of our our RAW (Rapid API Workshops) with a retail customer, a great question came up - what are the major differences between a content and transactional API?

Probably not a complete list, but in general:

Content APIs are more likely to be open, without sensitive information. Think of a search, media, or mapping API.    While the provider might want to track identity through API keys, these APIs often need no authentication, authorization, or encryption.    Search results may be highly cachable, which might be helpful to support high concurrency for bursts of demand for popular content.    Content APIs are also more likely to need throttling to protect the back-end and quotas to measure consumption - think about that grad student downloading your entire database one API call at a time.   Users might have some tolerance of downtime for free content that can easily be requested again.      Success for content APIs might be measured in terms of usage or engagement. (usage per consumer), so having  API usage analytics might be important.   If you can, make content APIs simple and easy to adopt with standards like REST.   

Transactional APIs  have sensitive data and therefore security needs go beyond identity and developer key level tracking  to include API authentication and authorization.   The data might need encryption and XML or API specific threat protection.    Instead of quotas, the back-end business logic might already contain all the controls you need to measure consumption and monetization.    There is probably no tolerance for downtime or lost transactions.   And of course success for transactional APIs can be measured in existing financial terms.

What's your experience in the difference between content and transactional APis?

Consider an API to unlock ‘dead data’ »

Book digitizing made the news last week, as Google announced a deal to digitize up to a million books from Italian libraries. Despite some author and publisher resistance, similar efforts are proceeding in Europe, including Norway’s Bookshelf project and Europeana.

What is the motivation for digitizing vast libraries of books?

Yngve Slettholm of Kopinor summarizes it well, “The vast majority of books are out of print and can be considered commercially dead.” “This creates an extra source of revenue for older books.”

What does this have to do with APIs?

How much ‘commercially dead’ data, content, or services is your company sitting on? Are you fully monetizing your assets? If you gave yet-known 3^rd parties access to data or content locked in your company, what new business opportunities and revenue streams would be created?

That’s what an open API does. It unlocks data and content, giving it the potential to be consumed in new, innovative ways, and monetized. Via an API you can get the broadest possible distribution for the lowest cost.

One example of this is BestBuy’s Remix program; BestBuy opened the Remix API and now BestBuy can be accessed via mobile apps and in new ways and places where its customers are. Another example is Sears which has opened it's vast product catalog, and TransUnion, which made its core credit report and credit monitoring services available to 3^rd party financial companies.

It is not always clear who will use the capabilities you expose or how. But one thing is for certain, if they are not accessible, they will not be used.

The book digitization efforts also hold some lessons for how to manage such a program, which I’ll cover in my next post.

SXSW discussion on API trends and adoption »

Ross Turk shot a great video panel at SXSW on trends in developer and API adoption.

Sam Ramji and Greg Brail from Sonoa, Laura Merling from Alacatel-Lucent, and Martin Tanlow from 3scale talk about what they're seeing in world of APIs, from the latest mobile and social apps to Alcatel-Lucent's Open API strategy for developers building on service provider networks.

API Tip of the Day: Self documenting code is an ideal »

Read the API carefully as you write it.  Does it read clearly?  Can new users make sense of it quickly?  Twitter’s “statuses/mentions” is a good example – you’ll HTTP GET the mentions of the user account you’re calling from.  Readable APIs mean easy-to-adopt APIs.

API Tip of the Day: Involve the users early »

Your API exists for the use of developers.  You’ll probably get it wrong the first time.  With a tight feedback loop between developers who use the API and those who build the API you can get it right quickly.  In turn these early users will help promote the API and help new users get productive.

Cloud computing is a double play »

Joe Weinman recently wrote that any business-focused CIO must first ask:  Why do cloud from a business perspective?

Joe makes the great point that technology will only be important if the business value is clear and compelling. CIO's have a small number of projects that they can really focus on in any given year, and major initiatives must have a compelling rationale or won't get supported by senior leadership. 

What could be more compelling to senior leadership than finding new revenue streams?

While senior IT leaders may still be concerned about being viewed as a cost center, and look to the advantages of cloud computing to reduce IT cost, savvy IT execs will find a way to help create new revenue streams for their company.  A great way to do that is to create new business channels by opening APIs.

So why not get a double play?

You can do both… have a  more business-relevant discussion with senior leadership about how to spur new revenue growth, as well as start to utilize cloud computing.  Sonoa’s API management solution is helping companies like TransUnion Interactive , SilverPop, and MySpace create new revenue streams by opening their APIs, and Sonoa is available both on premise as well as a ‘cloud service’.

API Tip of the Day: Don’t put verbs in your API »

The accepted verbs for REST APIs are contained in the HTTP Request: GET and POST.  Realistically PUT and DELETE are not always available due to IT standards for security on web servers.  Try to design your interface from the point of view of using just GET and POST as your verbs.

API Tip of the Day: Learn from the masters »

Joe Gregorio’s original column from 2004 is just as relevant today as it was then. It covers the four fundamental design questions that authors of a REST API must ask and answer. Find it here: http://www.xml.com/pub/a/2004/12/01/restful-web.html

API Tip of the Day: REST Means Exposing a Resource »

Ideally an API exposes a resource that can be manipulated.  While there may be multiple ways to organize those resources into categories and lists, at the end of the URL you are exposing a resource.  This is a core principle of the REST pattern.  Look at the problem from the resource’s perspective, not the perspective of the method.

Design your API for adoption with ‘true REST’ »

"The only reason you'd have only a SOAP API is because you hate 80% of your addressable market." - @sramji

There's usually little argument that a REST API is easier to use than a SOAP API.

But how important is it to be 'truly' or 'strictly' RESTful? That is, adhering to standard HTTP operations or 'verbs' - GET, PUT, DELETE, POST - on well defined resources, as opposed to the common practice of embedding 'verbs' or operations as methods in a GET URL.

Typically, security is cited as the big advantage of 'true REST' (with some good discussions here and here).

However, a truly RESTful API may help you boost developer adoption. For example, imagine a 'shopping cart' API:

While the above API isn't 'truly RESTful', it's not that hard to use.  But you do have to learn the individual operations and this can get cumbersome if there are a lot of them or as the API evolves.

Instead, this 'true REST API' may be easier to learn and predict as you use more features.

What if we want to list all the shopping carts in the system at any one time? We would add that via an HTTP GET to:

http://api.shopping.com/carts.xml

Query parameters can still serve a purpose - making it possible to specify additional options. For instance, imagine a very large shopping cart, and you want to "paginate" the results. To look at items 20-29, you might use a URL like:

http://api.shopping.com/carts/X.xml?start=20&;count=10

Bonus: True REST makes analytics easier

It's a lot easier to build reports that segregate and analyze traffic by URL than to build logic that tries to do this by methods or combination of methods. Good API analytics helps you optimize features, debug problems, and weed out traffic that can slow down your service.

What if your 'non-strict' REST API is already out there?

It might not be that big a deal if your API is very simple or 'read-only' API with information that isn't too sensitive (such as a free search API). Or you can map a non-RESTful API into a 'truly RESTful" API with custom code or API management tools thatperform API transformations.

* A note on POST VS. PUT

* One way to insert an item in the shopping cart is to use POST to update the shopping cart by sending it a new item. In this case, we are using POST to send the server an instruction that essentially tells it to insert some new content to the existing resource. This is why we use POST -- it is like an "update" in a database.

Alternately, we could add the item by using PUT to a new URL, such as:

PUT http://api.shopping.com/carts/X/items/Y.xml

But if we do this, then we need to somehow give our item some sort of unique URL by picking a value for "Y". This is kind of a strange thing to do, so it may be more natural to use POST and have the response include the URL for the new item, so that we may retrieve it later using GET or delete it using DELETE. Still, sometimes using PUT like this makes sense.

This all comes down to the difference between PUT and POST in the HTTP spec. POST modifies something that already exists, and how that thing is modified is up to the server. PUT replaces the entire contents of the URL with new data. Plus, like GET, HEAD, and DELETE, PUT is idempotent, which means that if you call it more than once, it has the same result every time, whereas POST may keep doing what you ask it to do over and over again

Yes, REST security does exist »

A recent article "Why REST Security Doesn't Exist." postulates, "REST does not have predefined security methods, so developers define their own."

Some good points in here (such as 'don't roll your own')  but I might not completely agree with the premise.

One of the fundamental principles of REST is that it builds on the HTTP protocol -- and the HTTP protocol very much does have "predefined security methods."
 
The basic HTTP protocol supports a way to plug in different security schemes. It also supports OAuth, two-way SSL, and many other mechanisms. Not only does HTTP allow for many security schemes, many of which like HTTP basic are defined by IETF standards, but it also supports a mechanism that allows a server to identify when a request was rejected, if the request was rejected because the security credentials were invalid or because an authorization check failed, and whether the rejection was permanent or temporary. HTTP also includes a mechanism that allows the server to issue a "challenge" that asks the client to re-send a request with a particular type of credentials if it has them. This all adds up to a security method that has proven quite robust over the years and has been extended with new methods such as OAuth when new problems arise.
 
Also, don't forget that different APIs demand different security requirements. An API that offers product catalog information, for instance, with no way to update the information, does not require strong authentication if the owner of the data intends that data to be public anyway. A simple "developer key" that uniquely identifies the sender of the request -- yes, a username "without a password" -- is just fine for that type of API because it is used to identify the user for various tracking purposes, and is not designed to prevent unauthorized users from gaining access to the data.

Cool article and great to see some discussion on this!

Mobile Location-Based Services - don’t forget the Telcos »

TechCrunch recently posted on a Juniper report on “Mobile Location Based Services"  This report taps on the potential for this new wave of powerful apps – like letting your phone geotag the video you just took and posting it to Facebook with a Google Maps link;  one-button dial to a nearby restaurant discovered through your social network; or dynamically billing for high-value media content via the operator.  

Companies like Google, Foursquare, and Nokia are mentioned as on the forefront of many of these services.

But don't forget the Telcos - they have rich location based services with network data and a huge customer base.  Exposing APIs and working with third party content and service providers is critical for telcos and network operators. 

But Telco APIs can be complex.  Our view is that the Telcos that focus on making these APIs as simple and accessible as those from consumer Web players can be winners in MLBS.  We talk about these and other issues in detail in our new Telco 2.0 Whitepaper.

Apigee vs. Sonoa ServiceNet for API management »

In a few years, most of your internet traffic might come in not through your website, but instead through mobile devices, tablets and affiliate partners accessing your services through APis.  For two years, we've been helping enterprise customers accelerate their API strategy and deployments with Sonoa ServiceNet.    

Last year we released Apigee as a tool for developers building apps with APIs.   Apigee provides free, self-service website for API analytics and protection. Hundreds of developers are using Apigee to monitor APIs they are consuming (such as the Twitter, Flickr, and Facebook APIs) or APIs that they are providing.   Here are some example apps. 

How do Apigee and Sonoa ServiceNet compare?

It's much like the difference between Google Analytics and Omniture. Apigee (like G analytics) is a free, self-service API analytics and management tool that provides coarse grained analytics and basic throttling policies.  Sonoa ServiceNet is an enterprise-scale API management platform that provides a rich policy framework for customized, complex policies and enterprise-levels of scalability.  While Apigee is available as a free service,  Sonoa ServiceNet is available as both an on-demand an on-premise (both hardeware and software) offerings.  Also, Apigee is itself an app built on the Sonoa ServiceNet platform.

So if you are a developer building an app that uses APIs and want basic API analytics and rate limiting in 5 minutes - give Apigee a spin.  If you are a product or engineering executive thinking about an enterprise API strategy or roadmap, consider these potential requirements and take a look at Sonoa ServiceNet for industrial-strength enterprise API management.

Why your sweet new feature needs an API. Now. »

Over the weekend, Twitter developer Alex Payne posted a tweet (since deleted) which sent such waves of speculation rippling through the Twitterverse that even Techcrunch felt the need to comment. Of course I'm interested in seeing the features that Alex wrote about, but I'm even more interested in Al3x's follow up tweet that noted, "We still release most everything API-first, of course."

Twitter, as much as anyone, has depended upon their API to drive adoption. This is the future of web services. The web IS the common ground, but as the web moves beyond browsers into devices like mobile phone apps, APIs are what allows it to happen. When you consider that 60% of all eBay listings are made via their API, then you have to appreciate how important APIs are.

If you want your developers to succeed, you have to empower them. Until you release most everything API-first, you aren't giving your developers the tools they need to push your platform forward. Take LinkedIn, for example. Their search API docs suggest that you cannot search LinkedIn by email address or phone number, even as an authenticated user. 

An email address is critical for disambiguating users. Imagine that I use a CRM system, like Highrise, and I want other users to be able to authenticate with LinkedIn to see who in the company might have a LinkedIn connection with a contact. What if that contact has a common name, and the only other information entered is an email address? The ability to identify specific people to see how you're connected is a primary use of the LinkedIn network. And yet third parties don't have access to these fundamental methods. Without this, their digital hands are tied. 

What's hot: releasing most everything API-first
What's not: leaving core functions out of any API

An API at Launch: Reflecting on the Google Buzz API »

Remember when videogames used to be released on different platforms at different times? One platform like the PlayStation version would lead, while other versions trailed behind. Then at some point, publishers realized they could make a bigger splash by landing all versions in stores at once—the simultaneous release.

Today on the web we expect an API to lag the launch of a new webservice. So it was interesting to see Google Buzz tout its API on day 1. On one hand the shiny new Buzz API is not much more than a stub, though on the other it is a promise to deliver a robust API.

But at this time, you can't even post an update to Buzz via the API. You also can't mute a buzz. You can't even search. It's read-only; all you can do with the API is get the Atom feed for a user's Buzz, which doesn't include important metadata like location information. You could argue, as Fred Wilson has, that any read-only API shouldn't even qualify as an API.

There ARE things to like about the promise of the Buzz API. It's a stake-in-the-ground to support existing standards like PortableContacts, the Social Graph API, and OAuth. Important protocols like PubSubHubbub, Salmon, and WebFinger should get a good boost too. Yay for open standards!

But there are some concerns too. In order to build on top of the Buzz API, you may have to become familiar with the disparate set of technologies listed above. And if you prefer JSON or raw XML responses to Atom, you'll have to wait (FWIW, format conversion is one of the many tricks that Sonoa's ServiceNet does right out-of-the-box).

What could Google do to encourage Buzz innovation? After all, isn't that a big reason why APIs get released? They could wrap the various standards they plan to support with a look and feel similar to the full Buzz API (whenever it lands). They could emulate the Twitter API, like Wordpress and Tumblr have done, but that brings up issues of licensing—something that Googler Dewitt Clinton has pursued with Twitter's Ryan Sarver. Still, that kind of familiarity would only drive Buzz adoption with developers already versed in Twitter. In some ways Buzz resembles FriendFeed more than Twitter. So why not map the FriendFeed API pattern onto the Buzz API, as Dave Winer has suggested (again not without issues)?

Not having API in 2010 is like a not having a website in 1997. How long will it be until everyone launches both of them at the same time?

(logo by Alex Gillis)

But can you hold the API to the SLA? »

Great article by Jonathon Feldman in Information Week recently with some steps for CIOs to take before getting into cloud computing.   One is to insist on SLAs from cloud providers, especially considering the natural tension from the provider's perspective between high-availability and low-cost operations.

Absolutely agree.  But to build on this - remember that scene from Seinfeld where Jerry is at the car rental counter -  "Anybody can *take* a reservation, the important part is to *hold* the reservation."

Often, cloud and API providers will agree to SLAs, but have limited means to enforce or verify the SLA is held. Many SLAs are just 'on paper' with minimal enforcement or monitoring.  This gets especially tricky if you need to discuss financial penalties. 

Consider how you will monitor, meter, and audit API traffic and content between you and your partners - from your side - in order to pinpoint problems, protect your organization, and especially if you need to enforce penalties based on SLA misses. 

APIs for connected devices - the rush is on »

My favorite Superbowl ad was for Vizio HDTV - it was great to see about a dozen leading Web APIs showcased right up there with Beyonce.

Yesterday Michael Zimablist posted on the New York Times Bits blog about how the NYT’s content must now support a growing range of devices like the Vizio, from web-connected printers to mobile apps to the new iPad.

And the Wall Street Journal's Martin Peers asks if the iPad’s e-book store is the new model for the television industry, citing how Netflix is streaming to over 50 different devices to build new audience. Blockbuster is pursuing a similar strategy to deliver it’s movies to “nearly every connected device.”

So the rush is on… just like a decade ago when companies scrambled for a ‘www’ address, companies today are rushing to create APIs - making their data and services available to execute a multi-channel, multi-device strategy.

Next: what are some considerations for your API in supporting an ever-expanding number of devices?

Why modern applications need an API proxy »

Structures of control are spontaneously generated in every environment and every wave of computing.

Today on the web we have a model where browsers are the single point of control for much of what happens, not just at the level of applications, but at the meta-application level as well. Not simply usage (“point-click-type”), but things about usage – who is the user (browser cookie), what are they using the app through (user agent), where did they come from (referrer), what can we infer about their behavioral state, and so on – as well as modifications of usage (browser add-ins, content filters, security modes, local caching for performance). To be sure, some of these things can be and are performed using infrastructure between the browser and the website (such as content filtering, security, and caching), but the guaranteed component is the browser.

This is one of the reasons that Google Analytics is so popular and useful – you can rely on it to tell you useful things about your traffic because it can rely on the browser as a predictable point of control. Including an invisible piece of content on your web page makes the browser fetch data from Google, implicitly sending information that enables Google to report on your usage.

For web and cloud APIs, what is the equivalent structure of control?

Currently there is no one point like the browser. This is for great reasons – APIs are all about reusing application or service logic and rendering it to different form factors: pure logic (built into an internal application computation), web UIs (part of a mashup), and most notably, client applications on a wide range of devices (from PCs to mobile phones, set-top boxes, and tablets like the iPad). These devices are in the early part of a boom that will see over 10 billion individual units in use, representing at least hundreds of unique hardware/software designs. The sheer utility of these internet-connected devices predicts that their usage will drive high demand for APIs rather than standard websites. There are initial specifications like BONDI that suggest a standard contract across all of these for “mobile web applications” that include interaction with the features of the local device (such as a camera or GPS) but they are years from broad adoption and don’t attempt to unify all API access down to a common control point.

Given that APIs are to application logic what RSS is for content, we know they will be very important; at least as important as the visible web that we use today and possibly more important. This suggests that the other things that are spontaneously generated in value-exchange environments like user/customer management, behavior analysis, content filtering, caching, and security – will show up for APIs as well.

The web API equivalent of the browser’s control structure is an API proxy.

This is a control point which unlike a web proxy is fully aware of API content, communications patterns, and able to drive the meta-application controls discussed above. An architecture like Google Analytics which is founded on a browser’s predictable algorithms cannot work in an API setting. The same rule applies to add-ons that modify usage – they can’t do so relying on the local device if they are to be widely adopted. But an API proxy – a server or service on the internet, sitting between the client (regardless of type) – is able to be that point of control. As traffic runs through it, meaningful data can be captured for immediate outcomes (block access, change the message, or respond from a cache) and later used for behavior analysis and business planning. Add-ons that modify usage of the API can be installed at this point (content filtering, adding new information such as advertising, or identity management). All of this can be done while adhering to the contracts of the APIs and supporting the web architecture and rules of HTTP-based applications, and without attempting to solve the logarithmically complex problem of modifications to all the world’s clients.

So API proxies are likely to be necessary for the sustained growth of web and cloud API usage. There are likely to be several nuances that end up differentiating the different implementations and providers of API proxies. The key is to start experimenting with them now in order to build better apps and stay ahead of the competition.

Telco API Management whitepaper with Alan Quayle »

Why aren't Web developers adopting Telco APIs?

We recently teamed up with Alan Quayle on a new Telco API Management whitepaper that discusses issues around API management adoption issues specific to telcos.  

As Alan points out, APIs have been fundamental to telcos for a long time- even the first switch in 1878 exposed an interface for people to make requests to set up telephone calls. 

But because Telco APIs aren't usually presented in a web-centric way, developers have been slow to adopt them vs. some of the other Web APIs.  Telcos need to adapt by making their APIs, tools and processes simpler and more suited to serving much larger numbers of web developers than before.    

We'd love your feedback, you can download the whitepaper here.

Test Driven Development with Web APIs »

Often when automating test driven development we don't take the time to create a solid log of what is happening in each test. This is especially troublesome for tests that involve live connections to web APIs.

1. Setup a new proxy (or multiple proxies) at Apigee
   
   
   
   
2. Setup your test initialization code to use the Apigee proxy URL
   

   require 'test/unit'
   require 'uri'
   require 'open-uri'
   require 'rubygems'
   require 'json'
   require 'active_support'
   require 'htmlentities'
   require 'twitter_search'
   class TwitterApiTest < Test::Unit::TestCase
     
     SEARCH_URL = 'http://twitter-growl-unit-test.apigee.com'
     TwitterSearch::Client::TWITTER_SEARCH_API_URL = "#{SEARCH_URL}/search.json"
     
     def test_twitter_search_api
       q = '@apigee'
       @search_tweets_client = TwitterSearch::Client.new 'Twitter Growl'
       @search_tweets_client.query(:q => q).each do |r|
         assert r.text.include?(q)
       end
     end
   end
   

3. Run your tests
    
   
   
   
4. use Apigee to better understand how your tests are using web APIs when they succeed and debug things when tests fail
   
   
   

Business logic vs. Business policy in cloud services and APIs »

As our customers move from websites to cloud APIs - we're seeing them separating business logic from business policy.

When you think about business logic, you probably imagine an application server or stored procedures, or custom code that accesses stored data and exposes it in a useful and manageable way – such as “show me all of a given customer’s purchases in this timeframe”.  

These pieces of code reflect the understanding of the fundamentals of the business - like customers, orders, tickets, requests – and how they interact.  In any business you build skill over time in understanding your domain and then build on top of that understanding to take the business further.  Business logic is generally stable, consistent across requests, and subject to careful modification.  After all, you wouldn’t want to put partly tested logic about bank balance transfers into production.

When you think about business policy, what comes to mind?

Maybe a stack of papers documenting HR or accounting guidelines, or maybe the elements of Sarbanes-Oxley requirements.  Policy is not about how you compute a specific result – like the customer’s purchase history – but about other factors like who can access it, how frequently, from what locations, how it’s rendered, how long the result is considered valid, and other “meta-logic” issues.  In any business you make changes to your policies frequently, as a way to meet the changing face of the market environment.  The nature of business policy is therefore to be dynamic, varied according to the request context, and subject to frequent experimentation.  For example, product managers need to continually test the value of different offers to given segments without having to redevelop the product.

When policies like security (“who can access this and what credentials do they have to show?”) and service level (“how many requests can be made in a given time interval?”) are collapsed into code responsible for business logic, the result is loss of agility, high cost to adopt new policy implementations (e.g. for identity and access schemes or to keep pace with increased hardware capacity), and confusion of concerns (rather than separation of concerns).

In the current era of cloud APIs, where these interfaces are being built as a direct link to what started as the backend for the website, many developers are realizing that what was really policy was built into their business logic layer.  They’re also seeing that adding new clients – going from enabling partners to enabling rich clients to enabling mobile device (such as iPhone or Android) applications – require many adjustments to how the request and response of an API are rendered, but should require no changes to the stable core of business logic.  

Supporting an API with a set of policies that activate based on the user (such as preventing or enabling access to different parts of the API – for example, only the development & testing methods but not production methods, or query methods only, or limiting requests to a certain transaction volume or financial value) and type of client (such as providing pagination and format translation for a mobile device) means that the risk of delivering the API is reduced, and the risk of negative impact to existing users of the API (typically the original website) are reduced.   Also,  the computational load for the policy processing can be moved to a low-cost, scale-out tier and away from the high-value application servers that host the business logic.

So what this seems to indicate is a movement towards a separation of concerns between logic and policy.  Responsibility for business logic processing lies in a stable and slowly changing layer, and policy is processed in a tier that allows for agile modification and enables the total computational result to meet the shifting needs of the business.

More specific examples will follow in the next blog.

Customers scraping your website? Time for an API. »

Sometimes opportunites are right under our nose.
 
I love Southwest Airlines.   But often you had to check in *exactly* 24 hours before a flight time to get better seating - else you were in the ‘B’ section.  So they introduced “EarlyBird” check-in, where for an extra $10 per flight they automatically check you in first.

Southwest is very good at observing and recognizing untapped customer value.

Web scraping: nuisance or untapped opportunity?
 
If you run a business on the web, you might experience web scraping.  Often legal and technical measures are put in place to stop it.
 
But before you try to stop scraping -  why are your customers doing it?
 
I'm working with a few companies who are re-architecting their website to be driven by web APIs.  While some use is nefarious,  these companies recognize web scraping might mean there is unlocked value in their web content!  By opening APIs, they are trying to provide legitimate access to the content with a business model.  And this is not a new idea. 

So look at the hoops customers jump through to get access to your data.  Also, is there valuable data that *isn’t* being accessed at all?  You might just find new revenue channels, additional distribution, or innovative partnerships.

And think about what your API needs to capitalize on this value.  You might need:

• visibility over who is using the APIs.  Just as you wouldn’t launch a website without web analytics, you shouldn’t be running an API without the ability to see and analyze who is using it and how.
• protection from XML specific attacks and misuse.
• differentiated access (think Southwest - business select, earlybird…)
• API scalability - what happens if your users increase by 10x? 100x?.
• API monetiziation (getting that good seat is worth $10!) if that is your goal.

 (thanks to Mark Watson for the pic!)

Flickr Photosets: Monitoring the Flickr API »

Because Apigee is a tool used by people who build things on top of APIs, we get to see a lot of cool mashups and tools — Flickr Photosets is one of them, built using the Flickr and Facebook APIs proxied through Apigee.

It's easy to see why this app has so many fans. It's the fastest and most intuitive Flickr app we've seen on Facebook. You can also view and share your comments.  And it's an impressive example of how community can make an app better — with both open source code available for any facebook developer and a rich community discussion forum for users to suggest improvements.

Brad Dougherty, the app developer, uses Apigee to monitor usage, errors, and response time from the Flickr API. We appreciate Brad's great feedback such as making it easy to identify (and rate limit) different URLs by API method calls specified by parameters - we've heard that across a few users, and we are working on it. We asked Brad for his comments on working with Apigee and for what he's learned through this project.

Although my app hasn't had any downtime since I started using Apigee, it's a great way to monitor that calls to the Flickr API are actually working, something I haven't been able to see in the past. The biggest lesson I've learned from this app is that it can be very taxing to keep up with the changes in Facebook's API. It's a battle to keep adding new features when you keep having to change things to keep up-to-date with Facebook.

The API is more than the new CLI »

This article by Lori MacVittie of F5 makes some good points that whoever becomes the de facto API in cloud infrastructure might win - and goes as far to say that the API replaces the CLI.

Generally agree but might take it a step further.  

Just as we drove a 'de facto' standard CLI at Cisco, de facto standard "infrastructure APIs" likely will emerge.   (Already seeing this happening with the AWS API)

But APIs represent a significant evolution.  Why?  CLI commands and output are unstructured.  API commands and output are structured. 

I can point to an experience I had at Cisco. There is a Cisco CLI command called ‘show BGP summary' that gives the status of BGP peer – a good window into the status of the complete routing infrastructure.  In one of the releases, we changed the display a little bit and all hell broke loose - a ton of P1s to fix.

Turns out nearly all the operators ran the command output thru scripts, parsed them and used output in ops. The small formatting change we introduced broke their operations. We were forced to roll back the change.

With APIs, the output is structured and it would have been possible to introduce additional information without breaking integrations.  

Even though Infrastructure APIs will take over some of what was integration with the CLI, a properly managed API allows evolution, migration and co-existence of multiple versions easily.

New features:  SSL support and “Remember-me” »

Recently we rolled out two new Apigee capabilities.

First, we've added SSL support for APIs that support the HTTPS protocol. Now you can create an Apigee proxy for an API secured by HTTPS—such as the PayPal API—just specify 'https' in the API URL when creating a proxy. You could also change an existing HTTP proxy to an HTTPS proxy. (Of course, the API in question must support HTTPS, else you'll get an error.)

We're also working on adding SSL support for API providers that want to map to their own domain URL (and therefore their own certificate). Stay tuned, and you can review and comment on our proposed SSL Solution at anytime on our community area.   (You will see that we've completed sections 1 and 4 of the proposal to date..)

Secondly, we've added 'remember me' functionality on log-in.  We'll remember your login for 2 weeks so you don't have to specify your password, just head to app.apigee.com to see your proxies. This also means you can bookmark a favorite chart.

Thanks for the continued feedback—we're prioritizing features based directly on your input!

Paypal X: Open API as an Indirect Strategy »

On Tuesday, the WSJ ran this article about the launch of Paypal X, Paypal’s developer program.

Great to see an API showcased as the heart of an industry leader's core business strategy - and called out as such in the top business newspaper. 

Paypal is executing an indirect strategy by opening their core services via APIs in order to have their payment capabilities more easily consumed in other applications and services.

Ten years ago, you could read articles like this about companies launching their websites to execute a *direct* business strategy… selling directly to customers.

Just as it was a decade ago, companies will slowly wake up and realize that they need to have an indirect business strategy online, either because they see the massive business opportunity ahead of them, or they see their competitors already executing an indirect strategy against them.

We see 'indirect online strategies' like PayPal and TransUnion Interactive as the beginning of an emerging API Economy.   A company without an API in 2009 is like a company without a website in 1997.

The Snow Report from the North Face: iPhone API monitoring »

As a bunch of skiers, we were excited to see the this app's traffic taking off when our first early winter storm moved in.

The Snow Report from the North Face gives you the best ski conditions around the world, while making it easy to quickly focus on what's local through favorites. 

We love the  UI, feature set, and use of notifications in this app.  The Twitter integration is killer - is a great way to plug into what's really happening so you can doublecheck the official reports.

FactoryLabs - the app developers - use Apigee to monitor API performance and usage for snow conditions, weather, and other data.  They've had great feedback on Apigee - specifically, we'll eventually let you specify fields in the payload that help you ID the developers or consumers using your app. 

We'd love to hear what's on your wish list also on our feedback forum and in the meantime check out the app!

Is Apigee ready for your Production App? »

One frequent question over the last six weeks is "if Apigee is still a preview product can I use it for my production app?"

Yes, you can use Apigee as a proxy for production APIs. Dozens of production apps run on Apigee today. (many of which we'll keep showcasing on this blog)

The current Apigee preview is built on a proven platform and architecture that delivers carrier-grade levels of uptime. Some specifics:

• Apigee's core is our Sonoa ServiceNet enterprise platform. This is the same API Infrastructure that runs over 60 financial services, media, and SaaS companies running critical apps at high traffic levels.
• From the load balancers to the database and everything in between, we built redundancy into every level of Apigee.
• We designed Apigee for zero downtime upgrades so that your proxies will stay up and running even as we apply fixes for bugs and enhance Apigee with new features.
• Our uptime targets for the preview are 99.9% for API proxies and 99% for the Apigee website.
• And we have operational teams in San Francisco, USA and Bangalore, India constantly monitoring Apigee so that we can respond 24x7 to any issues.

So even though Apigee is currently in private preview, we take your business very seriously and we've built the service to be enterprise-grade from the ground up.

Free vs. Open API: Is there a difference? »

"Open and Free" are often used interchangeably when talking about APIs.  But if you look at in terms of the "API Economy" -  open and free API are orthogonal.   

'Freedom' in the cloud is more typically discussed as your rights to move data from one service to another.    Application portability has also been raised as important but current application models are both early in their maturity curve and vary greatly.
 
Open and free are both crucial attributes in order for a market economy to grow.  There are many aspects of cloud computing but for the developers and users of cloud services, the atomic unit of the cloud is the API.    

The link between openness and economic growth is a deep subject that may provide clues for how to build a better cloud –  this paper makes me think a company’s APIs might represent the “export goods in which it has a comparative advantage”.

In the software industry open platforms have typically outperformed closed platforms in the long run due to the economies that develop on top of them, cementing those platforms’ place in a range of markets.

Open APIs are:

1)      Openly documented
2)      Available via self-service (i.e. developers can sign up and get a key on a website)
3)      Using open technologies (SOAP, REST, RSS)

A good example is bit.ly - a simple URL shortening service that also lets you see how many people have clicked on your shortened version of that URL.  It’s really useful if you want to both project important articles on the web and understand the reach of your projection.   Is it an Open API?

Test 1: Bit.ly's API is openly documented (here)
Test 2: It's available via self-service.  You can get an account and a key right away.
Test 3: It uses open technologies: It's a REST API (granted, it is a mix of verbs and nouns)

Open APIs lead to 3rd party innovation

Tweetdeck users put their bit.ly API key into Tweetdeck, which automatically uses their bit.ly account (indicated by the API key) to shorten URLs that are typed into tweets.  It made Tweetdeck better and probably increases the traffic to bit.ly.
 
This could also work on the iPhone application of Tweetdeck but it’s not yet implemented in the version I have.  Many iPhone applications work use one or more cloud APIs that provide access to services in a clean, machine-friendly way.

From the efficiency of innovation perspective, keep in mind that Twitter most likely never contacted Tweetdeck to use their API, nor did bit.ly (as far as I know).  The Tweetdeck guys simply built a killer application that uses those services via APIs rather than scraping their web sites.  In the last month, Tweetdeck also added Facebook and MySpace support via their APIs.

Open APIs lead to innovation, efficiency and reach through designing your core business service to be “remixed” is found through APIs – Tweetdeck users got new value through the app they like and Facebook and MySpace got a new stream of user-driven content, all without sales or business development teams engaging at the outset.

Next time: Free vs. Paid APIs...

(Sam Ramji is the VP of Strategy for Apigee.  Previously, Sam drove many of Microsoft's Open Source Initiatives)

API threat protection pack: 10 XML attack types to guard against »

The cost of IT security breaches has almost doubled from 2008 according to this piece via ComputerWorld Canada.

While we'd love to tell you this is just a problem for our Canadian friends - unfortunately we all need to understand API attack types.

(Remember in our Cloud security tech talks last week we saw that for breaches over a certain size you may even need to issue a press release!)

Here are 10 threats that we cover in our API threat protection policy pack. 

1. Malicious Code Injection:  exploits backend services that use SQL/LDAP/ XPATH/ XQuery statements from user-supplied input.  Servicenet ‘s Malicous Code Injection Detection policy can filter SQL,LDAP, XPATH, XQUERY injection or use Custom Regular Expression, XPATH and XSD technologies to filter the request further.   It also can integrate with anti-virus products to scan for virus in the API requests especially in the attachements or mime contents.
 
2. DOS Attacks: Denial of Service (DoS) intends to prevent an API or Service from serving normal user activity. These malicious attacks includes mega-message and entities attack, recursive element attack, request flooding, larger volume of  invalid requests etc.  The ServiceNet Message Payload protection policy detects various kind of DOS attacks and protect the backend from the attacker.
 
3. Service Information Leakage:  APIs can unintentionally leak information about their configuration, internal workings, or violate privacy through a variety of service errors.  For example verbose and informative error messages may result in data leakage, and the information revealed could be used to formulate the next level of attack. ServiceNet response Message control policies can customize fault/response message reaching the client which can weed out this attack.

4. Broken Authentication, Session id and Keys: Proper authentication, API key and session management is critical to service security. Flaws in this area most frequently involve the failure to authenticate (weak or multiple adhoc authentication schemes), weak session/key tokens that helps attacker to replay or fake the keys or tokens.   ServiceNet’s authentication and API key management policies  provides single point strong authentications and key generation techniques that frees-up API developer from attack risks.

5. Failure to protect API and corresponding Data access: Frequently, authorization is based only on base URI or operation of API.  An attacker can try passing various parameters to this API operation and get access to the data that he not authorized to access.  ServiceNet fle xible authorization policies supports authorization based on various request parameters/data not just URI or Operation name.

6  API Data snooping: Failure to encrypt sensitive API communications means that an attacker who can sniff traffic from the network will be able to access the conversation, including any credentials or sensitive information transmitted. Servicenet’s SSL or XML encryption policies can be used to secure the API data from getting snooped in the communication path.

7. API Request and Response tampering: The API data tampering attack is based on the manipulation of API request and response parameters exchanged between client and services in order to modify application data, such as user credentials and permissions, price and quantity of products, etc. Usually, this information is part of HTTP URI or Header or Body(XML or non-xml).  Servicenet’s SSL or XML signature policies can be used to secure the API request and response message from getting tampered in the communication path.

8. Request Burst: Spikes in API requests might bring down the backend server. Spike Arresting and caching helps the backend services to perform better under various load conditions.

9.  Auditing: If your API is going to be handling money, you may be required by law to adhere to certain security practices and regulations.  One important regulation is auditing every (full or part of) request or response from authorized and unauthorized users.  ServiceNet auditing policy supports very flexible way to log API audit data in various formats to different destinations like Local disk, NFS, Syslog, JMS or Web Services.

10. Threat Detection and Analysis:  Analyzing the threat data is important to find the failures and fix those failures on the API infrastructure. ServiceNet’s analytics policy provides capability to visualize and analyze various API errors or failures. It can also provide various patterns or rates of these failures that help an architect or developer to fix the problem in his or her API.

For more on API security and threat protection, check out our compliation of API roadmap issues - Is your API Naked?  And let us know if you like to see the demo of this policy pack in action.

(Senthil Doraiswamy is a product manager at Sonoa Systems.)

(And thanks misocrazy to for the photo..)

Social Mention: API monitoring for social sentiment and alerts »

Warning: don't try Social Mention unless you have 20 minutes - it's addicting.

Social Mention is a social media search platform that aggregates user generated content from across the universe into a single stream of information. 

It analyzes content on terms like apigee (score to the left) or Sam Ramji (screenshot below) and gives you alerts, feeds data and a sentiment scorecard. 

Think Google Alerts but for social media.

Jon, who built Social Mention, uses Apigee to monitor their social media API. We asked him for his insights and experience using Apigee (see below).

What insights into your app or other benefits have you gotten from working with Apigee?

Jon: I would say the best insight is general API usage and error rates.It's great to be able to track usage on an hourly basis and throttle accordingly.

What new Apigee feature would you most like to see from Apigee?

Jon:The ability to track individual users/developers of the api via query string key would be fantastic. Currently, there is no method to tell how many calls each api developer is making.

What mashup, app, or API do you admire?

Jon: I would have to say Twitter's API - it's simple, to the point and it works great. They also don't have restrictive usage policies or rate limit.

We're working hard to get per developer quota enforcement in soon for Jon and the rest of Apigee users that have been asking for this feature on our feedback forum.

Check it out!

Birdwatching on Facebook: Eyeing API response and error rates for a social app »

We spotted a great Facebook app in the wild.

Bird.im's Facebook app - apps.facebook.com/birdwatching - brings birdwatchers together to share and discuss their latest finds. 

This is a very well designed and full featured app - enabling sharing of bird photos, locations, discussions and connecting passionate enthusiasts with similar interests.   The Facebook platform at it's best.

The Birdwatching Facebook app provides an API for Facebook to consume when the user performs 'one-click' AJAX actions such as adding a bird, a country, or a photo for a spotting.  This streamlines entry creation by offering an alternative over a simple form and directly improves user engagement as users create and share more entries.   In the future, an iPhone app that enables entries from the field will also consume this API.

Hugo and the bird.im team (@birdim) use Apigee to measure API response rates and errors.  (see how Apigee calculates API response rates and API error rates in previous entries).

Thanks to Hugo for all the great feedback on our Apigee Feedback forum!

Shopdiggity.com:  API monitoring for user experience and monetization »

Need to compare prices for a blueray DVD ?  Or baby car seats?

Shopdiggity.com offers a streamlined comparison online shopping experience across hundreds of product categories and brands.

Shopdiggity has a fast, clean UI that makes browsing Shopzilla's massive product catalog more approachable.   It's a great example of an app that adds value and monetizes a publisher API and affiliate marketing program.

Christopher, Shopdiggity's owner, uses Apigee to monitor performance and response times from Shopzilla's API.

Some great feedback from Christopher -  we need to add more email alerts so that developers can respond to changes in API behavior.  For example, if the Shopzilla API has slow response time, an alert would help him adjust ad campaigns to balance between user experience and monetization.

Thanks to Christopher for the great example and the feedback on Apigee.   If anyone has more,  please hit us on Apigee's Get Satisfaction forum.

mLocal:  iPhone app API monitoring and analytics »

Apigee isn't only for API providers - if you use APIs in your mashup, mobile, or social app you can monitor those APIs as well.

Why?  You might want to find out before your users if an API is slow or down, leaving big holes in your app where content should be.  Or verify any terms of use or bill you get from an API provider.

For example, if you're an iPhone developer you know nobody will use a slow iPhone app.

Shorepoint systems is one iPhone shop using Apigee for this purpose on their iPhone apps. Their mLocal app is great for creating and sharing local listings.

mLocal makes heavy use of RESTful APIs for content and especially to communicate to a back-end content app hosted on AWS. Shorepoint uses Apigee for monitoring and debugging of these API calls between the iPhone client and AWS (in both QA and production), and especially to monitor response times and proactively find out if any API call is slowing down iPhone app performance. 

Thanks to Rajan of mLocal for all the feedback on Apigee and check out mLocal here!

How Apigee calculates API error rates (featuring Twitter Growl and the Detroit Tigers) »

Last time we showed how Apigee calculates API response time. This time we want to discuss how Apigee calculates API error rates.

Apigee provides charts and tables with API error rates by day, hour or minute with drill downs by API URL. 

The 'error rate' calculation itself is simply # of errors divided by # of requests. Errors include both any Apigee error (including those that you might configure) or any target error (such as a rate limit error from the Twitter API).

Let's show this with Brian's Twitter Growl mashup.  Twitter Growl provides alerts on topics using Mac's Growl alerts.   For example, Brian gets alerts for Tweets on Detroit and Michigan news.

Brian monitors Twitter Growl API activity using Apigee.  You can see he had some stretches in mid August and early September with high error rates.  

You can also drill into the days or hours where these spikes in errors occurred, such as this 3 day period in September - high error rates on a single IP that was being rate limited by the Twitter API.

What was happening over this time?   The Detroit Tigers - in a pennant race for the AL Central -  were swept in a 3 game series with the Royals, resulting in rate-limit busting volumes of angst-ridden tweets from the Detroit Twitterati.

So there you have it.  The Tigers are behind all the errors.   (not sure we needed Apigee to tell us that.) And they still have a 4 and 1/2 game lead, FWIW.

If you'd like Twitter Growl to keep you up to date about the Tigers or any other topics on Twitter, thanks to Brian - his version is a fork of a project by visnup on github

Speeds and Feeds: RSS feed management, validation, and performance »

We use this blog to talk about issues we see around securing, managing and scaling APIs and web services.  We also see many of these same issues and requirements with feeds.  Arguably, feeds - specifically RSS and Atom feeds - might just be the most common type of XML API.

Feeds are are growing beyond just being a great way to keep people abreast of changes to news or a blog and becoming a great way to aggregate or syndicate content to partners or customers or applications in general.

Our media customers, like MTV Networks, use RSS feeds management as a way to distribute updates about their content to customers and partners. Essentially, the RSS feed becomes a “catalog” of what they’re pushing out to their web site partners right now.   
 
The needs of feeds: customization, validation, and caching

In general, everything we said about API analytics, API traffic management and API scalability can apply to feeds that you offer or consume as well.     Do you know who is looking at your feeds, how the traffic changes over time, and what kind of response time your feeds are delivering? Do you have any way to limit access to your feed if it becomes tremendously popular? And while most feeds associated with blogs, for instance, have no access control so that they are open to the whole Internet, there are also feeds that need authentication and authorization just like any web service.

But aside from these other “web services” types of requirements, feeds have some special requirements that come up all the time – customization, validation, and caching.
 
We see feed validation issues frequently. Feeds are more likely to have risks with broken links, especially if you are aggregating frequently updated content from around the web.  Feeds often don’t match the proper XML schema or aren't valud XML, as often they are generated or manually assembled. Sometimes the feed provider needs to address these issues, and other times the feed consumer has to find a way to deal with the bad input.
 
Customization is another issue. The basic feed on your blog is pretty much the same for everyone – but for others that is not the case. Lots of companies use feeds to push content to their partners, and each partner may demand a different format. Sometimes a partner needs only certain items, or a certain number of items, or they need the feed transformed into different formats like RSS, MRSS, or Atom. It’s not unusual for a feed provider to have to provide 10 or 20 (or more) custom variations of the same feed if content syndication is important to the business.    You may want to drive these with existing profiles or business rules in SQL or LDAP.
 
Caching is the last, but certainly not the least. Sometimes feeds are just static HTML files served by a web server, but sometimes they’re not. And when they’re not, feeds can be slow. We’ve seen some infrastructure that takes over two seconds to produce a feed – that’s a long time. When that happens, a little caching can go a long way.

So think about feed management when considering API management or content syndication.  If you want to experiment with API management on on a feed, Apigee is a good way to get a feel for what stats  on feed usage, latency, data volumes, and error rates might offer.

Welcome Aboard:  API user management and onboarding »

(Continuing our blog series on Is your API Naked:  10 API Roadmap considerations).

Just like you have processes to add and maintain user accounts with your website, you'll need the same for developers using your API.     Making it easy for developers to quickly get API account details squared away is critical to reduce barriers to adoption.

And good user management will reduce your support costs and increase customer sat.  

This includes setting developers up with an account and any authentication credentials they need to work with your API.  But you'll also need UI and processes to add, maintain (such as reset) and revoke or delete these accounts.      You may also need to take into account how to hand out keys or OAuth keys, and even consider an approval process, if you are going that route.  (although you don’t necessarily need to use API keys).  

Also, consider whether you can make things simpler by defining rights around standard user profile.  For example, ‘bronze’ customers have access to certain APIs or requests vs. ‘gold’ customers.

Finally, consider any other information you can give developers that will make it faster and easier to work with you - to get their questions answered without calling you.   For example, usage statistics and alerts on versioning upgrades and outages. 

Do you need to start from scratch? 

Before building or buying this, consider if you might be able to extend some other existing user management UI and processes to work for developers requesting access to your API.  

(For example, we built our own user management system for Apigee freemium API management over a couple days with our existing salesforce.com setup.) 

Key to this decision is - how much integration do you need into your existing business processes?   Do you need to create accounts in your CRM system, and make sure you have enough developer information to map developers to applications, and customers? 

Consider the following user management questions for your API roadmap: 

For on-boarding developers

• Do you already have a way to manage user accounts that you plan to re-use for your API? Or have you considered it?
• If you have, do you also wish to offer OAuth keys?
• If you have no user management at all, what does a user need to do in order to sign up to use your API?
• Can they sign up quickly and easily using a web browser?
• Can you simplify things by defining ‘tiers’ of users?
• What kind of information do they need to have access to?

For maintaining and managing accounts

• Can you re-set their passwords?
• What user interface do you want to create for user management?
• Can users do it using a web site or is there some other way?
• Can you monitor their usage?  Can they monitor it themselves
• Can you revoke user accounts?
• Do you need to implement an approval or screening process?
• Do you need reporting and analytics around users – active developers, engagement and retention rates?

Integrating your API users into the rest of your business

• Does your developer activity need to get mapped into your sales, support, and ERP systems?
• Does your API key structure enable you to map developers to applications, your customers, and their end users?
• Does user data need to be integrated with existing profiles or user data systems?  Can you use existing SSO (single sign-on) systems?
• Can you create the right usage incentives by providing developer access to their own usage data?

How Apigee calculates API response time »

Earlier this week we showed how to caculate the latency that Apigee adds to your total API transaction (vs. not using Apigee as a proxy).

This is different than the API 'response time' that you see in your Apigee API analytics dashboard graphs and tables.

If you've set up your Apigee proxy and have run some traffic, you'll see graphs for "response time" in your Apigee dashboard.  (If you haven't seen a dashboard yet, these metrics include messages, developers, error rates, data in/out, and response time)

For example, the widget above and charts below are some of the data I see for the before/after latency test I ran against the Yahoo Local API in the previous  API latency blog entry,

This 'response time' is the roundtrip latency from time time your request enters Apigee - hits the target - waits for the target's response and that response leaves Apigee on the way back to your client.

Here is how your Apigee dashboard cacluates "response time".       For the timestamps:

    • T1 – when message arrives at Apigee
    • T2 – when message leaves Apigee to go to the target (in this case the Yahoo Local API)
    • T3 – when message arrives at Apigee from the taget
    • T4 -  when messages leaves Apigee to the Client

The Apigee 'response time' graph/table shows  T4 - T1 = Response times of the Apigee Proxy including time taken by the Target.  In this case, the Yahoo Local API response time was about 23 ms.

Hope this is useful and we'd love your comments or questins here or on our support and feedback forum.

API Scalability, part 2 - caching, rate limits, and offloading »

(Following from Tuesday's blog entry on API Scalability and Caching.

Last time we wrote about 3 things to think about when planning how to scale your API.

• Caching
• Rate limiting and threat protection
• Offloading expensive processing

and then talked about caching at length, so let's finish up with:

Rate Limiting and Threat Protection

Another aspect of scaling is just keeping unnecessary traffic away from your application servers and databases. Some of the techniques that we've discussed previously, such as rate limits and threat protection, apply here as well.

For instance, an API's performance can drop precipitously if a client, on purpose or by accident, sends too much traffic. A rate limit helps a lot here!

Bad requests can kill API performance too. XML threats, which we discussed in the last episode, are one example of a way that a bad request from a client can cause performance problems or even a crash on the server side. It's a lot easier to maintain scalability if you can stop these kinds of problems before they can hurt your servers.

Server Processing Offloading

Finally, consider the things that you can offload from your application server tier. The more you can offload to more efficient platforms, the less load your application servers have to handle. Plus, the more things you can offload, the simpler those application servers and their applications become, which means they're easier to manage and easier to scale.

For example:

SSL. Load balancers and ADCs like F5 and NetScaler products, not to mention web service proxies like Sonoa ServiceNet, can process SSL more efficiently than most application servers.

HTTP Connections. Those same products are highly optimized to handle tens of thousands of simultaneous connections from HTTP clients, and operate a smaller pool of connections to the back-end application servers. Offloading HTTP connection handling to another tier can free up a lot of server resources.

Authentication. If you perform authentication, a proxy like Sonoa ServiceNet can handle all your authentication for you, freeing your application servers to worry only about properly-authenticated requests. And if you're using SOAP, a product like ServiceNet can process many of the SOAP headers, such as WS-Security headers for authentication, then remove them so that the application server doesn't even need to see them.

Validation. If your API depends on XML input, it may run more efficiently if it only accepts valid XML requests. Turning on XML schema validation can hurt performance of most application servers - products like ServiceNet can do it more efficiently.

So to finishing up, key Questions to ask for your API scalability roadmap might include:

• What kind of volume are you expecting?
• Are you prepared if you get 10, 100, or 10,000 times that amount of volume with little warning?
• Do you have a way to shut a user off if they consume too much volume?
• Do you have a way to control API traffic in case you are unable to handle the volume (see Traffic Management)
• Are your back end servers capable of handling tens of thousands of concurrent connections?
• Are your back end services cacheable? Do you have a cache that you can use to reduce response times?
• Are you monitoring response times and tracking them to gauge customer satisfaction?

(next time:  API user management and oboarding)

Testing API latency and response time with Apigee »

There were some good comments last week on TechCrunch on the pros and cons of using a proxy for analytics and protection (or any operational or business policy) on your API.

Biggest concerns discussed were: latency, single-point-of-failure, and loss-of-control.   All great points.

We wanted to talk about latency first.  (and address the other two in a later post.)

A proxy definitely adds latency.   Both for the additional server hop and processing time of the proxy software.    So any proxy needs to minimize latency and add enough value (capability, time-to-market, etc.)  to justify this extra hop.
 
Our conservative estimate for Apigee is to expect 200-400 ms of latency.   This is mostly due to the extra hop and includes the 20-40 ms of latency due to Apigee's proxy 'think time.' (More detail our latency FAQ)

Your mileage might vary based on message size, the policies you are enforcing, and where you are hosted.  For example, our estimates are based on a 5K message size.  If you proxy Twitter with it's small messaages, your latency will likely be less, and if you are processing big message sizes (such as inserting ads into email), it will likely be higher.  
 
Test it yourself

Soon we'll introduce a tool to test your Apigee proxy's latency during the proxy setup process. In the meantime, you can test this yourself with Apache Workbench (or cURL) by:

1. Set up your Apigee proxy (or feel free to use my Yahoo Local API proxy in the steps below)
2. Open up a terminal.
3. Run a *before* test -  get the latency *without* apigee.  Run this Apache Workbench command (for 10 test requests). 

For this example, I'm using the Yahoo Local API's example API methods. 
 
ab -n 10 http://local.yahooapis.com/LocalSearchService/V3/localSearch?appid=YahooDemo&query=pizza&zip=94708&results=2

(This is an apache workbench command where -n 10 specifies 10 iterations)

You should get a results set in this format (where the "10" was for running the test 10 times). 

 So you can see - just hitting Yahoo Local without a proxy I get a latency of 250ms for all 10 requests.

4. Next, I get the latency *with* Apigee using my Apigee proxy URL.  (Feel free to use this URL yourself, don't worry, I rate limited it in Apigee)
 
ab -n 10 http://yahoo-local-1.apigee.com/LocalSearchService/V3/localSearch?appid=YahooDemo&query=pizza&zip=23662&results=2
 
In this case my results are:

In this case, my longest response with Apigee is 357ms.

5.  Subtract (3) from (4) and there is your approximate latency for the proxy.   Here the latency was roughly 357 ms - 250 ms = 107 ms for my 10 requests, on my verizon card outside Berkeley's Cafe Roma.  (thanks to Yahoo Local's great API for the recommendation.)
 
Run this a couple times to make sure your responses are consistent, and also mixing up your API query parameters so you don't accidentally compare a cached vs non-cached response time. For example, I changed zip codes in my Yahoo Local requests.

Turn up the volume: API Scalability with Caching »

(Part 7 in our blog series: 'Is your API Naked?: 10 API Roadmap considerations".

So far our discussion of APIs has focused on aspects like security, visibility, and data protection. But how do you make your API scale?

"Scale" means different things to different people, so let's narrow it down to the question of what to do as your traffic increases? Do you have a plan to handle 10, 100, or 10,000 times more traffic than your API is receiving today? 

The truth is that solving this problem at the high end can require fundamental changes to your architecture and code. The kind of engineering required to run an API that accepts a few requests per second is very different from what's required to scale to the size of Facebook or Twitter.  

Writing about all the dimensions of scalability and how to achieve them is a subject for a pretty long book. But here are a few specific things to think about:

• Caching
• Rate limiting and threat protection
• Offloading expensive processing

In this entry, we'll focus on caching.

Caching

Caching is a huge part of any scaling strategy. Whole products and web sites are built around caching as a fundamental architectural concept. Caching works because even in an API, usually much of the data that's returned is read. A good caching strategy can decrease latency by huge percentages, and improves throughput by taking load off expensive back-end servers and databases.

Typically, caching today is done in a few different places. Caching between the application server and database using a product like Coherence or memcached helps by reducing the number of database queries needed to serve an API request. Additional caching inside the application server code can further decrease throughput by making it possible to reassemble parts of an API response from pre-cached parts.

For instance, the response to a typical Twitter API call consists of an arrangement of many individual snippets of XML, each of which may have been cached to reduce the overhead needed to fetch data from the database and convert it into XML.

HTTP and CDN caching

Caching at the HTTP level is also very common. The HTTP protocol supports several headers that enable caching by proxy servers and of course by the web browser itself. This type of caching works fine for API calls. However, since it's based only on the HTTP request, it doesn't work for everything. Imagine a SOAP API, for instance, which includes an HTTP POST body describing the request - HTTP-only proxies cannot cache the response because they can't look inside the request and see what needs to be cached.

Similarly, CDNs like Akamai work by caching pieces of content all over the Internet so that users will receive it from a server near them. Caching strategies like this are great for big files that don't change often, but by design they are poor at data that changes more than every day or so.

Caching API responses

With APIs, it's also possible to add caching in between, by caching entire API responses. For instance, imagine a "getAccount" method on an API.  API management solutions like Apigee Enterprise can look at the parameters to the request and use them as a "key" to the cache. If a response already exists in the cache, Apigee Enterprise simply returns the exact response from the back-end server from its in-memory cache.

This cuts out all the latency to make an HTTP request to the application server, process the request, query the database, assemble the response, and so on, and unlike a cache that works only at the HTTP level, it can be configured to understand the semantics of the API so that it caches effectively and correctly. There are also ways to purge cache items programmatically, so that the "updateAccount" API method ensures that the next call to "getAccount" won't return stale data.

In other words, adding a intelligent caching layer between the client of an API and the back end application server adds more caching options that can increase scaling and decrease latency even further than some alternatives because this cache is closer to the API client. Plus, due to the magic of HTTP, XML and/or JSON, it's possible for this caching to be performed without any changes to the API client or server.

Summary: Up to 4 tiers of caching

This means that a highly scalable web site could use caching in different tiers, each with its own contribution to overall API performance:

• A cache like Coherence or memcached may be used between the application server and database to reduce database load.
• A similar cache may also be used from within the application server to reduce the overhead of assembling API responses.
• An API proxy like Apigee Enterprise can cache complete API responses, reducing load on the application server tier.
• A CDN like Akamai can provide an extra caching layer for large files that do not change often.

Up next: API Scalability through Traffic Management and Server offloading

APIs as Dark Matter: our vision for Apigee »

We'd like to share some of our thinking about APIs and why it motivates us to build Apigee into the world's best website for API analytics and management. 

The API Economy 

Web APIs are not mainstream, but they will be. The money being made from Amazon, Facebook and other APIs is just the beginning. Today, APIs are measured in hundreds - about 1,400 listed on ProgrammableWeb. Web UIs, on the other hand, are measured in tens of millions.

In the future, many more websites will provide APIs. And new companies will be formed with revenue models exclusively focused on web APIs. If we look 10 years out it's easy to imagine millions of web APIs.

Because APIs, by their nature, are networked together each additional API will amplify the value of existing APIs. That network effect will create an explosion of value that matches or exceeds the magnitude of today's web economy.

That's the API economy. It's going to be big. But we need some important stuff before it becomes a reality. 

APIs are the dark matter of the web 

We know they're out there. We know they're important. We can infer their existence from mashups and integrations - if we update Twitter on an iPhone it shows up on Facebook. However, we're not directly observing or effecting APIs. And until we do, APIs won't have the massive economic impact that websites have had.

Today, there are hundreds of ways to understand websites, privately (Omniture, Hubspot, Google Analytics, etc.) and publicly (Alexa, Compete, Comscore, etc.). In 1995, when Urchin was founded, that wasn't the case. Back then, there were few ways to understand how websites behaved or what people did with them. As a result, websites were often unreliable, they changed slowly and didn't always have the content people wanted. As the tools for understanding the web evolved, the web itself evolved and became more valuable.

Web APIs are about 10 years behind web UIs. Today, we can't benchmark APIs, we can't see which APIs are popular and we can't effect the way APIs behave without writing a bunch of code. 

APIs at Web Scale 

With Apigee we want to fix these problems. We want to make APIs at web scale a reality.

Our initial set of Apigee features is focused on protection and visibility. API providers can setup policies that enforce their terms of use and ensure uptime, acting as a circuit breaker that protect servers from overloading. Mashup developers can create heartbeat policies that monitor the APIs they rely on. With reports and analytics, API providers and mashup developers will gain visibility into their APIs.

Apigee users will be able to anonymously share their API data. Once API data are public, all of us will benefit from a global understanding of the API web. Just as we use Alexa, for example, to understand the UI web, we envision people using Apigee to understand the API web.

It's important that Apigee be a website and follow the rules of the web: Apigee has a simple pricing matrix with a free version, getting started takes about 2 minutes and Apigee will get better as more people use it.

To make the API economy happen we need tools like Apigee to work at web scale. Our company DNA and the technology Apigee uses - Sonoa ServiceNet's API Router - is all about high-scale networking. We've learned a lot from Sonoa enterprise customers about doing this at carrier grade levels of reliability and scale. 

Our Vision 

Apigee gives API providers and mashup developers the visibility, control and scale they need for their APIs. They will be able to share their data publicly so that all of us benefit from a better understanding of the API web. Once we evolve our tools for understanding APIs, we will see APIs themselves evolve and become more valuable.

We are at the beginning of the web API era. In the future, the value created by the API economy will rival that of today's web economy. 

Try it! 

You can take a look at how Apigee works now with YouTube demo videos: one for API providers and one for API Consumers - developers and mashup artists. You can also request an invitation to try the preview of Apigee today.

Introducing Apigee: Freemium, Self-Service API Management »

   Today we are delighted to announce Apigee. 

Apigee is a website that provides analytics, protection and control for APIs.  If you offer an API you can use Apigee to understand usage, protect your app, and enforce API terms of use.   If you are a developer using APIs in a mashup, mobile or social app, you can use Apigee to get visibility into the actual service levels of the APIs you consume.   

Apigee is freemium and self-service.  You can start using it in minutes and Apigee's Basic service is free for under 10,000 requests per hour.  Apigee is built on the same Sonoa ServiceNet technology that our enterprise customers use for industrial-strength API management.  You can think of it as a freemium, simplified subset of policy framework available in Sonoa technology.

Beginning today we are offering Apigee in a private, invitation-only preview.  Anybody can sign up for an invitation here.   More details are on the Apigee blog.

Show me the money: API Monetization »

(Next in our series on API roadmap considerations and following our last entry on "API business models")

At some point most API product managers are on the hook to demonstrate how the API results in downstream revenue.

And even if you never intend to charge for your API, you may be surprised by unexpected opportunities.

I used to be responsible for a number of ‘free open, take-it-or-leave-it APIs’ at a large Web portal. After a few months, the team was surprised to get a ton of calls from big companies that wanted to pay (a lot) for use of the API within their commercial products. 

One small detail -  they expected that the API could support some of the basic business terms of any commercial deal.  For example – could we:

• Meter service and give them a bill every month?
• Enforce a special rate limit give them an “SLA” (service level agreement)?
• Insert extra data fields if they were willing to pay for more licensed content?

We also needed some other basics:

• SOX compliant audit logs for each deal.
• Reporting on the cost of the data served by the API. (because we were licensing it from another company)
• The means to enforce similar terms on a ‘package’ of multiple APIs together in one deal

Unfortunately, all of these needed to be coded as they came up, so it was a big strain on development, PM and BD to tough to lock down and execute deals in time.

So – even if your API business model is already covered and baked in your API – you may want at least the option to support a slightly tailored or customized relationship.   You may need to quickly tailor the APIs syntax, protocol, priority, or content to a specific opportunity.

Once you have partners using an API, your business managers and partners will put your team under a lot of pressure - dollars are lost each day a business policy change takes to make.

So consider if you’ll ever need to abstract these policies – the difference between the API content and features that and the commercial business and operational terms of each deal – in an a separate API policy layer that you can manage quickly and independently of development cycles.  Or to make a single change across multiple APIs. (here's a case study on an API policy layer.)

Therefore, some questions to ask for your API roadmap might include:

General business and partner model questions:

• How is your business and revenue model supported by your API?  (are you looking for distribution and awareness only? Does the API drive a monetized transaction?  Will you ever charge for ‘pay by the API drink’?)
• How are your costs reflected in your API?  Do you pay for any of the data you are serving up?  If so, how do you keep track of this for the business and partner?
• Will large partners or deals surface where your team will need to change the API content, SLA, protocol or content?   Is there a partner that might want to pay enough and who is large enough to drive your team to move a way from “one size fits all?”
• Will you need to create ‘service tiers”?

Enforcing unique business and operational terms

• How will you meter service like a utility, so that you can bill partners and report data costs?
• What regulatory compliance will you need to support?  Do you need SOX-compliant audit trails by partner?  HIPPA?  PCI?
• How would you create and enforce a partner specific SLA, rate limit, or offer ‘priority access’ to a priority partner?
• If the partner wants any change in the API protocol or content – do you need to support a different API code-base?   Is there a way to create a transformation layer to handle and scale this?
• Do you need to buffer up or queue incoming requests?

Next time: API User management and onboarding

(and thanks to unhinderedbytalent for the photo - check out the detail on the full pic..)

API Business Models and Monetization »

Even with the success of APIs like Twitter, Amazon and Facebook, it can still be a struggle to articulate the value of opening an API to execs and other business folks whose support is needed. (Maybe this is why so many APIs are launched as skunkworks projects.) But we can start by identifying the business model.  Common ones with open APIs are:
 
APIs as a marketing channel

Kipp Bodnar argues than any CMO should consider an API to extend brand awareness and consumers’ perception by letting developers write applications to distribute your content. And this might be at a fraction of your online marketing budget. To measure ROI, you could start by looking at the number of interactions you are getting through APIs or by tracking the traffic boost to your website.  
 
APIs as distribution channel for your content

If your company has some valuable content or data, APIs are a natural way to increase syndication. Indeed data accessible through APIs is easy to retrieve and can be embedded in other websites and applications. Many consumer web products, such as Google's many search API products, use this effectively to distribute content all over the web, which in turn drives their main advertising business model.
 
APIs are the cheapest and fastest way to build applications

You would love to build applications on all the different platforms your customers use - iPhone, Blackberry, Pre, Facebook, MySpace..the list goes on.  While it may never be possible to cover every platform, with an open API developers can help you create applications much faster than your team might.  For example, Twitter has no shortage of apps for every platform. 
 
APIs to distribute services

SaaS companies often use an API to distribute additional services. Your API could be either free because it's part of their existing subscription (a great way to differentiate service from competitors) or as an 'add-on' service for incremental revenue. SpinVox Announced last week 600 Registrations for SpinVox API which converts voice to text. They charge 35c for a 30 second message. Apparently their pricing did not discourage developers.
 
APIs to let third-parties extend your product

The same way than you would not be able to build all type of clients for different platforms, you might not be able to build specialized solutions for each market segment and vertical. By opening APIs you might create an ecosystem of partners and developers that augment your core offering with specialized solutions and innovative ideas. This makes your offering much stronger for your customers. Saleforce does this well - Force.com and the App Exchange cover a rich spectrum of specialized solutions they might not be able to provide otherwise.
 
APIs make your business more sticky

There is no secret than in the enterprise industry software integration projects are expensive and once in place, integration code rarely changes. SaaS vendors and services providers that managed to get deeply integrated in their customers IT stack tend to stick. 
 
So, what is your API business model?

Up next:  Roadmap and technical considerations for API monetization.

Deja vu:  As went networking, now goes cloud.. »

Some people thought it was unusual for a bunch of networking guys to start a company that wants to simplify the cloud services governance world.    But we see parallels all the time between the evolution of networking technologies and web services (SOA, APIs, Cloud services, or whatever you want to call them).

One parallel -  constantly shrinking 'islands of complexity' .    When IP started happening in the network world, there were many complex technologies like SNA, IPX, CLNS, etc.   First IP was used as a much simpler way to connect these 'islands of complexity' because there were such high barriers to making the complex technologies work together.   Over time, the islands got smaller and smaller until it was just IP.

And now here we are in the services world -  SOA, WS-*, ESB/middleware - all this enterprise complexity being connected together aross domains -  by simpler and simpler standards such as REST and technologies from the consumer world.   LIght weight service infrastructure (or call it Cloud or API infrastructure) will be used to connect these islands of complexity until the islands shrink away - and simpler APIs for app2app communication become the de facto standard.

-Ravi is a co-founder and VP Engineering of Sonoa.  Ravi developed the BGP protocol and Cisco Express Forwarding in Cisco and while in Redback architected and led the development of the SmartEdge Service Gateway, a Subscriber Management and Edge Routing platform.

How is cloud computing related to SOA?  Case study on API Policy and Governance Patterns »

Last week, Scott Metzger of Truecredit.com gave a great case study presentation on how they opened their internal SOA as APIs for partners at the Burton Group Catalyst conference.  Specifically, the different policy and governance patterns.

Scott talks about the factors driving them to identify and implement a separate application agnostic layer for 5 major policy patterns including service access, routing, caching, transformations, and operations. (And more details of their implementation in this video)

Scott Metzger of TrueCredit Catalyst Presentation

Protect your API: Twitter’s Denial of Service Attack and API Security »

Lots of news on the Twitter attack last week. There’s general consensus that it was a distributed denial-of-service attack (DDOS) that it targeted a particular account.

DDOS attacks are tricky things. There’s no one technique, product, or protocol that will stop them. That’s what makes them so nasty. To defend against one, a company needs to be able to quickly take countermeasures at all the different levels of the protocol stack, including firewalls, routers, load balancers, and even in the application itself. And even more importantly, you need to have experienced, well-trained operations people who know how to quickly identify an attack and come up with a way to minimize its effects.

When a DDOS attack affects a web service – and there’s an good chance that the Twitter attack made use of the Twitter API – then proper traffic management at the API level is an important part of this stack of protection.

For instance, if a DDOS attack targets a particular user account, or API key, it may be possible to block those particular requests before they get to the back-end application servers, so that the effect of the attack is limited. Or, if the attack always sends some of the same parameters in an HTTP query string, or in the body of an HTTP POST, it may be possible to detect those patterns and reject the requests, again before they reach the back-end application servers. In the worst case, it may even be necessary to shut off access to a particular web service or web service operation, allowing other services to function as best they can while the attack is neutralized by other means.

In other words, just as it may be possible to stop a DDOS attack by blocking a range of IP addresses, or redirecting certain URLs, or configuring traffic shaping in a router or an ADC, it may sometimes be necessary to stop a DDOS attack closer to the application level. The ability to inspect requests at the web services level and take action based on the request content gives an operations team one more weapon to use against an attack.

The Twitter API already has a usage quota on each user account, a rate limit on each IP address, and extensive caching. Imagine how much easier a DDOS attack might have been if they didn’t even have those things.

-Greg

(Greg Brail is the CTO of Sonoa and writes on API Security topics such as API Keys, OAuth, and API Security Recommendations.) 

Tech Talk: API Security and Threat Protection »

Greg recorded a few whiteboard talks last month - this one is a good summary of recent posts on API Keys, API security recommendations, and OAuth best practices. 

(And here are some of our high-level API security features if you are looking into this.)

One size doesn’t fit all: API Versioning and Mediation »

(continuing our series on API roadmap considerations)

TrueCredit.com tells a story of calculating they would need thousands of IP addresses for all the different versions and flavors of their open API - to account for different variations and versions needed by partners.  

Even if you have a ‘one sized fits all’ API - you might need to be able to transform data, mediate terms or customize SLAs without coding each change or a creating a new version of the API. Reasons could include:

•    Protocol needs - A SaaS customer with a REST API had an important deal on the table with a bank, but the Bank insisted on a SOAP API with WS-Security.  Some SOAP shops want to offer a RESTful API because it’s easier for developers to work with. And you might need to transform between different syntaxes of SOAP, REST, or REST/JSON, etc...
•    Monetization – You might want to sell a premium version of your ‘one size fits all’ free API.   For example, a search API provider wanted to do a BD deal with it’s free API and needed to insert extra data ('enrich the API' as they called it) the partner wanted to pay for.
•    Standardization –  A customer of ours grew from offering 1 API to 40, and needed to add some standard fields to each API - enforcing some consistency without needing to coordinate a bunch of teams to write code.
•    Versioning - Ever used an API where you get an email every month asking you to upgrade to a new version?   TrueCredit wanted to provide API upgrades to customers that needed it while holding the API fixed for everybody else longer, to reduce versioning headaches.

So you may need to figure out how you to provide and manage different flavors or versions of the same API – or ‘mediate’ (or transform) API content and syntax.

Alternatives might be to support multiple APIs (painful), hold off as long as possible and push back on customers to snap to a ‘one size fits all’ model (more painful), or create a ‘mediation’ capability or layer that can transform between different ‘shapes of the API – protocol, data, version, credentials, etc.

(And going back to TrueCredit’s story at the top, this is what led them to think about an API gateway for mediation, caching, load balancing, and more.)

So ask if and when any of these issues might apply to your roadmap:

Will you need to mediate protocols?

• Will you need to offer more than one protocol or a different protocol?  (SOAP for enterprise customers? REST or JSON for developer adoption? )
• Would you ever need to map across different security or credential schemes?  (ex: from simple HTTP auth to WS-Security)
• Will you need to handle syntax issues across a particular protocol (SOAP 1.1 vs. 1.2, etc.)
• How important will it be to minimize API versions?