Big Broad Data: The role of Data APIs »

In previous posts on this topic of Big Broad Data, we looked at some of the reasons for and implications of enterprises shifting their focus from the “bigness” and technology hype of “Big Data” to breadth and diversity, signal extraction, analytics and deep insights.

The future is around the easy consumption, the flow and interaction of data, which drives a revolution in the world of Data APIs. The structure of the Data APIs becomes increasingly important.

Building an Information Halo around APIs

Let’s consider enterprises as one of two types from a data perspective: those for whom data is the core business and those who give data away to attract increased transactions to the core of their business. 

In the latter case, the data (information) itself is not necessarily monetizable, but it attracts people to the business.

For a great discussion about the notion of information halos around your core business, see Sam Ramji’s talk: Amundsen’s Dogs, Information Halos and APIs: The epic story of your API Strategy.

In both scenarios, data is a fundamental and critical part of an API strategy

Enterprises who are monetizing around data are beginning to plant flags in different domains. Weather, finance, real estate, Internet traffic and dozens or hundreds of other domains are forming.

The people building the data in any domain are doing so by collecting from disparate data sources. To build out any one domain, they’ve probably stitched together data from a large number of data sources, cleansed and standardized it before finally exposing it as an API.

What are companies using to collect and stitch the data?

A natural and familiar stitching technique is the linked data model (linkeddata.org). While linked data techniques are excellent at accessing individual data elements, I argue that this is not the model that these data providers need. Instead they need to crawl, bulk load, and access data in large quantities, before cleansing, standardizing, and delivering it.

If linked data is not the most effective method to stitch data together to create the domains (at the bottom of the stack), can linked data become the de-facto standard to express data out of the information halo (at the top of the stack and as the Data API for domains)?

The answer is probably yes – eventually. I think it is an unlikely scenario just yet. Today, the challenge is how to cleanse, standardize, unify and use the data in individual domains. Linked data techniques have the right characteristics to bring together data that have already been cleansed, standardized, and stitched but is not a great model to do the initial stitching. It will most likely become useful and common in the future when the inter-linking of domains becomes more important than it is today.

If linked data is not the approach to expose data as APIs, what is?

The school of thought to which I subscribe is one of schema-based access to data APIs patterned after relational models. Here are a few examples of data APIs, which highlight three common kinds of data access patterns.

* Primary key lookup - to get to a specific data element.
* Imposed hierarchy-based lookup - in which you have classes with hierarchy and in effect traverse the hierarchy to get to the data elements.
* Rectangular lookups - defined by typical relational lookups of rows and columns.

All of these techniques are being built around single data sources as opposed to massively linked data sources.

The structure and “RESTification” of  Data APIs

There are several approaches to Data APIs. In addition to the perspective that is Pragmatic REST for SQL Developers, there’s Microsoft’s OData approach. As I asserted in a recent talk at an OData Meetup event at Microsoft, OData is a step in the right direction but there are certain things OData needs to do to become the de-facto standard.

The “RESTification” of the Data APIs is a fundamental imperative and both the Pragmatic REST for SQL and the OData approaches are good starting points.

Whatever the solution is, it cannot be vendor specific. Data is too important, and the data revolution too fundamental for it to be associated with any one vendor.

OData technologies need to be available in all ecosystems, not just in the Windows Foundation Classes (WFC) library and the .NET Framework. Similarly pragmatic REST and other techniques cannot be available in Apigee or any other single vendor offering only.

Let the call to action be to come together as a community; get the best of the linked data and OData ideas and techniques together and transform the world with Data APIs.

The conversation has already started and we’d love to hear more of your perspectives and arguments over on api-craft.

Big Broad Data: Increasing the signal to noise ratio »

In my previous post about Making the shift from Big to Broad Data, I made the case for thinking about Big Data not so much as “Big” but as “Broad.” We looked at the explosion of new data sources in today’s economy, which are individually typically smaller and more diverse than the enterprise systems of record of the past. Data comes from a variety of sources like Twitter, Facebook, partners, tens and hundreds of apps (some built around your APIs), and more. 

To be responsive and make business decisions, an enterprise simply has to be responsive to data across many more sources than in the past.

Signal extraction and stitching data from diverse sources

Whenever you collect a lot of data, you collect a lot of both signal and noise. In fact, a lot of the Big Data approaches to date are focused on extracting the signal from the noise in the data collected from traditional enterprise sources.

As the data an enterprise collects has shifted from 5 to 7 primary "enterprise-centric" data sources (point-of-sales data sources, supply records, customer records, warehousing records, and so on) to hundreds of diverse and typically smaller sources (hundreds of apps, hundreds of social networks, business networks, and so on), the size of the data doesn’t matter nearly as much as the number and diversity of data sources you need stitch together to extract meaningful signal. 

Now with the footprints of customer and partner behavior spread across hundreds and maybe thousands of data sources (see Making the shift from Big to Broad Data), there's undoubtedly a smaller signal/noise ratio than before. Data need to be stitched together to ensure that the signal rises above the noise.

Successful enterprises will be those who understand and consolidate the data that they own as well as that which they can acquire. Today, the challenges are striking deals and forming partnerships to get access to hundreds, not handfuls of data sources. It's no longer about old styles of purchasing data or getting them from departments in your own organization.

Enterprises need to think about access and control; whether they need to push analytics to the data or push data to the analytics, and so on. In the past, bringing data to a central repository in the enterprise - whether a warehouse or other techniques - where it could be analysed was a less significant concern.

Data APIs will lead the way for easy data consumption, flow, and interaction

Fundamental questions are What are the mechanisms enterprises need to stitch data together from its own enterprise as well as syndicated and external data sources?

How will people interact with other people’s data? (We’ve got to understand the form in which this data will be exposed and how it will be consumed.)

In Web 1.0, techniques centered around Web crawling; in Web 2.0 it was about Web pages, AJAX and other "rich" interface technologies. I think that in Web 3.0, broad, diverse data will be accessed, consolidated, and correlated through the power of APIs.

Today, transactional and data APIs are the Ying and Yang of APIs and the API conversation is fairly dominated by transactional APIs (which achieve tasks like sending messages, making trades, getting credit information, and so on). But I think we’re looking at a revolution in the world of Data APIs because the future is around the easy consumption, the flow and interaction of data. As APIs are central to the evolution and handling of big, broad, diverse data, so is data central to the evolution of APIs. The structure of the data APIs becomes increasingly important.

In my next post, we'll take a look at some of the schools of thought around structure of Data APIs. We'll explore the different techniques companies are using to collect and stitch data to create individual domains and to expose data out of those domains. See The role of Data APIs »

Making the shift from Big to Broad Data »

In my previous post, I laid out why I think we need to move beyond the hype of Big Data technology and “bigness” to focus instead on the breadth and diversity of data, as well as signal extraction, analytics and deep insights from that broad data. 

Here we’ll delve into what we mean by "Broad Data" as well as some of the fundamental changes for businesses in today’s marketplace that compel the need to focus on breadth of data and on data stitching from disparate sources.

The shift of control to the edge of the enterprise

Social, mobile and cloud influences have caused enterprises to undergo a tectonic shift in how they do business with customers. The real value for an enterprise - the interaction with end users (customers) - has shifted one or two tiers away from the enterprise.  The control is shifting to social networks where people are talking about companies and products; to business networks where interactions are happening through partner channels; and to apps and the APIs they leverage.

The landscape for customer interaction with enterprises looked significantly different just a few years ago than it looks today. Data was controlled within the enterprise – all of the data that an enterprise gathered were collected when partners and customers interacted with systems produced and provided by the enterprise.

But today’s landscape reveals an expansion of the interaction with customers by one or two degrees from the core of the enterprise. The evolution of the apps and API economy has resulted in people using apps that may or may not have been created by the enterprise. Apps then are the vehicles that inform the enterprise about how customers and partners are interacting with them.

Factor in the influence of social networks, partner- and business- networks and the effect is amplified. Simply put, the enterprise is no longer in control of the data it needs to inform and make accurate business decisions. That’s the fundamental shift of interaction to the edge (and even beyond the boundaries) of the enterprise.

This shift in the market has a fundamental implication for the Big Data conversation. The number and variety of data sources is much more important than the volume that comes from any one source.

Big Data becomes Broad Data

Data is not by itself "Big". Aggregated fragments of small and contextually related data make for "Big" - more accurately - "Broad" data. Taking advantage of the breadth of the data, its variety, its dynamism, and its disparate sources is the real future.

Just a few years ago, the data an enterprise collected were collected from physical stores, Web sites, and partners and from 5 to 7 primary data sources. Data from point-of-sales data sources, supply records, customer records, warehousing records, and so on reflected all the interesting things happening with respect to an enterprise’s interaction with customers and partners.

Today the sources and types of data are expanding continuously - there are hundreds of new data sources, each generating data (which might be small or not-so-small) and definitely generating a smaller signal/noise ratio.

The shift is significant - from 100% of data captured from 5 or 6 sources to a scenario in which maybe less than 50% comes from those original sources. In time, I contend that the old enterprise sources may not even be the most important source.

The many new sources are much smaller and from a variety of relatively new sources: from Twitter, Facebook, partners, tens and hundreds of apps, some built around your APIs. The list goes on. This essentially defines the need for the shift from the deep and big focus of the old world to the broad and pervasive focus of the new world. This will allow businesses to focus on all of the new places in which there is the potential of a signal relevant to their enterprise.

Whenever you collect lots of data, you of course collect lots of both signal and noise. Next time, we’ll look at increasing the signal to noise ratio of broad data - Big Broad Data: Increasing the signal to noise ratio »

Big Data: Beyond the ‘Bigness’ & the Technology (video & slides) »

Thanks to all who participated in last week's Webcast, Big Data: Beyond the 'Bigness' & the Technology. We explored moving beyond the "bigness" and technology hype of the typical Big Data conversation to how  businesses need to respond to the explosion of new, disparate and dynamic data sources as social, mobile and cloud influences shift customer interaction to the edge of the enterprise.

The video (~35 min.) and slides are below. Thanks @jhingran.

We'd love to continue the discussion on the api-craft forum.

Big Broad Data: Beyond the “bigness” and the technology to extracting meaning »

The amount of data in our world has been exploding, and the concept of “Big Data” - collecting and analyzing large data sets—needs no introduction. It’s the buzzword of 2012 where IT is concerned.

There's been a focus on the business side of Big Data, which of course is a critical component of the discussion. Big Data is most certainly the next frontier for innovation, competition and productivity (McKinsey Global Institute, 2011).

However, a quick Google search, a track of #bigdata in your Twitter feed, or 5 minutes in a conversation with folks about "Big Data" will show you how the weight of the discussion focuses primarily on two things - first on technology and then on "bigness".

While both are important, I think that the focus on the technology and size is misplaced and causing us to miss the point that the depth of analysis of the data and the insights we get from them are the most important and valuable things.  

“What’s your tool set?”

Hardly a conversation happens in the big data space that doesn’t start with the pros and cons of Hadoop, No SQL, Cassandra, Hbase . . . the list goes on. Technology is of course extremely important because without it we couldn't determine the signal over the noise or handle large data sets. But the technology is almost commodity. (And of course, trying to get two of us technologists to agree on a technology is a whole different discussion.) 

"How big is your data anyway?"

Right behind the technology argument is the “Bigness” – the petabytes vs. terabytes argument. There are certainly technical complexities to dealing with petabytes of data. But terabytes and even kilobytes are big and more importantly they too hold valuable information.

Remember that a lot of the size will come from noise whether you’re dealing with kilobytes, terabytes, or petabytes. Big, noisy data is not valuable - the value will come from the signal that you can extract.  

Extracting Meaning

To successfully glean value from big data, we've got to pivot the discussion to focus on the breadth of the data, signal extraction, and deep insights. This should make us think about the areas above or below the technology and not on the technology itself. Bottom line - the data itself is the real gold – the new currency.

The disruptive technologies of social, mobile, and cloud that are transforming how we do business serve up the breadth of data. Data about a business' customers is available and interpretable in all kinds of new contexts. A customer that checked in at the gym on Foursquare before visiting a retailer is likely to be interested in sports stuff. You can imagine hundreds of similar examples.  

What's a good example of value from extracting signal over noise?  A Klout Score uses data from social networks to measure reach and influence. It is a signal extracted from a superabundance of tweets and other social interactions.

Deep Insight is about how people can take the output of the machines and convert it into business value. We might come to know that shopping cart abandonment is higher from apps on Android devices than on iPhone devices, indicating that Android apps are less persuasive.

There’s also a fundamental change for businesses because of the apps and API economy that compels the need to focus on breadth of data and data stitching from disparate sources.

I’ll talk more in upcoming posts about  "broad" data and "data stitching" as well as how Data APIs will lead the way in the exploding apps and API economy. We also discussed these topics in a Webcast last week. (video and slides here)

API Strategy and “Niches”: Are you missing an opportunity? »

A couple weeks ago Eventbrite launched an iPad payment card reader for event merchants to sell tickets and merchandise on-site.

However, as Austin Carr from FastCompany reported in this excellent article, Eventbrite built the system in-house instead of partnering with the obvious choice, Square.

Why?

Square is a very successful and innovative company - one of Fast Company's 50 Most Innovative Companies and is processing $4 billion in annualized payments. This could have been a classic 'niches' opportunity for Square... creating a new channel for their payment processing services by being embedded in Eventbrite applications. 

Eventbrite would carry the Square service to its events customers and event attendees, generating business for Square. This is a classic illustration of the new value chain model that savvy companies are executing towards with their API strategies.  Thinking about this as a possible missed opportunity for a company like Square, you should stop and consider, "what opportunities is my company missing by not having an API strategy?"

Austin nails the second major point later in his article as he writes:

Companies are exploring their ‘niches’… reaching out into the places where customers are, and selling in that context. Data and insights about those customer interactions, which are happening more and more at the edge of your enterprise through mobile apps and devices, are incredibly valuable.

Taking advantage of a data platform to give you and your API partners deep insights on your API transactions is table stakes for companies who are extending their reach and growing revenues via APIs.

Why APIs? (video & slides) »

Thanks to all who participated in last week's Why APIs Webinar in which Brian explored why APIs are important to your organization, which strategy is right for you (Internal, Partners, Customers or Open API), as well as how to map your API strategy to your objectives and target channels.

The video (~30 min.) and slides are below. Thanks @landlessness.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

API Facade Pattern: Technology for orchestration, transformation, compression, authorization »

In the previous post about technologies to implement an API facade, we covered technologies for handling versioning and caching scenarios as well as firewalls. This time we'll summarize the technologies to handle other common use cases. 

Transformation & Compression
To handle the common use case for apps, in general, of transforming XML to JSON or vice versa, you would use a transformation library (like XSLT) in the facade.

OAuth
Finally, let's look at how to handle authorization through the API facade. A request comes into the facade with an OAuth token or other indications of the authorization scheme. The facade can make the calls to the authorization system of record.

If the request is valid, the facade passes it to the core system; if invalid, the facade returns with an invalid response code.

Next - we'll take a look at the people considerations for implementing an API Facade.

API Facade Pattern: Technology for versioning, firewalls, caching »

In the previous post about technologies to implement an API facade, we covered setting up your test and production environment with DNS settings, a Cloud Platform, Web servers, app servers and an API Gateway. This time we'll summarize the technologies to handle some of the more common use cases including versioning, caching and securing with a firewall. 

Versioning & URL routing
Once you've got subdomain routing taken care of as discussed in our previous post Technology for Set Up, you can look at designing to handle multiple versions.

This is a similar scenario to subdomain routing but in this case you're doing URL parsing. Here for example, a request comes in for v2.

v1 of your facade may point at an old system, while v2 points at a new system. In this way, you have a simple way to shunt between two IP addresses and handle the scenario in which you need multiple versions of your API available.

Firewall
You want app requests coming through the API facade. You don't want anyone figuring out the IP address to which your facade is pointing and bypassing it. If the API facade is bypassed, you'll be unable to track the requests and unable to apply the API design logic you've built into your facade.

To counteract this, you create a firewall to block all the API traffic with the exception of the trusted IP address of the facade. That is, you ALLOW the IP address of the facade in firewall. Your system is then secure with all requests coming through the facade.

Geo DNS & Caching
In this scenario, we'll build out our facade functionality some more by adding a geo-distributed DNS. The DNS sends the app to a geographically close API facade, based on the source of the request.

Next - we'll take a look at technologies to implement other common patterns: orchestration across APIs, transformation and compression, and authorization.

Six Design Patterns for Highly Successful Mobile Apps »

For years, building web apps has required server-side code and a database. That was great for browsers, but then mobile apps came along and changed everything. Today apps appear on any number of devices, including browsers, and the same app needs to run seamlessly on multiple devices. 

Apps are fundamentally different from their web application counterparts. Traditional web applications run on a monolithic, server-side stack, which delivers content in the form of web pages. For the most part, browsers simply display this content. When a user interacts with a web page, requests are sent to the server for additional content.

In contrast, mobile apps run on a client. Instead of deploying program logic to a back-end server, the logic is housed on the client. This fundamentally changes the dynamic between client and server, making the client no longer merely a display mechanism.

Additional considerations for the development of mobile apps come in the form of the resource constraints imposed by mobile platforms. Mobile apps need to be as lightweight and agile as possible. In other words, you simply wouldn’t run a LAMP stack on an iPhone.

Another challenge for Mobile app developers is dealing with the cumulative traffic that can be generated by even a modest number of clients.  For example, many mobile apps connect to the cloud regularly throughout the day to perform authentication, upload and download data, or to perform geo-queries against a user's current location.  This can result in what amounts to a distributed denial of service attack (DDoS) against an underpowered service architecture.

An added consideration is the data-warehousing infrastructure necessary to deal with all the information being collected. For mobile apps to deliver cutting edge, rich interaction on mobile devices they must be supported by equally rich services and data storage mechanisms.

Enter the cloud.

By hosting rich services in the cloud, client apps can deliver the interactivity that users desire while maintaining a lightweight footprint. With the emergence of mobile PaaS (Platform-as-a-Service) solutions, the days of app developers having to find another developer with a different skill set to help or spending half or more of their development time building services, may soon be gone.

Everything from storing users, groups, roles, single sign-on authentication, social interactions, feeds, queries, connections between users and objects can be provided by a PaaS solution. This leaves the developers to focus on writing cool apps and not on the server code.

So what services do mobile developers need?  We think these 6 patterns are a great start.

1. User Management
Who are you? Who do you know who has the same app? Who do you know outside the app?
One of the most fundamental needs of app developers is user management - straightforward ways to handle users’ (customers’) access control, groups, roles, subscriptions, upgrades, reminders, and so on across multiple devices and apps.

Internet ID and social connectivity also relate to user management. Developers are being rewarded for building identity into apps - from showing pictures of users to bridging to other identities like Twitter and Facebook.

2. Connected & Social Interactions
Who do you follow? Who are your friends? Who will you invite?

In-app and extra-app interactions are all core to mobile apps and devices. For example, an app might bridge out to do a notification or invite a user to join.

In the past, this would have required a generous helping of custom server-side code. Development platforms that provide activity streams, linked profiles, and so on facilitate the development of connected and social interactions.

3. Activity Streams
Activity streams are a kind of static query on moving data. They take connections and turn them into activities, adding a social layer to data and opening up real-time collaboration.

Popular activities vary depending on context but activity streams enable activities such as status updates, check-ins, comments, or other broadcasts to fellow users off of any object or off of multiple objects.

4. Sync Content & Data
Apps need to be able to share data with users, and import, export, and sync application data with third-party applications. The ability to sync data across devices allows developers to create apps, which provide consistency of experience. Think about your favorite “shopping list” or “to do” app that syncs across your laptop, your iPad, and your mobile phone!

5. Location, location, location
Geo-location or Location-Based Service (LBS) apps attach real-world locations to mobile devices and are one of the fastest growing types of apps. You can establish when and where you receive information, track your peers and friends in real-time by location and proximity, find and provide information about things and friends near you.

6. Analytics
App developers need to be able to monitor usage and analyze data to drive both app functionality and business intelligence. Important services include real-time activity processing, behavior tracking and targeting.

Analytics on individual apps are powerful and useful for decision-making about functionality, usage, troubleshooting problems, and so on.  But it’s a game changer when the insights from aggregated data - possibly from multiple apps a developer has created - allow developers to turn what they’re doing into a business and evolve it over time.

API Facade Design Pattern: Technology for set up »

In previous posts in our series about the API facade pattern, we looked at the basic steps to implement an API facade as well as at common patterns.

This time, we'll start exploring the technologies at the heart of implementing an API facade. We'll begin with the set up involving DNS, Cloud Platform, Web server, app server, API Gateway and subdomain routing.

DNS Provider, Cloud Platform

In the spirit of test-driven development, the first thing to set up is our test environment. The first piece of technology we'll need is a DNS Provider.

Set up a CNAME entry, which points to our test facade. A good choice for the subdomain is api-test.

For the sake of our example, we're assuming a Cloud Platform technology because it's the most complicated case and will allow us to explore the most options. It's definitely simpler if everything is behind your firewall.

Web server, app server, API Gateway

Once you have the DNS and the Cloud Platform set up, you are ready to implement the first patterns, errors and data stubs.

To ensure that you have a solid foundation for test-driven development, HTTP codes and error responses need to be in place; you need to be able to stub out data to support mock=true and raise={HTTP code}, and so on. To do so, you need either a static Web server, or an app server for more dynamic content, or an API Gateway (the Swiss army knife for this scenario) in the Cloud Platform.


Subdomain routing and production environment

The next step is to set up your production environment. You'll add a new CNAME entry in the DNS so that the poduction subdomain is api. In our example below, we've pointed api to the same Cloud Platform as in our test environment, but of course you can have different test and production targets if you like.

In the API facade, you'll specify the IP address of the target system.

Note that the error capability is also here in the production environment. Just as it is important in the test environment, you need to ensure verbose error messages and comprehensive codes in the production environment.



With these pieces in place, you'll have requests coming in to api-test and api. In addition to the target IP address, the facade has a shunt that knows where the subdomain is and understands where to point - for example to the data stub server for test or to an internal system for production.

Next time we'll look at the technologies that support versioning and URL routing, firewalls, caching, and more.

The API Facade Pattern: People »

Thanks to all who participated in the fourth and final episode in the Webinar Shorts series on the API Facade pattern.

The fourth episode covers people considerations - the team structures, the roles and responsibilities and the politics - for building and using an API facade. The video (~22 min.) and slides are below. Thanks @landlessness.

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

API Facade Common Patterns: For internal & external systems »

In the Webinar and blog posts about the API Facade pattern, we've talked about API facades that make it easy to provide access to internal systems, and a lot of companies use a facade in this mode. 

Another use of the same pattern is to easily consume APIs from external systems.

All of the same issues and considerations come into play with internal and external systems. We've seen a number of cases in which core businesses rely on external services. We've also seen cases where those service providers change their pricing models and with that the business that is tied to the service provider can see its profit margins shrink, it's SLAs deteriorate, or other business relationship change. A facade pattern can help mitigate risk in cases like this.

With a facade pattern in place, apps that consume the external services can expect a canonical model to consume the APIs. Also, if implemented through the facade, the external services can easily be plugged and replaced.

All of the same approaches to implementing common patterns come into play for the external scenario - starting with building out the errors facade and the data stubs - and you've created your canonical model of service consumption.

API Facade Common Patterns: Versions & data formats »

In this post in our series about common patterns in the API facade, we'll look at patterns for designing versions and data formats. 

Versions

Best practices and principles for versioning your API are described in RESTful API Design: Tips for versioning. Here we'll focus on designing for a scenario in which you need to support more than one version. This is common scenario especially in certain phases of your API's life cycle.

The way to handle this with a facade is to design it such that regardless of which request comes into the facade, you have a shunt in place that points the request to the proper internal system, which serves the response.

Data Formats

Different developers have different expectations for formats. For example, an HTML5 developer might want JSON responses while a Java developer might depend on libraries to handle SOAP requests and responses.

Let's look at how a facade handles this for a developer who needs SOAP. Take a developer who does a POST to get a collection of accounts. The facade mediates the POST into the more complicated internal system and returns SOAP. This is a simple scenario not unlike the URL mapping scenario. There's no real data format mediation happening here.

A more complex mediation happens for example when the developer does an HTTP GET and wants JSON in the response. The facade maps SOAP to JSON on the response, probably using XSLT. Note that the developer has no knowledge of the complexity or the mediation and only knows that they are getting the right format data returned for their app.

Data formats is the last pattern in our summary of the 5 common API facade patterns. We've covered errors, data stubs and URLs in previous posts.  Hopefully you've seen how a handful of common patterns allows you to surface simple interfaces to complex systems in a number of contexts, and in predictable and reusable ways. 

Simple interfaces to complex systems

One reaction to the facade pattern is that it adds a layer of complexity to the technology stack. Whether or not an organization uses the facade pattern, there is complexity that must be addressed between the app developer and the API. Often this complexity is buried within individual systems and difficult to track. So there is complexity on top of unknowns - which is bad. The facade pattern actually helps to create order where there was none.

Next time, we'll finish our review of common patterns by looking at how the API Facade is useful as a front for external systems in addition to the internal systems we've discussed in this series.

API Facade Common Patterns: Data stubs & URLs »

In our series about the API facade we've already looked at the basic steps to implement an API facade and started examining common patterns by looking at errors.

This time, we'll look at a couple more common patterns: data stubs and URLs.

Data Stubs

In the same way we designed for errors with the facade unconnected to any back-end systems, you can stub out what response data would look like, and have the facade return that to you.

In the same way we designed the forced raise for errors, you can force the mock. Setting mock = true, you have a shunt in the facade that returns the stub. Again, we're looking at predictable behavior to do test driven development.

It's a good idea to only support the mock attribute on the test server and to raise an error if the mock parameter is included in production.

URLs

I think of the URL as the strongest affordance for a well-designed API. This is where the facade pattern begins to shine.

The goal is something like this - an app developer wants to do something as simple as see a collection of accounts by doing an HTTP GET on /v2/accounts. However, the internal system may be far more complex than /v2/accounts, like this Salesforce URL.

Doing URL mediation through the facade, you can show the developer on the outside the simple /v2/accounts interface while keeping the complexity of the internal system behind the facade.

Supporting limited clients

Here's a scenario to consider in the context of the URL pattern. Certain clients have limitations on the HTTP methods that they support.

Take for example a client app that doesn't support HTTP DELETE.

You can handle this through the facade by making the method an optional parameter. As the request comes into the facade, the facade changes the HTTP method from GET to DELETE. It also strips the method=delete query parameter and translates to original request of the backend system.

Next time, we'll finish our review of common patterns by looking at versions and data formats.

API Facade Common Patterns: Errors »

We've recommended that the best solution to the problem of exposing complex back-end systems' functionality in a way that's useful for app developers is to implement an API facade pattern.

In a previous post we looked at the basic steps to implement an API facade. In the next few posts, we'll examine common design patterns and the problems that the different patterns solve. We'll cover six common patterns: errors, stubs, URLs, versioning, data formats, and internal and external systems.

Errors                                   "When I say errors, you say test-driven development"

Test-driven development involves building test cases and when they fail they help guide developers towards creating the right app. Because the black box is more stringently enforced with a Web API, this model and the errors are more important in the world of APIs and apps than in other areas of software development. 

Design the HTTP codes and responses you want

Start with a facade, not connected to any internal systems yet and then get the error codes right. In previous Webinars, we talked about the 8 error codes we think are most important (shown below too). (The important error codes may vary for your domain.)

Get the error messages in place so that you can start to build your test suite, including the HTTP error codes and the payload response.

Control over the facade - raise the error

Sometimes you want to explicitly cause a raise to happen. Here's a useful trick and pattern we created for a large API provider which worked well. It involves putting a raise query parameter in your HTTP request.

It will raise that HTTP code. When you are building your test suite, this allows you to ensure your app logic handles exceptions properly.

Warning - don't let this make it's way to your production servers.

Test & implement - plug internal system into the facade

You've designed your set of HTTP codes from the outside in. You have a big internal system, which let's say was built on the .NET framework. Microsoft has an extension of the HTTP status codes - 449 Retry With. You will want to map the 449 to something more aligned with what mobile developers are familiar with today. To do so, you can implement a lookup table and transform the 449 code into a 404 error.

So, you started with the design intent - the HTTP codes and responses. You've exerted some controls over the facade by explicitly forcing the raise. And finally, you've tested it all by plugging internal system into the facade.

Next time, we'll look at other common patterns including data stubs and URLs.

“APIs: A Strategy Guide” O’Reilly author webinar & free chapter »

Last week O'Reilly hosted a webinar by the authors of the new O'Reilly book "APIs: A Strategy Guide."  Video and slides are below. 

The book is an overview of API strategy for business executives and this webinar dives into both public and private API strategies.

Thanks to O'Reilly, @daniel_jacobson, @gbrail and @danwoodscito.

Courtesy of O'Reilly, a free book chapter is posted here.

The API Facade Pattern: Technology »

Thanks to all who participated in the third episode in the Webinar Shorts series on the API Facade pattern. The third episode covers technology for an API facade. The video (~15 min.) and slides are below. Thanks @landlessness. 

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

The API Facade: Common Patterns »

Thanks to all who attended the second episode in the Webinar Shorts series on the API Facade pattern. The second episode covers common patterns for an API facade. The video (~20 min.) and slides are below. Thanks @landlessness.

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

API Facade Design Pattern: A simple interface to a complex system »

In the first post about the API facade, we concluded by recommending that the best solution to the problem of exposing complex back-end systems' functionality in a way that's useful for app developers is to implement an API facade pattern. This time we'll look at the basic steps to implement an API facade.

This pattern gives you a buffer or virtual layer between the interface on top and the API implementation on the bottom. You essentially create a façade – a comprehensive view of what the API looks like from the perspective of the app developer and end-user of the apps they create.

The developer and the app that consume the API are on top. The API façade isolates the developer and the application and the API. Making a clean design in the facade allows you to decompose one really hard problem into a few simpler problems.

Implementing an API façade pattern involves three basic steps.

1 - Design the ideal API – design the URLs, request parameters and responses, payloads, headers, query parameters, and so on. The API design should be self-consistent.

2 - Implement the design with data stubs. This allows application developers to use your API and give you feedback even before your API is connected to internal systems.

3 - Mediate or integrate between the façade and the systems.

Three Small Problems  

Using the three-step approach you’ve decomposed one big problem to three smaller problems.  If you try to solve the one big problem, you’ll be starting in code, and trying to build up from your business logic (systems of record) to a clean API interface. You would be exposing objects or tables or RSS feeds from each silo, mapping each to XML in the right format before exposing to the app. It is a machine–to-machine orientation focused around an app and is difficult to get this right.

 One Big Problem

Taking the façade pattern approach helps shift the thinking from a silo approach in a number of important ways. First, you can get buy in around each of the three separate steps and have people more clearly understand how you’re taking a pragmatic approach to the design. Secondly, the orientation shifts from the app to the app developer. The goal becomes to ensure that the app developer can use your API because the design is self-consistent and intuitive.

Because of where it is in the architecture, the façade becomes an interesting gateway. You can now have the façade implement the handling of common patterns (for pagination, queries, ordering, sorting, etc.), authentication, authorization, versioning, and so on, uniformly across the API. (This is a big topic, which we'll delve into later.)

Other benefits for the API team include being more easily able to adapt to different use cases regardless of whether they are internal developer, partner, or open scenarios. The API team will be able to keep pace with the changing needs of developers, including the ever-changing protocols and languages. It is also easier to extend an API by building out more capability from your enterprise or plugging in additional existing systems.

Next time, we'll look at common facade patterns.

API Facade Design Pattern: What is it & do I need it? »

The first episode of our Webinar Shorts series on the API Facade Design Pattern is an overview of the idea and theory of API facades. The video and slides are here.

In this post, we'll look at the design problem that arises for API designers when back-end systems of record are too complex to expose directly to application developers. Many businesses find that they need to craft solutions to deal with exposing complex systems.

The advantages of back-end systems of record is that they are stable (have been hardened over time) and dependable (they are running key aspects of your business), but they are often based on legacy technologies and not always easy to expose to Web standards like HTTP. These systems can also have complex interdependencies and they change slowly meaning that they can’t move as quickly as the needs of mobile app developers and keep up with changing formats.

In fact, the problem is not creating an API for just one big system but creating an API for an array of complementary systems that all need to be used to make an API valuable to a developer.

The goal of an API Facade Pattern is to articulate internal systems and make them useful for the app developer. Let's start by looking at a few anti-patterns that we’ve seen and why we believe they don’t work well.

The Build Up Approach

In the build-up approach, a developer exposes the core objects of a big system and puts an XML parsing layer on top.

This approach has merit in that it can get you to market with version 1 quickly. Also, your API team members (your internal developers) already understand the details of the system.

Unfortunately, those details of an internal system at the object level are fine grained and can be confusing to external developers. You’re also exposing details of internal architecture, which is rarely a good idea. This approach can be inflexible because you have 1:1 mapping to how a system works and how it is exposed to API.  In short, building up from the systems of record to the API can be overly complicated.

The Standards Committee Approach

Often the internal systems are owned and managed by different people and departments with different views about how things should work. Designing an API by a standards committee often involves creating a standards document, which defines the schema and URLs and such. All the stakeholders build toward that common goal.

The benefits of this approach include getting to version 1 quickly. You can also create a sense of unification across an organization and a comprehensive strategy, which can be significant accomplishments when you have a large organization and a number of stakeholders and contributors.

A drawback of the standards committee pattern is that it can be slow. Even if you get the document created quickly, getting everybody to implement against it can be slow and can lack adherence. This approach can also lead to a mediocre design as a result of too many compromises.

The Copy Cat Approach

We sometimes see this pattern when an organization is late to market – for example, when their close competitor has already delivered a solution. Again, this approach can get you to version 1 quickly and you may have a built-in adoption curve if the app developers who will use your API are already familiar with your competitor’s API. However, you can end up with an undifferentiated product that is considered an inferior offering in the market of APIs. You might have missed exposing your own key value and differentiation by just copying someone else’s API design.

Solution – The API façade pattern

The best solution starts with thinking about the fundamentals of product management. Your product (your API) needs to be credible, relevant, and differentiated. Once your product manager has decided what the big picture is like, it’s up to the architects.

We recommend you implement an API façade pattern. This pattern gives you a buffer or virtual layer between the interface on top and the API implementation on the bottom. You essentially create a façade – a comprehensive view of what the API should be and importantly it is the view from the perspective of the app developer and end user of the apps they create.

Next time, we'll look at the basic steps to implement an API facade pattern and then at some common facade patterns.

OAuth: The Big Picture - Free e-Book »

The API Facade: Overview »

Thanks to all who attended the first episode in the Webinar Shorts series on the API Facade pattern.

The first episode is an overview of the idea and theory of API facades. Here are the video (20 min.) and slides. Thanks @landlessness

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum. 

Insights from API data: The human factor »

In the last post after our Webinar Visibility at the Edge - Deep Insights from your API, I talked about what it means to get 360 degrees of visibility from your API. This time, I'll talk about using that 360 view to get deep insights.

Q: It is one thing to keep all the data, it is quite another to gain insights from it.

We've all seen examples of enterprises saying  . . . "let's first store it, we'll worry about its uses later." Is that a good strategy? Is there a danger of collecting a lot of the data that is not very actionable or useful?

If you are generating 1000 TPS, you'll generate around 200 terabytes of data per year. We know that storage costs are going down but if you're storing 200 TBytes in a landfill fashion, you're looking at spending hundreds of thousands of dollars per month.

The basic question becomes - for the cost of collection and storage what value are you getting? You have to get incremental value for incremental investment. The only way to get this incremental value is to not "store and forget." Instead "store, analyze, synthesize, get insights" and then determine what else to store creating a tight loop between data, analysis, and business decisions.

Q: Are there any easy on-ramps to doing this?

There are certain technological considerations that help get started.

Cloud model: Any enterprise looking at an API strategy should look at a cloud-based strategy, either with the help of people who know how to do it or with your own people. It is no secret that getting infrastructure set up is not easy in any sized enterprise.  Enterprise IT infrastructure has a 6-12 month procurement cycle and you have to plan it carefully. But how to plan in this space? You cannot know what API traffic will be or what value you'll get from the data 6-12 months ahead of time.

In a cloud model, you can avoid showing up-front costs without showing the value.

Out-of-the-box API data. Consider the kind of out-of-the-box API data you want to expose to your business users, which can show instantaneous value. It may be primarily operational value, but still instantaneous data that shows value.

Contextual data. Don't boil the ocean. Start with critical data from the enterprise that is “close” to the API. Consider what’s your core context, and expand slightly around the API data, restricting the scope of what you take in.  In other words, take a judicious and iteritive approach to incorporating context data, which should help avoid paying the cost of context without knowing the value it adds.

Q: Is all that sufficient to affect the business?

You need data, you need analytics, and most importantly, for data and analytics to be relevant, you need to derive insights from these two.

Analytics is about running machine learning and computations on large amounts of data.

Insights is about how people can take the output of the machines and convert it into business value.

I believe that machines cannot replace human beings in deriving the insights that are critical to driving and pivoting your business. Deep insights come from API data, contextual data, machine analytics, and people augmenting the machine intelligence.



Q: What are the organizational challenges to succeeding here?

It is important for those embarking on API journey to convince people to invest in APIs but also in analytics. I've made the case in the past that enterprises should make analytics and APIs peers in their strategy. This might mean an 80:20 effort for APIs:analytics. Making an investment in this culture change will pay dividends as you'll be able to show value far earlier than if you spend 100% on APIs up front and zero on analytics.

Another organizational challenge can be that not all API strategies are fully sanctioned arms of their enterprises. Many are nascent and need nurturing. People can still be wary of "opening up" back-end systems to let people interact with them.

Like many things in the corporate world, APIs and analytics need executive sponsorship. Of course, they have to deliver value, but without the executive sponsorship, the skeptics might kill it before it has a chance to prove itself.

Q: Can an API provider do something in the design of an API to make it more "insight friendly?"

I assert it's not a good idea to try to design for insights or to think about insight as an end goal. In this world, you're either in the business of APIs, or in the business of running a business.

If you're in the former, you never want to optimize for insights. Instead, you want insights to help you optimize your APIs for developers, for building apps, for app end users, and so on.

if you're in the latter camp, where APIs are generating a large amount of traffic and transactions of your business, then insights into the business of APIs now become insights into your overall business. Again, you don't want to, and probably can't effectively design for insights. Rather let the insights optimize your business.

That said, you can definitely think ahead to collecting those out-of-the-box data that will give you a jump start.

Q: Should you collect data yourself or gather it from systems already collecting it?

First work with people who have put APIs in front of data - figure out if you can use the data in this way. If not, and if you think you can derive a huge amount of value from the data, you want to be able to be able to slice and dice it so you might want a copy of it for yourself.

Insights from API data: 360 degrees of visibility »

Thanks to all who participated in the Webinar: Visibility at the Edge - Deep Insights from your API. Check out the video and slides here.

In the last post, I talked about what we mean by visibilty at the edge of your business and the concept of 360 degrees of visibility. This time, I'll talk about the data that gets you the 360 degree view.

Q: What’s needed to make this 360 degree visibility happen?

There are a few things.

- First obviously, you need to be collecting data from the APIs.
- Second is collecting data that adds context to the API data. 
- Thirdly, you need analytics that model and predict both the business and operational metrics.

Let's elaborate on data that adds context. Visibility into APIs is the best aperture into enterprises but just API data is insufficient - you need a mechanism to collect other "contextual" data. 

Say a user is interacting with a Telco API. A cell phone number is flowing through that API. Other than that phone number, there's a huge amount of context that is not captured in that API:

- who's number is it?
- what do you know about that person?
- which app is that interaction happening through?
- Are they using only your APIs, or others; are they happy with your API or not? . . .

Knowing the mobile number and knowing the name, address, payment history of the person using that number, and so on, are two very different things.

Clearly visibility into APIs gives you valuable operations information, and even business impact information like the number of products purchased. 

But wouldn't it be interesting and useful to know in what context that product was purchased, in which context the app was built, and so on? API data is great for operational and business visibility. Having contextual data greatly enriches the information for business visibility. Ideally you can collect enough contextual information around the APIs so that your analytics is no longer just looking at the bits and bytes of APIs.

Q: Does this combination of API data and context data dictate a need for a BIG DATA solution for companies?

Yes. Let's look at the two factors that I think are driving to a big data solution. The first is the volume and chattiness of API traffic, and the second is the new ways in which you need to interact with data. 

Think about how your APIs are designed. Are they optimized first for back-end efficiency or for easy consumption?

They should be (and probably are) optimized for consumption with the understanding that the back-end optimization can come later. Optimizing for back-end systems first can lead you down a path where you build a system that's efficient from a back-end perspective, but not necessarily one that is attractive for developers to consume and adopt. 

This necessity to optimize for consumption tends to lead to API traffic that is chatty and voluminous. Even a small sized enterprise that has 1000 TPS can easily end up with 200 terabytes of data every year. That's not huge, but it's also not small for most companies. 

Just as important as the volume of data is how you interact with that data. Traditional data warehouses were built to answer questions you know you have and repeatedly.

The new big data solutions are about determining the questions to ask. In this scenario, it's not just the volume that is at issue - but the variety and nature of questions you need to ask. In an API, app, and customer-centric world you don't really know the patterns and questions at the outset.

Next time, we'll talk about analytics and how to get those insights.

Insights from API data - visibility across the value chain »

Thanks to all who participated in last week's Webinar: Visibility at the Edge - Deep Insights from your API. The video and slides are here.

As promised on the Webinar, we'll write out some of the Q&A we got into in the live session. We'll start with what we mean by visibility at the edge and why we think it's important.

Q: What do we mean by visibility at the edge?

 in the past, enterprises could focus on controlling their interactions with customers through physical presence or websites. Now people are interacting more and more with core businesses through apps. 

Although the interaction has moved farther away and businesses are no longer in control of the interaction in the same way, customers are more important to the business than ever before. More data is coming in from developers and end users through apps and APIs.

Q: Clearly, to get such visibility, we will collect lots of data.

Before we get into the data part, are there examples of the value such visibility might provide?

Think about the number of different constituents around API interactions (even non-API interactions). There are the people transacting with your business, developers building apps, teams in enterprises responsible for the building and the success of APIs, teams responsible for the operations of APIs, and business users interested in the top and bottom lines. 

In addition to various constituents, there are lots of interactions through multiple channels with different systems, including billing, catalog, procurement, order management, and so on.

The biggest challenge is how to collect the disparate perspectives you get through these channels and generate a 360 degree sense of it. If you believe in the apps economy, you believe this will all get standardized around the APIs "interface."

Bottom line, your APIs and the management of APIs gives you visibility into a larger set of interactions than any one back-end system can.

If you harness that visibility, regardless whether you run operations for the back-end systems, your business is to make the APIs successful, or you are a business person focused on the top and bottom line, you can get a lot of insights into your business. For example: 

- How well is the API being adopted?

- What is the response time per request?

- Does operations need to allocate more resources at peak times?

- What impact is an API having on my bottom line?

and much more . . .

But there's something even more transformative one can get from API data. Instead of thinking only about what the enterprise or the API team learns from API data, think about what developers and the app end users can learn from the API data.

For example, if you are a developer, you'd like to know what impact you're having on the business whose API you are using to build your app. 

Visibility into the APIs gives insights for all of the constituents of the value chain, therefore getting us closer to a 360 degree and actionable view.

For more on the business implications, check out this recent post Engage customers where they are - at the edge of your enterprise.

Next time, we'll talk about what else is needed to make this 360 degree visibility a reality.

Visibility at the Edge - Deep Insights from your API (video & slides) »

Thanks to all who participated in last week's strategy webinar:
Visibility at the Edge - Deep Insights from your API

Thanks to our speakers @jhingran and @brianpagano.

Here are the slides and video. Also, check out our follow-up posts this week in which our speakers further explore the topics of this Webinar hour. 

We'd love more of your thoughts, insights, or questions on the api-craft forum.  Or tune in to the new IRC channel #api-craft.

Unlocking business opportunity with APIs and mobile strategies »

I sat down with Brian Gracely (@bgracely) and Christian Reilly (@reillyusa) of Bechtel recently to talk about the evolving API and app economy. We had a terrific discussion about the adoption of mobile devices and how open access to data is unlocking tremendous new business value. Check out the Cloudcast.net podcast.

Christian gives us terrific insight and first-hand knowledge of why it makes sense for a big traditional enterprise to use APIs.  It starts with mobile devices driving the strategy. Mobile devices have evolved from toys in the enterprise to real business tools for people at every level in today's enterprise. We see the same story play out for all kinds of enterprises - from large-scale construction, to retail, to healthcare, to Telco, and others. Data entry and collaboration on job sites and in offices around the globe are now driven by apps on smartphones and tablets.  This increases immediacy and accuracy of data capture and project management, reduces cost and risk, and improves compliance.

In a pre-API world, enterprises were mired in large applications that were bound behind physical and digital walls.  As mobility grew exponentially and with it different expectations from consumers took hold, these large, legacy, internal applications simply couldn't cut it from a usability or user acceptance point of view. An application accessed from a tablet or smartphone through a browser or remote terminal is not  an acceptable experience, as people get used to cool apps for all aspects of their lives on their mobile devices.

In our conversation, I think Christian aptly captured the new environment as one in which "people don't want apps - they want information."

This realization drives a whole different mindset. In addition to the demand for a different experience on mobile than on desktop, enterprises also have to think differently about how to provide access to the information people needed. APIs were the obvious solution. Enterprises can expose read/write access selectively with APIs and they don't have to replicate the entire app.

Adopting an API strategy allows enterprises to do things like focus their investment on making robust APIs, which makes it easy for developers to build apps. Enterprises can leverage the creativity of (in-house or external) developers to build light-weight UIs (which can be very simple interfaces) that allow users to get and put data from anywhere and from numerous devices. They no longer have to be sitting in an office close to the "enterprise applications" and systems of record like ERP or billing systems.

In this scenario, IT became responsible for exposing the business as a platform, driving a fundamental change in the speed at which business can be done. Instead of IT projects taking years, they were now taking months. As a result, enterprises experience higher level of fluidity and flexibility - much more rapid and experimental development cycles and take advantage of natural feedback loops. This all directly relates to an enterprise's revenue and profitability.

Simply, this means that the constant underlying platform is the API. Apps, while critical for the last mile, are a more fluid component, available for rapid change, augmentation with other apps, or disposal.

There are many examples of companies who understand how to apply and invest in APIs in order to support the type of consumer and partner engagement that's the mark of the economy we're seeing in front of us, and they are being directly rewarded in revenue and profit.

Managing Big Data »

In my recent Data Analytics & APIs post, I talked about analytics as a peer to your API, and using analytics to gain insights to drive success of your API strategy.

An API strategy quickly gets an enterprise to a place where they have big data sets. As defined by Wikipedia, "Big Data consists of datasets that grow so large that they become awkward to work with using on-hand database management tools."

No question that Big Data comes with big opportunity for valuable insights but also with challenges and questions about data storage, how to effectively manage the growth of data sets, methods for understanding data analytics and gleaning the most valuable insights from it.

Many people have commented, "big data is great, but before I get an ounce of value out of it, I am stuck with the cost of storing and managing it."* This is an understandable pain. However there are some simple solutions to manage this pain. Once managed, the gains from Big Data are easily achieved.

Let’s take a look at an enterprise with APIs that are getting requests at about 1000 TPS.  Even if, conservatively, each request generates 1K of data, we are looking at ~100GB/day, or ~300TB/year, for just the raw information.  While people tend to focus on the “storage” costs of managing this kind of information, it is almost always the people costs that dominate.  Then add the complexity of running analytical tasks, and the simple “systems management” problem (not the byte storage costs) will tend to be overwhelming.

I have argued in the past that APIs benefit from cloud deployment.  Today I will argue that the analytics on APIs also benefits tremendously from cloud deployment.

People Costs

Let’s first tackle the people costs. Managing hundreds of TB is not for the faint of heart.  Clearly, when you have such a large volume, one would like to use the cheapest storage, which often times is significantly more unreliable than SAN storage. The storage needs to be backed up.  The hundreds of servers need to be monitored. When bad things happen (and when is it that bad things never happen? ), corrective action needs to take place. Many of the Big Data technologies are immature, and have a fast release/patch cycle, so someone has to worry about patch management too.  The list goes on and on.

One soon realizes that it is often better to have some experts “manage” this for you – typically a cloud delivered service.  The service provider has in-house data management experts, might even have in-house data scientists – this allows you to focus on the business you have to run, and the value that Big Data will deliver, rather than all the blocking and tackling that needs to be done managing that data.

API Burstiness

APIs by nature are “bursty” in use. How does one provision the Big Data infrastructure that is needed to support this (over provision for peak, or under provision for average)?

It is absolutely clear that instantaneous provisioning of new data and storage is not an easy model for any enterprise to execute on, let alone a small to medium sized enterprise. Failure to provision leads to enterprise amnesia^#. The data is either lost forever, or can only be analyzed days or weeks after the fact, when the new systems are provisioned.

It therefore makes perfect sense to look at a cloud delivery model for your API analytics.

I am not suggesting that there aren’t issues to deal with as you make these decisions. For example, issues about data security are always top of mind. However, as cloud-delivered services mature, these issues are less and less bothersome.  The benefits are very little pain in moving your API analytics to the cloud, and not having to worry about all the headaches of managing the infrastructure, so that you can focus only on the gains.

* Hadoop Has Promise but Also Problems- Jessica E Vascellaro, Wall Street Journal
# Enterprise Amnesia: Organizations Have Lost Their Minds Jeff Jonas

Data Analytics & APIs »

There's an explosion of developers building mobile and web apps. There's also an explosion in APIs as enterprises adopt API strategies to open up access to their back-end systems to enable developers. As a result, more transactions flow from end users and developers to enterprises increasing reach, revenue, and profit for enterprises.

In this short video, I talk about how to think about analytics as a peer of your APIs and the role analytics plays in having your API strategy succeed.

Every enterprise expects to see increasing rate of transactions (sales, subscriptions, etc.) over time. API-delivered  transactions are going to contribute an increasingly larger fraction to this top or bottom line.

For an enterprise that is early in its API journey, a small fraction of total transactions come from APIs.

For an enterprise in a later stage of its API program, APIs contribute a much larger fraction of the total transactions.

For all these companies, analytics on APIs benefit many constituents including developers, API managers, operations teams, and business managers.

For the early adopters - the companies embarking on their API journey - analytics optimize the business of APIs. IT and API metrics like throughput, availability, latency, errors etc. all help improve your APIs which results in an increased rate in the growth of your API strategy.

For the seasoned enterprises, analytics become even more important to optimize the business of the enterprise. Leveraging business-level analytics on APIs -- which products are purchased, which are tagged as favorite, which API resources are most used, etc. -- the enterprises in which APIs already contribute a high fraction of transactions will also see an uptick in the rate and relative contribution of API business to the bottom line.

Irrespective of the phase or size of your API program, your business will benefit from analytics.

OAuth: Implementing OAuth 2.0 »

In a recent OAuth post, we recommended that if your API can require HTTPS, use OAuth 2.0.  Otherwise, use OAuth 1.0a.

How should you use OAuth 2.0?

There are three types of credentials in OAuth 2.0.

BEARER TOKEN: This type requires SSL. A bearer token is a big random number. Bearer tokens are easier for developers than OAuth 1.0a because they don’t have to write the signature. This is the most straightforward of the credential types to implement.  Use it!

MAC TOKEN: Like OAuth 1.0a, uses signature instead of SSL. The spec is still changing so I recommend that you wait.

SAML: Makes sense if the API Team and the developers really want SAML. The OAuth standards team is working on ways to use OAuth with SAML but that option is still changing. I recommend that you wait to implement SAML.

In short, I recommend that you use bearer tokens, as it is the simplest to implement.

How many legs?

I often get this question – how many legs does OAuth have and how many legs should I use?

Let’s look at the 3-legged and 2-legged terminology that’s developed around OAuth.

3-legged OAuth refers to the fact that there are three entities - a user, a client (application), and a server (service) involved. Both the user and the app have credentials.

2-legged OAuth refers to the case where the OAuth signature method is being used to identify just the application – that is, to identify just the client and not the user. This is often done using SSL. But the OAuth 1.0 signature can be used.

In my opinion, 2-legged OAuth uses only a part of OAuth and 3-legged OAuth is really OAuth.

If your API needs to authenticate users, you should use 3-legged OAuth. It has benefits for the API Providers and above all, it will mean improved security and better end user and consumer experiences with web and mobile apps.

Would love to hear your thoughts or questions over on the API Craft Google groups.

OAuth: Is it worth the effort? »

In the previous few blog posts, I’ve talked about what OAuth is and when and why I recommend it.

This time, let's explore some of the reasons why OAuth is more cumbersome and complicated for developers than plain passwords as well as some recommendations to help you make the decision between OAuth 2.0 or 1.0a.

Because OAuth eliminates password sharing between web apps, and password storage on mobile devices, and importantly for the sake of your end-users’ experience and security, I believe it is worth the effort.

How many versions of OAuth and which one should you use?

OAuth 1.0 The first "production" version of OAuth 1.0 didn't actually do authentication delegation correctly and as a result, the spec itself had to be patched. No one should be using OAuth 1.0 now.

OAuth 1.0a Not an IETF standard but stable and well understood.
Uses a signature to exchange credentials between client and server. This means that you get API security without SSL (not only is the password never exchanged but the OAuth token is never exchanged – it’s encrypted inside the signature). But coding the signature right is tricky.

OAuth 2.0 Actively under development in the IETF.
It is getting more stable and changes less with each draft – core flows are unlikely to change. Supports a ‘bearer token’, which is easier to code than the signature but requires SSL. However, most APIs should be using SSL by default anyway. Optional specs support signatures, SAML, and so on, but those specs are not yet stable.

The authentication dance is painful

There are a lot of great sites where you can get the details of the OAuth 1.0a authentication flow chart, so I'm not going to tackle that here. It’s not simple. It gets a little simpler with bearer tokens in OAuth 2.0 but because of the security requirements and the fact that you have to exchange identity without a password, it's always going to be a little complex.

Signatures are painful

The good news is that in OAuth 2.0 signatures are optional. You can do SSL and use a bearer token.

As I mentioned above, getting the OAuth 1.0a signature algorithm right is difficult. If you’re going to use OAuth 1.0a, I recommend you use one of the many libraries that are available.

Where do you store the credentials on the client?

There is no magic. You have an OAuth token and secret that needs to be on your mobile device and accessible.

You could encrypt it, but then you need access to the key that decrypts it.  You could also break them into smaller pieces.

What should you do? 

Be patient. If your API can require HTTPS, use OAuth 2.0 with bearer tokens. (Use the latest draft of 2.0.) Otherwise, use OAuth 1.0a.

It's possible to build an effective API right now using either a current draft of OAuth 2.0 (see Facebook) or OAuth 1.0a (see Twitter).

Bottom line, with your end-users’ experience and security top-of-mind, I believe OAuth is worth the effort.

Next time: Implementing OAuth 2.0

OAuth: For server-to-server APIs? »

In my last post, OAuth: Why it's good for API providers, I talked about why I recommend OAuth for API providers when they are exposing APIs for web or mobile apps.

This time, a short follow up to look at a couple of scenarios in which OAuth might not be the best solution.

APIs designed to be used only by servers.

When the only clients of your API are servers, OAuth is not the best solution. Having a separate set of authentication credentials for each app is a nice feature of OAuth, but for server-to-server use, the need to log in securely using a browser gets in the way.

Simply assigning a unique password to each app is probably sufficient. Two-way SSL is another good, albeit cumbersome approach.

I believe that OAuth is unnecessary for APIs that are used by a small number of internal or partner systems, and which are used by servers and not by end users who have passwords.

However, think ahead! If you discover that those APIs that are only accessed by servers today are useful by other types of clients (like web or mobile) tomorrow, then you'll need to support OAuth.

Are there other scenarios for which OAuth is unnecessary or even a bad idea?

Just like the server-to-server scenario, I think that OAuth doesn’t make sense in the following scenarios:

-       Anything that requires commercial levels of trust. For example, when your security model requires the capabilities of a PKI infrastructure.

-       One-time tokens. OAuth is a lot of complexity and machinery to make one API call.

Bad ideas include creating your own ‘version’ of OAuth or creating something that’s like OAuth but different. Sticking with standards, and focusing your development efforts on creating great apps seems like a better idea than rolling your own security scheme.

Next time: We'll talk about some OAuth complexities, and why it's worth the effort.

OAuth: Why it’s good for API providers »

In OAuth: The valey key metaphor and OAuth: Flow for mobile apps, we talked about why OAuth is good for users - how it allows users to grant third-parties access to their web services or mobile apps without sharing their passwords.

This time, why OAuth is good for API providers whether they are exposing APIs for web apps or APIs designed for mobile apps.

OAuth means that Web apps that expose APIs don’t have to share passwords.

There are two alternatives

-       The security risk of typing your password into every single web app you use – an unacceptable risk!

-       Some sort of universal single sign-on mechanism (run by a third party that everyone agrees upon and trusts) – it doesn’t exist!

Therefore, for all web apps that expose APIs to other web apps - we recommend OAuth.

OAuth eliminates the need to store a password on a mobile device adding a layer of security when an API is used by mobile apps built by untrusted developers for a public API.

An OAuth token is hard to guess. It is tied to a particular app and device. It can be revoked without affecting other devices and apps.

OAuth allows the API provider to revoke tokens for an individual user, for an entire app, without requiring the user to change their original password. This is critical if a mobile device is compromised or if a rogue app is discovered.

OAuth future proofs the authentication process. Because users’ browsers are redirected to a place that’s under the control of the API team to get the OAuth token, the API team can control what to do at that point in the workflow. 

In other words, OAuth is flexible enough to support any of your company’s specific workflows (think SSL, multi-factor authentication, terms of service, changes to terms of service, and so on…) without requiring a change to the client or server.

Therefore, for APIs designed for mobile and native apps – we recommend OAuth

Next time: OAuth in a server-to-server API scenario.

OAuth: Flow for mobile apps »

In my previous post, I talked about how OAuth allows users to grant third-parties access to their web services without sharing their passwords.

In that previous example, our user (Bob) accessed his Twitter account through the bit.ly web site.

This time, let's look at what happens when Bob is using a mobile app instead of a web app.

It’s pretty much the same for a mobile app as a web app. When you fire up your mobile app and want to access Twitter, you don’t want to have to give the mobile app your Twitter password and store it on your phone.

In the same way as happens for the web app,  the phone app opens a browser window and directs Bob to login to Twitter (and authorize bit.ly to use his Twitter account) - using OAuth.

The phone app never sees Bob’s Twitter password.

The phone app uses its OAuth credentials to retrieve credentials for Bob. It stores these locally. In this way,

OAuth allows this one phone app access to the Twitter service on behalf of one user (Bob).

This all happens through your browser. There are both advantages and disadvantages to this:

- The advantage of this is that the app never sees your password.

You don’t have to trust the person that built the app or worry about losing your phone (as much).

If Bob loses his phone, he can log into Twitter and revoke the credentials that Twitter gave the phone app. (A thief cannot uncover the password, only the token.)

- The disadvantage is that the app has to open up a browser window. This possibly breaks the flow of a nicely designed UI for a mobile app.

There are ways in OAuth around this but you eliminate many of the security reasons for OAuth in doing so. Twitter and Facebook for example, have made the choice to have everybody log in through the browser.

Next time: Why OAuth is good for API providers.

OAuth: The valet key metaphor »

OAuth has taken off as a standard way for apps and websites to handle authentication. But OAuth is a confusing spec that can be hard to pin down.

I wanted to talk a little about what is OAuth and when you should use it for your API – hopefully pin it down a little in a few blog posts. I covered a lot of this in OAuth: The Big Picture. Check out the video and slides! 

Let’s start with what is OAuth and why it came about.

The developers of OAuth set out to solve the problem that services and passwords don’t mix well as you start to combine apps and mash them up.

Imagine, in today’s environment of web APIs and mobile apps, if every web site that used an API from another web site had to share and store that web site password. Soon you’d have the proliferation of your passwords all over the Internet with every service you’ve used from Facebook, to Twitter, to Skype, to your bank account. Do you trust them all with your password? 

What is OAuth?

It is another way to authenticate to a service - a security protocol that allows users to grant third-party access to their web resources without sharing their passwords.

OAuth supports this “delegated authentication” between web apps using a security token called an "access token." Rather than relying on a single password as the master key for every app that accesses an API, OAuth uses this token.  

An OAuth token gives one app access to one API on behalf of one user.

Eran Hammer-Lahav, a spec author for OAuth, compares the token to a valet key.  This is an apt metaphor.

For most people, their car is one of their most valuable possessions, valued in tens of thousands of dollars.  They are convenient places to leave our other valuables like computers and clothing.  Yet we are sometimes required to give them to a parking attendant or valet whom we’ve not met before. A valet key solves the problem – it’s an access token with limited rights that can operate the vehicle but not grant access to the trunk or glove box.

For great information about the history of OAuth, see Eran Hammer-Lahav’s guide. 

How does it work?

Simply, there are three entities (legs) to consider for an OAuth scenario:

• The user of a service – let’s call him Bob
• A client (a Web app, a mobile app) – let’s take bit.ly as an example client
• The server (where the service runs) – let’s take twitter as an example

How does our user Bob interact with Twitter through his bit.ly account?

1 - Bob visits bit.ly on the web, which uses a service provided by Twitter. Bob already has logins for bit.ly and Twitter.

2 - Behind the scenes, bit.ly uses it’s OAuth credentials to begin the authentication process for Bob.

Bit.ly redirects Bob temporarily to twitter.com to log in. (bit.ly never sees Bob’s Twitter password).

(The page hosted by twitter that asks if an application can access Twitter on your behalf is probably familiar to most of us by now.)

Note that twitter shows Bob what rights the app is asking for, and importantly what the app will not be able to do – in this case, see that bit.ly cannot access Bob’s direct messages or see his twitter password. 

3 - If this sign in is successful, bit.ly uses its own OAuth credentials (token) to retrieve credentials for Bob (that’s the valet key that allows bit.ly to use Twitter on Bob’s behalf).

4 - Bit.ly stores Bob’s credentials along with Bob’s account. They allow him to use bit.ly and only bit.ly to access Twitter.

Why is it important?

What if bit.ly is hacked and someone steals all the passwords. Bob will have lost his bit.ly password but the thieves don’t have his Twitter password.

The Twitter API team, knowing that bit.ly has been hacked, can decide to revoke bit.ly OAuth credentials. Now everyone that uses bit.ly can’t use twitter. This is extreme but can be important when an app is severely comprised so you don’t have to change your passwords.

On the other hand, if Bob decides to change his Twitter password, the token that bit.ly has for Bob that allows him to access Twitter is unaffected.

Bit.ly never had Bob’s twitter password so he doesn’t have to do anything at bit.ly – and he will be able to access twitter through bit.ly next time he tries.

Next time: The OAuth flow for mobile apps

API Trends: What to expect in 2012 »

Thanks to all who participated in last week's webinar:  "API Trends: What to expect in 2012" with Sam @sramji, Anant @jhingran, and Brian @brianpagno.

Here are the video and slides. Thanks for a great interactive session.

We'd love to hear more of your thoughts and questions; please join the conversation on the api-craft forum.

“Mobile APIs - Optimizing APIs for many devices” Webinar - video & slides »

Thanks to all who participated in yesterday's webinar - " Optimizing APIs for Many Devices"  with Sam @sramji, Brian @brianpagno, and Greg @gbrail.

Here are the video and slides. We'd love to hear more of your thoughts and questions; please join the conversation on the api-craft forum.

Yin and Yang of APIs and the Cloud - Part 2 »

In a recent post I talked about how you need APIs as your business grows; how services need to integrate directly to a companies' business processes through APIs rather than indirectly through a portal or Web site. Check it out here.

This time, because APIs need to scale, I talk about how you need the Cloud to effectively manage those APIs and enable developers to be successful using your APIs to build their apps.

You need a cloud to effectively manage your APIs.

Let's start by comparing API usage in the traditional enterprise scenario with API usage in the growing API economy around social and mobile applications.

In a traditional enterprise model it takes years to build out the enterprise ecosystem of apps and the life span of an app is often measured in decades. Traditional enterprise apps connected to back-end systems are typically designed to carefully shrink and grow their IT capacity. Change is slow, IT requirements are predictable but capacity needs to be available to support maximum loads.

In the new API economy - in a world of mobile and social apps development - we're working in an environment of rapid innovation and a usage model in which we cannot predict or preconfigure capacity. IT needs are not as predictable as in the traditional model. Burstiness or spikes in requests happen as a result of a number of things: API access and usage changes from within apps, app users come and go, apps can get popular for a few weeks, create a surge in demand, then subside.  It's a rapidly changing environment in which you need to be able to scale and respond to rapid bursts of requests that create periods of peak system usage.

Hosting your APIs in the cloud offers the ability to provision and deprovision as needed. In other words, you can avoid spending at the peak rate - rather spend at an average rate and take advantage of cloud elasticity and burst as required in times of peak load. Several enterprise API customers already leverage the Cloud for variable capacity.

Should the cloud be internal or external to your business? Inside or outside your firewall?

Unless your app developers and the APIs they leverage are entirely within the walls of your enterprise, I recommend that your APIs are hosted in a cloud outside your firewall. There are a few reasons:

- The developers who will use your APIs - your audience -  are external to your enterprise and supporting these developers is priority.

- Those developers are familiar with using public APIs. Developers are busy building social and mobile apps meaning that they're comfortable in an environment in which they connect to public APIs offered by Facebook, Twitter, Twilio, SimpleGeo, and so on. 

- It is common for developers to use more than one provider's API in building their apps. If APIs are behind your firewall, you make it more difficult for developers to access those APIs as they need to navigate different firewalls and different back-end systems for the different APIs they want to use. This will adversely impact developer adoption of your APIs. 

In summary, you need a Cloud to be able to scale and respond to the dynamic nature of today's API-driven apps. The Cloud should be on the Web outside your enterprise firewall to effectively support your customers - app developers.

Yin and Yang of APIs and the Cloud »

Cloud services need APIs. APIs need clouds. They are Yin and Yang. If you are working with one, you are working with the other.

First the Yin – if you have a cloud why you need APIs?

One of the ways cloud services (IaaS, PaaS, or SaaS) achieve scale is through customer self-service.

This usually means a customer portal, through which early customers – usually small companies or a department of a larger company – set up and manage their service.

At this scale, “integration” is achieved through people using the portal to interact with the cloud service, which in turn,  ‘integrates’ into the business process.

At some point, the use of the cloud service grows to serve larger companies or a larger cross-section of a company, or both. Now, “people-level” integration through a portal is not enough -  the cloud service must plug in or integrate directly to the companies business process.

At this point, you need an API. 

Tim Madewell from Innotas, a leading provider of SaaS PPM (project portfolio management) services makes this point well.

“In the early days of our company, our average customer had 25-30 users.  But as we started to grow and the SaaS market became more mature, we started working with larger enterprise customers and the product needed to evolve.  When we landed our first 5000-user account, we found that one of many requirements was that we enable integration from their back-end CRM, HR, and billing system.  Exposing the API where the customers can ‘come and get it’ worked well for large customers.”

The transformation that Tim talks about is the change from “here is a web portal that you can use to govern your IT processes,” to “here are the set of entities that we manage, and now you can interact with them in the flow of your processes.”  This is the transformation that is leading thousands of non-cloud enterprises to move from “here is a web site where you can deal with us” to “here are the APIs and now you can work with us.”

This is a big shift for the enterprises, and a big shift for the cloud SaaS providers.  A similar shift is happening to the IaaS and PaaS providers. An example is GoGrid's cloud control APIs where an enterprise can manage cloud provided computing and storage resources in the same seamless fashion as managing their own, internal IT-provided resources.

At IBM, we used to say that hybrid cloud management is a critical future need and that integration will become a big challenge with the proliferation of clouds (see my post).

I did not realize then was that this requires clean (hopefully REST-based APIs) on the cloud provider side.  I also did not fully realize that this requirement extends all the way into the IaaS, PaaS, and SaaS layers.  Companies like Innotas have taught me that indeed, the cloud, like other enterprise systems and applications, needs APIs.

Next, the Yang. I will discuss how all APIs need to scale, and therefore need the cloud to be effectively managed. Check out Part 2.

“Bigger, Better Business with OAuth” API strategy webinar - video & slides »

Thanks to all that attended yesterday's webinar "Bigger, Better Business with OAuth."  

Unfortunately, we had a Webex outage which cut the webinar audio off after a few minutes, but thanks to @sramji and @landlessness for finishing out the webinar for the video (and slides below) after the outage.

Video and Slides - “Developer segmentation for your API”  strategy webinar »

Thanks to all that attended last week's API strategy webinar "Developer Segmentation for your API" on developer community and market size. (And thanks to our speakers @sramji  and @landlessness).

Here are the video and slides (and as always, you are free to use under Creative Commons).

“Your API is not a website!” - webinar video and slides »

Thanks to those that joined us for last week's API how to webinar presentation "Your API is not a website".   (and thanks to our speakers @gbrail and @brianpagano)

Video and slides are below!

Video and Slides - “Boss, we need an API!”  strategy webinar »

Thanks to all that attended last week's API strategy webinar "Boss, we need an API". (And thanks to our speakers @sramji, @brianpagano, and @landlessness).

Here are the video and slides (and as always, you are free to use under Creative Commons).

Join us this Thursday for our next API webinar "Your API is not a website!" by @gbrail - you can sign up here!

Video and Slides: Is your API naked?  API Platform and Ops Considerations »

Thanks to all that attended last week's API Best Practices Webinar #5 "Is your API Naked? API Platform and Operations Considerations" (and thanks to our presenters @gbrail and @landlessness). Video and slides are below.

Our next API webinar, "Your API Sucks!  Why developers hang up and how to stop that" with @landlessness and @earth2marsh, is June 14th at 11am PST (sign up here!)

(And you can see all our API best practices webinars to date here)

Video and Slides: API Metrics - What to Measure »

Thanks to all that attended last week's API Best Practices Webinar #4, API Metrics - What to Measure (and thanks to our presenters @brianpagano and @landlessness). Video and slides are below.

Our next API webinar, "Is your API Naked? API Technology and Ops Considerations" with @landlessness and @gbrail, is June 14th at 11am PST (sign up here!)

Social Commerce as part of ‘Distributed Commerce’ through APIs »

There’s been some buzz about ‘social commerce’ this year, with a great WSJ article a couple weeks back: Retailers Embrace Social Commerce.

However, there’s not really such a thing as ‘social commerce’.

What these companies are doing is executing a ‘distributed commerce’ strategy.To really understand this profound shifft, it's worth watchign Sam Ramji's talk on  evolving business models in his Web 2.0 Strategy presentation: Darwin’s Finches, 20th Century Business and APIs.
 
A key point in this talk is how business is migrating from a direct to indirect model. 

The web revolution that gave birth to Amazon and eBay was based on a shift to a direct business model… retailers selling directly via their websites. 

Today businesses are figuring out how to execute an indirect model.  They are trying to put their product/service/content where their customers are. 

To use some retail-speak, the nuance is that commerce is now about more than about 'selling in' a channel, it is about selling ‘through’ a channel.

So where are customers?

Increasingly, consumers are on the move and on their smart phones.  They are on tablet computers like the iPad and on game consoles (xbox, ps3) and set-top boxes (roku, etc), and filling their car at gas stations. 

Consumers are also spending time on social networks… which brings us back to ‘social commerce’.

Borrowing Sam’s Finches metaphor, retailers are exploring their ‘niches’… reaching out into the places where customers are, and selling in that context.

So a Facebook store is just one niche, one place to reach customers where they are.   Gamestop is another retailer with a Facebook store.  However, Gamestop is also on mobile device apps and other social networks.  They are executing a distributed commerce strategy well.  In the same issue as the social commerce article, WSJ reported that Gamestop’s profit rose 6.9% on higher sales.

Another example of distributed commerce is Shopsavvy, a mobile app that searches for products based on location and brings back pricing, availability and product attributes.  Retailers want to be included in the shopsavvy app to have their products discovered by consumers in that venue.  Shopsavvy itself is exposing its engine to developers who are finding niches of consumers via wine-specific apps, “Lego apps”, etc

In the post-web world, forward-thinking retailers are executing distributed commerce strategies to reach consumers where they are, including on social networks. Where are your customers?  And what is your distributed commerce strategy to reach them?

Globalization, Black Swans, and APIs »

Sam Ramji gave this talk at GlueCon on delivering APIs globally.  Considerations include:

• distribute locally
• serve elastically
• specialize universally

To see what Sam is talking about (and how Black Swans play into it) check out the slides below.  To deliver APIs globally, also check out our API Delivery Network - a CDN for APIs. 

Video: RESTful API Design (Pragmatic, not Dogmatic) »

Recently Brian Mulloy (@landlessness) and Marsh Gardiner (@earth2marsh) hosted a webinar on API design and Pragmatic REST. The video of the recording and the slides are below.

Our next API webinar - 10 Patterns in Successful API Programs - is this Thursday, May 19th at 11am PST, with Brian and Greg Brail @gbrail. Interested in the topic? Then you should sign up now!

Mapping out your API Strategy: Webinar slides and recording »

Thanks to everyone that attended yesterday's API Strategy Workshop Webinar (and thanks to our presenters @sramji and @landlessness)

The slides are below and here is the full recorded webinar. 

Join our next API webinar, "Pragmatic REST:API Design Fu", May 5th at 11am PST.  You can sign up here. 

Free Webinar: Mapping Out Your API Strategy »

Ever need to explain why APIs are so powerful to someone at your company?  Need an easier way to think about the different API strategy options?

Join us for a Webinar - “Mapping Out Your API Strategy” - this Wednesday, April 20 at 11 am PST / 2 pm EST.

It's free (of course) and you can sign up at this registration page.

Brian Mulloy and Sam Ramji will talk about the API market, and propose some ways of thinking (and cool frameworks) for how you can think about API strategy vs. your objectives.  

Punctuated Equilibrium, Celestial Navigation, and APIs - Web 2.0 2011 API Strategy talk »

Sam Ramji from Apigee, along with Dan Jacobson and Michael Hart from Netflix, recently gave this API strategy talk at the Web 2.0 Expo in San Francisco. 

It includes frameworks, best practices and lessons learned to help in thinking of your API strategy from a business model, architecture, and data perspective.   

Why Punctuated Equilibrium and Celestial Navigation? See the slides below and look out for the full video shortly. 

Using your existing systems to create new value with APIs »

Companies need to wring value out of their investments; find new ways to serve customers or create new value with their current systems, products, supply chains and partnerships. 

I keep going back to water analogies.  My house has a plumbing infrastructure that was paid for a long time ago.  I recently found new value in that infrastructure, an automatic sprinkler system.  Again the system reminds me of an API.  Through this one common interface I can connect many endpoints (sprinkler heads) and my yard has never looked better.  I have some endpoints that cover the lawn, some that concentrate on shrubs, others take care of the plant beds.  The point is that I tapped in to my existing water pipes and found new value in a 35 year old system. 

Can you do the same with your enterprise systems?

Let's take a common strategy like a mobile device strategy.  First we have to understand what information customers need when using a mobile device like a smartphone.  Certainly location and product come to mind.  Maybe information about their account or orders.  These little gems of information can be exposed through the API interface and carried to many different endpoints; today a smartphone, tomorrow a tablet PC or connected car.  Let's call these 'functional APIs" since they carry a function to the user through a mobile device. 

The next step to realizing a functional API is to map the data elements and functionality of the new API to their location in the existing systems.  These data elements may map to databases, ERP systems, other internal services, etc.  In fact, a single API may map to one or more internal systems.  Any holes that are discovered will have to be filled. 

For example, if you are designing an API and define features or data that aren't available through existing systems, you will have to provide that new system or new data set.  But, in the interest of embracing the legacy systems, you should be able to reuse the majority of what you have by extending it outward through the new abstract API.

Leveraging what you have through an API can reduce your development cost, give you speed to market and also give you more flexibility in the future to integrate with more devices and partners.  Leveraging my existing residential water system yielded a "yard of the month" award, perhaps this strategy can beautify your business as well.

Next we'll talk about the details of integrating with existing systems.   And ots on this in our recent whitepaper "APIs: A Profit Interface - using APIs to grow your business.

What’s an API?  It’s Money, Baby! »

So I'm on my magical patio again consuming a cold beer and thinking about retail customers and the systems that serve them.

I've been asked too many times in my career to replace legacy systems with bright shinny new retail systems. Now, I'm a bit of a nostalgic guy. My first rule of technology is "embrace the legacy". Good thing because not only has that saved millions of dollars for the companies I've worked for, it's also made them millions of dollars.  

You see, I've found that the fastest path to innovation is through understanding the past; the solutions to even the hardest problems are always obvious with hindsight. Irony at its best. You don't need to see into the future, you need to see into the past.

Freedom to Innovate
With that in mind, you can free yourself from the excuses of not innovating. You won't need millions of dollars in new systems, a hoard of new employees or a 18 month roadmap to change the way you do business in the new multichannel web. Want to reach mobile phones, app stores, set-top boxes, hoards of partners and affiliates? All you need is a way to tap into all the goodness of your enterprise systems and capabilities. All you need is an API, or as I call it, A Profit Interface.

That's the first step - abstract yourself from the legacy systems with an API - technically an Application Programming Interface (a technology blast from the past). Abstracting from the legacy is a bit like asking a girl to drive you to the prom but not making her your official date. You want to play the field and in this age of retailing that means smartphones, tablets, e-book readers, kiosks, game platforms, connected cars and internet everywhere. APIs let you get there.

Getting Everywhere, Fast
Imagine having a single connector, to say your legacy product catalog, that all those new screens could access quickly.  And when the next sensational device arrives, you just snap it in - no coding necessary. How many more sales could you make if your customers had access to your products and services from anywhere?  What if another retailer could access your product catalog and offer your complementary items to their customers? That's money, baby!

I know because the innovations I've seen and guided at companies opening APIs have made millions of dollars. I even got a couple of logo coffee cups for my efforts. One more beer, slides into my logo beer cozy. I forgot i got a cozy too.

I really wish I could pay for all this beer with my phone. I always forget my wallet in the car. Hmmmmm..... how to do that without millions of dollars in hardware upgrades? Maybe someone needs to contact the beer distributors about embracing their legacy systems...

Where will your customers be this Christmas? »

Keith Morrow has a great article on TechTarget series on how an API can create a disruptive impact on a retailer's ability to reach customers beyond traditional web browser channels.

One key point is that an API can be driven by a relatively small team. 

Yet an API has a disproportionate impact on the business, and this impact scales with low marginal costs.  

This can be an especially effective strategy for retailers that might live in world of thin margins, limited resources, and the constant pressure of the peak holiday season. 

For more on Keith's retail API experience, we commissioned him to write a Retail 2.0 API Strategy whitepaper discussing Retail API opportunities, challenges of launching with tight resources, and best practices.

You can download the whitepaper here.

Darwin’s Finches, 20th Century Business, and APIs: Evolve Your Business Model »

What do APIs have in comon with Darwin's observations on evolution, the 20th century garment district, and the Kobayashi Maru?

Sam Ramji makes the case for APIs in his much written about web 2.0 talk  - watch and listen to the full talk or just flip through the slides - both below.

Paypal X: Open API as an Indirect Strategy »

On Tuesday, the WSJ ran this article about the launch of Paypal X, Paypal’s developer program.

Great to see an API showcased as the heart of an industry leader's core business strategy - and called out as such in the top business newspaper. 

Paypal is executing an indirect strategy by opening their core services via APIs in order to have their payment capabilities more easily consumed in other applications and services.

Ten years ago, you could read articles like this about companies launching their websites to execute a *direct* business strategy… selling directly to customers.

Just as it was a decade ago, companies will slowly wake up and realize that they need to have an indirect business strategy online, either because they see the massive business opportunity ahead of them, or they see their competitors already executing an indirect strategy against them.

We see 'indirect online strategies' like PayPal and TransUnion Interactive as the beginning of an emerging API Economy.   A company without an API in 2009 is like a company without a website in 1997.