API Facade Pattern: Webinar shorts full series »

In March, we tried something a little different with our Webinars - a series of short presentations (~30 min. each).

Our first "shorts" series covered the API Facade Design Pattern. Here's a roll-up of the links to the video and slides for all 4 Webinars. 

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

Check out the full series of blog posts around API Facades.

API Facade Pattern: Technology for orchestration, transformation, compression, authorization »

In the previous post about technologies to implement an API facade, we covered technologies for handling versioning and caching scenarios as well as firewalls. This time we'll summarize the technologies to handle other common use cases. 

Transformation & Compression
To handle the common use case for apps, in general, of transforming XML to JSON or vice versa, you would use a transformation library (like XSLT) in the facade.

OAuth
Finally, let's look at how to handle authorization through the API facade. A request comes into the facade with an OAuth token or other indications of the authorization scheme. The facade can make the calls to the authorization system of record.

If the request is valid, the facade passes it to the core system; if invalid, the facade returns with an invalid response code.

Next - we'll take a look at the people considerations for implementing an API Facade.

API Facade Pattern: Technology for versioning, firewalls, caching »

In the previous post about technologies to implement an API facade, we covered setting up your test and production environment with DNS settings, a Cloud Platform, Web servers, app servers and an API Gateway. This time we'll summarize the technologies to handle some of the more common use cases including versioning, caching and securing with a firewall. 

Versioning & URL routing
Once you've got subdomain routing taken care of as discussed in our previous post Technology for Set Up, you can look at designing to handle multiple versions.

This is a similar scenario to subdomain routing but in this case you're doing URL parsing. Here for example, a request comes in for v2.

v1 of your facade may point at an old system, while v2 points at a new system. In this way, you have a simple way to shunt between two IP addresses and handle the scenario in which you need multiple versions of your API available.

Firewall
You want app requests coming through the API facade. You don't want anyone figuring out the IP address to which your facade is pointing and bypassing it. If the API facade is bypassed, you'll be unable to track the requests and unable to apply the API design logic you've built into your facade.

To counteract this, you create a firewall to block all the API traffic with the exception of the trusted IP address of the facade. That is, you ALLOW the IP address of the facade in firewall. Your system is then secure with all requests coming through the facade.

Geo DNS & Caching
In this scenario, we'll build out our facade functionality some more by adding a geo-distributed DNS. The DNS sends the app to a geographically close API facade, based on the source of the request.

Next - we'll take a look at technologies to implement other common patterns: orchestration across APIs, transformation and compression, and authorization.

API Facade Design Pattern: Technology for set up »

In previous posts in our series about the API facade pattern, we looked at the basic steps to implement an API facade as well as at common patterns.

This time, we'll start exploring the technologies at the heart of implementing an API facade. We'll begin with the set up involving DNS, Cloud Platform, Web server, app server, API Gateway and subdomain routing.

DNS Provider, Cloud Platform

In the spirit of test-driven development, the first thing to set up is our test environment. The first piece of technology we'll need is a DNS Provider.

Set up a CNAME entry, which points to our test facade. A good choice for the subdomain is api-test.

For the sake of our example, we're assuming a Cloud Platform technology because it's the most complicated case and will allow us to explore the most options. It's definitely simpler if everything is behind your firewall.

Web server, app server, API Gateway

Once you have the DNS and the Cloud Platform set up, you are ready to implement the first patterns, errors and data stubs.

To ensure that you have a solid foundation for test-driven development, HTTP codes and error responses need to be in place; you need to be able to stub out data to support mock=true and raise={HTTP code}, and so on. To do so, you need either a static Web server, or an app server for more dynamic content, or an API Gateway (the Swiss army knife for this scenario) in the Cloud Platform.


Subdomain routing and production environment

The next step is to set up your production environment. You'll add a new CNAME entry in the DNS so that the poduction subdomain is api. In our example below, we've pointed api to the same Cloud Platform as in our test environment, but of course you can have different test and production targets if you like.

In the API facade, you'll specify the IP address of the target system.

Note that the error capability is also here in the production environment. Just as it is important in the test environment, you need to ensure verbose error messages and comprehensive codes in the production environment.



With these pieces in place, you'll have requests coming in to api-test and api. In addition to the target IP address, the facade has a shunt that knows where the subdomain is and understands where to point - for example to the data stub server for test or to an internal system for production.

Next time we'll look at the technologies that support versioning and URL routing, firewalls, caching, and more.

The API Facade Pattern: People »

Thanks to all who participated in the fourth and final episode in the Webinar Shorts series on the API Facade pattern.

The fourth episode covers people considerations - the team structures, the roles and responsibilities and the politics - for building and using an API facade. The video (~22 min.) and slides are below. Thanks @landlessness.

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

API Facade Common Patterns: For internal & external systems »

In the Webinar and blog posts about the API Facade pattern, we've talked about API facades that make it easy to provide access to internal systems, and a lot of companies use a facade in this mode. 

Another use of the same pattern is to easily consume APIs from external systems.

All of the same issues and considerations come into play with internal and external systems. We've seen a number of cases in which core businesses rely on external services. We've also seen cases where those service providers change their pricing models and with that the business that is tied to the service provider can see its profit margins shrink, it's SLAs deteriorate, or other business relationship change. A facade pattern can help mitigate risk in cases like this.

With a facade pattern in place, apps that consume the external services can expect a canonical model to consume the APIs. Also, if implemented through the facade, the external services can easily be plugged and replaced.

All of the same approaches to implementing common patterns come into play for the external scenario - starting with building out the errors facade and the data stubs - and you've created your canonical model of service consumption.

API Facade Common Patterns: Versions & data formats »

In this post in our series about common patterns in the API facade, we'll look at patterns for designing versions and data formats. 

Versions

Best practices and principles for versioning your API are described in RESTful API Design: Tips for versioning. Here we'll focus on designing for a scenario in which you need to support more than one version. This is common scenario especially in certain phases of your API's life cycle.

The way to handle this with a facade is to design it such that regardless of which request comes into the facade, you have a shunt in place that points the request to the proper internal system, which serves the response.

Data Formats

Different developers have different expectations for formats. For example, an HTML5 developer might want JSON responses while a Java developer might depend on libraries to handle SOAP requests and responses.

Let's look at how a facade handles this for a developer who needs SOAP. Take a developer who does a POST to get a collection of accounts. The facade mediates the POST into the more complicated internal system and returns SOAP. This is a simple scenario not unlike the URL mapping scenario. There's no real data format mediation happening here.

A more complex mediation happens for example when the developer does an HTTP GET and wants JSON in the response. The facade maps SOAP to JSON on the response, probably using XSLT. Note that the developer has no knowledge of the complexity or the mediation and only knows that they are getting the right format data returned for their app.

Data formats is the last pattern in our summary of the 5 common API facade patterns. We've covered errors, data stubs and URLs in previous posts.  Hopefully you've seen how a handful of common patterns allows you to surface simple interfaces to complex systems in a number of contexts, and in predictable and reusable ways. 

Simple interfaces to complex systems

One reaction to the facade pattern is that it adds a layer of complexity to the technology stack. Whether or not an organization uses the facade pattern, there is complexity that must be addressed between the app developer and the API. Often this complexity is buried within individual systems and difficult to track. So there is complexity on top of unknowns - which is bad. The facade pattern actually helps to create order where there was none.

Next time, we'll finish our review of common patterns by looking at how the API Facade is useful as a front for external systems in addition to the internal systems we've discussed in this series.

API Facade Common Patterns: Data stubs & URLs »

In our series about the API facade we've already looked at the basic steps to implement an API facade and started examining common patterns by looking at errors.

This time, we'll look at a couple more common patterns: data stubs and URLs.

Data Stubs

In the same way we designed for errors with the facade unconnected to any back-end systems, you can stub out what response data would look like, and have the facade return that to you.

In the same way we designed the forced raise for errors, you can force the mock. Setting mock = true, you have a shunt in the facade that returns the stub. Again, we're looking at predictable behavior to do test driven development.

It's a good idea to only support the mock attribute on the test server and to raise an error if the mock parameter is included in production.

URLs

I think of the URL as the strongest affordance for a well-designed API. This is where the facade pattern begins to shine.

The goal is something like this - an app developer wants to do something as simple as see a collection of accounts by doing an HTTP GET on /v2/accounts. However, the internal system may be far more complex than /v2/accounts, like this Salesforce URL.

Doing URL mediation through the facade, you can show the developer on the outside the simple /v2/accounts interface while keeping the complexity of the internal system behind the facade.

Supporting limited clients

Here's a scenario to consider in the context of the URL pattern. Certain clients have limitations on the HTTP methods that they support.

Take for example a client app that doesn't support HTTP DELETE.

You can handle this through the facade by making the method an optional parameter. As the request comes into the facade, the facade changes the HTTP method from GET to DELETE. It also strips the method=delete query parameter and translates to original request of the backend system.

Next time, we'll finish our review of common patterns by looking at versions and data formats.

API Facade Common Patterns: Errors »

We've recommended that the best solution to the problem of exposing complex back-end systems' functionality in a way that's useful for app developers is to implement an API facade pattern.

In a previous post we looked at the basic steps to implement an API facade. In the next few posts, we'll examine common design patterns and the problems that the different patterns solve. We'll cover six common patterns: errors, stubs, URLs, versioning, data formats, and internal and external systems.

Errors                                   "When I say errors, you say test-driven development"

Test-driven development involves building test cases and when they fail they help guide developers towards creating the right app. Because the black box is more stringently enforced with a Web API, this model and the errors are more important in the world of APIs and apps than in other areas of software development. 

Design the HTTP codes and responses you want

Start with a facade, not connected to any internal systems yet and then get the error codes right. In previous Webinars, we talked about the 8 error codes we think are most important (shown below too). (The important error codes may vary for your domain.)

Get the error messages in place so that you can start to build your test suite, including the HTTP error codes and the payload response.

Control over the facade - raise the error

Sometimes you want to explicitly cause a raise to happen. Here's a useful trick and pattern we created for a large API provider which worked well. It involves putting a raise query parameter in your HTTP request.

It will raise that HTTP code. When you are building your test suite, this allows you to ensure your app logic handles exceptions properly.

Warning - don't let this make it's way to your production servers.

Test & implement - plug internal system into the facade

You've designed your set of HTTP codes from the outside in. You have a big internal system, which let's say was built on the .NET framework. Microsoft has an extension of the HTTP status codes - 449 Retry With. You will want to map the 449 to something more aligned with what mobile developers are familiar with today. To do so, you can implement a lookup table and transform the 449 code into a 404 error.

So, you started with the design intent - the HTTP codes and responses. You've exerted some controls over the facade by explicitly forcing the raise. And finally, you've tested it all by plugging internal system into the facade.

Next time, we'll look at other common patterns including data stubs and URLs.

The API Facade Pattern: Technology »

Thanks to all who participated in the third episode in the Webinar Shorts series on the API Facade pattern. The third episode covers technology for an API facade. The video (~15 min.) and slides are below. Thanks @landlessness. 

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

The API Facade: Common Patterns »

Thanks to all who attended the second episode in the Webinar Shorts series on the API Facade pattern. The second episode covers common patterns for an API facade. The video (~20 min.) and slides are below. Thanks @landlessness.

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum.

API Facade Design Pattern: A simple interface to a complex system »

In the first post about the API facade, we concluded by recommending that the best solution to the problem of exposing complex back-end systems' functionality in a way that's useful for app developers is to implement an API facade pattern. This time we'll look at the basic steps to implement an API facade.

This pattern gives you a buffer or virtual layer between the interface on top and the API implementation on the bottom. You essentially create a façade – a comprehensive view of what the API looks like from the perspective of the app developer and end-user of the apps they create.

The developer and the app that consume the API are on top. The API façade isolates the developer and the application and the API. Making a clean design in the facade allows you to decompose one really hard problem into a few simpler problems.

Implementing an API façade pattern involves three basic steps.

1 - Design the ideal API – design the URLs, request parameters and responses, payloads, headers, query parameters, and so on. The API design should be self-consistent.

2 - Implement the design with data stubs. This allows application developers to use your API and give you feedback even before your API is connected to internal systems.

3 - Mediate or integrate between the façade and the systems.

Three Small Problems  

Using the three-step approach you’ve decomposed one big problem to three smaller problems.  If you try to solve the one big problem, you’ll be starting in code, and trying to build up from your business logic (systems of record) to a clean API interface. You would be exposing objects or tables or RSS feeds from each silo, mapping each to XML in the right format before exposing to the app. It is a machine–to-machine orientation focused around an app and is difficult to get this right.

 One Big Problem

Taking the façade pattern approach helps shift the thinking from a silo approach in a number of important ways. First, you can get buy in around each of the three separate steps and have people more clearly understand how you’re taking a pragmatic approach to the design. Secondly, the orientation shifts from the app to the app developer. The goal becomes to ensure that the app developer can use your API because the design is self-consistent and intuitive.

Because of where it is in the architecture, the façade becomes an interesting gateway. You can now have the façade implement the handling of common patterns (for pagination, queries, ordering, sorting, etc.), authentication, authorization, versioning, and so on, uniformly across the API. (This is a big topic, which we'll delve into later.)

Other benefits for the API team include being more easily able to adapt to different use cases regardless of whether they are internal developer, partner, or open scenarios. The API team will be able to keep pace with the changing needs of developers, including the ever-changing protocols and languages. It is also easier to extend an API by building out more capability from your enterprise or plugging in additional existing systems.

Next time, we'll look at common facade patterns.

API Facade Design Pattern: What is it & do I need it? »

The first episode of our Webinar Shorts series on the API Facade Design Pattern is an overview of the idea and theory of API facades. The video and slides are here.

In this post, we'll look at the design problem that arises for API designers when back-end systems of record are too complex to expose directly to application developers. Many businesses find that they need to craft solutions to deal with exposing complex systems.

The advantages of back-end systems of record is that they are stable (have been hardened over time) and dependable (they are running key aspects of your business), but they are often based on legacy technologies and not always easy to expose to Web standards like HTTP. These systems can also have complex interdependencies and they change slowly meaning that they can’t move as quickly as the needs of mobile app developers and keep up with changing formats.

In fact, the problem is not creating an API for just one big system but creating an API for an array of complementary systems that all need to be used to make an API valuable to a developer.

The goal of an API Facade Pattern is to articulate internal systems and make them useful for the app developer. Let's start by looking at a few anti-patterns that we’ve seen and why we believe they don’t work well.

The Build Up Approach

In the build-up approach, a developer exposes the core objects of a big system and puts an XML parsing layer on top.

This approach has merit in that it can get you to market with version 1 quickly. Also, your API team members (your internal developers) already understand the details of the system.

Unfortunately, those details of an internal system at the object level are fine grained and can be confusing to external developers. You’re also exposing details of internal architecture, which is rarely a good idea. This approach can be inflexible because you have 1:1 mapping to how a system works and how it is exposed to API.  In short, building up from the systems of record to the API can be overly complicated.

The Standards Committee Approach

Often the internal systems are owned and managed by different people and departments with different views about how things should work. Designing an API by a standards committee often involves creating a standards document, which defines the schema and URLs and such. All the stakeholders build toward that common goal.

The benefits of this approach include getting to version 1 quickly. You can also create a sense of unification across an organization and a comprehensive strategy, which can be significant accomplishments when you have a large organization and a number of stakeholders and contributors.

A drawback of the standards committee pattern is that it can be slow. Even if you get the document created quickly, getting everybody to implement against it can be slow and can lack adherence. This approach can also lead to a mediocre design as a result of too many compromises.

The Copy Cat Approach

We sometimes see this pattern when an organization is late to market – for example, when their close competitor has already delivered a solution. Again, this approach can get you to version 1 quickly and you may have a built-in adoption curve if the app developers who will use your API are already familiar with your competitor’s API. However, you can end up with an undifferentiated product that is considered an inferior offering in the market of APIs. You might have missed exposing your own key value and differentiation by just copying someone else’s API design.

Solution – The API façade pattern

The best solution starts with thinking about the fundamentals of product management. Your product (your API) needs to be credible, relevant, and differentiated. Once your product manager has decided what the big picture is like, it’s up to the architects.

We recommend you implement an API façade pattern. This pattern gives you a buffer or virtual layer between the interface on top and the API implementation on the bottom. You essentially create a façade – a comprehensive view of what the API should be and importantly it is the view from the perspective of the app developer and end user of the apps they create.

Next time, we'll look at the basic steps to implement an API facade pattern and then at some common facade patterns.

The API Facade: Overview »

Thanks to all who attended the first episode in the Webinar Shorts series on the API Facade pattern.

The first episode is an overview of the idea and theory of API facades. Here are the video (20 min.) and slides. Thanks @landlessness

Check out the videos and slides for all 4 episodes.

As always, we'd love more of your thoughts, insights, or questions on the api-craft forum. 

RESTful API Design: API Virtualization »

Last time, we looked at why you might consider complementing your API with an SDK or code libraries. In the series so far, we've covered a lot of tips and tricks for designing pragmatic RESTful APIs. 

You may be asking -  

How do I follow all these best practice guidelines and still maintain and iterate my APIs? 

What should I be thinking from an architectural perspective in terms of implementing these best practices?

Add an API virtualization layer 

I recommend you give yourself a buffer or virtual layer between the interface on top and the API implementation on the bottom.

The app that consumes the API is on top. The API virtualization layer isolates the application and the API. Make a clean design for the app on top and turn the problem into an integration problem.

 In other words, work at integrating your design on top of the key APIs through the virtualization layer.

Don't start in code, and try to build up from your business logic to a clean API interface.

API Virtualization »

We've seen an architectural pattern emerge over the last two years that deals with the expanding variety of computing devices and the exploding number of APIs used in modern applications. This pattern is API Virtualization - applying a virtual layer above the API itself to deal with the different concerns that come into play when delivering an API to different device and machine endpoints as well as across a range of different classes of business relationships.

I gave this presentation at GlueCon as a Prezi. The prezi is available here, and I've put the audio with it in the screencast below.

One of the big shifts that has occurred in application development is the move from libraries to APIs for use of pre-written code. APIs enable access to not just logic but data and the the computational power required to render that data appropriately. The Facebook and Twitter APIs are good examples of this, but there are many more including Paypal's X.com, Ebay, Amazon, Netflix, Sears, Blockbuster, Best Buy, MTV and Google.

While applications like Siri which use 20 or more web APIs are still unusual, most applications use several APIs to deal with access to content, social networks, events, identification, payment, and other concerns. These APIs change frequently, as do the applications that consume them, resulting in a web of dependencies and causing chaos in the development lifecycle.

API Virtualization enables these different concerns and dependencies to be isolated from the application or API provider, resulting in a cleaner development process and higher-performing applications.

API Versioning Hell? Think API Virtualization »

We've seen an architectural pattern emerge over the last two years that deals with the expanding variety of computing devices and the exploding number of APIs used in modern applications.  This pattern is API Virtualization - applying a virtual layer above the API itself to deal with the different concerns that come into play when delivering an API to different device and machine endpoints as well as across a range of different classes of business relationships.

Sam Ramji gave this presentation at GlueCon as a Prezi.  The prezi is available here, and we've put the audio with it in the screencast below.

One of the big shifts that has occurred in application development is the move from libraries to APIs for use of pre-written code.   APIs enable access to not just logic but data and the the computational power required to render that data appropriately.  The Facebook and Twitter APIs are good examples of this, but there are many more including Paypal's X.com, Ebay, Amazon, Netflix, Sears, Blockbuster, Best Buy, MTV and Google. 

While applications like Siri which use 20 or more web APIs are still unusual, most applications use several APIs to deal with access to content, social networks, events, identification, payment, and other concerns.  These APIs change frequently, as do the applications that consume them, resulting in a web of dependencies and causing chaos in the development lifecycle.

API Virtualization enables these different concerns and dependencies to be isolated from the application or API provider, resulting in a cleaner development process and higher-performing applications.

Tradeoffs in XML data transformations »

Daniel Jacobson of NPR posted a fascinating piece about how NPR tackles a common problem – what’s the best way to render content on a variety of devices, from modern web browsers with top-notch CSS implementations that look almost like typesetting (like Safari) to mobile phones using WAP to low-end devices like HD Radio receivers that don’t understand anything but plain ASCII text.

NPR’s clever solution is to strip markup out of the text and store it in a database table, indexed by position in the text document. To re-generate the content for a particular device, their software queries the database and re-applies the markup tags to the content according to what device it is rendering to.

This takes me back to the original reason SGML was invented and made an ISO standard in 1986. The idea was to describe the semantic meaning of text, and then to let a computer program figure out how to render it for human consumption.

SGML was a little over-engineered for that purpose, however, so a bunch of smart people got together in 1996 and invented XML. XML then begat technologies like HTML, XSLT, and CSS.

So today, instead of writing something like:

<h1 class=”headline”>This is a headline</h1><p class=”byline”><b>By I.M.A. Reporter</b></p><p class=”paragraph”>And here is my first paragraph with something in <i>italics</i>.</p>

XML lets us write:

<main_headline>This is a headline</main_headline><byline>By I.M.A. Reporter</byline><p>And here is my first paragraph with something in <i>italics</i>.</p>

The difference is that my second example isn’t HTML – it’s part of a document that uses an XML schema that’s up to me, and when writing it I don’t care if I’m coding for an HTML browser or for a car radio – I just have to identify when I’m writing a headline, or a byline, or a caption, and so on. I can now use XSLT or another transformation technology to transform this XML into very simple HTML for a simple browser, or into very complex HTML with links to a CSS stylesheet for a more sophisticated browser, or just into plain text. And if I decide that part of my XML schema should look just like HTML (like I did above with the “p” and “i” tags) then that’s fine too.

Other approaches and tradeoffs

NPR’s approach has a lot of benefts. Depending on your business and situation, this might mean lot of database processing, which could to be expensive to scale in either licenses or capacity.  Caching helps a lot in this case, since once is content there’s no need to do it again.

You could also solve this problem by writing the original content in very simple HTML or XML (in whatever schema one desires) and then by using something like XSLT to transform the content for each input device. This solution might be CPU-intensive but might compare favorably vs. database operations depending on what you are doing. Plus, XSLT processing can be easily scaled across thousands of parallel nodes if necessary without buying any more database licenses.  

If development resources and cycles are the constraint, a dedicated policy layer can help.  In the case of our Sonoa ServiceNet technology - you could configure transformation policies that leverage XPath or XSLT from within our proxy.   This might also make it easier to add and validate 3rd party APIs or feeds from outside your own database.   You can also handle other types of mediation such as versioning or protocol transformations, if that is in your use case, such as some of our Sonoa media and consumer web services customers do.

TrueCredit.com API case study »

Scott Metzger, CTO of TrueCredit.com was kind enough to take some time to talk about their Consumer Connect API program and some of the technical challenges that they have addressed using Apigee's API Gateway.

Scott wanted to make life easier on his development team as they ramped up their number of APIs, partners and traffic volumes.  Here, he describes how he uses the technology as a 'policy layer' to provide API analytics, fine-grained data protection, and caching in an API Gateway. In this case, Apigee Enterprise is deployed on-premise virtualized software.

We're very excited to be working with Scott and TrueCredit, and check out the full TrueCredit Case study.