Introduction

Rutter’s vision is to build software that simplifies how companies share core business data. Anyone working with businesses from fintech lenders to e-commerce marketplaces to dropshippers needs access to data shared from accounting, e-commerce, and payment processing platforms and more. To bring this vision to life and truly be the single, universal API, we need to integrate with every platform our customers need in a scalable and reliable way - hundreds and eventually thousands of times over.

Risks of Building Integrations

Today, many companies brute force through the process of building integrations, investing hundreds of engineering hours. They copy and paste the code from one integration and edit it for a new one. Everything that’s built is platform-specific and as these integrations are added, they’re prone to breaking errors, and uneven quality across integrations. Dealing with all these issues pull developer time away from building new integrations or investing in other product functionality on the roadmap.

Even when we first started building integrations 2 years ago, we encountered problems with the brute force way:

• Our customers work with merchants that have massive amounts of real-time data – synchronizing millions of orders, thousands of SKUs, and various other metadata made completing the initial data fetch take days, sometimes weeks.
• Merchants and SMBs on a platform run different versions of that platform, each version with a slightly different API. Some merchants used plugins or added customization, changing their data in a way that wasn’t reflected in the platform’s API documentation.
• Each platform API would return similar, but slightly different errors when encountering a rate limit or internal issue. This made error handling a nightmare across our 25+ platforms. Initial platform connectivity was also an issue especially apparent with self-hosted or firewalled storefronts.
• Adding a new platform required duplicating a lot of old code, and we felt there was a better abstraction for handling data fetching, API pagination, and persistence of data.
• Additional features or quality of life improvements were built specific to a single platform, and had to be rebuilt for every additional integration.

From working with customers the past year, we’ve also learned that any company building integrations care about two things:

• Platform Coverage - This means being able to build and support the long tail of integrations that will allow them to support existing and new businesses and access the core business data.
• Quality & Reliability of Integrations - This means the integrations work the way one expects it to work, and have the highest uptime and healthiest connections.

Introducing Rutter Odyssey

“A man who has been through bitter experiences and travelled far enjoys even his sufferings after a time” — Homer, The Odyssey

Having seen the bitter challenges of building these integrations first-hand and hearing the pain from our customers, we developed the infrastructure that enables us to build integrations 10x faster while maintaining the quality of these integrations. This infrastructure is Rutter Odyssey.

Odyssey is a code-level abstraction that enables Rutter to more efficiently build out and maintain a new platform integration. By moving error handling, fetch, refresh and persist logic, monitoring and other integration features into their own abstraction, our engineers only need to build the data schema and mappings per platform in order to roll out a full integration. This lets our team push out an integration with 85% less code, guarantee a high bar of integration quality across the board, and streamline the way we deal with errors and maintenance improvements.

Key Capabilities

1. Error Handling - After building out 25+ integrations and running over thousands of connections through our platform, we’ve seen hundreds of different errors across platform and type. Every time we see an error, we adapt to handle it by mapping it to a universal error code.  Based on the universal error code, we can automatically take an action, manually take one, or alert our customers that they need to take one.  Frequently, we just update our system to automatically resolve these errors as they come in, saving our customers time.  Even if these errors are surfaced to a customer, we deliver them a clear action item to solve the issue, helping them avoid the nuances of each platform. For example, an API or integration that hasn’t dealt with a rate limit will throw an error when it sees one. Rutter knows the rate limit exists and when the platform can be called again, so we’ll wait and call when we know we can (while also disabling any unhealthy connections).
2. Configurable Data Refresh - Because we started in the e-commerce space where the quantity of data is vast (millions of orders and thousands of SKUs), we quickly learned that our customers each have different priorities in how quickly they need data, what endpoints are most important, and what can be sacrificed in favor of speed. Take an online lender, for example, who relies on delivering terms within 48 hours. For this reason, we built the infrastructure to allow our customers to choose what data they need, over what time period, and in what order for each platform they integrate into. For example, for some of our most high volume clients, we fetch 3 months of orders data, then 1 year of transactions data, and finally all products data. We get this configurability for minimal work by mapping Rutter entities to every platform’s entities and Rutter Odyssey handles the rest.
3. Visibility Into Sync Status - Since we focus on customized data refresh, it’s important to make sure the details of each refresh are visible to our customers. In our dashboard, you can see exactly how much data was refreshed and what rows were added. And if there’s a refresh issue? Platform outages happen, but we track them proactively across our customer base, so that we know about them before you have to.

Adding New Platforms

With Rutter Odyssey, adding a new platform is as simple as generating the underlying schema and building the platform-to-Rutter data mapping. The three core components of Odyssey, the Fetch Client, the Persist Client, and the Refresh Client, give us all of the structure around fetching, refreshing, and persisting data out of the box.

Fetch Client

The Fetch Client is an abstraction that allows us to easily call third-party APIs and re-use logic for pagination, handling HTTP success/error codes, and transformation of XML and SOAP APIs to RESTful responses. In the past, engineers would need to write a completely new set of code to send HTTP requests to fetch data and paginate through API responses, but we noticed that most APIs had a similar offset/limit or cursor-based pagination scheme. The Fetch Client allows an engineer to plug in the URLs of new platform endpoints, and skip rewriting code to send HTTP requests, paginate, and handle errors.

In addition to dealing with all of our third party data, the fetch client is also responsible for our initial data fetch. When a merchant or business platform connection is created on Rutter, our client attempts to pull all historical data on the business. To do this, Rutter has to deal with figuring out what data to pull in what order (Initial Fetch Configuration) and what permissions of data access were granted by the end business (Permissions Management).

Persistence Client

The Persistence Client is an abstraction that allows us to save & de-duplicate data into our Aurora instance. The Company and Storefront Rutter data models are heavily relational and hide schema resolution complexity from the end user. For example, every Rutter Product contains a variants field containing multiple variants which are each represented by a variant_id, but some platforms internally represent variants as part of their internal products table, while other platforms do not represent variants as their own entity and instead nest them into the product object. To generate a cohesive data model, Rutter stitches together all of the different relationships between underlying platform data models to conform to the Rutter schema.

As Rutter continues to refresh and persist data, this relational mapping lets us detect data deletions, do data diffing reliably, and identify what webhook updates to send to our end users.

Refresh Client

The Refresh Client is an abstraction that uses both the Persistence Client and Fetch Client to fetch and save platform data to our database.

The Refresh Client parses a customers’ configuration file, looks within our system to understand how much data we have already, and generates an execution path to complete the next data synchronization job. By figuring at what pointer to start fetching data, Rutter can guarantee freshness of data without blowing through underlying platform rate limits.

The Refresh Client also handles data sync frequency, which determines how often to refetch data. It contains fallbacks to retry data fetches should a rate limit be reached, and also handles OAuth Token Management and Connection Health, refreshing the auth tokens when needed and sending a webhook to the end developer when connection health changes or is marked inactive.

10 Platforms in 10 Days

Don’t just take our word for it. Over the next 10 business days, we’re going to announce 10 new platform integrations. These platforms span e-commerce, accounting, and our newly released set of APIs for subscription billing platforms that our customers have been asking for.

We’re excited to show the power of Odyssey and introduce new platforms to our customers while maintaining the highest bar of reliability. At Rutter, our goal is to have the highest platform coverage for our customers and have the highest quality APIs.

Stay tuned to see what platforms we’re adding!