Implementing a usage-based billing system: integration deep dive (part 2)

This is the second of a two-part series where we go behind the scenes of our recent experience in rolling out new pricing plans, along with a revamped usage-based billing system.

In the previous post, we discussed what usage-billing is, various considerations and tradeoffs in buying a usage billing provider vs building it in-house, and our decision to ultimately choose Orb as our usage-based billing partner after evaluating alternatives in the market.

In this post, we take a closer look at the implementation details of building the new billing system, what the integration with Orb broadly looks like, and our learnings from it. Although it is based on our experience integrating Orb specifically, many of these implementation concepts and learnings can be applied to other usage-based services.

How the billing system works

The new billing system is made up of three disparate systems integrated together, in which Orb sits in the middle as a usage-based billing provider between Knock’s application system and payment processors, such as Stripe or Bill.com. Each system owns a distinct domain of responsibilities in the overall billing system.

Knock is responsible for:

• Synchronizing customer and subscription data
• Checkout (upgrades or downgrades) and billing management
• Reporting usage events to Orb

Orb is responsible for:

• Managing subscription data as a single source of truth
• Metering usage over each billing cycle
• Sending subscription usage alerts
• Generating and sending invoices

Payment processors are responsible for:

• Handling customer payment information
• Payment processing or invoice settlements

Synchronizing customer and subscription data

Knock synchronizes necessary data throughout different integration points in the billing system, so each integrated system can perform the functions it is responsible for.

There are two major pieces of data Knock integrates with Orb: customer and subscription data.

• Customers represent unique accounts or organizations created in Knock. They are the billing entities to which usage events can be attributed.
• Subscriptions represent the recurring charges associated with each customer based on the plan they are on and their current usage. Subscriptions are created in Orb and have a monthly billing cycle; usage events are metered over this billing cycle and then invoiced according to the pricing details of its plan.

Even though both Knock and Orb need to understand what plan each customer is on (and every customer must be assigned to a plan), Knock’s system focuses only on the plan’s features for application purposes whereas Orb only cares about the price details of the plans for billing purposes.

While Orb is the single source of truth for all subscription-related details, payment processors hold one piece of important and sensitive data: customers’ payment information. By using Stripe Elements, we can capture and store payment information directly with Stripe, which means neither Knock nor Orb needs to touch this sensitive piece of data.

In practice, this data synchronization flow looks like this:

• Create a corresponding customer on both Stripe and Orb when a customer signs up for a new account in Knock. The customer in Orb is created with a reference to the Stripe customer id, so Orb can later bill the customer directly via Stripe on behalf of Knock.
• Set the customer on an initial free tier subscription (our “Developer” plan) and create a subscription on Orb with the free plan. It’s important to create a subscription in Orb even for customers on the free plan so we can track usage across all customers regardless of the plan they are on.
• Schedule a subscription plan change with Orb when customers upgrade or downgrade their plan from Knock’s dashboard
• Update payment method info on Stripe when customers add or update a payment method from Knock’s dashboard

Checkout and billing management

One trade-off of using Orb as a usage-based billing provider is that you can no longer rely on the pre-built Stripe Checkout and Billing Portal to handle your checkout flow and billing management pages. Even though you are using Stripe to process payments, you are no longer using Stripe’s subscription model for billing directly and therefore not able to use the hosted payment page.

This meant creating a custom built page for managing billings and checkouts in our dashboard, which involved the following parts:

• Capturing credit card details. For this we used Stripe Elements which did most of the heavy lifting in creating the credit card form, in conjunction with the Setup Intents API. The Setup Intents API is optimized for collecting customer’s payment credentials for future payments without immediately charging them. It is well suited for usage-based services like Knock that charges a recurring but potentially different amount each month.
• Showing the current usage. Here we read directly from the Orb API to show the usage and the cost for the current billing period, taking advantage of the upcoming invoice API to do so.
• Handling plan upgrades and downgrades. Except for upgrading to the Enterprise plan, Knock customers can freely upgrade or downgrade their plan on their own. There are a few subtle but important details to be mindful in how this is handled:
  • Plan upgrades are effective immediately in Knock, so we first need to ensure a payment method is on file before proceeding. Then we make an immediate plan change request to Orb for the customer’s subscription, which should kick off a charge for the plan upgrade. Upon success we update the customer account in Knock’s system for the upgraded plan features. One tricky part here is when a customer upgrades to a higher paid tier plan from a lower paid tier plan in the middle of the billing period, as we need to apply a prorated credit toward the new plan purchase so to account for the remaining period in the billing cycle purchased under the current plan. Thankfully for us, Orb takes care of this prorating and crediting part automatically as part of the Schedule plan change API.
  • Plan downgrades are handled differently than upgrades, as downgrades get scheduled to take effect at the end of the current billing period. Therefore, when a customer downgrades their plan, it entails a two-step process: First, we make a scheduled plan change request to Orb for the end of the customer’s billing period. Then, when the plan downgrade change actually takes place on the scheduled date, we receive a subscription.plan_changed webhook event from Orb where we update the customer account in Knock’s system for the downgraded plan features.
• Showing past invoices. Once again, we take advantage of the Orb’s API by reading directly from the List invoices API to fetch all historical invoices. Each invoice payload in the response includes the URL to the invoice PDFs generated by Orb as part of the billing process.

Reporting usage events to Orb

Orb, like most usage-based billing providers, requires you start sending events that denote usage happening in your system that a customer could potentially be billed for (a “billable metric”). For us at Knock, our primary billable metric is a message sent, which corresponds to a single notification being sent to a single recipient during the execution of a workflow.

In building our usage reporting system, we created a new Kinesis consumer on-top of an existing stream for message events. We batch deliver our message.sent records to Orb via this consumer using their events ingestion API. Given the critical nature of reporting events correctly, we took advantage of using the message_id as an idempotency key when sending events to ensure that we can only ever report a message sent at most once.

At Knock, we use Elixir heavily in our backend, which means we have the amazing Broadway library at our disposal for consuming and processing these message events in a highly concurrent and robust manner.

Sending subscription usage alerts

Given we offer a capped bucket of free usage on our Developer plan, having customers understand when they reach a threshold of usage is critical to ensure we don’t cause any interruptions to their notifications being sent.

Fortunately, Orb provides a webhook for this exact case where we specify a webhook to be triggered at 50%, 75%, and 100% of usage on their plan for the current period. We use this usage alert to then set a banner that is visible within the dashboard, as well as sending them an email notification (powered by Knock, of course 😎).

Generating and sending invoices

At the end of each billing period, Orb automatically calculates an invoice amount based on the subscription’s plan and the usage for the period, then charges the customer via Stripe (or opens a new invoice if using bill.com).

This billing process takes place continuously over a given month, as customers’ subscriptions can be on different billing cycles depending on when they upgraded to a paid plan. Orb takes care of this bookkeeping of starting and closing billing periods, metering usage toward correct periods, and kicking off charges when needed.

For our part, we ingest webhook events to notify our customers when an invoice is charged. The invoice.issued payload includes the link to the PDF invoice Orb generated, and we send an email notification powered through Knock to do so (using our attachments feature in order to attach the PDF to the email itself).

Challenges and learnings

Plan based feature differentiation

Like many SaaS products, we use a tiered model for our subscription plans, where each plan includes the features of the plan preceding it and includes additional features unlocked at the current tier and above.

Interestingly, none of the usage-based billing providers on the market allow you to capture this concept of features enabled or disabled at a plan level and instead push this to your application to implement. In our case, that meant separating the plan data into two parts: the concept of plan features are defined as static plan structs in Knock’s system, whereas all the details of plan prices are defined on Orb.

%Plan{
  id: "free-june-2022",
  type: :free,
  display_name: "Developer",
  features: %Plan.Features{
    channel_limit: nil,
    message_sent_limit: 10_000,
    data_retention_days: 30,
    knock_branding_required: true,
    custom_branding_allowed: false,
    tenant_preferences_allowed: false
  }
}

In the short-term, in order to opt certain customers into features that their plan does not grant (like for proof-of-concept trials and such) we turn to feature flagging. In the future, this is where we may need to build a small internal tool in order to set plan feature overrides per customer.

Handling retries in our usage reporting

With any critical system that needs to deliver data to a third party service we also had to consider retries and service failures during execution, as well as observability to understand the current state at a glance.

We had some unexpected service issues during peak load and while backfilling a substantial amount of usage that resulted in some of our usage event requests being dropped. To guard against any unexpected failures in the future, we have built in a mechanism to capture these errors and replay our events to ensure they’re delivered.

We now run a regular cron job to replay any usage events that we reported as having errored and have monitors to alert our team given the critical nature of this request.

Migrating existing customers and aligning plans

We’ve been charging customers manually each month since very early in our journey of running Knock. As we’ve experimented with different pricing models and charges over time we had a number of customers on different pricing schemes that we needed to account for while migrating them. We were able to model these entirely as plan price overrides though within Orb, making the process of modeling these different plans much easier.

Deciding on what to do about our existing customers as we moved to a new pricing model was easy for us – we chose to honor the existing pricing and lock in the rate that was agreed. However, actually moving them from our ad-hoc invoicing to our new streamlined system was a bit more involved, with a good-old-fashioned spreadsheet helping keep us on the straight-and-narrow.

Modeling complex Enterprise plan pricing terms

At the top of our pricing plans, we offer the Enterprise plan for customers with large-scale volumes and most feature rich requirements. We often work with our Enterprise customers and tailor the plan to meet their needs, including the ability to offer volume-based discounts.

This typically involves modeling bespoke pricing and discount terms that diverge from the non-Enterprise plan tiers, which can get quite complex very quickly. Again, here we lean into Orb’s price override functionality to keep this process manageable.

Instead of creating a new custom plan for every individual Enterprise customer, which gets unwieldy and hard to manage over time, we can apply various pricing and discount terms for the customer on top of the “base” Enterprise plan using the Orb’s dashboard UIs.

Any changes made to the customer’s subscription plan from Orb’s dashboard sends a subscription.plan_changed webhook event to Knock, so we can update the customer account accordingly.

Supporting manual invoice approval & settlement workflows

For the vast majority of our customers, the billing process is automatic with little friction whereby customers are charged automatically at the end of each billing cycle using the latest payment method on file (via Stripe Connect).

For certain large Enterprise customers, however, this automatic charge process might not fit well with their existing accounts payable process and might require a manual invoice approval and settlement workflow using a provider like Bill.com.

Luckily, Orb provides an integration with Bill.com as well as Stripe Invoicing out of the box, which means we do not have to add more complexity or support additional integrations in our billing system. The only change required is to update payment_provider of the customer in Orb as using bill.com or stripe_invoice.

Do note that in order to create a Bill.com integration in Orb, your Bill.com account first needs to have the API access granted which requires a paid plan.

In closing

Overall, we are happy with the new billing system we put in place now, and our decision to power it with a 3rd-party usage-based billing provider, Orb.

Despite a few trade-offs and challenges, being able to offload many of the complexities involved in billing infrastructure to Orb, and leveraging their flexible out-of-the-box price models really helped us put together a billing system that feels both solid and flexible for our use case without getting bogged down in minutiae of usage-based billing details.

Having a robust billing system is important for any usage based services — yet the reality is it does not necessarily add more value to our customers. At this early stage of a business we are in, we want to remain laser focused on building what makes Knock great and delivering more value to our customers. We believe using Orb helped us do that and was the right decision for us.

Lastly, we want to thank the Orb team for their excellent support throughout this process and being a great partner to us!