Extending Ocelot's QoS

For the past few months I’ve been working on the dotnetcore microservice backend for a new mobile application. One of these services is a Backend for frontend(BFF) powered by Ocelot. Ocelot is an API Gateway for dotnetcore. It provides a lot of functionality out of the box and most of the work is done in JSON configuration files.

The Problem - Configurable Retry Policy

Policy supports a powerful retry policy which I used in a number of the other services and wanted to add to the BFF. I also wanted to vary the values depending on what ReRoute I was calling depending on the service, e.g.

• Known slow services, or those doing a lot of computations have a larger timeout.
• We don’t want to make multiple retries to 3rd party services that have a cost value per request.
• Flakey legacy services that are simple lookups have a low timeout but a number of retries.
• Downstream services that are not idempotent (POST or PUT) don’t have a retry.

Quality of Service

It is trivial to add basic QoS capability to Ocelot, via the wonderful Polly project.

public virtual void ConfigureServices(IServiceCollection services)
{
  services
    .AddOcelot()
    .AddPolly();
}

Simple JSON configuration allows you to set a timeout and/or circuit breaker pattern by giving you 3 nobs to twiddle - ExceptionsAllowedBeforeBreaking, DurationOfBreak and TimeoutValue.

Implementation

ReRouteKey

The first problem I had to solve was to identify which ReRoute is being called, from within the DelegatingHandler. As this will be in the same request I would be able to modify the incoming HttpContext, specifically by adding a value to the HttpContext.Items collection. Fortunately Ocelot offers a few different middleware lifecycle functions that can be implemented.

I implemented the PreQueryStringBuilderMiddleware which comes at the correct time in the pipeline (after the ReRoute has been identified and the context has been build) and also has access to the DownstreamContext object which has everything we need (HttpContext and ReRouteKey).

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
  app.UseOcelot(configuration =>
  {
    configuration.PreQueryStringBuilderMiddleware = (ctx, next) =>
    {
      // set Item and call next
      ctx.HttpContext.Items["ReRouteKey"] = ctx.DownstreamReRoute.Key;
      return next.Invoke();
    };
  }).Wait();
}

Now that “ReRouteKey” is set in the HttpContext.Items collection it will be available to all further handlers in the pipeline. To access the HttpContext within a DelegatingHandler we need to inject IHttpContextAccessor. Note that there is no need to explicitly call services.AddHttpContextAccessor(); as Ocelot already does this.

DelegatingHandler

I couldn’t see a nice/clean/maintainable way to extend the QoS settings, either in code or in the JSON configuration, so I opted to implement it in a DelegatingHandler. By default the lifecycle of a DelegatingHandler is tied to a specific ReRoute which is required so that the same circuit breaker is applied for all requests to a particular route. I tried to keep the DelegatingHandler as simple as possible, all that it was responsible for was:

1. Find which ReRoute is being called.
2. Get the PollyOptions (timeout, retry count etc) for ReRoute.
3. Configure a Polly AsyncPolicyWrap<T> using PollyOptions. This same Policy will be used for all requests to the relevant ReRoute.

An over simplified example of this handler is below:

public class PollyRetryBreakerHandler : DelegatingHandler
{
  private readonly AsyncPolicyWrap<HttpResponseMessage> _circuitBreakerPolicy;

  public PollyRetryBreakerHandler(IHttpContextAccessor contextAccessor)
  {
    var pollyOptions = GetPollyOptions(context.GetDownstreamReRouteKey())
    _circuitBreakerPolicy = CreatePollyPolicy(pollyOptions);
  }

  protected override Task<HttpResponseMessage> SendAsync(
    HttpRequestMessage request,
    CancellationToken cancellationToken)
  {
    return _circuitBreakerPolicy
      .ExecuteAsync(async ct => await base.SendAsync(request, ct),
        cancellationToken);
  }

  private static GetPollyOptions(string reRouteKey)
  {
    // return PollyOptions for ReRoute.
  }

  private static AsyncPolicyWrap<HttpResponseMessage> CreatePollyPolicy(PollyOptions pollyOptions)
  {
    // return configured Polly policy.
  }
}

All that needs to be done now is for the PollyRetryBreakerHandler to be configured on the ocelot.json file for the relevant ReRoutes.

Demo

I’ve made up a simple demo of it available on github. This is a really simple example with a few elements ommited to keep it as simple as possible. For example, in my real-world app I have a “Polly” key that contains the values for each of the ReRoutes, plus a fallback key that is used if there is no override. The JSON looks like below and is mapped to Dictionary<string, PollyOptions> for ease of lookup.:

"Polly": {
  ":default:": {
    "ExceptionsBeforeBreak": 5,
    "BreakDurationMs": 1500,
    "RetryCount": 2,
    "RetryBackOffMs": 200,
    "TimeoutMs": 4000
},
  "Foo": {
    ...
  }
}