Operations in action–Marking WCF interface as down for maintenance

As many of us deploy our shiny web services and expose them to the world (or just our apps), we invariably encounter these pesky maintenance windows. During these times, a database, other web services or any other IO dependent tasks cannot be performed.

Wouldn’t it be nice to tell the caller of your web API that the operation is currently unavailable? It can get pretty ugly if we don’t solve this. If we simply bring down the whole endpoint, connecting clients will experience a pile-up of timed out connection attempts. If we leave it up, every operation attempted would experience it’s own slow excruciating failure, with the same IO timeout pile-up, this time on your server and often bringing the server to it’s knees with too many doomed connection requests queued up. My game plan shaped up to :

1. Each service operation shall return a standard response, exposing some status flag
2. A configuration controls whether services are to be marked as unavailable
3. A WFC extension will take care of returning the standard response with proper flag when so configured, but let the regular response return under normal conditions.

The requirement that each operation returns a standard response may seem peculiar. You may have created

string GetUserName(string id);
DateTime GetUserBirthdate(string id);

The thing is, when operations fail, you have no way to signal the caller except for smelly nulls or throwing exceptions. Although Soap Fault Exception can do the trick, I find it distasteful to throw a Client Fault exception because exceptions are more costly, and validation of request data often enough finds client faults. For that and other reasons, I use code that looks like the following:

[DataContract(Namespace = "...")]
public class ServiceResponse
{
    [DataMember]
    public string Error { get; set; }
    [DataMember]
    public ResponseStatus Status { get; set; }
}

Where the status is an enumeration:

[DataContract(Namespace = "...")]
[Flags]
public enum ResponseStatus
{
    [EnumMember]
    None = 0,
    /// <summary>
    /// Operation completed without failure
    /// </summary>
    [EnumMember]
    Success = 1,
    /// <summary>
    /// General failure
    /// </summary>
    [EnumMember]
    Failure = 2,
    /// <summary>
    /// Client request not valid or not acceptable
    /// </summary>
    [EnumMember]
    ClientFault = 4,
    /// <summary>
    /// Server failed processing request
    /// </summary>
    [EnumMember]
    ServerFault = 8,
    /// <summary>
    /// The underlying service is not available, down for maintenance or otherwise marked as non-available.
    /// </summary>
    [EnumMember]
    BackendFault = 16,
    /// <summary>
    /// Convenience value for client fault failure comparison
    /// </summary>
    ClientFailure = Failure + ClientFault,
    /// <summary>
    /// Convenience value for server fault failure comparison
    /// </summary>
    ServerFailure = Failure + ServerFault,
    /// <summary>
    /// Convenience value for backend failure comparison.
    /// </summary>
    BackendFailure = Failure + BackendFault
}

One may also abstract the ServiceResponse to an interface, allowing any response object to implement the interface rather than inherit the base response. For this post, let’s just go with the base class.

Now the signature of every operation would be an object derived from ServiceResponse. Rather than a fragmented GetName, GetBirthdate etc – a chatty interface anyway – we would expose:

public class GetUserResponse: ServiceResponse
{
    [DataMember]
    string Name{get;set;}
    [DataMember]
    DateTime Birthdate {get;set;}
    // whatever else a user profile has..
}
// then the operation signature becomes
[ServiceContract]
public interface IMyService
{
    [OperationContract]
    GetuserResponse GetUser(string id);
    // and other operations
}

Now that we have that out of the way, you get the payoff: we can define a fail fast attribute to decorate operations we know rely on some back-end which may be turned off on us. We’ll utilize the IOperationBehavior extension point of WCF, allowing us to specify behavior on an operation by operation basis.

I’ve created an attribute implementing the IOperationBehavior. It replaces the operation invoker with my own implementation when ApplyDispatchBehavior is called. All other IOperationBehavior methods remain blank.

public class FailFastOperationAttribute : Attribute, IOperationBehavior
{
    public void Validate(OperationDescription operationDescription) { }
    public void ApplyDispatchBehavior(OperationDescription operationDescription, DispatchOperation dispatchOperation)
    {
        var returnType = operationDescription.SyncMethod.ReturnType;
        dispatchOperation.Invoker = new FailFastOperationInvoker(dispatchOperation.Invoker,returnType);
    }
    public void ApplyClientBehavior(OperationDescription operationDescription, ClientOperation clientOperation) { }
    public void AddBindingParameters(OperationDescription operationDescription, BindingParameterCollection bindingParameters) { }
}

The finishing piece is to implement the operation invoker. It will check a special configuration, and based on that would either invoke the underlying operation as the stock implementation would have, or construct a new response with the failed flags set.

public class FailFastOperationInvoker : IOperationInvoker
{
    private readonly IOperationInvoker _operationInvoker;
    private readonly Type _returnType;
    public FailFastOperationInvoker(IOperationInvoker operationInvoker, Type returnType)
    {
        _operationInvoker = operationInvoker;
        _returnType = returnType;
    }
    #region IOperationInvoker Members
    public object[] AllocateInputs()
    {
        return _operationInvoker.AllocateInputs();
    }
    public object Invoke(object instance, object[] inputs, out object[] outputs)
    {
        object result;
        if (Config.ShouldFailFast())
        {
            outputs = new object[0];
            // construct response of the type the specific method expects to return
            result = Activator.CreateInstance(_returnType);
            // mark the response as fail fast failure
            result = (result as ServiceResponse).Error = "Not available";
            result = (result as ServiceResponse).Status = ResponseStatus.Failure|ResponseStatus.BackendFault;
        }
        else
        {
            result = _operationInvoker.Invoke(instance, inputs, out outputs);
        }
        return result;
    }
    public IAsyncResult InvokeBegin(object instance, object[] inputs, AsyncCallback callback, object state)
    {
        return _operationInvoker.InvokeBegin(instance, inputs, callback, state);
    }
    public object InvokeEnd(object instance, out object[] outputs, IAsyncResult result)
    {
        return _operationInvoker.InvokeEnd(instance, out outputs, result);
    }
    public bool IsSynchronous
    {
        get { return _operationInvoker.IsSynchronous; }
    }
    #endregion
}

A method for determining if the API should by up or down hides behind the Config.ShouldFailFast() call. Read your app setting, check a file, do what you like to make that determination.

The next thing is manufacturing an instance of a response object. Here we need to create the same type or a type assignable to the one the formal method expected. Note that that type would need to have a parameter-less constructor for this to work. Since all my service DTO are plain POCO, this is rarely a restriction. With this code in place, all we need to do is decorate specific methods as [FailFastOperation] and bingo!