+N Consulting, Inc.

MongoDB’s ObjectId() has some nice sequential properties. One of the interesting ones is the fact that the most significant 4 bytes are a timestamp with seconds granularity.

Suppose you want to query your collection for items created on or after a certain date. Since the timestamp portion can be constructed (milliseconds since epoch), and the rest can be manufactured (zero would be fine) we can now write a function to generate what the ObjectId would be or be just higher or lower than:

var past =  new Date((new Date()).getTime() - (90 * 24 * 60 * 60 * 1000));
var stamp = ObjectId(Math.floor(past.getTime() / 1000).toString(16) + "0000000000000000");

The stamp object contains an ObjectId with a value representing the floor value of any object ids generated 90 days ago to the seconds granularity. Using the stamp value, we can then write a query for objects created on or after that time.

While this value may not be suitable for exact reporting (as the rounding may exclude or include some values because of the lack of granularity) it is well suited to finding records inserted at or around that time, such as retiring older records etc.

MongoDB’s engine can log quite a bit of useful detail. Whether high-transaction rate or verbose, the log can get quite large.

While setting the log mode to append helps you retain the old / existing log, mongo does not currently have a facility to rotate the log at prescribed times or when a size limit is reached. In other words, the log will grow indefinitely.

There are 2 ways to have the engine release current file and start a new one:

1. SIGHUP
2. Issue a command to mongod via a client connection

The first option, available on Unix variants, is issued like so:

killall -SIGUSR1 mongod

This would force log rotation on all instances of mongod on that machine.

The second option requires a connection to mongo. The mongo shell is capable of running in non-interactive mode in 2 ways: using eval command line expression or running a named JavaScript file. Let’s pick the eval method, since we only have one command to send. Since the log rotate command needs to be issued from the admin namespace, we specify that on the connection string directly:

mongo localhost/admin -eval "db.runCommand({logRotate:1})"

The result of either of these methods is that mongod will take its current file, say /var/log/mongo/mongod.log and rename it to be suffixed with a date/time stamp such as /var/log/mongo/mongod.log.2012-01-31T12-34-56 if invoked January 31st at 12:34:56 .

The next sticking point is that you may want to compress down that file, and clean out older log files. There are some tools out there, Logrotate being one, but I decided to write a small shell script:

#!/bin/bash

### log rotate
mongo localhost/admin -eval "db.runCommand({logRotate:1})"

### compress newly rotated

for f in /var/log/mongo/mongod.log.????-??-??T??-??-??;
do
        7za a "$f.z" "$f"
        rm -f "$f"
done

### remove files older than x days
find /var/log/mongo/mongod.log.????-??-??T??-??-??.z -ctime +14 -delete

You might like a different compression library, 7z works for me. It produces the .z suffix by default, so the cleanup step looks for that.

Notice how the find command is issued against the file creation time, and configured to delete files older than 14 days here. Your rotation policy and deletion may require otherwise. You can run this as often as you wish to keep files small and granular, or less frequently to get logs covering extended periods. It is sometimes useful to spelunk a single larger file instead of scrolling through several hourly files to track a single event.

This solution takes care of time-triggered rotation, and is not sensing file size in any way. But it should be easy enough to modify the script to only log rotate if the current mongod.log is larger than some predefined size.

Happy admin!

The new Windows Azure Portal looks great, but has moved things around a bit. This post serves as note to self and others:

How do I set a custom domain name for blob/ table/ queue?

1. Go to the new portal https://manage.windowsazure.com/
2. Click the “Storage” item on the left (icon reminiscent of a table or spreadsheet)
3. Click on the one of your storage items for which you want to create a custom domain
4. Click the “configure” tab (you are in “dashboard” by default)
5. Click the “manage domain” icon on the bottom action bar (look all the way at the bottom between “manage keys” and “delete”)
6. Enter the full domain name you want to have point to the storage “bob.mydomain.com” (assuming you own mydomain.com)
7. Set up a CNAME record in your DNS server for the domain you own as instructed
8. Validate the CNAME entry (may need a bit of time to propagate, so let it).

Steps 6-8 described here: http://www.windowsazure.com/en-us/develop/net/common-tasks/custom-dns/

As many of us deploy our shiny web services and expose them to the world (or just our apps), we invariably encounter these pesky maintenance windows. During these times, a database, other web services or any other IO dependent tasks cannot be performed.

Wouldn’t it be nice to tell the caller of your web API that the operation is currently unavailable? It can get pretty ugly if we don’t solve this. If we simply bring down the whole endpoint, connecting clients will experience a pile-up of timed out connection attempts. If we leave it up, every operation attempted would experience it’s own slow excruciating failure, with the same IO timeout pile-up, this time on your server and often bringing the server to it’s knees with too many doomed connection requests queued up. My game plan shaped up to :

1. Each service operation shall return a standard response, exposing some status flag
2. A configuration controls whether services are to be marked as unavailable
3. A WFC extension will take care of returning the standard response with proper flag when so configured, but let the regular response return under normal conditions.

The requirement that each operation returns a standard response may seem peculiar. You may have created

string GetUserName(string id);
DateTime GetUserBirthdate(string id);

The thing is, when operations fail, you have no way to signal the caller except for smelly nulls or throwing exceptions. Although Soap Fault Exception can do the trick, I find it distasteful to throw a Client Fault exception because exceptions are more costly, and validation of request data often enough finds client faults. For that and other reasons, I use code that looks like the following:

[DataContract(Namespace = "...")]
public class ServiceResponse
{
    [DataMember]
    public string Error { get; set; }
    [DataMember]
    public ResponseStatus Status { get; set; }
}

Where the status is an enumeration:

[DataContract(Namespace = "...")]
[Flags]
public enum ResponseStatus
{
    [EnumMember]
    None = 0,
    /// <summary>
    /// Operation completed without failure
    /// </summary>
    [EnumMember]
    Success = 1,
    /// <summary>
    /// General failure
    /// </summary>
    [EnumMember]
    Failure = 2,
    /// <summary>
    /// Client request not valid or not acceptable
    /// </summary>
    [EnumMember]
    ClientFault = 4,
    /// <summary>
    /// Server failed processing request
    /// </summary>
    [EnumMember]
    ServerFault = 8,
    /// <summary>
    /// The underlying service is not available, down for maintenance or otherwise marked as non-available.
    /// </summary>
    [EnumMember]
    BackendFault = 16,
    /// <summary>
    /// Convenience value for client fault failure comparison
    /// </summary>
    ClientFailure = Failure + ClientFault,
    /// <summary>
    /// Convenience value for server fault failure comparison
    /// </summary>
    ServerFailure = Failure + ServerFault,
    /// <summary>
    /// Convenience value for backend failure comparison.
    /// </summary>
    BackendFailure = Failure + BackendFault
}

One may also abstract the ServiceResponse to an interface, allowing any response object to implement the interface rather than inherit the base response. For this post, let’s just go with the base class.

Now the signature of every operation would be an object derived from ServiceResponse. Rather than a fragmented GetName, GetBirthdate etc – a chatty interface anyway – we would expose:

public class GetUserResponse: ServiceResponse
{
    [DataMember]
    string Name{get;set;}
    [DataMember]
    DateTime Birthdate {get;set;}
    // whatever else a user profile has..
}
// then the operation signature becomes
[ServiceContract]
public interface IMyService
{
    [OperationContract]
    GetuserResponse GetUser(string id);
    // and other operations
}

Now that we have that out of the way, you get the payoff: we can define a fail fast attribute to decorate operations we know rely on some back-end which may be turned off on us. We’ll utilize the IOperationBehavior extension point of WCF, allowing us to specify behavior on an operation by operation basis.

I’ve created an attribute implementing the IOperationBehavior. It replaces the operation invoker with my own implementation when ApplyDispatchBehavior is called. All other IOperationBehavior methods remain blank.

public class FailFastOperationAttribute : Attribute, IOperationBehavior
{
    public void Validate(OperationDescription operationDescription) { }
    public void ApplyDispatchBehavior(OperationDescription operationDescription, DispatchOperation dispatchOperation)
    {
        var returnType = operationDescription.SyncMethod.ReturnType;
        dispatchOperation.Invoker = new FailFastOperationInvoker(dispatchOperation.Invoker,returnType);
    }
    public void ApplyClientBehavior(OperationDescription operationDescription, ClientOperation clientOperation) { }
    public void AddBindingParameters(OperationDescription operationDescription, BindingParameterCollection bindingParameters) { }
}

The finishing piece is to implement the operation invoker. It will check a special configuration, and based on that would either invoke the underlying operation as the stock implementation would have, or construct a new response with the failed flags set.

public class FailFastOperationInvoker : IOperationInvoker
{
    private readonly IOperationInvoker _operationInvoker;
    private readonly Type _returnType;
    public FailFastOperationInvoker(IOperationInvoker operationInvoker, Type returnType)
    {
        _operationInvoker = operationInvoker;
        _returnType = returnType;
    }
    #region IOperationInvoker Members
    public object[] AllocateInputs()
    {
        return _operationInvoker.AllocateInputs();
    }
    public object Invoke(object instance, object[] inputs, out object[] outputs)
    {
        object result;
        if (Config.ShouldFailFast())
        {
            outputs = new object[0];
            // construct response of the type the specific method expects to return
            result = Activator.CreateInstance(_returnType);
            // mark the response as fail fast failure
            result = (result as ServiceResponse).Error = "Not available";
            result = (result as ServiceResponse).Status = ResponseStatus.Failure|ResponseStatus.BackendFault;
        }
        else
        {
            result = _operationInvoker.Invoke(instance, inputs, out outputs);
        }
        return result;
    }
    public IAsyncResult InvokeBegin(object instance, object[] inputs, AsyncCallback callback, object state)
    {
        return _operationInvoker.InvokeBegin(instance, inputs, callback, state);
    }
    public object InvokeEnd(object instance, out object[] outputs, IAsyncResult result)
    {
        return _operationInvoker.InvokeEnd(instance, out outputs, result);
    }
    public bool IsSynchronous
    {
        get { return _operationInvoker.IsSynchronous; }
    }
    #endregion
}

A method for determining if the API should by up or down hides behind the Config.ShouldFailFast() call. Read your app setting, check a file, do what you like to make that determination.

The next thing is manufacturing an instance of a response object. Here we need to create the same type or a type assignable to the one the formal method expected. Note that that type would need to have a parameter-less constructor for this to work. Since all my service DTO are plain POCO, this is rarely a restriction. With this code in place, all we need to do is decorate specific methods as [FailFastOperation] and bingo!

As documented here on MSDN the PropertyItem object does not have a public constructor.

What to do then when you want to add a property to an image, using Image.SetPropertyItem(..) method?

This post suggests you create some bank of all property items you want, hold it memory and clone from it.

A commenter on that blog suggested using reflection: Get the non-public parameter-less constructor and invoke it. Notable downside for this approach is reliance on internal implementation of the object. True. I’ll risk it though.

In my implementation, I added a helper method which simply generates the PropertyItem using System.Activator like so:

public static PropertyItem CreatePropertyItem(int id, int length, short exifType, byte[] buffer)
{
    var instance = (PropertyItem)Activator.CreateInstance(typeof(PropertyItem), true);
    instance.Id = id;
    instance.Len = length;
    instance.Type = exifType;
    instance.Value = buffer;
    return instance;
}

Pretty clean and simple. Under the covers, Activator will use some reflection to create the instance, but would also utilize some caching and speed written by not-me. I like not-me code because it means ** I** don’t have to write it.

Since one of my my upcoming talks at http://socalcodecamp.com is on the subject of reflection, this all falls neatly into place.

In a previous posting, I discussed replacing the stock MVC serializer used for the JsonResult exposed when you use the controller method Json(..) instead of View.

This was all find and dandy. But how – you may wonder – can I call actions that contain complex parameters? Do I need a special binder? Should I write my own?

The answer is mostly “no”. As Phil Hack blogged, the ASP.NET MVC framework already contains value providers that take care of that for you. All you need to do is ensure that your Javascript calls your action using some very specific header and mime types.

Here is a function that may help you

<script type="text/javascript">
       var postToAction = function (sender) {
           var json = $('textarea#' + sender.data).val();
           $("textarea#myResponse").val('working...');
            $.ajax({
                url: document.location.href,
                type: 'POST',
                dataType: 'json',
                data: json,
                contentType: 'application/json; charset=utf-8',
                success: function (data) {
                    var replyText = JSON.stringify(data);
                    $("textarea#myResponse").val(replyText);
                }
            });
        };
    </script>

The special sauce here is the dataType and contentType together with the POST method. The rest is pretty much how jQuery wants to be called.

On line 6 you will note that I’m POSTing to the same URL as the browser is on – you may want to fiddle with that.

We call this function by hooking up a button, something like:

$("button#myButtonId").click('myTextAreaWithJsonContent', postToAction);

In your own implementation you will no doubt compose some object in JavaScript, or create a Json object. In the code above, I’m simply using Json formatted string: Line 3 gets the value from a text area element. If you compose a complex object and want to send it, you will need to convert it into a Json string. jQuery 1.5.1 – which I happen to use – already contains this facility, JSON.stringify(x) would do the trick. You can see it here used on line 12, for the inbound response which I simply stringify and hydrate a response text area. But this is only demo code – you will wire up the success (and failure function- right?) to your own logic.

But back to the core issue: Yes, MVC supports submitting Json and hydrating my controller action, but how do I persuade it to use my custom chosen super duper serializer rather than the stock one?

The answer is: Create a custom ValueProviderFactory and instantiate your favorite serializer in there. A bit of inspection on the MVC source code on CodePlex.com reveals the stock implementation.

Here’s a modified version, which isolates the serialization in a clear way:

public class MyJsonValueProviderFactory : ValueProviderFactory
{
    public override IValueProvider GetValueProvider(ControllerContext controllerContext)
    {
        if (controllerContext == null)
        {
            throw new ArgumentNullException("controllerContext");
        }
        if (!controllerContext.HttpContext.Request.ContentType.StartsWith(
                "application/json", StringComparison.OrdinalIgnoreCase))
        {
            return null;
        }
        object value = Deserialize(controllerContext.RequestContext.HttpContext.Request.InputStream);
        if (value == null)
        {
            return null;
        }
        var bag = new Dictionary<string, object>(StringComparer.OrdinalIgnoreCase);
        PopulateBag(bag, string.Empty, value);
        return new DictionaryValueProvider<object>(bag, CultureInfo.CurrentCulture);
    }

    private static object Deserialize(Stream stream)
    {
        string str = new StreamReader(stream).ReadToEnd();
        if (string.IsNullOrEmpty(str))
        {
            return null;
        }
        var serializer = new JavaScriptSerializer(new MySpecialTypeResolver());
        return serializer.DeserializeObject(str);
    }
    
    private static void PopulateBag(Dictionary<string, object> bag, string prefix, object source)
    {
        var dictionary = source as IDictionary<string, object>;
        if (dictionary != null)
        {
            foreach (var entry in dictionary)
            {
                PopulateBag(bag, CreatePropertyPrefix(prefix, entry.Key), entry.Value);
            }
        }
        else
        {
            var list = source as IList;
            if (list != null)
            {
                for (int i = 0; i < list.Count; i++)
                {
                    PopulateBag(bag, CreateArrayItemPrefix(prefix, i), list[i]);
                }
            }
            else
            {
                bag[prefix] = source;
            }
        }
    }
    
    private static string CreatePropertyPrefix(string prefix, string propertyName)
    {
        if (!string.IsNullOrEmpty(prefix))
        {
            return (prefix + "." + propertyName);
        }
        return propertyName;
    }
    
    private static string CreateArrayItemPrefix(string prefix, int index)
    {
        return (prefix + "[" + index.ToString(CultureInfo.InvariantCulture) + "]");
    }
}

It all really boils down to the same ceremony as the default implementation, except on the line
var serializer = new JavaScriptSerializer(new MySpecialTypeResolver()); lets to use our own special serializer. Woot!

To use this instead of the built in one, you will modify your global.asax.cs Application_Start() to include something like

var existing = ValueProviderFactories.Factories.FirstOrDefault(f => f is JsonValueProviderFactory);

if (existing != null)
{
    ValueProviderFactories.Factories.Remove(existing);
}

ValueProviderFactories.Factories.Add(new MyJsonValueProviderFactory());

where the built in one gets removed and my custom one gets added. Pretty straightforward.

With this technique and the one described in my previous post, you are ready to use the full power of MVC as an API supporting a nice strongly typed parameter for the back end developer and supporting fully customizable JSON in and out of methods. No real need for other frameworks or technologies for the serving jQuery needs.

Depending on your methods, you may even get away with one set of actions serving both form posts and jQuery invocation.

Happy coding!

Oh, I remember it like it was yesterday: Dot-com, no DBA, late nights, a pale developer walking into my office head hung low and mumbling something about restoring backup because he ran an update query with no where clause.. the good old days.

Zoom forward a decade.

A developer is cranking away on some MongoDB query generation in code. Not wanting to make any rookie mistakes, she uses the Query.EQ builder, which would surely create the correct syntax. Better than her own string.Format() at least!

Or so she thought.

The issue is a small peculiarity with the Query methods. The formal signature takes a string and a BsonValue:

public static QueryComplete EQ(
    string name,
    BsonValue value
)

So a some call path would get to a method which would create the query. Lets say

public string CreateFooQuery(string value)
{
    var query = Query.EQ("Prop1",value);
    return query.ToString();
}

Did you spot the issue? I didn’t. She didn’t either.

The issue is that the Query.EQ(field, value) method would boil down a value “xyz” to the MongoDB query {“Prop1”:”xyz”} just fine. It would also boil down an empty string “” to the query {"Prop1":""} without flinching. But if you supply null - and yes, strings can be null - then the query becomes {}. Yiekes (!!!): The null query. Yes, the one that matches EVERY document!

Now take that query as a read query - you got all documents returned. Use it for an update - might as well have some good backup strategy!

Note to self: write more unit tests. Ensure that every code path you visit have been visited before and the expected values are returned.

Sigh!

For various reasons you may find that the default JsonResult returned by invoking the controller method such as

return Json(data);

Is unsuitable for the consumer. The main issue most often encountered is that this method uses the JsonResult which in turn uses the JavaScriptSerializer with no access to the JavaScriptTypeResolver.

This means that you can provide that serializer a parameter specifying you own custom type resolver.

Other issues, such as maximum recursion depth and maximum length and type resolvers can be simply configured in web.config. See Configuring JSON Serialization section on MSDN.

Back to the core problem though.

To override the JsonResult, we would need to do 2 things:

1. Create our own custom JsonResult implementation
2. Tell the controller to use ours rather than the stock one.

A new JsonResult is needed because the base one hard codes the construction of the JavaScriptSerializer.

So here we go. Some CTRL+C, CTRL+V later from the open source MVC on Codeplex gives us

public class TypedJsonResult : JsonResult
{
    public override void ExecuteResult(ControllerContext context)
    {
        if (context == null)
        {
            throw new ArgumentNullException("context");
        }
        if ((JsonRequestBehavior == JsonRequestBehavior.DenyGet)
            && string.Equals(context.HttpContext.Request.HttpMethod, "GET", StringComparison.OrdinalIgnoreCase))
        {
            throw new InvalidOperationException("JsonRequest GetNotAllowed");
        }
        var response = context.HttpContext.Response;
        response.ContentType = !string.IsNullOrEmpty(ContentType) ? ContentType : "application/json";
        if (ContentEncoding != null)
        {
            response.ContentEncoding = ContentEncoding;
        }
        if (Data != null)
        {
            var serializer = new JavaScriptSerializer(new BiasedTypeResolver());
            response.Write(serializer.Serialize(Data));
        }
    }
}

You will note that we’re still using the JavaScriptSerializer, but this time we’re controlling its construction and decided to give it our own type resolver. More on that type resolver in a bit.

Next, we want to our controller(s) an easy way to choose our TypedJsonResult rather than the stock one. Luckily, the controller boils down the call Json(data) and several other signatures to a call to a virtual signature which we may simply override, like so:

protected override  JsonResult Json(object data, string contentType, Encoding contentEncoding, JsonRequestBehavior behavior)
{
    return new TypedJsonResult { Data = data, ContentType = contentType, ContentEncoding = contentEncoding, JsonRequestBehavior = behavior };
}

That’s it! On line 3 you will notice we return our custom Typed JsonResult, whereas the stock implementation would have returned the JsonResult.

If this new behavior is desired everywhere, then you would probably want to place this override in a base controller of your own, and have all your controllers inherit it.

Other powers this bestows on you is of course using some other serializer altogether, such as Json.NET or whatever you fancy.

Now back to my own type resolver. You could, after all, use the SimpleTypeResolver built into the framework, which works quite well. However, it introduces fairly long type names – frowned upon by my clients consuming this Json on other platforms, and also doesn’t enable me to map my own type names to a type of my choice. Enter the BiasedTypeResolver.

private static readonly Dictionary<string, Type> _KnownTypes;
static BiasedTypeResolver()
{
    _KnownTypes = new Dictionary<string, Type> ();
    var appDomain = AppDomain.CurrentDomain;
    foreach (var assembly in appDomain.GetAssemblies().Where(a => a.GetName().Name.EndsWith(".ValueObjects")))
    {
        foreach (var type in assembly.GetTypes().Where(t => !t.IsInterface && !t.IsAbstract))
        {
         _KnownTypes[type.Name] = type;
        }
    }
}

public override Type ResolveType(string id)
{
     var result = Type.GetType(id);
     if (result != null || _KnownTypes.TryGetValue(id, out result))
     {
         return result;
     }
     throw new ArgumentException("Unable to resolve [" + id + "]", "id");
}

public override string ResolveTypeId(Type type)
{
    if (type == null)
    {
        throw new ArgumentNullException("type");
    }
    return type.Name;
}

This resolver spelunks specific assemblies only (those named [whatever].ValueObjects which are my naming convention for POCO public objects) and catalogs them into a dictionary by short type name.

It doesn’t know how to resolve 2 types of the same name if they only differ by namespace, but then again I’d ask “how come you would define 2 classes of the same exact name in your solution?” You can define type names to whatever suites your needs.

The resolver’s responsibility is twofold: Given a string , return the System.Type that corresponds to it. Given a type, return a name for it. The former is used during deserialization, the latter when serializing.

Now you may not need or like this particular type resolver implementation. But now that you can inject your own, possibilities are limitless. These can range to configuration based type resolution, versioning decisions base on some revision upgrades etc.

Note also that upon deserialization, the type resolver is only called when a type discriminator exists in the JSON stream. That is, a complex type that doesn’t contain “__type”:”foo” will be serialized by the JavaScriptSerializer by matching the target member name rather than the resolver. This is nice because the JSON can contain strategically placed type discriminators for polymorphic reasons on some members, but be left terse and bare otherwise.

Hopefully, this helps you, or me-of-the-future when the gray cells got filled with the next great thing..

Happy Coding!