Fluent asynchronous API 3: the helper library

In this third post, I’ll show the inner workings of the little helper library that I wrote to build fluent asynchronous APIs. As you’ll see, it’s not simple, but analyzing it does, I hope, give interesting insights about asynchronous programming.

In the first post in this series, I showed an example of an API call:

dump
  .write('The file readme.txt contains:')
  .fromFile('readme.txt')
  .write('End of file...')
  .then(function() {
    // do something else
  });

With regular Node-style asynchronous calls, the same code would look like this:

dump
  .writeSync('The file readme.txt contains:')
  .fromFile('readme.txt', function(err, text) {
    dump.write('End of file...', function(err) {
      // do something else
    });
  });

The helper library will have to transform the API calls on the fly from the first sample to the second, or something that behaves like it.

The helper is a mix-in, that can be applied to an object by calling the “flasync” function with the object as a parameter. In order to maintain a queue of pending callbacks, a _todo array is added to the object, as well as a _isAsync Boolean to keep track of whether any asynchronous method calls have been made yet:

function flasync(thing) {
  thing._todo = [];
  thing._isAsync = false;

The second part of the puzzle is the private method that will call the next callback and manage _todo and _async:

thing._nextTask = function nextTask() {
  if (thing._todo.length === 0) {
    thing._isAsync = false;
    return;
  }
  var nextTask = thing._todo.shift();
  try {
    thing._isAsync = true;
    nextTask(function (err) {
      thing._nextTask(err);
    });
  }
  catch(err) {
    // If there's an error, we call onError and stop the chain.
    thing._todo = [];
    if (thing._onError) {thing._onError(err);}
    else {throw err;}
  }
};

The function does some simple error handling, but the heart of the method is getting the next task from the queue, then executes it, passing in a function that will call thing._nextTask again. It’s important to remember this convention: anything that goes into the queue must be a function that takes a “done” callback as its only parameter.

The simplest way to get something onto the queue is to call the “then” method:

thing.then = function then(callback) {
  thing._todo.push(callback);
  // Ask the first task to execute if it doesn't already exist.
  if (!thing._isAsync) {
    thing._nextTask();
  }
  return thing;
};

“Then” must be called with such a function that itself takes a “done” callback. The method pushes it onto the queue, then checks the _isAsync flag. If the queue is not yet processing asynchronous calls, the first task in the queue gets started. When that task determines that it’s time to call “done”, it does so, which triggers the next task, and so on until the queue is empty.

The “async” method, which is used to add a chainable asynchronous method to the API (see second post), is not so different:

thing.async = function async(method) {
  return function asyncMethod() {
    var args = Array.prototype.slice.call(arguments);
    args.unshift(thing);
    var bound = Function.prototype.bind.apply(method, args);
    return thing.then(bound);
  };
};

The return value from an invocation of “async” is a wrapped version of the function that was passed in. The code is not very easy on the eyes because it has to manipulate the arguments array and curry it into a bound version of the method. All there is to it however is just that: create a version of the method call, complete with its list of arguments and “this” context, that can be stored (using thing.then) and executed later, but with a single parameter that is the “done” callback.

In summary, the API user calls api.foo(arg1, arg2, arg3), which gets stored as a function(done) for _nextTask to call. When that function does get called, it calls into the method the API author defined and passed into async: a function with signature function(arg1, arg2, arg3, done), that calls “done” when appropriate (see second post).

It gets more convoluted for synchronous methods:

thing.asyncify = function asyncify(method) {
  return function asyncified() {
    if (!thing._isAsync) {
      // If not async yet, just do it.
      return method.apply(thing, arguments);
    }
    // Otherwise, bind it and enqueue it
    var args = Array.prototype.slice.call(arguments);
    args.unshift(thing);
    var bound = Function.prototype.bind.apply(method, args);
    thing.then(function(done) {
      bound.apply(thing);
      done();
    });
    return thing;
  };
};

The exact same currying happens here, in order to be able to store the function call with its arguments and context, but we also need to transform the synchronous method into an asynchronous one. This is done by simply wrapping the call into a function(done) that calls “done” right after calling the method.

And that’s probably enough for this post, but before I finish it, I have to make a little confession. I did my research before I went ahead and wrote this helper, but for some reason I did not think about promise libraries, in particular Q. I had studied Q before, but had somehow seen it as just a different way to call asynchronous APIs. As such, I had dismissed it as less comfortable than the “async” library. It’s a question of taste from this point of view, and my preference was going to async, because it’s closer to Node’s style. What I completely missed was Q’s potential as an API for API developers. There is a way to use Q that is very close to what I’ve been re-inventing here.

I’m happy I spent the time to write this library and the accompanying posts, of course, because it gave me the opportunity to dig quite deep into asynchrony in JavaScript. It was definitely a very useful learning experience for me, and I hope these posts have been useful to you too. I will however throw all that code to the trash for the next post, and will attempt to rewrite it all using Q instead. Stay tuned…