Future Orchard Part 1: Introducing Tokens

After a long phase of cleanup on the new Orchard 2.0, we are now busy designing new features. We are focusing on a few foundational pieces, and on enabling e-commerce on top of the platform. In this post, I'm going to expose the basics of the preliminary design for one new foundational piece: Tokens.

You could technically confuse Tokens with a fancy String.Format, or with a very lightweight templating solution, but you'd be slightly wrong in both cases. Its usage is what sets it apart from both. You will use tokens whenever you need to build a string by inserting named environmental variables into a string that has placeholders. That is it. No code, no loops, no ifs, just formatted substitution, but really the keyword here is environmental, and by the end of this post it should be clearer what we mean by that.

Let's start with trying to implement Mad Libs. If you have the following Mad Lib:

_____________! he said ________ as he
  exclamation            adverb

jumped into his convertible ______ and
                              noun

drove off with his ___________ wife.
                     adjective

Somebody could pick at random: "Praise the Flying Spaghetti Monster", "vigorously", "left nipple" and "smelly", and we would get the following weird sentence: "Praise the Flying Spaghetti Monster! he said vigorously as he jumped into his convertible left nipple and drove off with his smelly wife."

What's fun and creative about it is that the "format string" and the tokens that go into the placeholders come from different persons, and none of them know in advance what the other will write.

Similarly, Tokens in Orchard 2.0 can come from many sources, and the list is fully extensible. The way you use the API is by first injecting an ITokenizer dependency. Once you've done that, you can call Replace to start formatting strings:

_tokens.Replace(
  @"{MadLib.Exclamation}! he said {MadLib.Adverb} as he
    jumped into his convertible {MadLib.Noun} and
    drove off with his {MadLib.Adjective} wife.",
    null);

Without doing anything else, this will only yield a rather meaningless "! he said  as he jumped into his convertible  and drove off with his  wife." Being used to String.Format as you are, you may be a little frustrated by this and be tempted to directly inject the values for the tokens as an additional parameter to Replace. After all, that null does look a little suspicious. Well, if that is what you want, you should probably continue to use String.Format. But this is not what we're trying to do here. What we want is for whatever other player is available to provide us with values to inject. This is where the second part of the puzzle appears: token providers.

public class MadLibTokenProvider : ITokenProvider {
  public void Describe(DescribeContext context) {
    context.For("MadLib", T("Mad Lib"), T("Random words for Mad Libs."))
      .Token("Exclamation", T("Exclamation"), T("A random exclamation."))
      .Token("Adverb", T("Adverb"), T("A random adverb."))
      .Token("Noun", T("Noun"), T("A random noun."))
      .Token("Adjective", T("Adjective"), T("A random adjective."));
  }

  public void Evaluate(EvaluateContext context) {
    context.For<IMadLib>("MadLib", () => new MadLib())
      .Token("Exclamation", madLib => madLib.GetExclamation())
      .Token("Adverb", madLib => madLib.GetAdverb())
      .Token("Noun", madLib => madLib.GetNoun())
      .Token("Adjective", madLib => madLib.GetAdjective());
  }
}

public class MadLib : IMadLib {
  public string GetExclamation() {
    return PickOneOf(
        "Praise the Flying Spaghetti Monster",
        "Holy rusted metal, Batman",
        "!@#$%");
  }

  public string GetAdverb() {
    return PickOneOf(
        "vigorously",
        "cheerfully",
        "alarmingly");
  }

  public string GetNoun() {
    return PickOneOf(
        "left nipple",
        "rocket",
        "jacket");
  }

  public string GetAdjective() {
    return PickOneOf(
        "smelly",
        "pink",
        "absurd");
  }

  private string PickOneOf(params string[] strings) {
    return strings[(new Random().Next(strings.Length))];
  }
}

public interface IMadLib {
  string GetExclamation();
  string GetAdverb();
  string GetNoun();
  string GetAdjective();
}

The Describe part is pure metadata that will be used to provide users hints on what tokens can be used. The second part, Evaluate, is doing the actual evaluation of the tokens. It's accepting tokens of type "MadLib" and of names "Exclamation", "Adverb", "Noun" or "Adjective". The specific implementation should be fairly simple to understand as it's just picking at random in lists of strings.

Because we now have a token providers that knows how to handle all four of our Mad Lib tokens, and because our provider is picking them at random, we have recreated the fun of Mad Libs.

Decoupling the token values from the formatting site is more than a cosmetic improvement. Now our tokens are available everywhere. I can reuse the MadLibTokenProvider for other Mad Libs of course:

_tokens.Replace(
  @"My {MadLib.Noun} is {MadLib.Adverb} {MadLib.Adjective}.",
null);

Or I can reuse them to add some fun to an e-mail notification, although the notification module knows nothing of Mad Libs (note that the string wouldn't be hard-coded but rather come from configuration):

_tokens.Replace(
  @"{MadLib.Exclamation}, {Site.Admin.FullName},

A new {MadLib.Adjective} comment has been published on
{Site.Name} by {Comment.Author}: {Comment.Text} <a href="
"{Comment.AdminUrl}"">Click here to moderate</a>.", new {Comment: newComment});

In addition to this mix of tokens from various providers, something interesting and new is happening here. We have a number of tokens that are just environmental, such as Mad Libs, the site name, and the full name of the site administrator, but the comment object is provided as local context at the formatting site.

You may have noticed how in our Mad Lib token provider, we provided an expression that creates a new MadLib object when evaluating token values:

context.For<IMadLib>("MadLib", () => new MadLib())

This defines the default, or global, or environmental MadLib to use, but if you needed an IMadLib implementation that was choosing in a different list of words, for example words that actually make some kind of sense in this context, all you would have to do is override the default by just providing it in your context object:

_tokens.Replace("...my Mad Lib...",
  new {MadLib: mySpecialMadLibVocabulary});

This new special object would become the argument to the Lambda expressions for the token evaluation, and it will just work, without a change in the token provider.

In summary, token providers are exposing a vocabulary of tokens that can be used everywhere with the guarantee that any given token is always going to work, even though the values themselves can be global or contextual. The feature effectively gives users a predictable and stable vocabulary that they can use in dynamically populated strings.

Next time, I'll show some more subtle features of the new Tokens API, such as how to build deep graphs of tokens.

15 Comments

  • Have you seen HTML5 microdata? Is it an option for your tokens?

  • @Bruno: I have, but I'm not sure how this would relate to it.

  • What's the use case for this? I don't think I've seen a CMS try to solve this problem before (at least not in the core package). Considering all of the other issues that need to be addressed in Orchard, I think the time would be better spent making it easier to use, not building in complex, rarely used functionality that does not solve an obvious pain point.

    J.Ja

  • @Justin: a typical use case is URL generation, a feature in very high demand, or user-customizable alerts. If you haven't seen a CMS try to solve this problem before, then you have never looked at Drupal. They have a token module that does exactly that and that is used pretty much everywhere in the platform.
    You are mistaken if you think that this is the only thing we are going to do for Orchard 2.0. We are also looking at the "other issues". Also, it's open source, so you are more than welcome to contribute.

  • You're doing great job with Orchard.

    We're a web agency, built at least 3 CMS from scratch, used many ready e-commerce and CMS platforms. Orchard is the first platform where our developers are happy to work with. Many of the problems we faced are already considered and resolved in a clever way. Especially the modular design, we have been dreaming of this for 10 years but not able to make a fully modular product. Thanks to Orchard, finally we have it.
    We're not only using it in our website projects, but also learn new technologies and apply best practices on other projects we are developing.

    Keep up the good work guys.

    Cheers
    Gokhan

  • Very Cool. Cannot wait for v2.0!

  • I'm probably not the only one who isn't familiar with MadLibs, is there any chance you could have another try at explaining what the problem-being-solved is?

    Thanks

  • Seems cool. hoping can see more scenarios the tokens are targeting to in part 2.

  • I noticed that tokens were recently added to the source dev branch and couple of days later completely removed. Is it just some refactoring taking place or this feature was really dropped?

  • Refactoring. You'll actually notice the code that was removed was not consistent with the code in this post.

  • @Rob: yes, I know mustache, and I participated to the design of jQuery templates. This feature is not a templating solution. The point is to get tokens from the environment. The actual replacement of the tokens in the string is a detail, and in the places where we are going to use this, there is no need for complex template structures, just token replacement.

  • @Bertrand In your second post on this you mention creating an e-mail using these tokens. Surely knowing whether some data value is empty or missing is essential when creating an e-mail message, or even looping through a list for some simple values to output. Mustache is "logic less", and very simple.

    The simple implementation of your token system should work well for really simple things your currently throwing at it, but after it is put to numerous real world tests, I would imagine there will need to be some kind of simple logic and you'll have to extend it anyway.

  • @Rob: Drupal proves otherwise. It's real world enough for me. If you need to write loops and tests, does this really belong in a user/admin-provided template? There is a templating solution in Orchard already, it's Razor. This is not what Tokens are for.

  • the reach of a feature, is always hard to grasp :)

    i particularly like the injection of a tokens per context, but as per example, is newComment a model that gets registered by a some generic token provider on the Replace method? haven't looked at the code for this feature yet!

  • @Pedro: newComment is a piece of context provided by the code using the token API. It is a plain dynamic object. A token provider that understands the "Comment" token will inpect it to extract token values.

Comments have been disabled for this content.