The desperate quest for doing it 'right'

This morning I ran into an interesting design decision. The problem at hand isn't that interesting, I've solved it a lot of times before. The interesting thing is that this problem isn't always solved the same way. It goes like this: do you tell an element which is inside a container (which can be inside another container) to exclude (remove) itself from its container or do you tell the container to exclude (remove) the element? This might sound simple enough, but what is the right thing to do here? And if one is chosen, on what ground is that approach the right thing and is that always the case, no matter what the scenario might be? No, "It depends" doesn't cut it, for the sole reason that every single day probably millions of developers around the world are, in any state of desperation, searching for the right thing to do, be it for this or other problems. Check the various Q&A sites, the various newsgroups and above all, the wide range of developer blogs, articles and twitter channels, and you'll see that a lot has been, is and will be discussed about that single concept: the right thing.

When I was confronted with the decision outlined above (more on that below), I wondered how a developer with decades of experience in the trenches like myself still has to wonder about this somewhat small decision and isn't capable of instantly choosing one over the other. Is it the big fear deep in all of us that if we make the wrong decision it might haunt us and eventually will bring us down? Looking at myself, with the massive code base this decision will be part of taken into account, it does bug my mind: if I pick the wrong decision, it might hurt the system, my company, and everyone depending on that. If I have these kind of questions, there must be others with the same question, wondering the same thing: what's the right thing to do. Looking at all the blogs, articles, answers given to similar questions on various Q&A sites, indeed, there are many many people out there wondering that same thing: either showing that by giving advice how to do the right thing (dear reader, if you now start to wonder if this is a recursive blog post, you're probably right), or by asking what it might be.

So I wondered: isn't this quest to find what the right thing is actually haunting our profession and why exactly is this? Why do we care so much? And more importantly: can we ourselves solve this?

This post was partly triggered also by a blog post I read this morning by Patrick Smacchia, where he shows that a toolkit written by Jeremy D. Miller called StructureMap has cyclic dependencies between namespaces and Patrick tries to make a case that this kind of coupling is apparently not that great to have. Reading the post I wondered why anyone would give a hoot about such a thing. Don't get me wrong, I like solidly written software which allows great maintainability and extensibility without a lot of effort, but I couldn't help wondering who decides what's right and wrong and why we should care about these kind of 'rules'. A lot of these rules make sense simply because they are based on common sense, however I still have the feeling that the vast majority of these rules only work in a given scenario, however in many situations the boundaries of these scenario's are omitted, be it deliberately or by mistake. The pitfall is that if these scenario boundaries aren't given, the rule at hand starts to look like a rule which can be applied always as it apparently is a rule which is one of the ones based on common sense and is the right thing to do, as it has no boundaries/scenario given where it does work so it should work always.

With the rewrite of LLBLGen Pro's designer for v3.0 using .NET 3.5, I'm trying to do some things differently compared to what I did in v2.x. One of these things is a completely different set of data-structures to store meta-data. These data-structures give a lot of freedom to reason about the meta-data and as everything is event/observer controlled, it's very loosely coupled. However, I too ran into cyclic dependencies of namespaces (inside a common root namespace in the same assembly). When I detected this, I wondered... "should I correct this" ? But then I thought: "Why? Will the sky fall down if I leave these few cycles in?". So I did what I always do in the case when I have to make a design decision: make a pro/con list and decide what's the better option based on that list, document that decision and why you took it (so you can always read back why a decision was made, the most important thing about design documentation), move on. In this particular case, I didn't see any advantage of refactoring the code to obey some rule which is only important if you're going to split up assemblies (which I'm not planning to do in this case) so I left it in.

You might wonder why I didn't present to you right away the pro/con list of the decision I started this post with. The main reason is that I wanted to show you that there is no such thing as the right thing if there's no context given, or better: if your situation isn't known. This is very important. Today, and in the years to come, you'll likely be exposed to articles, blog posts, books, lectures and what not, written by generous people who simply want to help you out, which will tell you what's the right thing to do. I'd like you to keep one single thought in mind, whenever you read such an article, post etc.: Does the scenario this rule, this good advice, applies to actually match my scenario? If not, be careful to apply that advice without proper thinking it through. Software engineering isn't an exact science: there's no such thing as a formula where you put something in and the result is calculated, our profession is about building an executable form of what's been described as the functionality of a system, however there are no turn-key solutions to make that possible: every situation is different.

Let's go back to the decision I started with and give it some context, a scenario, the situation it occurred in. This might help you with what you thought instantly what the right thing was when you read the first paragraph. In LLBLGen Pro v3, the user can add meta-data obtained from multiple databases to a single project and choose which elements to obtain from these databases, e.g. which tables, which schemas etc. See the screenshot below for an impression.

Database Meta Data retrieval wizard, step 2
LLBLGen Pro v3 alpha: Step 2 of meta-data retrieval wizard.

The screenshot above shows how the user obtains the meta-data, a simple click-through wizard with some fancy selection/filtering (not shown). The ability to select elements, also raises the question: "what if I selected a couple of tables I don't want to see anymore in my project?". Or in other words: I want to be able to exclude elements (tables in this example) from the project later on without actually removing them from the database. In the scenario presented to the user this looks something like this:

Context menu Catalog Explorer to exclude two tables
LLBLGen Pro v3 alpha: Context menu (incomplete, not all features have been added yet!)

The above screenshot shows the ability to exclude the selected tables (Customer and Employee in this case) from the project. This means that the objects representing these two tables are removed from the meta-data data-structure of the Project (through a controller which asks the Project to exclude the elements at hand which will delegate the call further) and all mappings to these elements are cleared. This feature works, it removes the tables from the Tables collection in the schema object representing the schema the tables are in and as everything is observer-aware, events are raised which are picked up by mapping objects which clear themselves if they have the elements excluded as their target, all undo-able thanks to Algorithmia.

When I wanted to implement the feature on the Schema node I ran into a slight problem: a Schema isn't a Schema Element like a table, view or stored procedure, it is a container, although it is contained in a Catalog. I had to refactor the code I had to also support these different elements (catalog, schema, database meta container, which all aren't a schema element). This gave me the hint that the whole setup might not be correct and I should simply create an interface like IExcludable or something, implement that on all elements which are excludable and call an Exclude method on that. Sounds logical and feng shui-compliant with Common Sense Software Engineering (CSSE), don't you think?

However this runs into a tiny problem: does every element know its container? Is the container of a table the schema (its logical container) or the Tables collection in the schema object (its physical container). To be able to work with meta-data, it's essential that a table knows its schema. However it doesn't need to know that it's in a Tables collection. A schema knows its parent catalog, but a catalog doesn't know it's parent container as it's not something it should be aware of (it doesn't have a logical container, although it has a physical container in the Project). Is it better to tell the element that it should remove itself from its parent's container, or is it better to tell the parent that a contained element has to be removed? Example: do you tell the schema that it should remove itself (which in turn will force the schema to tell its logical container (catalog) to remove the schema) or do you tell the catalog that a schema has to be removed?

The interface idea sounds great, but requires that elements know their container (also for the catalog). It gives the freedom to implement the logic inside the elements which it is all about instead of in code outside the data-structures. On the other hand, code outside the data-structures and placed in a method in the Project class, which contains the meta-data, which controls the calls to the right parent is also tempting as it also knows the container of the catalog, something the catalog itself doesn't know. Yes, the Catalog owns the Schemas collection, but does it own the Schema elements inside the Schemas collection?

So it comes down to:

  1. Interface route. This route requires little code in Project, as an IExcludable is passed into a method on Project by the controller and the project simply calls the Exclude method on the object and the object has to take care of it being excluded (removed) from its parent. We've to make sure the Catalog knows its container as well which is outside its reach at the moment (other assembly) so logically not the right thing to do. Instead for the catalog, this requires some if/else code to call the container of the catalog as well.
  2. Manager code route. This route uses some switch/case statement in Project and based on the element to exclude, it calls the proper container to exclude the element. This looks straight forward, but places knowledge what the parent of which element is, inside the Project class instead of the element itself. This could be less ideal when for example the parent type changes and you have to hunt for all the references to that parent and change that code everywhere.

One thing to note is that, as you can see in the second screenshot, multiple target database types can be in the same project. Telling a project to exclude a given table object isn't a matter of asking the single container of catalogs to remove the given table object, one first has to find the proper database specific store which contains the catalog with that table. A table itself doesn't have this knowledge of course, its parent schema's parent catalog does, at least the ID of the database.

Still convinced what you picked as the right thing to do in the first paragraph is the right thing, or is it more complicated than what you initially thought? I know this scenario is very specific but that's precisely the point: the question presented is very simple and likely you've made this same decision a lot of times before, as I have too. Yet, the specific scenario, the specific context the decision has to be made in makes things less trivial than it initially looked like.

I don't believe in do this and it will be all right kind of advice without a firm context description, so I'm not going to give you one. Instead, I'm giving you advice on how you yourself might be able to find the right decision in similar and other situations you will run into: make a pro/con list of each alternative, eventually prioritize these pro/con items if you will, and simply look at the lists and make a decision based on them, make the decision which makes rationally the most sense, considering the pro/con lists. That's it. Make a decision, based on rational reasoning and cold hard facts and document it, implement that decision and move on. Don't let your decision be lead by what looks like a turn-key 10-step process to get it right which is applicable to any (and thus also yours) scenario/context. There's no such thing, every situation, every scenario is different, yours is too. Perhaps a guide tells you to componentize everything which will take you 2 weeks to complete and in the end 99% of the components are used by only one other component. Did you gain anything by that? It might be you actually made things more complex than the situation you had before you componentized everything. Who has to deal with that extra complexity? The people who gave you advice in a generic "Use ABC with XYZ and everything shall be great" article, or you?

In the end, what matters is that you a) made a decision and b) you documented the reason why you made that decision and didn't take one of the the alternative(s). If for example, after a year or two it turns out your decision wasn't the best, based on the knowledge you have at that moment, you can always check the design decision documentation you've made of the decision, and conclude you did make the right decision back then, but based on other, perhaps incomplete (compared to the situation after two years) knowledge/information. That's life. As my wise mother always says: "If you'd know everything up front, it's not hard anymore to get rich".

What I decided? For this particular case I chose the interface route with the if/else for Catalog in the Project method. Yes, it's perhaps not that pretty due to the if-statement but the alternative isn't that great either and based on the pro/con items of both alternatives, the interface route seems the best choice. In this context, at this moment, with the information at hand.

Should you do the same thing in the situation where you have to make the same decision? That's not a conclusion you should draw from this post, instead you should take my advice, make the pro/con list and decide for yourself what to do. It might be you make the same decision, it also might be you pick an alternative. That's ok, you have the pro/con list plus the reasoning to prove you made the right decision at that moment. That's what matters.


  • >but I couldn't help wondering who decides what's right and wrong and why we should care about these kind of 'rules'

    Did I wrote that the StructureMap design choice is bad or wrong? I wrote verbatim from the post:

    >the code structure is entangled, there is no care for componentization nor any identifiable layer.

    This is fact, not judgment. From such a graph (shown in the original post), the team and even tier outside the team (like code consumer of a Fx) can discuss design choices and compromises based on fact, not based on rumor, hunches, guess, rhetoric or impression.

  • What about having a status on each of the contained objects - an IStatus that defines an interface containing a status and a status changed event? You could define your states and have the container manage it's items anyway it wants based on the item's status.

  • @Patrick: as I said per email as well: displaying an image, showing 'facts' will tell a story. Omitting info might make the story told by the data presented look like it's the truth while it might be slightly besides the truth.

    @Spiledpinchvictim: interesting idea! So in short you want objects to signal to observers (e.g. their containers) that they are obsolete and should be removed. In fact I have something like that in place (INotifyElementRemoved) which raises an event that the object has been removed (or should be) so references to it should be cleaned up. Why didn't I think of that... :)

  • As you seem to be a regular reader of Jeremy's (D. Miller) blog, you should be familiar with the term Reversibility. I believe that concept should be an important aspect of your decision as well. In a few words, it says that you should not spend a lot of time trying to find the right decision if it is very easy to change that decision later on. Don't get me wrong, I do recognize your dilemma and run into it almost daily, but this concept alone has prevented me from stalling in an analysis paralysis quite often.

  • If your presentation (the treeview) doesn't reflect the model (your schema/table hierarchy), you should build a view model that match the presentation structure...

    In this view model, every node would know its parent, only for presentation adaptation purpose. It would give a strong support for the logical containers even if they don't belong to your model hierarchy.

    This way you don't have to tweak your model for presentation purpose.

    Don't know if it's the 'right' thing to do but decoupeling your concerns will make your code more supple.

  • @Think before coding: the treeview does reflect the model, that's it purpose, the user can browse the model through the UI. I see what you're getting at, though the visual representation is actually simpler here: the physical model is presented as such, the thing is that not every element in the hierarchy knows what its parent is. It's not coupling that's part of the problem, because the elements of physical hierarchy internally (the model objects) aren't always knowing their containers, (due to decoupling of their defining namespaces/assemblies), this has nothing to do with the visual representation of things, that's just a way to show what we're dealing with here.

    The point of the post however isn't to focus on that detail, it's about the fact that if something is presented as 'do this, it will make things better', it requires context, and it also requires that that context matches the context you're dealing with. If not, the advice is not useful. Furthermore, people should always think for themselves (e.g. think before coding ;)) to find out what is the best approach, based on arguments they define/recognized themselves.

    Like, 'decouple your concerns, it will make your code better'... why? (I'm not arguing it wouldn't, I just use it as an example) A person should think about why that is, and what the pro/con's are of decoupling and tight coupling and what's better for a given situation. THEN the developer makes wise decisions and the software is maintainable (as decisions are based on reasoning) and not a hodgepodge of 'good' advice brought by random people on the net.

  • So if I understand this article and the comments correctly, these are the important nuggets:

    Context - key to success is your understanding of the problem at hand.

    Gray - there’s no right or wrong. Track (involve your audience) the decision making process.

    Reversibility - a decision that is easy to reverse should be dealt with in a pragmatic manner.

    I agree with all of these points and I’d like to add that "decision making" is much bigger and complex than we touched in this article. Frans was clearly wearing his white thinking hat ;)

    Have a great weekend all!

  • "So I wondered: isn't this quest to find what the right thing is actually haunting our profession and why exactly is this?"

    In 'Zen and the Art of Motorcycle Maintenance', Robert Pirisg write a lot about how his students are crippled by a compulsion to do 'the right thing' at the expense of their creativity.

    When I first started programming I was using Flash 4 which nobody really took seriously. It was a lot of fun and as long as something worked people were pretty amazed whatever the state of the code beind it.

    As Flash has evolved and I've got into more industrial quality technologies, I've lost some of my ability to reason and have confidence in what I do because I've become infected with the need to things the 'right way'.

  • Frans,

    I think you misunderstood Think Before Coding's point about a view model.

    Imagine that your Model contains this interface:

    public interface IHasChildren
    where T : IHasChildren
    #region Properties (public)
    IEnumerable Children { get; }

    Now, for presentation purposes, you may find situations where it would be VERY helpful to be able to nagivate from child to parent (say when a UI event is fired on a child, we may have to do something with the parent).

    Since it may not be possible or it may simply be difficult to modify all of the classes that have the IHasChildren interface, we can add a decorator to the View Model, which is used only in the presentation layer:

    public class ParentChildDecorator : IHasChildren
    where T : IHasChildren
    #region Fields (private)
    private readonly T _item;
    private readonly ParentChildDecorator _parent;

    #region Constructors
    public ParentChildDecorator ( T item, ParentChildDecorator parent )
    _item = item;
    _parent = parent;

    #region Properties (public)
    public IEnumerable<ParentChildDecorator> Children
    foreach ( var child in _item.Children )
    yield return new ParentChildDecorator( child, this );

    public ParentChildDecorator Parent
    return _parent;

    #region IHasChildren Properties
    IEnumerable IHasChildren.Children
    return Children.Cast( );

    Now you can use the following type of code to interact with any object that implements IHasChildren in a bidirectional fashion using the decorator above:

    var item = new TestItem {
    Children = new[ ] { new TestItem( ) }

    var viewModelItem = new ParentChildDecorator( item, null );

    Debug.Assert( viewModelItem.Children.First().Parent == viewModelItem );

    Suffice it to say that a View Model is a set of classes/interfaces that exist only at the presentation layer to make the necessary adjustments to the underlying Model to make it VERY EASY to program the presentation layer. There are plenty of good write ups on the View-View Model-Model-Controller pattern out there.

  • @Mark: I think I understood what he said, I'm not denying that a set of UI specific classes for display purposes only is a solution, however my pragmatic gut-feeling tells me that writing a hierarchy of classes for just UI display purposes (and we're talking about a lot of classes here), is a bit overkill if I can solve the particular problem with 1 if statement.

    (the article wasn't about the UI specific problem as well ;) it was about the question whether chosing a less 'popular' option( because it seems 'not done' in the eyes of the preachers on the internet and books/conferences) is something you should do because it fits your situation or should you drop it because someone else might say "what a bunch of **** code!", as it doesn't match -coding style.

  • Anyone from an Agile background, like myself, will happily write "ugly" code on the first pass that gets the job done and makes our tests pass.

    Once the tests are passing, we can go through the "messy" bits and look for anything that would benefit from immediate refactoring.

    From an Agile perspective, the cost-benefit analysis of making these refactorings should always be at the forefront of one's thought processes. This goes against the dogmatic it-must-be-this-way type of thinking.

    Sounds like you're either moving towards, or already endorse, Agile techniques.

  • It's interesting that most of the comments focus on the problem and not on what I thought was your main point: that "context" is everything. I'm reminded of Andy Hunt's book "Pragmatic Thinking and Learning," where he points out that the difference between an expert and a novice is the ability to 'adjust' a rule (idiom/design pattern/etc) based on context. Thanks for the peek at another real-life problem and the dilemmas that we developers go through...

  • I'm interested in your thoughts on software development being an exact science. Is building a bridge an exact science? An automobile? Yes, the comparison is apples to oranges, but (I'm sure I am not the first to say this) they are both fruit, are they not? (please don't "flame me" - I have out there thoughts but I'm not trying to waste anyone's time)

  • @yohoe1101: I don't see building a bridge as science, it's an application of science so a bridge can be build. In that light, writing a program isn't science, it's applying science so the software is doing what it should do. At least I look at it that way. Others see it as a creative process which has little to do with science at all. I find that a little too farfetched.

    The science behind what we apply in software engineering every day, I think that's exact, as in: you can prove that things are correct or not. How that is applied in writing actual software is up to the engineer. Similar to how physics and math is applied in building a bridge.

Comments have been disabled for this content.