Problems with RewritePath and Search Engines.

We just recently launched the new version of LearnVisualStudio.NET. The implementation uses the RewritePath method to serve up dynamic pages based on templates. All of our pre deployment testing showed that this was all working without a hitch. Once we deployed to production however it became clear which test cases we had left out. We didn't test how the site responded when getting crawled by search engines.

Shortly after deployment I started to get an inbox full of error messages that said: "Cannot use a leading .. to exit above the top directory". Apparently this was only happening when a search engine was crawling the site. Hence the error spam. Well needless to say we started to worry about our page rankings. So ironically, a quick  google search on the error message rendered this blog post: http://todotnet.com/archive/0001/01/01/7472.aspx?P...

I followed the instructions on how to create a .browser file and got the problem fixed (mostly). I couldnt leave well enough alone though and had to find out more about why this problem had occured. As it turns out it has nothing to do with HtmlTextWriter vs Html32TextWriter. The problem lies within the System.Web.HtmlControls.HtmlForm's GetActionAttribute method. You can see it in the stack trace from the error dump:

Stack trace:    at System.Web.Util.UrlPath.ReduceVirtualPath(String path)
   at System.Web.Util.UrlPath.Reduce(String path)
   at System.Web.Util.UrlPath.Combine(String appPath, String basepath, String relative)
   at System.Web.HttpResponse.ApplyAppPathModifier(String virtualPath)
   at System.Web.UI.HtmlControls.HtmlForm.GetActionAttribute()
   at System.Web.UI.HtmlControls.HtmlForm.RenderAttributes(HtmlTextWriter writer)
   at System.Web.UI.HtmlControls.HtmlControl.RenderBeginTag(HtmlTextWriter writer)
   at System.Web.UI.HtmlControls.HtmlForm.Render(HtmlTextWriter output)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.HtmlControls.HtmlForm.RenderControl(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.Control.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.Page.Render(HtmlTextWriter writer)
   at SeriesTemplate.Render(HtmlTextWriter writer)

I started looking at the code for those methods using Reflector and noticed something peculiar in "HtmlForm.GetActionAttribute()". Take a look at this code:

private string GetActionAttribute()
{
      string text1;
      VirtualPath path1 = this.Context.Request.ClientFilePath;
      VirtualPath path2 = this.Context.Request.CurrentExecutionFilePathObject;
      if (object.ReferenceEquals(path2, path1))
      {
            text1 = path2.VirtualPathString;
            int num1 = text1.LastIndexOf('/');
            if (num1 >= 0)
            {
                  text1 = text1.Substring(num1 + 1);
            }
      }
      else
      {
            path2 = path1.MakeRelative(path2);
            text1 = path2.VirtualPathString;
      }
      if ((CookielessHelperClass.UseCookieless(this.Context, false, FormsAuthentication.CookieMode) && (this.Context.Request != null)) && (this.Context.Response != null))
      {
            text1 = this.Context.Response.ApplyAppPathModifier(text1);
      }
      string text2 = this.Page.ClientQueryString;
      if (!string.IsNullOrEmpty(text2))
      {
            text1 = text1 + "?" + text2;
      }
      return text1;
}

 

Now, I noticed the code path that led to the exception was in the "ApplyAppPathModifier". The logic of this code seems to imply that it should call ApplyAppPathModifier if the app is configured to use a "Cookieless" forms authentication. More over the FormsAuthentication.CookieMode seemed to be what was controlling this. Inspection of the default settings for forms authentication indicated that the default is "UseDeviceProfile". See MSDN docs: http://msdn.microsoft.com/en-us/library/1d3t3c61....

So what was happening is the search engines were connecting with a browser agent string that ASP.NET didnt detect as supporting cookies. Now our rewritten pages are all public pages and are not authrorized by forms authentication. It is simply the fact that we have forms auth enabled that triggered this. For our situation it was sufficient to just set the cookiless attribute of the <forms> element in the web.config to "UseCookies". This was alot easier than concocting a .browser file with every search engine string we could find. The implications for our site is that we could not support cookiless authentication (which we don't).

My guess is that ApplyAppPathModifier is trying to back up the requested url 1 directory since it uses a sessionid as the top level folder. If the site isnt really using cookiless mode then it tried to back up out of the top level folder. Hence the exception.

So to sum up. If you are mixing forms auth with url-rewriting, even if your rewriting is limited to public pages, cookiless forms authentication can cause problems.