Fixing a database with too many orphaned blobs

Hello again,

This time, I'm gonna tell you about an interresting problem we ran into this week.

We had a customers solution, that contained around 6GB worth of media items (I know that, because we just loaded them into the solution using serialization).

Things where working fine - we removed them a few time to test our import tool (using the little known dbbrowser gem in Sitecore), and finally we where ready to backup the database and move it to it's final destination.

Entering the folder on the MSSQL server, looking at the .mdf file, showed that the database was taking about 25GB of diskspace.....

So clearly something was wrong - so tried shrinking the database, but it only said around 200MB of free space in the database, so that's not gonna work.

The next thing to try, was running the database cleanup tool inside Sitecore, which ran for a while, and then bugged out with a timeout exception.
This however indicated that whatever it was doing, it seems to be doing it on quite alot of data.

It seems the problem is, that this tool fails if there is too many orphaned blobs in the database - orphaned blobs can happen in a few cases - like removing media items using the dbbrowser it seems. (Overwriting media folder using the package installer might also cause this).

Talked to a co-worker, which had this timeout problem before, and got a SQL script from Sitecore Support, that does what the cleanup tool does, but works better, since it is run from inside SQL Server Management Studio.

Tried running it, and after around 90 minutes, it was done, and the database could be shrinked down to 7GB - weee.

However, the script had one problem, it was just standing there running, not really posting any info on what it was doing, how far it had come etc.

So, I took the SQL script, made a small C# program that does the same thing, but gives a percent indicator for how far it has come, and how long it expects it to take to finish running.

The best part of this is, that I'm allowed to share it with you guys - so here it is, my little tool, that fixes the problem in a friendly way.

You are welcome to disassemble it, if you like me are unsure about just running code downloaded from the internet.

Templates and inheritance of layouts... And why you should never do it

Okay, it's been a while since my last blog post, but I've been busy with other stuff.

Anyway, the other day, I ran into an interresting problem on one of the solutions we have made for a customer a while back.

It seems like that if you are inheriting from templates that has a layout (and sublayouts) defined, you might end up with a really weird acting Presentation Details window.

Here is what we had - these four templates:

Article Main Site
Article Sub Sites
News Article Main Site
News Article Sub Sites

All four templates had their presentation details set on their standard values, like they should.
The Main Site templates was using the layout "Main Layout" and the Sub Sites templates was using the "Sub Sites Layout" layout.

Now comes the problem - the customer was wondering how all the sudden, when they created news articles on their subsites, the layout of the news article was using the layout from the main site.

So, what to do? Well ofcouse, you just open the Sitecore backend, navigate to one of the news articles from a subsite, that acts weird, and takes a look at the Presentation Details, to see what is defined there.... To my surprise - it shows the "Sub Sites Layout" as the layout being used on this item.

So now we have an item showing layout A in the backend, but using layout B when rendering... Okay, maybe something hasn't been published correct, so I took a look at the web database - looks the same...

Back to the master database, going to the News Article Sub Sites template, and then to it's standard values - looking at Presentation Details here, everything looks correct... So the item is showing the wrong layout - so back to the news folder, and create a new test news item, so we know that layout hasn't been changed.... That one is also broken!

As you might imagine by now, I was getting pretty puzzled by this, untill somebody suggested that maybe it was a problem with inheritance...

He was right - it turned out, that "News Article Main Site" was inheriting from "Article Main Site", which seemed to work, since they where supposed to have the same presentations, so no problem showing up there.
However, "News Article Sub Sites" was inheriting from "Article Main Site", which meant that it inherited totally different presentations, resulting in this weird behaviour.

The solution was to switch the inheritance to inherit from the right template.

This is (one of) the reason that you should never inherit from a template with defined Presentation Details - it will break badly.

Instead, what you should do, is create a template for each field you want to have on your templates, and then inherit the fields from these templates (so you only have to define them once) - and then copy Presentation Details from template to template - this means that you have maintain them for each template, but thats better than having to debug this problem, and trying to clean up after it.

Sitecore 7.2, Media picker, The core database and alot of headaches

Okay, this is an interresting bug.

It seems like the new Media picker in Sitecore 7.x (the SPEAK version) has a bug - you can trigger it this way:

1: Create a template with an image field.
2: Set the source property of the field to a folder inside the media library.
3: Create an item based on this template, and click Browse on the image field.

Notice that no images from the folder is displayed.

To workaround this, the folder structure should be published to the core database (this is the real bug)......

Now, this workaround causes big problems with Sitecore 7.2, since the publishing framework has been changed, which means adding the core database as a publishing target is dangerous.

Here is a simple guide to trigger this problem:

1: Install clean Sitecore 7.2 initial release
2: Create "core" publishing target
3: Click the "Publish Item" button on the Core item
4: Select the /sitecore/layout/Layouts item
5: Click the "Publish Item" button on the Layouts item.
6: Press F5 to reload Sitecore
7: Open the content editor - you will now see "The layout cannot be found".
8: Open /sitecore/admin/dbbrowser.aspx , open the core database and see that the Layouts item no longer has any children.

This means, that if the Layouts (there are other items that is right as dangerous) item gets published to the core database - every layout defined the core database is wiped, and Sitecore is completely broken.

So, if you are using Sitecore 7.x (which you should, since it is great!), here is a better workaround untill the image picker can get fixed.

Create an include file, give it a name, os it gets loaded after Sitecore.Speak.config (Like Support.406666.config) (406666 is the issue number for this issue), and add this to it:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <overrideXmlControls>
      <override xmlControl="Sitecore.Shell.Applications.Media.MediaBrowser" with="/sitecore/client/applications/Dialogs/SelectMediaDialog">
        <patch:delete />
      </override>
    </overrideXmlControls>
  </sitecore>
</configuration>

This changes the media picker, so it reverts to using the old one, that works.

Fixing Webforms for Marketeers saving dates in different formats

We had an issue a while ago, with a customer where their webforms started saving dates in different formats in the database, which caused the exported data to be mangled.

Sitecore Support found the solution - it seems like the DatePicker control has a bug, where the DatePicker is missing the correct adapter.

The solution is simple - create your own DatePicker class, that extends the builtin DatePicker class, like this:

[Adapter(typeof(DateAdapter))]
public class DatePicker : Sitecore.Form.Web.UI.Controls.DatePicker
{

}

Compile the code, and then go to this item in the master db: "/sitecore/system/Modules/Web Forms for Marketers/Settings/Field Types/Simple Types/Date Picker"

Here you change the Assembly and Class fields to match this class, save the item and publish it.

Done! - really, that is all there is to it - do this, and now your dates show up correct.

Adding new renderings to an item from code

Starting with Sitecore 6.4 and forward, renderings in Sitecore is saved as deltas (differences between the standard values and the "full" rendering) on the item.

This means, that we need to do some work if we want to read, change and save the renderings.

Here is how to both read the renderings, and how to save the deltas back to the item:

public static IEnumerable<RenderingDefinition> GetRenderings(Item item, DeviceItem device)
{
    string xmlLayout = XmlDeltas.GetFieldValue(item.Fields[FieldIDs.LayoutField], XmlDeltas.WithEmptyValue("<r />"));

    LayoutDefinition layoutDefinition = LayoutDefinition.Parse(xmlLayout);
    DeviceDefinition deviceDefinition = layoutDefinition.GetDevice(device.ID.ToString());

    return deviceDefinition.Renderings.ToArray().Cast<RenderingDefinition>();
}

By using the GetFieldValue on the XmlDeltas class, we get the combined layout from the standard values combined with the delta.

Now, to be able to add a new definition to the item, here is what you have to do:

public static void InsertRendering(Item item, DeviceItem device, int index, RenderingDefinition rendering)
{
    string xmlLayout = XmlDeltas.GetFieldValue(item.Fields[FieldIDs.LayoutField], XmlDeltas.WithEmptyValue("<r />"));

    LayoutDefinition layoutDefinition = LayoutDefinition.Parse(xmlLayout);
    DeviceDefinition deviceDefinition = layoutDefinition.GetDevice(device.ID.ToString());

    deviceDefinition.Insert(index, rendering);

    using (new EditContext(item))
    {
        XmlDeltas.SetFieldValue(item.Fields[FieldIDs.LayoutField], layoutDefinition.ToXml());
    }
}

Here we get the definition for the specified device, and just insert the rendering into the passed index, moving the other renderings down the list.

Finally we edit the item, and tells it to set the value of the layout field on the item to the xml defining the entire layout.

The secret is in how SetFieldValue works, since it, behind the scenes, generates the delta between the standard values and the passed xml, and only saves that to the item.

Using this as the base, will allow you to change the renderings, since you can use the same logic to create a ChangeRendering function.

Rendering controls from a placeholder on another item

From time to time, it makes sense to render the content that is defined on another item, as if it was defined on the item the user is viewing.

To do this, we have created an custom placeholder, that can be inserted like any other ASP.NET control, and then configured to get the renderings from somewhere else.

First we need to define a base class for all user controls inserted on a control, to both help getting the datasource, and being able to change it from outside of the control itself.

This can also be done for webcontrols, but I'll leave that to you to figure out how to do that :)

Here is how the base class looks:

public abstract class BaseSublayoutUserControl : UserControl
{
    private Item _datasourceItem;

    private NameValueCollection _renderingParameters;

    protected Item Datasource
    {
        get
        {
            if (_datasourceItem != null)
            {
                return _datasourceItem;
            }

            try
            {
                Sublayout sublayout = (Sublayout)Parent;
                if (!string.IsNullOrEmpty(sublayout.DataSource))
                {
                    _datasourceItem = Sitecore.Context.Database.GetItem(sublayout.DataSource);
                }
            }
            catch (NullReferenceException)
            {
                _datasourceItem = null;    
            }
            catch (InvalidCastException)
            {
                _datasourceItem = null;    
            }
            catch (InvalidOperationException)
            {
                _datasourceItem = null;
            }

            return _datasourceItem;
        }

        set
        {
            _datasourceItem = value;
        }
    }

    protected NameValueCollection RenderingParameters
    {
        get
        {
            if (_renderingParameters != null)
            {
                return _renderingParameters;
            }

            _renderingParameters = string.IsNullOrEmpty(Attributes["sc_parameters"]) ? new NameValueCollection() : HttpUtility.ParseQueryString(Attributes["sc_parameters"]);

            return _renderingParameters;
        }
    }

    internal void ForceDatasourceItem(Item newDatasource)
    {
        _datasourceItem = newDatasource;
    }

    internal void ForceParameters(string parameters)
    {
        Attributes["sc_parameters"] = parameters;
    }
}

Here both the parameters and the datasource is exposed, so it is easy to get them from the control.
Then every usercontrol that is to be inserted using the page editor should just inherit this class instead of the normal UserControl class, and all is fine.

Now that we have the base class done, we need to implement the placeholder - this requires two classes, the placeholder itself, and what is known as a ControlBuilder, which makes it possible to configure how the control behave when inserted.

Here is the code for the control builder class:

public sealed class RemoteLayoutRendererControlBuilder : ControlBuilder
{
    public override bool AllowWhitespaceLiterals()
    {
        return false;
    }
}

Then, we just need to control that does all the work:

[ControlBuilder(typeof(RemoteLayoutRendererControlBuilder))]
[ToolboxData("<{0}:RemoteLayoutRenderer runat=\"server\" />")]
public class RemoteLayoutRenderer : Control
{
    public RemoteLayoutRenderer()
    {
        Device = Sitecore.Context.Device;
    }

    public Item Item { get; set; }

    public string PlaceholderKey { get; set; }

    public DeviceItem Device { get; set; }

    protected override void CreateChildControls()
    {
        base.CreateChildControls();

        if (Item == null || string.IsNullOrEmpty(PlaceholderKey))
        {
            return;
        }

        if (string.IsNullOrEmpty(Item[FieldIDs.LayoutField]))
        {
            return;
        }

        IEnumerable<RenderingReference> renderings = from rendering in Item.Visualization.GetRenderings(Device, false)
                                                        where rendering.Placeholder.EndsWith(PlaceholderKey, StringComparison.InvariantCultureIgnoreCase)
                                                        select rendering;

        int i = 0;

        foreach (RenderingReference rendering in renderings)
        {
            Control control;

            try
            {
                switch (rendering.RenderingItem.InnerItem.TemplateName)
                {
                    case "Sublayout":
                        control = GetSublayout(rendering, i++);
                        break;
                    // Hint - here might be a good place to handle webcontrols ;-)
                    default:
                        continue;
                }
            }
            catch (TargetInvocationException)
            {
                continue;
            }
            catch (InvalidCastException)
            {
                continue;
            }

            if (control != null)
            {
                Controls.Add(control);
            }
        }
    }

    protected override void OnInit(EventArgs e)
    {
        base.OnInit(e);

        Load += RemoteLayoutRendererLoad;
    }

    private void RemoteLayoutRendererLoad(object sender, EventArgs e)
    {
        EnsureChildControls();
    }

    private BaseSublayoutUserControl GetSublayout(RenderingReference rendering, int controlIndex)
    {
        if (string.IsNullOrEmpty(rendering.RenderingItem.InnerItem["Path"]))
        {
            return null;
        }

        BaseSublayoutUserControl control = Page.LoadControl(rendering.RenderingItem.InnerItem["Path"]) as BaseSublayoutUserControl;

        if (control == null)
        {
            return null;
        }

        if (!string.IsNullOrEmpty(rendering.Settings.DataSource))
        {
            control.ForceDatasourceItem(Sitecore.Context.Database.GetItem(rendering.Settings.DataSource));
        }

        if (!string.IsNullOrEmpty(rendering.Settings.Parameters))
        {
            control.ForceParameters(rendering.Settings.Parameters);
        }

        control.ID = ID + "_dynamic_" + controlIndex;

        return control;
    }
}

Now we can just insert this like any other ASP.NET control, and define which placeholder to load the controls from, and then set the Item property to the item they are defined on.

As long as the user controls inherit from our base class, we can configure their datasource and parameters, so they don't know that they have been pulled in somewhere else.

Create a custom Sitecore cache

I have been experimenting with creating a custom Sitecore cache, to be able to cache the result of some calculations that happens all the time.

In this case, we happen to quite often lookup which of the sites defined in Sitecore, that matches a certain criteria (has certain sub items).

In solutions with many sites, this quickly starts to slow the solution down - but there is a simple solution, which is caching the result :)

So first, we have to implement the cache, which requires two classes - the cache (called SiteCache) and a cache manager (called SiteCacheManager).

Here is the code for the SiteCache:

internal class SiteCache : CustomCache
{
    public SiteCache(string name, long maxSize) : base(name, maxSize)
    {
    }

    public bool IsSiteInCache(string siteName)
    {
        object obj = GetObject(siteName);

        return obj != null;
    }

    public void AddSiteToCache(string siteName)
    {
        SetObject(siteName, true, sizeof(bool));
    }
}

And here is the code for the SiteCacheManager

internal static class SiteCacheManager
{
    private static readonly SiteCache Cache = new SiteCache("SiteCache", StringUtil.ParseSizeString("100KB"));

    public static bool IsSiteInCache(string siteName)
    {
        return Cache.IsSiteInCache(siteName);
    }

    public static void AddSiteToCache(string siteName)
    {
        Cache.AddSiteToCache(siteName);
    }

    public static void Clear()
    {
        Cache.Clear();
    }
}

The way it works is pretty simple, since the manager is static, as soon as Sitecore starts, it creates a new instance of the SiteCache cache.
Then, to use this, we create a simple function to test if sites are valid:

public bool IsSiteValid(string siteName)
{
    if (SiteCacheManager.IsSiteInCache(siteName))
    {
        return true;
    }

    bool siteValid = RunSiteTest(siteName);

    if (siteValid)
    {
        SiteCacheManager.AddSiteToCache(siteName);
    }

    return siteValid;
}

This way, we always look in the cache first, and if the site isn't there, we do the normal tests, and if the result is that the site is valid, we add it to the cache, so we can avoid the test the next time.

The reason we do this, is because the cache might have been flushed, so we need to always do the check if the site is missing from the cache.

So this works best, if most of the things you test on is valid - if most of the sites in the solution for this is returning false from the test, the amount of time saved will be minimal.