Understanding SharePoint: SPRequest

I am going to put together a series of ‘Understanding SharePoint’ posts and cover various important parts of SharePoint which are not very well understood or for which there is little or no information on the net. The first post will be about possibly the most important class in SharePoint called SPRequest – an internal class from the SharePoint object model.

It may come as a surprise to you (which only shows how little it has been written about this) but most of the ‘important’ things in SharePoint are actually done through unmanaged code and the SharePoint .NET object model provided by the Microsoft.SharePoint.dll is a sort of a wrapper to this unmanaged world. Understanding how your code does what you want it to do can be very helpful when troubleshooting issues, fixing bugs, doing performance tuning and writing solid code in general so I believe that every SharePoint developer should know about SPRequest.

As you are aware the gateway to doing anything in SharePoint is the SPSite object. In order to get to any other objects that reside in a site collection such as SPWeb, SPList, SPListItem, SPFile and others we first need to obtain an SPSite and there is a very good reason for that. Every SPSite instance holds a reference to the unmanaged WSS world in the means of a field member of type SPRequest which is called m_Request. The internal SPRequest class has an unmanaged reference to a COM object called SP.SPRequest and having a ClassID of BDEADEE2-C265-11D0-BCED-00A0C90AB50F which is implemented in and exposed by the OWSSVR.DLL class library.

The SP.SPRequest COM object exposes almost 400 basic operations and almost everything you do with the Microsoft.NET managed SharePoint object model that reads from or writes to the ContentDatabase (including data, fields, content types, list schemas, etc) will actually go via this unmanaged COM object. Even more the OWSSVR.DLL is actually an ISAPI extension registered in IIS and its methods can be called directly via an HTTP request to /_vti_bin/owssvr.dll. Many of the Office applications (Word, Excel, InfoPath, SharePoint Designer etc) are using HTTP calls to OWSSRV directly in order to integrate with a remote SharePoint server. It won’t be too much of an exaggeration to say that OWSSRV.DLL is the soul and body of WSS 3.0. It comes historically from SharePoint ver 1.0 from the days before Microsoft.NET when web applications were developed using technologies like ISAPI and DCOM. For more information on OWSSRV and the WSS architecture have a look at the Overview of the SharePoint Team Services Architecture in MSDN.

Let’s now have a look at the public ISPRequest interface which is implemented by the SP.SPRequest COM object and is available for .NET via the SPRequest class. The list below shows only a fraction of the ~400 actions exposed by this COM object and demonstrates what sort of operations are done through the SPRequest class. As you can see this is pretty much everything in WSS:

[

    ComImport,

    Guid(“BDEADEBE-C265-11D0-BCED-00A0C90AB50F”),

    ComConversionLoss,

    TypeLibType((short)0x100),

    SuppressUnmanagedCodeSecurity,

    InterfaceType((short)1)

]

public interface ISPRequest

{

    …

    void GetFieldsSchemaXml(…);

    void GetViewsSchemaXml(…);

    …

    string CreateList(…);

    …

    void DeleteList(…);

    …

    void QueryUserInfo(…);       

    …

    string AddField(…);

    void RemoveField(…);

    void UpdateField(…);

    …

    void GetMetadataForUrl(…);

    void DeleteView(…);

    void UpdateView(…);

    void CreateView(…);

    …

    void AddOrUpdateItem(…);

    void DeleteItem(…);

    void GetAttachmentsInfo(…);

    …

    void CrossListQuery(…);

    void RenderViewAsHtml(…);

    …

    void UpdateUser(…);       

    …

    void BackupSite(…);

    …

    void CreateSite(…);

    void DeleteSite(…);

    void OpenSite(…);

    void SetSiteProps(…);

    void GetAllWebsOfSite(…);

    …

    void UpdateFileOrFolderProperties(…);

    void PutFile(…);

    …

    void GetFile(…);

    …

    void CheckOutFile(…);

    void CheckInFile(…);

    …

    void CreateWeb(…);

    void SaveWebAsTemplate(…);

    …

    void OpenWeb(…);

    string ApplyTheme(…);

    …

    int AddRoleDef(…);

    …

    void AddWebPart(…);

    …

    void CreateListViewPart(…);

    …

    void GetTimeZoneInfo(…);

    …

    void AddMeeting(…);

    …

    void GetSiteQuota(…);       

    …

    void RegisterEventReceiver(…);

    …

    void InvokeTimerJob(…);

    …

    void GetListContentTypes(…);

    …

    void AddWorkflowToListItem(…);

    …

    void SetGhostedFile(…);       

    …

    int AddNavigationNode(…);       

    …

}

To see the full list of methods exposed by SPRequest you can open up this class in Reflector. In order to do this you first need to get the Microsoft.SharePoint.Library assembly from the GAC. To get a physical copy of this file you can go Start -> Run -> type ‘C:\WINDOWS\ASSEMBLY\GAC_MSIL’  -> press ‘Enter’. This is a little trick that will open up the physical store of the GAC assemblies. And remember you must do this from Start -> Run – it will not work otherwise. So once you have it opened, navigate to Microsoft.SharePoint.Library\12.0.0.0__71e9bce111e9429c and copy the file to a place you can open it with Reflector.

[Updated 23 Jan]

ISPRequest is actually a COM interface and the public SPRequestInternalClass from the Microsoft.SharePoint.Library.dll is a wrapper to it. The ISAPI extension  /_vti_bin/owssvr.dll can be used to do various things as documented in the Windows SharePoint Services URL Protocol.

Even that the SPRequestInternalClass is public it is actually undocumented according to the Microsoft SharePoint Escalation Engineer Stefan Gossner. (See his comment below and the link to his blog Documented / Undocumented API – Why Should I care?).  Stefan also points out that any SharePoint database which is modified using direct access to this undocumented API will become an unsupported state as if you would have modified the database using direct SQL queries. So be warned!

[/Updated 23 Jan]

So now that we know that when we call the SharePoint .NET object model almost everything is done via unmanaged code the next question is what is the implication of this. Well firstly this means there are unmanaged objects being created all the time, and those objects cannot be cleaned up automatically by the .NET garbage collector. This is exactly why it is so important to call Dispose() or Close() on an SPSite when we have finished using it as we need to free those unmanaged resources. If we don’t do this they will cause unmanagedmemory leaks and will severely slow down the server. We need to call Close()/Dispose() with every object that implements the IDisposable interface and from the ‘common’ SharePoint objects this are both the SPSite and SPWeb class instances.

If you try to allocate too many SPRequest objects (create too many SPSites) without releasing them you will start getting medium severity trace messages in the ULS Log similar to the one below:

“Potentially excessive number of SPRequest objects (10) currently unreleased on thread 23. Ensure that this object or its parent (such as an SPWeb or SPSite) is being properly disposed. This object will not be automatically disposed.”

There are two registry keys you can use to control the ULS trace log related to allocating and disposing SPRequest objects. Firstly the warning threshold can be controlled via the LocalSPRequestWarnCount value in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions. You should however always try to have a minimal number of SPSite objects created at the same time and should always dispose them as soon as you’ve finished working with them. Secondly to get the stack trace of the not correctly disposed SPRequests (which is not enabled by default) you can set a value of 1 for SPRequestStackTrace  in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\HeapSettings registry key.  You can monitor the ULS trace messages in real time using the very light-weight SPTraceView utility. For detailed information and best practices on disposing SharePoint objects see the following MSDN article: Best Practices: Using Disposable Windows SharePoint Services Objects

Because creating too many unmanaged objects is a serious concern for a server application, Microsoft have implemented a couple of ways for those SPRequest objects to be cached and reused internally whenever this is possible. However this caching might cause additional problems such as: Reusing a cached version when you don’t really want this or attempting to reuse a cached version of the SPRequest on a thread different from the thread it has been created on. Before looking into those potential issues lets see how the caching of SPRequests works.

For managing the SPRequests SharePoint gets the help of two internal classes – SPRequestManager and SPRequestContext.  All SPRequest objects created in the managed world of Microsoft.NET are tracked and cached per process using an SPRequestManager object which singleton instance is saved in the HttpContext for web applications or in the ThreadContext of the SPFarm for non web applications (such as OWS Timer jobs, command line application such as custom STSADM commands or rich clients). This also means that this caching is per thread. The code below shows how the SPRequestManager is created and associated with the running thread:

private static SPRequestManager Instance

{

    get

    {

        SPRequestManager manager = null;

 

        if (HttpContext.Current == null)

        {

            manager = SPFarm.ThreadContext[typeof(SPRequestManager)] as SPRequestManager;

        }

        else

        {

            manager = HttpContext.Current.Items[typeof(SPRequestManager)] as SPRequestManager;

        }

 

        if (manager == null)

        {

            manager = new SPRequestManager ();

 

            if (HttpContext.Current == null)

            {

                SPFarm.ThreadContext[typeof(SPRequestManager)] = manager;

                return manager;

            }

 

            HttpContext.Current.Items[typeof(SPRequestManager)] = manager;

        }

 

        return manager;

    }

}

 Additionally to this the SharePoint Object Model may be also caching the opened SPSites when running in a web context. This caching is done with the help of the SPRequestContext class which single instance is stored in the HttpContext per server thread that processes the request.

internal static SPRequestContext GetCurrent(HttpContext context)

{

    if (context != null)

    {

        string str = “HttpHandlerSPRequestContext”;

 

        SPRequestContextreqContext = (SPRequestContext)context.Items[str];

 

        if(reqContext== null)

        {

            reqContext= new SPRequestContext(context);

            context.Items[str] = reqContext;

        }

 

        returnreqContext;

    }

 

    return null;

}

The site instances will be cached only if they have not been created with a farm admin account (when SPSite.m_bAdministratorOperationMode is false) . Please note that runing code using SPUtility.RunWithElevatedPrivilegies() method doesn’t mean your code will have farm administrator privileges. Whether a user is a farm administrator or not is determined by the SPFarm.CurrentUserIsAdministrator() method. An account will be a  farm admin if:

– It is the identity of the application pool used by the SharePoint Central Administration web application

– The account has been added to the SharePoint Administrators site group of the Central Administration web application

– The farm is being provisioned and the account is a local administrator or the Local System account

When you are using  SPUtility.RunWithElevatedPrivilegies() this will simply revert the current user identity to the identity of the current application pool, and if this identity is not a farm administrator then your SPSite _will_ be cached in the SPRequestContext. This is done when the SPSite object is created:

private void SPSiteConstructor(

    SPFarm farm, Guid applicationId, Guid contentDatabaseId, Guid siteId,

    SPUrlZone zone, Uri requestUri, string serverRelativeUrl, bool hostHeaderIsSiteName,

    Uri redirectUri, Pairing pairing, SPUserToken userToken)

{

    …

 

    if (SPSecurity.ImpersonatingSelf && (userToken == null))

    {

        // We will enter here if the site is created inside SPUtility.RunWithElevatedPrivilegies()

        this.m_bAdministratorOperationMode = SPSecurity.ProcessIdIsGlobalAdmin;

    }

 

    …

 

    if (!this.m_bAdministratorOperationMode)

    {

        // The site is cached here and may be reused when activating features

        SPRequestContext.RegisterSite(this);

    }

 

    …

}

The following diagram shows at a high level the relationship between the various objects working with SPRequests:

 

As you see one of  the classes I added on the above diagram is the SPFeatreManager class. This class can be actually called from the unmanaged world to activate features when you are provisioning a WSS site and this is when a cached SPSite may get reused. Here is what will happen every time you create a site or a web from a Web-based SharePoint application (web part, aspx page or SharePoint itself) if the WSS site definition you are using has some features to be activated:

– Before you can initiate a site creation first there must be an SPSite object created. This will be either done by the SharePoint UI code behind or by you directly creating the SPSite. If the user identity used to create the site is not a farm administrator then the SPSite will be cached by SiteId and by UserToken in the current SPRequestContext.

– You now initiate a new site creation for example by calling SPWebCollection.Add()

– The SharePoint Object Model calls the corresponding unmanaged function of the SP.SPRequest COM object to create the site. This is ISRequest.CreateWeb()

– Based on the WSS site definition to be used for the site, there may be features to be activated. The SP.SPRequest COM object makes calls back to .NET using the ISPFeatureManager COM visible interface of the Microsoft.SharePoint.SPFeatureManager managed class in order to activate those features. It possibly calls EnsureTemplateAssociatedWebFeaturesActivated(). This results in any existing SPSitescached in the same HttpRequest for the same user to be reused. The SPFeatureManager.EnsureSite() method will reuse the cached SPSite or create a new one if there is no cached (when not running in HttpContext).

Now if you are using custom site definitions and depending on how complex are they i.e. how many features do you have and what they are doing, in some cases because the cached SPSite will be reused, you may get some very weird errors particularly when trying to provision two sites at the same time. Some of those errors which I have seen at least a couple of times each include:

———————————————————————————————————————————-

System.InvalidOperationException : You cannot invalidate the SPRequest object while it’s in use.
Source: Microsoft.SharePoint; Help Link: 
Stack Trace:    at Microsoft.SharePoint.Library.SPRequest.ReleaseResources()
    at Microsoft.SharePoint.SPRequestManager.Release(SPRequest request)
    at Microsoft.SharePoint.SPWeb.Invalidate()
    at Microsoft.SharePoint.SPWeb.Close()
    at Microsoft.SharePoint.SPSite.Close()
    at Microsoft.SharePoint.SPSite.Dispose()

———————————————————————————————————————————-

System.Runtime.InteropServices.COMException : Attempted to make calls on more than one thread in single threaded mode. (Exception from HRESULT: 0x80010102 (RPC_E_ATTEMPTED_MULTITHREAD))
Stack Trace:    at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListsWithCallback(String bstrUrl, Guid foreignWebId, String bstrListInternalName, Int32 dwBaseType, Int32 dwBaseTypeAlt, Int32 dwServerTemplate, UInt32 dwGetListFlags, UInt32 dwListFilterFlags, Boolean bPrefetchMetaData, Boolean bSecurityTrimmed, Boolean bGetSecurityData, ISP2DSafeArrayWriter p2DWriter, Int32& plRecycleBinCount)
                       at Microsoft.SharePoint.Library.SPRequest.GetListsWithCallback(String bstrUrl, Guid foreignWebId, String bstrListInternalName, Int32 dwBaseType, Int32 dwBaseTypeAlt, Int32 dwServerTemplate, UInt32 dwGetListFlags, UInt32 dwListFilterFlags, Boolean bPrefetchMetaData, Boolean bSecurityTrimmed, Boolean bGetSecurityData, ISP2DSafeArrayWriter p2DWriter, Int32& plRecycleBinCount)

———————————————————————————————————————————-

ExeName: w3wp.exe, Machine: localhost, Severity: Unexpected, Product: ‘Windows SharePoint Services’, Category: ‘General’
ERROR: request not found in the TrackedRequests. We might be creating and closing webs on different threads. ThreadId = 24, Free call stack =    at Microsoft.SharePoint.SPRequestManager.Release(SPRequest request)
    at Microsoft.SharePoint.SPWeb.Invalidate()
    at Microsoft.SharePoint.SPWeb.Close()
    at Microsoft.SharePoint.SPContentType.CloseWebAsNecessary()
    at Microsoft.SharePoint.SPWeb.Close()
    at Microsoft.SharePoint.SPSite.Close()
    at Microsoft.SharePoint.SPSite.Dispose()

———————————————————————————————————————————-

All of the 3 errors suggest that a cached SPSite may have been reused from a different thread, when it actually shouldn’t have been. It is very difficult to say why those errors do occur but I believe it has to do with the SPRequestContext caching some SPSites which are being reused when activating features. One of the facts that supports this theory is that you don’t get those errors when you are creating the sites from a non HttpContext application such as a custom STSADM command for example. In that case there is no HttpRequest and no SPRequestContext class i.e. no SPSites are being cached and reused. One of the times when I was getting those errors and was trying to resolve the problem I found a couple of post in Internet from people having the same problem but there were no resolutions. Almost everyone experiencing those errors was saying that they don’t get the errors when their code was called by a console application (custom STSADM command).  I wasn’t able to resolve the issue and it left as a known issue in the product. If you have experienced the same problem and have managed to nail it down I would love to hear about your findings/resolution. To mitigate the risk of getting those errors I would suggest that you should:

– Avoid using custom WSS site definitions if they include custom feature receivers that contain complex code

– Consider provisioning your sites from a non HttpContext application whenever possible. OWSTimer jobs count as non HttpContext way of provisioning sites

Also remember that you may be getting those errors for a different reason that is completely unrelated to the SPRequestContext cache. For example if you really are trying to reuse SPWeb or SPSite on a different thread or if you are not disposing properly the SPSite/SPWeb.

And finally lets recap the story about the SPRequest. Almost all read/write content/metadata operations that you do using the .NET SharePoint object model use unmanaged class to do the job. This unmanaged class is called SP.SPRequest and is used from .NET via the SPRequest internal class. The class is implemented and exposed by the OWSSRV.DLL library and has almost 400 methods. Because the SPSite and SPWeb objects have a reference to unmanaged objects you must keep the number of SPSite/SPWeb objects that are active at the same time to a minimum and should dispose them as soon as you have finished using them. You can monitor the creation and disposal of SPRequest objects via the ULS log. In order to minimize the number of SPRequests used in a HttpContext application, SharePoint uses internal cache. It is possible your SPSite objects to be reused when you are creating a site or a web from a web application which site/web is based on a custom WSS site definition that contains custom feature receivers. In rare cases you may be getting errors because of this so if you are using custom WSS site definitions then keep the complexity of your custom feature receivers to a minimum. Consider creating your sites from a non HttpContext application such as an STSADM command, Windows Forms application or a OWSTimer job.

I hope you’ve enjoyed this not very short reading. My next post in the “Understanding SharePoint” series, which will not necessarily be my next post, will be about how asynchronous Event Receivers work in SharePoint and what traps people can get into.

The column name that you entered is already in use or reserved. Choose another name.

This is an annoying one. SharePoint will not allow you to create a site column from the UI (using the _layouts/fldnew.aspx page) if the name of the column is one of:

constructor, concat, join, pop, push, reverse, shift, slice, sort, splice or unshift

even if these site columns do not exist in the SharePoint site. If you try to create one of these fields you will get an error message saying that:

The column name that you entered is already in use or reserved. Choose another name.

Hmmm … Well after investigating this one it turned out to be a genuine bug caused by a JavaScript artifact. The way the fldnew.aspx page checks at the client side if a column already exists is by having all existing columns stored in a JavaScript Array called g_FieldName and then checking if the column that user tries to create is already in the array. This is done by the following code:

…

g_FieldName[“E-Mail”.toLowerCase()] = 1;

 

g_FieldName[“Type”.toLowerCase()] = 1;

…

function doesFieldNameConflict( strName )

{

      if(g_FieldName[strName.toLowerCase()])

            return true;

      else

            return true;

}

The problem with this code is that for some unknown to me reason which is probably related to the way the JavaScript VM is implemented, if you create a new Array called arr and then try to read arr[“shift”] for example, this will return the following string:

 

And this happens on both Internet Explorer and Firefox. And this will happen for all strings that are names of methods of the JavaScript Array object and this is case sensitive. Because SharePoint does the check in lower case, then only the methods that are entirely in lower case will be a problem and the list of SharePoint site columns you won’t be able to create from the UI given at the top is the list of all methods of the Array object that are fully lower case.

So how to get around this. Well first of all this is a bug that should be resolved by Microsoft. They could either store everything in upper case (no JavaScript methods are fully upper case) or can change the function to

function doesFieldNameConflict( strName )

{

      if(g_FieldName[strName.toLowerCase()] == 1)

            return true;

      else

            return true;

}

The problem with the first function is that g_FieldName[strName.toLowerCase()] will be evaluated to true for anything which is not 0, false, null or unknown. In the case of arr[“shift”] it will return the string shown on the screen-shot above, which will be evaluated to “true”. So explicitly testing for “1” will resolve the bug.

In the meantime if you need to create a site column with one of those names you could do it using code or features. And if you wonder how I discovered this issue – well I wanted to create a site column called “Shift” as in work shift which I consider as a totally legitimate use case.

Custom ‘Presence’ With SharePoint Person Fields

Not that you will want to do this often, but I discovered today that the “Presence” control displayed with every Person on a SharePoint site can be actually completely customized. Usually it looks something like this:

According to the MSDN documentation for NameCtrl:

A developer can create a custom control for displaying presence information if it has the same ProgID (Name.NameCtrl.1), uses the same method or property names, and provides the same functionality that is described in this reference. Functions can be created that handle the event in which the online status of a specified person changes.

What this means is that if you don’t have Office 2007 installed (or another version of the name.dll) then you can implement your own ActiveX control (in .NET or VB6 or C++) and name the COM class “Name.NameCtrl“. Then automatically this control will be called by SharePoint and will display whatever you want.

This could be handy for example when your customers don’t use Outlook as an email client or Office Communicator as an instant office messenger.