Large List

Best Practices for LARGE SharePoint Lists and Documents Libraries

SharePoint lists and document libraries are not designed to handle large numbers of items, in fact as soon as you add more than a few thousand you will see a warning within the List Settings Page that says:

This list or library contains a large number of items. Learn about managing a large list or library and ensuring that items display quickly.

As part of my product suite I consistently have lists with hundreds of thousands of items in them and have learned many valuable lessons in making these kind of numbers workable.

Hopefully some of what i learned can help you.

1. First off do not use the DataView webpart (from the SharePoint designer) or any xml based rendering webpart as they first grab all the items in the list and then apply the XLST to render and you will face both memory and huge performance problems. The regular list webpart works fine even for large datasets, but you will not be able to get total list counts regardless.

2. If you are planning on sorting or filtering by any columns be sure to add them to the lists indexed fields in the list settings page.

3. SharePoint is a database driven application and the list data is thus stored in database tables. The way it stores them is very denormalized as each columns data is stored in a single row, which means if you have 2 list items with 3 columns you get at least 6 rows of actual sql data. This adds up very fast and will quicky cause both disk and index fragmentation. Always ensure that your logical disks are defragmented once a week and use the SQL Management studio to create an index maintenence plan for all your databases.

4. For Developers using the Object model:

a. Getting counts: never reference the list item collections count property as that will trigger a complete retrieval of all the items in the list and possibly throw memory errors. If you need the list count use list.count instead of list.items.count

b. Item Access: never access the list item collection by index position as it will re-retrieve each item when accessed.

for (int i=0;i< list.items.count;i++) // bad

SPListItem li = list.items[i]; //bad

instead use the standard enumeration model to walk though your collection when needed

foreach (SPListItem li in list.items) //good

d. Paging through large numbers of list items: use the SPQuery and the RowLimit property to page through your larger list collections

SPQuery query = new SPQuery()
query.RowLimit = 10000;
do
{ 

SPListItemCollection myItems = list.GetItems(query);

for (SPListItem li in myItems)
{
//do something here
}
query.ListItemCollectionPosition = myItems.ListItemCollectionPosition;

} while (query.ListItemCollectionPosition != null);

e. Deleting large numbers of list items (new): use SPWeb.ProcessBatchData whenever possible. Taking above advice here is my idea of the best model for item deletion.

SPList lst = SPContext.Current.Web["MYLIST"];
SPQuery query = new SPQuery();
query.ViewFields = "";
query.IncludePermissions = false;
query.RowLimit = 20000;

// first get all the id's to be deleted
ArrayList al = new ArrayList();
do
{
SPListItemCollection myItems = lst.GetItems(query);

foreach (SPListItem item in myItems) // note use enumeration
{
if (item.ID == skipId) continue;
al.Add(item.ID.ToString());

}
query.ListItemCollectionPosition = myItems.ListItemCollectionPosition;
} while (query.ListItemCollectionPosition != null);

StringBuilder sbDelete = new StringBuilder();
sbDelete.Append("");
string listguid = lst.ID.ToString();
int bcount = 0;

for (int i = 0; i < al.Count; i++)
{
if (bcount > 1000)
{
sbDelete.Append("");
lst.ParentWeb.ProcessBatchData(sbDelete.ToString());
sbDelete = new StringBuilder();
sbDelete.Append("");
bcount = 0;

}
bcount++;
sbDelete.Append("");
sbDelete.Append("" + listguid + "");
sbDelete.Append("" + al[i].ToString() + "");
sbDelete.Append("Delete");
sbDelete.Append("");
}

sbDelete.Append("");
lst.ParentWeb.ProcessBatchData(sbDelete.ToString());

I hope this helps. If you have any further advice please comment on this post and i will incorporate it for everyone. THANKS!

Joel's list of more performance related information

 also see http://go.microsoft.com/fwlink/?LinkId=95450&clcid=0x409  for an earlier white paper on the subject

Custom Rss feed webpart in Sharepoint 2010

Here are the steps and the code snippet for creating a custom webpart to display items or news via rss feed.

1. Open the VS and add a new blank solution.
2. Now in your Solution say “RssFeedPackage” add a new item then choose add a webpart. Lets name it as RSSFeedWebPart.
Please note : You can directly create a webpart solution and do not necessarily have to add it as an item. I did this because I wanted to add some more features in the same solution.
3. Next, you need to add some logic to the RSSWebPart WebPart. Edit the RSSWebPart.cs class to include the code below. I have added a custom property to receive the Rss feed url for the webpart.

Code goes like this :

using System;

using System.ComponentModel;

using System.Runtime.InteropServices;

using System.Web.UI;

using System.Web.UI.WebControls;

using System.Web.UI.WebControls.WebParts;

using Microsoft.SharePoint;

using Microsoft.SharePoint.WebControls;

namespace RssFeedPackage

{

[ToolboxItemAttribute(false)]

public class RSSFeedWebPart: WebPart

{

[WebBrowsable(true)] [Personalizable(PersonalizationScope.Shared)]

public string RSSUrl

{

get; set;

}

protected override void RenderContents(HtmlTextWriter writer)

{

RSSFeed feed = new RSSFeed(RSSUrl);

HyperLink newLink = new HyperLink();

foreach (RSSItem singleRssItem in feed)

{

newLink.Text = singleRssItem.Title;

newLink.NavigateUrl = singleRssItem.Href;

newLink.Target = “rssSite”;

newLink.RenderControl(writer);

writer.WriteBreak();

}

base.RenderContents(writer);

}

}

}

//Code for RSSFeed class

internal class RSSFeed : List

{

internal RSSFeed(string RssURL)

{

try

{

XmlDocument rssDoc = new XmlDocument();

XmlTextReader xRead = new XmlTextReader(RssURL);

rssDoc.Load(xRead);

XmlNodeList xNodes = rssDoc.SelectNodes(“./rss/channel/item”);

foreach (XmlNode xNode in xNodes)

{

this.Add(new RSSItem(xNode)) ;

}

}

catch (Exception)

{

this.Add(new RSSItem());

}

}

}

//Code for RSSItem

internal class RSSItem

{

public string Title { get; internal set; }

public string Href { get; internal set; }

internal RSSItem()

{

Title = “Feed not available at this time” ;

Href = “~” ;

}

internal RSSItem(XmlNode xNode)

{

Title = xNode.SelectSingleNode(“./title”).InnerText;

Href = xNode.SelectSingleNode(“./link”).InnerText;

}

}

4. Now build and deploy the webpart.

5. Finally, Insert the webpart in your webpart page and specify the Rssurl in Edit this WebPart -> RssUrl box.