Handling Error 429 in DocumentDB

I am working on one of these IoT projects at the moment and one of my requirements is to stream events from devices into DocumentDB.  My simplified architecture looks like this. 

 

Event Hub to Worker Role to DocumentDB

 

 

Each collection in DocumentDB is provisioned with a certain amount of throughput (RU == Request Units).  If you exceed that amount of requests you will receive back a 429 error “Too Many Request“. To get around this you can

 

1. Move your collection to a more performant level (S1, S2, S3)
2. Implement throttling retries
3. Tune the requests to need less RUs
4. Implement Partitions

#1 makes sense because if you are hitting DocumentDB that hard then you need to get the right “SKU” to tackle the job (but what if S3 is not enough)
#2 means capture the fact you hit this error and retry (I’ll show an example)
#3 in an IoT scenario could be taken to mean “Slow down the sensors”, but also includes indexing strategy (change to lazy indexing and/or excluding paths)
#4 IMHO the best and most logical option. (allows you to scale out.  Brilliant)

 

Scenario

My scenario is that I want to take the cheapest option so even though I think #4 is the right option, it will cost me money for each collection.  I want to take a look at #2.  I want to show you what I think is the long way around and then show you the “Auto retry” option.

 

The Verbose Route

 

        public async Task SaveToDocDb(string uri, string key, dynamic jsonDocToSave)
        {
 
            using (var client = new DocumentClient(new Uri(uri), key))
            {
                var queryDone = false;
                while (!queryDone)
                {
                    try
                    {
                        await client.CreateDocumentAsync(coll.SelfLink, jsonDocToSave);
                        queryDone = true;
                    }
                    catch (DocumentClientException documentClientException)
                    {
                        var statusCode = (int)documentClientException.StatusCode;
                        if (statusCode == 429) 
                            Thread.Sleep(documentClientException.RetryAfter);
                        //add other error codes to trap here e.g. 503 - Service Unavailable
						else
                            throw;
                    }
                    catch (AggregateException aggregateException) when (aggregateException is DocumentClientException )
                    {
                            var statusCode = (int)aggregateException.StatusCode;
                            if (statusCode == 429)
                                Thread.Sleep(aggregateException.RetryAfter);
							//add other error codes to trap here e.g. 503 - Service Unavailable
                            else
                                throw;
                        }
                    }
                }
            }
        }

 

 

The above is a very common pattern TRY…CATCH, get the error number and take action.  I wanted something less verbose and would have automatic retry logic that made sense. 

 

The Less Verbose Route

 

To do it I had to add another package to my project.  In the Nuget Package Manager Console

Install-Package Microsoft.Azure.DocumentDB.TransientFaultHandling

 

**
This is not a DocumentDB team library and it is unclear as to whether it is still being maintained.  What is clear is that the DocumentDB team will be bringing this retry logic into the SDK natively.  This will mean a lighter more consistent experience and no reliance on an external library.
**

Now add some using statements to the project

using Microsoft.Azure.Documents.Client.TransientFaultHandling;
using Microsoft.Azure.Documents.Client.TransientFaultHandling.Strategies;
using Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling;

And now instead of instantiating a regular DocumentClient in your application you can do this.

 

private IReliableReadWriteDocumentClient CreateClient(string uri, string key)
{
ConnectionPolicy policy = new ConnectionPolicy()
{
ConnectionMode = ConnectionMode.Direct,
ConnectionProtocol = Protocol.Tcp
};
var documentClient = new DocumentClient(new Uri(uri), key,policy);
var documentRetryStrategy = new DocumentDbRetryStrategy(RetryStrategy.DefaultExponential) { FastFirstRetry = true };
return documentClient.AsReliable(documentRetryStrategy);
} 

 

Summary

You may never hit the upper ends of the Request Units in your collection but in an IoT scenario like this or doing large scans over a huge collection of documents  you may hit this error and you need to know how to deal with it.  This article has provided you with two ways to handle this need.  Enjoy

 

My thanks go to Ryan Crawcour of the DocumentDB team for proof reading and sanity checking

Deleting Multiple Documents from Azure DocumentDB

I have been using DocumentDB a lot recently and wanted to share with you something that is harder to do than it should be.  When I say harder I mean you have to type more to get what seems a really easy thing to do.  I will also tell you how to do this more efficiently but because of cost sensitivity I couldn’t do it that way.

Scenario

I have a collection of documents.  These documents are being streamed into DocumentDB from a worker role which is reading from Azure Event Hubs.  As you can imagine I get a lot of documents in a relatively short space of time.  The size of a DocumentDB collection is 10GB.  I only want to use one collection.  My situation is that really I only need to keep two days worth of data in the collection at any one time.  My requirements therefore are

• Maintain only one collection
• Retain only two days worth of data
• Remove documents on a schedule.

For point #3 I am using a .net StopWatch object and that is really simple.  Having only one collection is also very simple so really it comes down to

How do I delete a bunch of documents from DocumentDB?

First Attempt

//setup
string databaseId = ConfigurationManager.AppSettings["DatabaseId"];
string collectionId = ConfigurationManager.AppSettings["CollectionId"];
string endpointUrl = ConfigurationManager.AppSettings["EndPointUrl"];
string authorizationKey = ConfigurationManager.AppSettings["AuthorizationKey"];
//connection policy
ConnectionPolicy policy = new ConnectionPolicy()
{
ConnectionMode = ConnectionMode.Direct,
ConnectionProtocol = Protocol.Tcp
};
//Build our selection criteria
var sqlquery = "SELECT * FROM c WHERE " + ToUnixTime(DateTime.Now).ToString() + " - c.time > 172800";
//Get our client
DocumentClient client = new DocumentClient(new Uri(endpointUrl), authorizationKey, policy);
//Database
Database database = client.CreateDatabaseQuery().Where(db => db.Id == databaseId).ToArray().FirstOrDefault();

//Get a reference to the collection
DocumentCollection coll = client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == collectionId).ToArray().FirstOrDefault();
//Issue our query against the collection
var results = client.CreateDocumentQuery<Document>(coll.DocumentsLink, sqlquery).AsEnumerable();
Console.WriteLine("Deleting Documents");
//How many documents do we have to delete
Console.WriteLine("Count of docs to delete = {0}", results.Count().ToString());
//Enumerate the collection
foreach (var item in results)
{
// Console.WriteLine("Deleting");
client.DeleteDocumentAsync(item.SelfLink);
}
//How many documents are still left
var postquery = client.CreateDocumentQuery<Document>(coll.DocumentsLink, sqlquery).AsEnumerable();
Console.WriteLine("Count of docs remaining = {0}", postquery.Count().ToString());
Console.ReadLine();

You may be expecting the result of the second count to be 0.  Unless you have 100 documents or less as the result of the first query then you are going to be disappointed.  We enumerate through the result of our first query getting a reference to each document and deleting it.  Seem fine?  The problem is that DocumentDB only returns 100 documents to us at a time and we didn’t go back and ask for more.  The solution is to execute our query and tell DocumentDB to re-execute the query if it has more results.  You can see a visual example of this when you use Query Explorer in the portal.  Down the bottom, under your query after execution you will find something like this.

The solution.

Here is the code that asks if there are more results to be had and if there are then can we go get the next batch

//setup
string databaseId = ConfigurationManager.AppSettings["DatabaseId"];
string collectionId = ConfigurationManager.AppSettings["CollectionId"];
string endpointUrl = ConfigurationManager.AppSettings["EndPointUrl"];
string authorizationKey = ConfigurationManager.AppSettings["AuthorizationKey"];
//Connection Policy
ConnectionPolicy policy = new ConnectionPolicy()
{
ConnectionMode = ConnectionMode.Direct,
ConnectionProtocol = Protocol.Tcp
};
//build our selection criteria
var sqlquery = "SELECT * FROM c WHERE " + ToUnixTime(DateTime.Now).ToString() + " - c.time > 172800";
//client
DocumentClient client = new DocumentClient(new Uri(endpointUrl), authorizationKey, policy);
//database
Database database = client.CreateDatabaseQuery().Where(db => db.Id == databaseId).ToArray().FirstOrDefault();

//Get a reference to the collection
DocumentCollection coll =
client.CreateDocumentCollectionQuery(database.SelfLink)
.Where(c => c.Id == collectionId)
.ToArray()
.FirstOrDefault();
//First execution of the query
var results = client.CreateDocumentQuery<Document>(coll.DocumentsLink, sqlquery).AsDocumentQuery();

Console.WriteLine("Deleting Documents");
//While there are more results
while (results.HasMoreResults)
{
Console.WriteLine("Has more...");
//enumerate and delete the documents in this batch
foreach (Document doc in await results.ExecuteNextAsync())
{
await client.DeleteDocumentAsync(doc.SelfLink);
}
}
//second count should now be 0
var postquery = client.CreateDocumentQuery<Document>(coll.DocumentsLink, sqlquery).AsEnumerable();
Console.WriteLine("Count of docs remaining = {0}", postquery.Count().ToString());

The key is this statement

var results = client.CreateDocumentQuery<Document>(coll.DocumentsLink, sqlquery).AsDocumentQuery();

using AsDocumentQuery allows us to know if we have more results.

The Easy Way

The easy way and most definitely the proper way to do this is to use Partitions in DocumentDB.  A Partition is essentially a collection.  There are different types of partition but for this example I would have used a range partition over time.  When I wanted to delete documents I would have just simply dropped a partition.  My partitions would have been based on dates.  I would always have had 2 full partitions (full meaning closed for data) and one partition (current) that was filling with data

In the example above collection #1 and #2 would be closed for data as we are filling collection #3.  Once collection #3 is full then we drop collection #1, add collection #4 and make that the one that is accepting data.

Conclusion

This is simple to do when you know how but it does seem like a long winded approach.  I would like to see something a little less verbose.