DocumentDB client 1.14 kills running process

Question

After upgrading DocumentDB client to version 1.14.0 (CosmosDB changes) my application terminates unexpectedly when running in Azure as cloud service. I have no problems running it locally against the latest version of the emulator. I also do not get any problems when running it locally against Azure CosmosDB instance. That seems to point to some issue with cloud service environment (downgrading to previous version 1.13.4 works).

Here are the errors from Event Log:

Faulting application name: Application.exe, version: 1.0.21.0, time stamp: 0x59196ef2
Faulting module name: DocumentDB.Spatial.Sql.dll, version: 1.14.8.3, time stamp: 0x59125a74
Exception code: 0xc000001d
Fault offset: 0x0000000000074753
Faulting process id: 0x1754
Faulting application start time: 0x01d2cd9ce28ddb4f
Faulting application path: C:\Resources\directory\08707c89190040c7b18ae72ed40addee.Host.LocalStoreDirectory\orleans-backend\1.0.21\Application.exe
Faulting module path: C:\Resources\directory\08707c89190040c7b18ae72ed40addee.Host.LocalStoreDirectory\orleans-backend\1.0.21\DocumentDB.Spatial.Sql.dll
Report Id: da991426-1eff-4d89-bf21-118198848fef
Faulting package full name: 
Faulting package-relative application ID: 

Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Runtime.InteropServices.SEHException
   at Microsoft.Azure.Documents.ServiceInteropWrapper.GetPartitionKeyRangesFromQuery(IntPtr, System.String, Boolean, Boolean, System.String[], UInt32[], UInt32, Microsoft.Azure.Documents.PartitionKind, IntPtr, UInt32, UInt32 ByRef)
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfoInternal(Microsoft.Azure.Documents.SqlQuerySpec, Microsoft.Azure.Documents.PartitionKeyDefinition, Boolean, Boolean)
   at Microsoft.Azure.Documents.Routing.PartitionRoutingHelper.GetProvidedPartitionKeyRanges(Microsoft.Azure.Documents.SqlQuerySpec, Boolean, Boolean, Boolean, Microsoft.Azure.Documents.PartitionKeyDefinition, Microsoft.Azure.Documents.Query.QueryPartitionProvider, System.String, Microsoft.Azure.Documents.Query.QueryInfo ByRef)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<TryGetTargetPartitionKeyRangeAsync>d__23.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<TryGetTargetPartitionKeyRangeAsync>d__23, Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<TryGetTargetPartitionKeyRangeAsync>d__23 ByRef)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.TryGetTargetPartitionKeyRangeAsync(Microsoft.Azure.Documents.DocumentServiceRequest, Microsoft.Azure.Documents.DocumentCollection, Microsoft.Azure.Documents.Query.QueryPartitionProvider, Microsoft.Azure.Documents.Routing.IRoutingMapProvider, Microsoft.Azure.Documents.Routing.Range`1<System.String>, System.Collections.Generic.List`1<Microsoft.Azure.Documents.Query.CompositeContinuationToken>)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<ExecuteOnceAsync>d__f.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<ExecuteOnceAsync>d__f, Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ExecuteOnceAsync>d__f ByRef)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.ExecuteOnceAsync(Microsoft.Azure.Documents.IDocumentClientRetryPolicy, System.Threading.CancellationToken)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<>c__DisplayClass2+<<ExecuteInternalAsync>b__1>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<>c__DisplayClass2+<<ExecuteInternalAsync>b__1>d__4, Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<<ExecuteInternalAsync>b__1>d__4 ByRef)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<>c__DisplayClass2.<ExecuteInternalAsync>b__1()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1+<>c__DisplayClass2+<<ExecuteAsync>b__0>d__4[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.BackoffRetryUtility`1+<>c__DisplayClass2+<<ExecuteAsync>b__0>d__4[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<<ExecuteAsync>b__0>d__4<System.__Canon> ByRef)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1+<>c__DisplayClass2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<ExecuteAsync>b__0()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1+<ExecuteRetry>d__1b[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.BackoffRetryUtility`1+<ExecuteRetry>d__1b[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ExecuteRetry>d__1b<System.__Canon> ByRef)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].ExecuteRetry(System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.Func`3<System.Exception,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Microsoft.Azure.Documents.ShouldRetryResult>>, System.Threading.CancellationToken, System.Action`1<System.Exception>)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1+<ExecuteAsync>d__a[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.BackoffRetryUtility`1+<ExecuteAsync>d__a[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ExecuteAsync>d__a<System.__Canon> ByRef)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].ExecuteAsync(System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, Microsoft.Azure.Documents.IRetryPolicy, System.Threading.CancellationToken, System.Action`1<System.Exception>)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<ExecuteInternalAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext+<ExecuteInternalAsync>d__8, Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ExecuteInternalAsync>d__8 ByRef)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.ExecuteInternalAsync(System.Threading.CancellationToken)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase+<ExecuteNextAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase+<ExecuteNextAsync>d__4, Microsoft.Azure.Documents.Client, Version=1.14.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ExecuteNextAsync>d__4 ByRef)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.ExecuteNextAsync(System.Threading.CancellationToken)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1+<ExecuteNextPrivateAsync>d__2d`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()

I've run into another related problem: when running DocumendDB client 1.13.4 against the latest local emulator I get the following exception:

Newtonsoft.Json.JsonReaderException
{"Unexpected character encountered while parsing value: ≻. Path '', line 0, position 0."}
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfoInternal(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Routing.PartitionRoutingHelper.GetProvidedPartitionKeyRanges(SqlQuerySpec querySpec, Boolean enableCrossPartitionQuery, Boolean parallelizeCrossPartitionQuery, Boolean isContinuationExpected, PartitionKeyDefinition partitionKeyDefinition, QueryPartitionProvider queryPartitionProvider, String clientApiVersion, QueryInfo& queryInfo)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.<TryGetTargetPartitionKeyRangeAsync>d__1e.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.<ExecuteOnceAsync>d__a.MoveNext()

Clearly there has been a breaking change in handling partition keys.

Answer 1

This is an issue with DocumentDB SDK:

To summarize, we identified a critical issue in the .NET SDK v1.14.0 release on x64 machines that dont support SSE4 instructions. We are working on a patch, but meanwhile please use https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/1.13.4 or earlier. We will post an update when a patch is available.

The issue has been fixed in 1.14.1

https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/1.14.1

Answer 2

I've encountered the same problem. For me it only happens when calling IDocumentQuery.ExecuteNextAsync.

Adding a new and fetching a single document works fine.

I reported it here as you mentioned, Jakub: https://github.com/Azure/azure-documentdb-dotnet/issues/259