Azure Webjob HTTP 409 Conflict errors

Question

I had a C# WebJob that was working nicely with the alpha WebJob api. I just updated it to the beta release, and after fixing connection strings and namespaces I get a HTTP 409 (Conflict) error when the JobHost tries to connect.

Here is the call stack:

Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.Protocol.TableOperationHttpResponseParsers.TableOperationPreProcess(Microsoft.WindowsAzure.Storage.Table.TableResult result, Microsoft.WindowsAzure.Storage.Table.TableOperation operation, System.Net.HttpWebResponse resp, System.Exception ex)   Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.InsertImpl.AnonymousMethod__2(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, System.Net.HttpWebResponse resp, System.Exception ex, Microsoft.WindowsAzure.Storage.OperationContext ctx) Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync<Microsoft.WindowsAzure.Storage.Table.TableResult>(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, Microsoft.WindowsAzure.Storage.RetryPolicies.IRetryPolicy policy, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.Execute(Microsoft.WindowsAzure.Storage.Table.CloudTableClient client, Microsoft.WindowsAzure.Storage.Table.CloudTable table, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.CloudTable.Execute(Microsoft.WindowsAzure.Storage.Table.TableOperation operation, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Storage.SdkCloudStorageAccount.Table.GetOrInsert<Microsoft.Azure.Jobs.Host.Runners.HostEntity>(Microsoft.Azure.Jobs.Host.Runners.HostEntity entity) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Runners.HostTable.GetOrCreateHostId(string hostName)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHostContext.JobHostContext(string dashboardConnectionString, string storageConnectionString, string serviceBusConnectionString, Microsoft.Azure.Jobs.ITypeLocator typeLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)   Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.GetHostContext(Microsoft.Azure.Jobs.ITypeLocator typesLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)  Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(System.IServiceProvider serviceProvider) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(Microsoft.Azure.Jobs.JobHostConfiguration configuration) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost()    Unknown

Exception text:

[Microsoft.WindowsAzure.Storage.StorageException] {"The remote server returned an error: (409) Conflict."} Microsoft.WindowsAzure.Storage.StorageException

packages.config:

package id="Microsoft.Azure.Jobs" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.Core" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.ServiceBus" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Bcl" version="1.1.7" targetFramework="net45" package id="Microsoft.Bcl.Build" version="1.0.14" targetFramework="net45" package id="Microsoft.Data.Edm" version="5.6.0" targetFramework="net45"
package id="Microsoft.Data.OData" version="5.6.0" targetFramework="net45" package id="Microsoft.Data.Services.Client" version="5.6.0" targetFramework="net45" package id="Microsoft.Net.Http" version="2.2.19" targetFramework="net45"
package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.3" targetFramework="net45" package id="Newtonsoft.Json" version="6.0.4" targetFramework="net45" package id="System.Spatial" version="5.6.0" targetFramework="net45" package id="WindowsAzure.MobileServices" version="1.3.0-alpha4" targetFramework="net45" package id="WindowsAzure.ServiceBus" version="2.4.2.0" targetFramework="net45" package id="WindowsAzure.Storage" version="4.2.0" targetFramework="net45"

Any ideas?

Answer 1

Changing the jobRecurrenceFrequency in webjob-publish-settings.json to "Hour" worked for my https://error404.atomseo.com project!

I had the same problem and it turned out that publish process failed because I set it to recur every 10 minutes while the app was meant to run in free tier. As MS describes here:

https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-deploy-webjobs/

one can deploy with all frequencies other than those defined in minutes.

Answer 2

I did some more research and could see that this can happen if there are concurrent requests to create queue/table with same name. Azure seems rejecting those requests with custom 409 http code which is meant for applications.

To reproduce even in the latest Azure SDK, just have a webjob to process queues and start the JobHost only after having many requests in queue. In the Job function Get the blob as TextWriter and write into it. Makes sure the blob 'logs/webjob1' is not present when the JobHost starts to see the issue.

public static void ProcessQueueMessage([QueueTrigger("testqueue1")] string inputText,
       [Blob("logs/webjob1")]TextWriter writer)
        {
            writer.WriteLine(inputText);
        }

I just had 4 items in the queue when I started JobHost and was able to see the issue from Azure. Used free hosting.

Answer 3

I was getting the same error code

2018-08-18T23:06:02.2102822Z ##[error]Conflict
2018-08-18T23:06:02.2119417Z ##[error]Unable to retrieve connection details for Azure App Service : KaktusWatch. Status Code: 409 (Conflict)
2018-08-18T23:06:02.2150200Z ##[section]Finishing: Deploy Azure App Service

In my case the webjob-publish-settings.json was obsolete.

I had to fill in interval and jobRecurrenceFrequency settings parameters.