Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use database sharding with EF Core and C#"

I'm currently in the process of converting my 6 years old C# application to .NET Core v3 and EF Core (and also using Blazor). Most of it is working except for the Sharding part.
Our application creates a new database for each client. We use more or less this code for it: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-scale-use-entity-framework-applications-visual-studio
I'm now trying to convert it to EF Core, but get stuck at this part:

        // C'tor to deploy schema and migrations to a new shard
        protected internal TenantContext(string connectionString)
            : base(SetInitializerForConnection(connectionString))
        {
        }

        // Only static methods are allowed in calls into base class c'tors
        private static string SetInitializerForConnection(string connnectionString)
        {
            // We want existence checks so that the schema can get deployed
            Database.SetInitializer<TenantContext<T>>(new CreateDatabaseIfNotExists<TenantContext<T>>());
            return connnectionString;
        }

        // C'tor for data dependent routing. This call will open a validated connection routed to the proper
        // shard by the shard map manager. Note that the base class c'tor call will fail for an open connection
        // if migrations need to be done and SQL credentials are used. This is the reason for the 
        // separation of c'tors into the DDR case (this c'tor) and the internal c'tor for new shards.
        public TenantContext(ShardMap shardMap, T shardingKey, string connectionStr)
            : base(CreateDDRConnection(shardMap, shardingKey, connectionStr), true /* contextOwnsConnection */)
        {
        }

        // Only static methods are allowed in calls into base class c'tors
        private static DbConnection CreateDDRConnection(ShardMap shardMap, T shardingKey, string connectionStr)
        {
            // No initialization
            Database.SetInitializer<TenantContext<T>>(null);

            // Ask shard map to broker a validated connection for the given key
            var conn = shardMap.OpenConnectionForKey<T>(shardingKey, connectionStr, ConnectionOptions.Validate);
            return conn;
        }

The above code doesn't compile because the Database object doesn't exist in this way in EF Core. I assume I can simplify it using TenantContext.Database.EnsureCreated(); somewhere. But I can't figure out how to modify the methods, which to remove, which to change (and how).

Of course, I've been searching for an example using sharding and EF Core but couldn't find it. Does anybody here has done this before in EF Core and is willing the share?

I'm specifically looking for what to put in startup.cs and how to create a new sharding/database when I create a new client.

like image 211
Paul Meems Avatar asked May 06 '20 10:05

Paul Meems


People also ask

How do you use database sharding?

Range-based sharding is the simplest sharding method to implement. Every shard holds a different set of data but they all have the same schema as the original database. In this method, you just need to identify in which range your data falls, and then you can store the entry to the corresponding shard.

Can we do sharding in relational databases?

Sharding, also known as horizontal partitioning, is a popular scale-out approach for relational databases. Amazon Relational Database Service (Amazon RDS) is a managed relational database service that provides great features to make sharding easy to use in the cloud.

Can sharding be done on SQL?

Unfortunately, monolithic databases like Oracle, PostgreSQL, MySQL, and even newer distributed SQL databases like Amazon Aurora do not support automatic sharding. This means manual sharding at the application layer has to be performed if you want to continue to use these databases.

How do I shard a SQL Server database?

Divide the data store into horizontal partitions or shards. Each shard has the same schema, but holds its own distinct subset of the data. A shard is a data store in its own right (it can contain the data for many entities of different types), running on a server acting as a storage node.


1 Answers

In EF.Core just resolve the shard in OnConfiguring. EG

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    var con = GetTenantConnection(this.tenantName);

    optionsBuilder.UseSqlServer(con,o => o.UseRelationalNulls());

    base.OnConfiguring(optionsBuilder);
}

Note that if you have a service or factory that returns open DbConnections, then you'll need to Close()/Dispose() them in the DbContext.Dispose(). If you get a connection string or a closed connection then DbContext will take care of closing the connection.

ASP.NET Core best-practices probably call for injecting an ITenantConfiguration service or somesuch in your DbContext. But the pattern is the same. Just save the injected service instance to a DbContext field and use it in OnConfiguring.

like image 150
David Browne - Microsoft Avatar answered Oct 14 '22 00:10

David Browne - Microsoft