Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Saving a large graph with Entity Framework 5

I know that it is not recommended to use Entity Framework for bulk inserting since it adds them one by one. (Example 1, example 2.)

How about attaching one object that has a large graph underneath it, e.g. 100Ks of objects? - Specifically:

  • What are the (time/memory) performance considerations for using EF5 to do this?
  • Are there any best practices for storing large graphs?
  • Are relational databases even a good idea for storing large graphs?

A bit of background:

  • The graph is created using EF5 POCOs, then attached at the root and saved. The graph is always new data, never updates. The DB we use is SQL Server 2012.

  • The actual creation process can take tens of seconds for large cases (100Ks of objects), so if the attach/save process takes 10% of that time, it won't have a significant affect on the overall time performance.

like image 232
Danny Varod Avatar asked Oct 03 '22 17:10

Danny Varod


1 Answers

As no one have answered for some reason, even commonly known here EF expert, then I'll try to ask.

Recently, I've dealed with complex model with large amount of cycle referencing and numbers of relationsbetween entities. And large amount of data, yes, just as yours or bigger. I've tried many things and none were convenient for me, so I've ended with using Fastest Way of Inserting in Entity Framework. It helped me a lot a year ago and you haven't mention it in your question post somehow.


But if you haven't got many complex relations between entities and have large amount of completely new data to insert just one time, then the fastest way as you have already mentioned is SqlBulkCopy because it is extremely fast and simple in this case.
Using CodeFirst approach, for example, you could firstly create database structure and then simply insert your data.

Hope it helps! No guess if using NoSQL storage will be fast or convinent in your case as I've never used those solutions. May be experienced colleagues will add some about it.

like image 126
Fedor Avatar answered Oct 07 '22 18:10

Fedor