Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Bulk insert to ElasticSearch with NEST

I try to add 100k products to elasticsearch, but when i try i get: {"Validation Failed: 1: no requests added;"}

My code:

        var Node = new Uri("......");
        var ConnectionPool = new SniffingConnectionPool(new[] { Node });
        var Config = new ConnectionConfiguration(ConnectionPool)
                    .SniffOnConnectionFault(false)
                    .SniffOnStartup(false)
                    .SniffLifeSpan(TimeSpan.FromMinutes(10));
        var Client = new ElasticsearchClient(Config);

        var AllProducts = Product.GetProducts();
        var SPl = AllProducts.Split(100); // Split into 100 collections/requests

        var COll = new List<ElasticsearchResponse<DynamicDictionary>>();

        foreach (var I in SPl)
        {
            var Descriptor = new BulkDescriptor();

            foreach (var Product in I)
            {
                Descriptor.Index<Product>(op => op.Document(Product));
            }

            COll.Add(Client.Bulk(Descriptor));
        }

AllProducts contains a list of this object:

public class Product
{
 public int AffiliateNr { get; set; }
 public string AffiliateProductId { get; set; }
 public int BrandNr { get; set; }
 public string Currency { get; set; }
 public string IndexId { get; set; }
 public decimal Price { get; set; }
 public long ProductNr { get; set; }
 public string Title { get; set; }
}

So,

  1. Where can i set the name of the index?
  2. Why did i get, Validation Failed: 1: no requests added;?
  3. IndexId is my id for the product. How do i tell Elasticsearch to use this id? Or must i specify a id?
like image 939
mrcode Avatar asked Jun 04 '15 11:06

mrcode


1 Answers

Refering to your previous problem, you can use IndexMany for indexing the data. Now as per your question in comment, you can specify the id which elastic search will use. see the below example.

  ElasticType(IdProperty = "<fieldName>")]
    public class ClassName
    {

if you dont want to specify any Id to elastic search, create a dummy field dummyId(nullable) and put it in "IdProperty". Elastic search will auto assign the value to it if it null.

Edit : From 2.3 onwards, Its

[ElasticsearchType(IdProperty = "<fieldName>")]
like image 150
Shivang MIttal Avatar answered Oct 13 '22 00:10

Shivang MIttal