Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Contains on list is too slow, how to improve?

Tags:

c#

list

linq

I have a list of objects that I wish to reduce to only the ones that have properties contained within a separate list.

List1 is a list of simple strings.

List2 is a list of objects containing two string-properties; A and B.

All items where A and B are not present in List1, should be removed.

This process is very time-dependent and needs to be as quick as possible. Currently I have the following implementation;

var List1 = new List<String>() {"Around", "9000", "strings"}; //List of about 9000 strings
var List2 = databaseList.ToList(); //Around 2.5 million objects

var reducedList = new HashSet<Object>();            
foreach (var item in List2)
{
    if(List1.Contains(item.A) && List1.Contains(item.B))
    {
        reducedList.Add(item);
    }
}

This process takes around 7 seconds to complete, which for my current requirements is too slow.

I have tried running this using LINQ, but gives the same result, around 7 seconds.

var reducedList = List2.Where(r => List1.Contains(r.A)).Where(r => List1.Contains(r.B)).ToList();

Any suggestions to what I can do to improve this?

EDIT: I am unable to do this on the SQL side of things, since the 9000 strings that I need to compare against cannot be "translated" into and SQL-query, but will go above the allowed 2100 input parameters which are allowed in our SQL Server setup.

like image 690
Sander Avatar asked Dec 03 '15 09:12

Sander


1 Answers

I have not tried it but probably it would increase the performance.

var List1 = new List<String>() {"Around", "9000", "strings"};
var List2 = databaseList.ToList(); //Around 2.5 million objects

var reducedList = List2.RemoveAll(i => !List1.Contains(i.A) && !List1.Contains(i.B)).ToList();
like image 59
Mohit S Avatar answered Sep 28 '22 17:09

Mohit S