Looking for text file search algorithm for very large projects in C#

Question

I'm trying to write a search algorithm in C# for a very large project (+700MB) and, due to my lack of experience, my code runs for over 30 hours before it ends/finds error. I'm not asking for anyone to do it for me, but to point me to an algorithm so I may study, learn and implement it.

The project: I have a text file with many keys and values (like a load file), let's call it INI file. I also have a huge project with many files in many folders. For each of these INI keys I have to:

1) Search the whole project (not the INI file) to make sure every key is being loaded. Each value will be loaded, individually, with a function named LoadIniValue(). If a value is not called in one of these functions, I will have to eventually remove it from the INI file.

2) In this LoadIniValue() function you also have the container the value will be stored in, passed as a parameter. Using that parameter as the new search key, is it used anywhere (except in the constructors or destructors) in the project? If the parameter is not used (evaluated, read, etc,) I will have to eventually remove it from the INI file.

The goal of this project is to have a clean INI file with no unused parameters.

I truly hope this will be considered a valid question and someone will be able to provide a clue.

Thanks so much to this community for helping me learn. You guys are really awesome!

Answer 1

You need an indexing agent. Don't think you need to design it by yourself from scratch. I would use Lucene to index these files and use its APIs to get the output.