How to get online HTML-tables down locally automatically using C#

Question

Okay to keep it short:

I have some different websites with tables containing information that i would like to have "local" to query.

I have searched for possibilities and i have some ideas myself.

1. In Excel i found a function where i can navigate to a webpage and copy the data from a table. Problem is this only happens once. The data in the tables will be updated every week so i need Excel to be updated automatically every time i open my program.

2. I could use a crawler, but then i would have to write a different solution for every table and find a way to save it.

I have a MySQL database right know containing many of the information i need in my program, so if any solution requires a database that totally acceptable.

About my program: Will be written in C#, first as a local program and then to a MVC project. Suggestions for both projects are very welcome and if you need more information just commented, and i will try and describe it some more. :)

EDIT! 1

I am very sorry that I am didn't from the beginning enlightened you about which tables i were talking about, but when I started this question I still needed to find all the tables. Now, however, I have taken a few of them out to show you guys how different types of tables I have to work with. About the project, it should tell you that the program I have planned to make is only for private use and not to sell. I am not aware about the rules on crawling on public sites, so thats why i keep it private.

Table 2 Table 3

As you can see, it's a lot of soccer data showing in very different ways, so i need to know which way is best for me to gather the data because i believe it to be easier to design med database with this knowledge.

Answer 1

Anders, Excel has an inbuilt way to fetch the data and you have to do that once. Next time you have to simply refresh the query. Please see this link.

html parsing of cricinfo scorecards

FOLLOWUP

Try look at this page: soccernet.espn.go.com/stats/_/league/eng.1/… There are 3 tables but it dosent seem that excel detect them. :( – Anders Gerner 7 mins ago

In this particular website, if you view the source, you will see that the table doesn't have an ID. All the three table have the same class "tablehead". If you want, in the workbook open event, loop though all the tables and extract the data. Your job is made easier as all the 3 tables have the same class.

Alternatively, you can also do this

In Excel, click on File | Open, and in the Dialog box, directly type the URL which you mentioned below. You will notice that Excel stacks up the data neatly :)

You can in fact write a small macro/code which opens a temp workbook and then opens the URL and then simply extract the tables from the temp workbook into your workbook. My estimate is that on a good internet connection, the entire process should not take more than 15 secs to complete