Automation: Push data from local CSVs to Kusto db tables

Question

I have different data downloaded from a web page and saved as .csv files on my local directory. I would like to push that data to the tables that I've created on Kusto Explorer and ideally I want this process to be scheduled to take place everyday automatically because that data from the website is updated daily.

For example:

Here I have 3 .csv files:

1.csv
2.csv
3.csv

and here I have 3 tables created on Kusto to store each csv data respectively:

Table_1
Table_2
Table_3

The question is: Is it possible to automate and schedule the data push/load from local CSVs to Kusto? The data push/load type is Replace for now but might need to be Append in the future.

I was recommended Flow (https://preview.flow.microsoft.com/en-us/) to check if any similar templates were set up already. However, I was not able to find any and am new to these tools so I need help from someone who knows about this. Thanks so much in advance. Please let me know if you need any further information.

Answer 1

if you want to control the schedule:

• you can use any of the ingestion client libraries to automate this: https://learn.microsoft.com/en-us/azure/kusto/api/

• or, if you prefer simply invoking a readymade utility, there are also LightIngest, and Kusto Ingestion Tools (KIT)

an alternative approach would be:

1. have the CSV files you generate uploaded to cloud storage once they're created.

2. setup a data connection for the storage container they're uploaded to, so that any file that gets uploaded to it is automatically ingested into your cluster: https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid