Limiting the number of concurrent tasks running

Question

So I come accross this issue with go a lot. Let's say I have a text file with 100,000 lines of text. Now I wanna save all these lines to a db. So I would do something like this:

file, _ := iotuil.ReadFile("file.txt")

fileLines := strings.Split(string(file), "\n")

Now I would loop over all the lines in the file:

for _, l := range fileLines{
  saveToDB(l)
}

Now I wanna run this saveToDB func concurrently:

var wg sync.WaitGroup

for _, l := range fileLines{
  wg.Add(1)
  go saveToDB(l, &wg)
}

wg.Wait()

I don't know if this is a problem or not but that would run 100,000 concurrent functions. Is there any way of saying hey run 100 concurrent functions wait for all of those to finish then run 100 more.

for i, _ := range fileLine {
  for t = 0; t < 100; t++{
    wg.Add(1)
    go saveToDB(fileLine[i], &wg)
  }
  wg.Wait()
}

Do I need to do something like that or is there a cleaner way to go about this? Or is me running the 100,000 concurrent tasks not an issue?

Answer 1

I think the best approach for this would be to keep a pool of worker goroutines, dispatch the work for them in channels, and then close the channel so they would exit.

something like this:

// create a channel for work "tasks"
ch := make(chan string)

wg := sync.WaitGroup{}

// start the workers
for t := 0; t < 100; t++{
    wg.Add(1)
    go saveToDB(ch, &wg)
}

// push the lines to the queue channel for processing
for _, line := range fileline {
    ch <- line
}

// this will cause the workers to stop and exit their receive loop
close(ch)

// make sure they all exit
wg.Wait()

and then the saveFunction looks like this:

func saveToDB(ch chan string, wg *sync.WaitGroup) {
    // cnosume a line
    for line := range ch {
        // do work
        actuallySaveToDB(line)
    }
    // we've exited the loop when the dispatcher closed the channel, 
    // so now we can just signal the workGroup we're done
    wg.Done()
}