Treating nils in sort function

Question

I don't know how to handle nils my sort function gets.

When I have this checking in it, table.sort crashes after some calls.

if a == nil then
    return false
elseif b == nil then
    return true
end

With this error:invalid order function for sorting. But according to the documentation, sort function should return false, if a goes after b. True otherwise.

If I remove remove that code, it of course crashes of indexing nils.

Answer 1

This has little or nothing to do with nil values in the table. The error message is generated if the comparison function itself is invalid. From the documentation for table.sort:

If comp is given, then it must be a function that receives two table elements, and returns true when the first is less than the second (so that not comp(a[i+1],a[i]) will be true after the sort).

In other words, comp(a,b) must imply not comp(b,a). If this relation does not hold, then the error "invalid order function for sorting" will likely raised. (Note that it might not be raised in all cases.)

In order to be more helpful, we really need to see the whole function passed to table.sort.

Answer 2

To put all nil values at the beginning of the array:

  function mycomp(a,b)
    if a == nil and b == nil then
      return false
    end
    if a == nil then
      return true
    end
    if b == nil then
      return false
    end
    return a < b
  end

To put all nil values at the end of the array:

function mycomp(a,b)
  if a == nil and b == nil then
    return false
  end
  if a == nil then
    return false
  end
  if b == nil then
    return true
  end
  return a < b
end