Extract upwards pointing lane lines

Question

We try to detect lane lines on a running track (and with this information the upcoming direction).

We currently use the following (simplified) steps:

1. Binary: transform the input with a binary threshold

2. Cropped: crop the region of interest (currently just the lower half of the image)

3. Canny: detect edges and group them with HoughLinesP

4. Upward Lines & Closing: only keep lines with an extreme slope - ignore horizontal lines

5. Result: find connected components and fit quadratic function through each line

This works in general (examples: straight.png, left.png) but has issues if for example a horizontal connection line wasn't removed (example: problems.png - bottom right & bottom left). In such a case the two lines and the connection get interpreted as one connected component.

As our point of view can tilt a lot from left to right (camera mounted on a running person) it is quite hard to define a slope threshold for upwards pointing lines.

Is there a better way to get rid of non upwards pointing track lines? As the current solution with canny, hough transform and slope filtering is not optimal for curves and sometimes doesn't work at all (as described above).

Would it be possible to get from the cropped image straight to the separated lines through morphology operations? Similar to this example. I know we have no strict horizontal lines which makes any fitting kernel a lot more complex (I assume).

Currently, we try to use perspective transformations to get a bird view on the track. This should help to distinguish between horizontal and upward pointing lines.

Another small issue is too short lines which result in inaccurate approximated quadratic functions (most left line in problems.png and straight.png). This might be easily solvable (by requiring a minimal pixel count for a component to be counted as track line) and should not be part of this question.

EDIT (answer questions from vlad_tepesch)

What is with the curves? should they be part of your lane models or not?

We clearly want to detect curves of the running track - as this is needed for the direction estimation. But we want to ignore (remove) the horizontal connection curve on the bottom of the problems.png example.

4./6. rectification - camera distortion

I will take another look at rectification, until now I postponed it, as I thought it isn't that important.

7.3 how many pixel if overlap between segments is ignored (group pixel count)

Just draw all lines from a group and check the non zero pixel count?

10. lanes normally are not quadratic - look int clothoids instead - this may overkill so may be use a 3rd order polynomial instead

I get your point. Clothoids are out of scope at the moment, but I will keep them in mind. How would you go with the direction estimation if we use more complex fitting? Currently we just take the first coefficient of the 2nd oder polynomial to estimate the bending direction (and the degree of curvature to differentiate between curves and straight parts).

optional more advanced

Good point, we already thought about averaging results over multiple frames. I also keep that proposal in the back of my mind.

Straight

Left

Problems

Answer 1

It seems that your real problem is the filtering and the connecting step. because you filter out a lot of horizontal components, then for connecting of the unfiltered nearly vertical lines only sub optimal segments are left.

What is with the curves? should they be part of your lane models or not? At first I assume not, but it only has some influence during the grouping stage.

I would suggest the following steps

1. crop
2. binary (why wasting processing time on binarization of image parts that getting cropped)
3. canny (maybe try a vertical Sobel to get only horizontal edges)
4. consider rectification to correct the camera distortion
5. HoughLinesP
6. if you decided against a rectification of input image then rectify at least the coordinates of the output lines
   From a processing time point of view this may be the more desirable option - however the hough line detector will be produce less optimal results depending on the strength of your cameras distortion
7. connect lines regardless of their directions
   1. i would first try the following
   2. for each segment calculate a line (point, direction)
   3. find lines with similar parameters and check the distance of the segments
   4. if close enough, group them into a line group
   5. at the end you should have a few line groups with some properties you may calculate to estimate the quality
      1. how many lines/segments belong to the group
      2. how many pixel belong to the group (sum of length of all line segments)
      3. how many pixel if overlap between segments is ignored (group pixel count)
      4. the leftmost/rightmost pixel on the line group (some kind of group line segment)
      5. fill ratio (group line length/group pixel count)
   6. after that grouping the line segments and their assignment to the groups are not required any more, but i would keep the calculated properties.
8. if the bends should be part of the lane then i would try look for groups that continue the found groups at the end and merge them.
   This should prevent merging of crossing lines.
9. filter the groups to your needs
10. fitting.
    1. lanes normally are not quadratic
    2. look int clothoids instead
    3. this may overkill so may be use a 3rd order polynomial instead

optional more advanced

1. keep all the groups at the end
2. do filtering only at the output level
3. in the next frame match the old groups against the new one and do some tracking and plausibility checks of your models.