Algorithm for smoothing edges of an open 3D mesh

Question

I have a 3D mesh which represents a surface with some rough boundaries which I would like to smooth:

I am using a half edge data structure for storing the geometry so I can easily iterate over the boundary edges, vertices and faces. I can also quite easily determine whether a given pair of edges is a convex/concave using a dot and cross product.

What would be the best approach for smoothing the edges out, so they form a continuous, curvy line, rather then the sharp pattern seen in the pictures?

Answer 1

1. compute angle between two neighboring faces

   I call it ada as abs delta angle. If it is bigger then threshold it means this point is edge. You can compute it as max of all angles between all edge lines. In 2D it looks like this:

   

   in 3D mesh there is more then 2 lines per point so you have to check all combinations and select the biggest one

   ada=max(abs(acos(n(i).n(j)))
   

   where n(i),n(j) are normal vectors of neighboring faces where i != j

2. identify problematic zones

   so find points where ada > threshold and create a list of these points

3. filter this list

   if this point is too far from any other (distance>threshold) then remove it from list to preserve geometric shape

4. smooth points

   you have to tweak this step to match your needs I would do this:

   find a group of points in the list which are close together and apply some averaging geometric or numeric on them for example:

   pnt(i)=0.5*pnt(i)+0.25*pnt(i-1)+0.25*pnt(i+1)
   

   this can be applied repetitive

   

   blue and red dots are original points, green dots are smoothed points