Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to normalize disparity data in iOS?

Tags:

ios

swift

depth

In WWDC session "Image Editing with Depth" they mentioned few times normalizedDisparity and normalizedDisparityImage:

"The basic idea is that we're going to map our normalized disparity values into values between 0 and 1"

"So once you know the min and max you can normalize the depth or disparity between 0 and 1."

I tried to first get the disparit image like this:

let disparityImage = depthImage.applyingFilter(
    "CIDepthToDisparity", withInputParameters: nil)

Then I tried to get depthDataMap and do normalization but it didn't work. I'm I on the right track? would be appreciate some hint on what to do.

Edit:

This is my test code, sorry for the quality. I get the min and max then I try to loop over the data to normalize it (let normalizedPoint = (point - min) / (max - min))

let depthDataMap = depthData!.depthDataMap
let width = CVPixelBufferGetWidth(depthDataMap) //768 on an iPhone 7+
let height = CVPixelBufferGetHeight(depthDataMap) //576 on an iPhone 7+
CVPixelBufferLockBaseAddress(depthDataMap, CVPixelBufferLockFlags(rawValue: 0))
// Convert the base address to a safe pointer of the appropriate type
let floatBuffer = unsafeBitCast(CVPixelBufferGetBaseAddress(depthDataMap), 
    to: UnsafeMutablePointer<Float32>.self)
var min = floatBuffer[0]
var max = floatBuffer[0]
for x in 0..<width{
    for y in 0..<height{
        let distanceAtXYPoint = floatBuffer[Int(x * y)]
        if(distanceAtXYPoint < min){
            min = distanceAtXYPoint
        }
        if(distanceAtXYPoint > max){
            max = distanceAtXYPoint
        }
    }
}

What I expected is the the data will reflect the disparity where the user clicked on the image but it didn't match. The code to find the disparity where the user clicked is here:

// Apply the filter with the sampleRect from the user’s tap. Don’t forget to clamp!
let minMaxImage = normalized?.clampingToExtent().applyingFilter(
    "CIAreaMinMaxRed", withInputParameters: 
        [kCIInputExtentKey : CIVector(cgRect:rect2)])
// A four-byte buffer to store a single pixel value
var pixel = [UInt8](repeating: 0, count: 4)
// Render the image to a 1x1 rect. Be sure to use a nil color space.
context.render(minMaxImage!, toBitmap: &pixel, rowBytes: 4,
    bounds: CGRect(x:0, y:0, width:1, height:1), 
    format:  kCIFormatRGBA8, colorSpace: nil)
// The max is stored in the green channel. Min is in the red.
let disparity = Float(pixel[1]) / 255.0
like image 378
Jimmy Avatar asked Nov 05 '17 00:11

Jimmy


1 Answers

Will's answer above is very good, but it can be improved as follows. I'm using it with depth data from a photo, it's possible that if the depth data doesn't follow 16-bits, as mentioned above, it won't work. Haven't found such a photo yet. I'm surprised there isn't a filter to handle this in Core Image.

extension CVPixelBuffer {

func normalize() {
    CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
    
    let width = CVPixelBufferGetWidthOfPlane(self, 0)
    let height = CVPixelBufferGetHeightOfPlane(self, 0)
    let count = width * height

    let pixelBufferBase = unsafeBitCast(CVPixelBufferGetBaseAddressOfPlane(self, 0), to: UnsafeMutablePointer<Float>.self)
    let depthCopyBuffer = UnsafeMutableBufferPointer<Float>(start: pixelBufferBase, count: count)

    let maxValue = vDSP.maximum(depthCopyBuffer)
    let minValue = vDSP.minimum(depthCopyBuffer)
    let range = maxValue - minValue
    let negMinValue = -minValue

    let subtractVector = vDSP.add(negMinValue, depthCopyBuffer)
    let normalizedDisparity = vDSP.divide(subtractVector, range)
    pixelBufferBase.initialize(from: normalizedDisparity, count: count)

    CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
}

}

like image 104
RPSM Avatar answered Oct 03 '22 11:10

RPSM