Depth Perception with 2 Images / Computer Science / Forums

Forums

4hv.org :: Forums :: Computer Science

« Previous topic | Next topic »

Depth Perception with 2 Images

1 2 next

Move Thread

LAN_403

Arkin

Thu Feb 25 2010, 12:07AM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

I am trying to write a program to correlate 2 images from a stereoscopic set of cameras at a known distance and angle, etc. However, i have a question regarding the correlation of objects in the image.

My thought on how to approach this is to have an image, run an edge detect, find which points on each image correlate (i.e, of the same object) and find the total amount these points moved, and calculate the depth. My problem is the correlation.

Image:

The red lines are from one Image, and the White are from the other image.

In gimp, i
1) resized the photo to 800x600 (want it to be fast enough for real time, eventually)
2) converter the image to gray scale
3) ran a Sobel edge detect at 2
4) converter to a binary image

How should i relate x with y? Should i use the surround point's relative/absolute position (e.g, point x also has a point at x+2 and x-4)? Corner Detection? Something else?

If I used relative position, how would i know which points correlate to which points? Should the points be contiguose or separate?.

Any help would be appreciated.

Carbon_Rod

Thu Feb 25 2010, 12:29AM

Registered Member #65 Joined: Thu Feb 09 2006, 06:43AM
Location:
Posts: 1155

The Intel OpenCV lib has most of these algorithms:

...it also can handle the intrinsic properties of the camera.

Cheers,

Arkin

Thu Feb 25 2010, 02:25AM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

Thanks! Ile need to see if i can get that to work with java.

Steve Conner

Thu Feb 25 2010, 09:49AM

Registered Member #30 Joined: Fri Feb 03 2006, 10:52AM
Location: Glasgow, Scotland
Posts: 6706

The textbook way of correlating images is just to multiply (the luminance value of) every pixel in one image by every pixel in the other. You don't make any assumptions about what objects they contain, and you end up with a third image that has bright spots expressing the amount and location of the correlation. This is of course very computationally intensive, although the FFT or wavelets can help. There are examples in "The Scientist and Engineer's Guide To Digital Signal Processing", which was available as a free e-book last time I checked.

You're trying to save computing power by taking an ad-hoc shortcut: you're using edge detection as if it were object detection. But edge detection is not object detection, and that's why you can't decide which points belong to which objects.

Maybe you can perform the correlation in the way I described above, but after edge detecting the images and reducing them to two colours. Then you can use AND instead of multiply, and crank out 32 or 64 pixels at a time depending on your processor.

hboy007

Thu Feb 25 2010, 01:11PM

Registered Member #1667 Joined: Sat Aug 30 2008, 09:57PM
Location:
Posts: 374

Arkin,

that is a very interesting subject which I'm working on, too. However, there are many different approaches that differ in performance. The easiest thing that comes to my mind is (after preprocessing, of course)

1) be sure the images are at the same scale (prompt user or do a rough matching analysis of the stereoscopic image pair to eliminate the scale parameter)

2) find blocks of high contrast or divide one of the images into blocks

3) evaluate the blocks including their immediate neighbours, eliminate blocks with linear degeneracy (a line may have contrast but you can shift a line segment along a straight line with no extremum of the correlation function )

4) now that you have blocks that can be considered unique (locally), you can do a multiscale analysis of the correlation function, so perform a power-of-2 downscaling (1/2, 1/4, 1/8 will be fine) of the blocks. The maximum for the correlation of a block of image A within a range of blocked image content in image B sets the search boundaries for the next higher resolution scale, which improves performance.

As to the search location, you can set up a 2d grid of arrays in which you store the resulting translation vectors for the valid correlations (threshold) you found and do some statistics. The arithmetic mean of the (delta x, delta y) vector components can be used as a preset but you will have to the standard deviation into account - which means: a grid point representing a zone where occlusion by a foreground object occurs or where many different depth values are present (imagine a line of pillars with the camera viewport aligned almost in parallel) has a higher variation of possible translation vectors so the preset predictablilty is weakened and you have to correlate with a larger area just to make sure.

You end up with a field of translation vectors which you can use to recover the depth information.

quoting Steve who cuts the long story short:
"and crank out 32 or 64 pixels at a time depending on your processor."

Bjørn

Thu Feb 25 2010, 04:15PM

Registered Member #27 Joined: Fri Feb 03 2006, 02:20AM
Location: Hyperborea
Posts: 2058

Fairly modern CPUs (since 2000) can do 128 bits at a time.

Here is the link to the "The Scientist and Engineer's Guide To Digital Signal Processing" files :

Carbon_Rod

Fri Feb 26 2010, 09:08AM

Registered Member #65 Joined: Thu Feb 09 2006, 06:43AM
Location:
Posts: 1155

Arkin wrote ...

Thanks! Ile need to see if i can get that to work with java.

IIRC Java VMs support imported native binary execution in a special area.
Google should yield a wrapper template someplace...

Since v2, the lib supports OpenMP video-card parallel processing etc.

ROS packages include the OpenCV lib along with other stuff:

or a commercial solution:

Note the solid aluminum mount for the cameras....

Cheers,

Arkin

Sat Feb 27 2010, 01:56PM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

Ok, so i am trying to get OpenCV to work in a program using OpenCV.jar. Netbeans compiles it succesfully, but when i start the program, and try to use it, the program freezes and gives me this:

!!! required library not found : no OpenCV in java.library.path
Verify that the java.library.path property is correctly set and 'libcxcore.so', 'libcv.so', 'libcvaux.so', 'libml.so', and 'libhighgui.so' are placed (or linked) in one of your system shared libraries folder

Exception in thread "Thread-3" java.lang.UnsatisfiedLinkError: hypermedia.video.OpenCV.capture(III)V
	at hypermedia.video.OpenCV.capture(Native Method)
	at hypermedia.video.OpenCV.capture(OpenCV.java:945)
	at touchcam3.ImageManager.getImage(ImageManager.java:165)
	at touchcam3.TouchCam3View.setOriginalImage(TouchCam3View.java:484)
	at touchcam3.TouchCam3View.load(TouchCam3View.java:295)
	at touchcam3.TouchCam3View.run(TouchCam3View.java:84)
	at java.lang.Thread.run(Thread.java:619)

Where do i put the *.so files? I tried putting links in the folder with the OpenCV.jar (Netbeans' dist/lib folder), and in the same directory.

EDIT: nvm, turns out there is an additional so file not mentioned :)

Carbon_Rod

Sun Feb 28 2010, 10:53AM

Registered Member #65 Joined: Thu Feb 09 2006, 06:43AM
Location:
Posts: 1155

In *nix the ".so" are more efficient if you have multiple processes working on similar tasks with the same lib.
Yet static linking ".a" libs with SSE support turned off ensures the massive program binary will function on most systems.

Note, Java generally has some fundamental issues with timing...
...even if the perceived performance can sometimes near C++ with Boost lib.

Kind of lucky those ".so" lib files should keep the performance related issues negligible.

Arkin

Sun Feb 28 2010, 05:29PM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

What do you mean by
"IIRC Java VMs support imported native binary execution in a special area."

1 2 next

Moderator(s): Chris Russell, Noelle, Alex, Tesladownunder, Dave Marshall, Dave Billington, Bjørn, Steve Conner, Wolfram, Kizmo, Mads Barnkob