3D reconstruction / General Science and Electronics / Forums

Forums

4hv.org :: Forums :: General Science and Electronics

« Previous topic | Next topic »

3D reconstruction

1 2 next

Move Thread

LAN_403

Arkin

Thu Mar 04 2010, 01:58AM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

I was reading this for some background for my stereo vision project. However, i don't think i am understanding this right. As i see it, the article is simple saying that 2 different images are put on to the same plane, Basically virtually "turning" the cameras so they are aligned, both pointing straight.

Is this correct? Does this mean if i start with 2 CCD cameras already coplanar, image rectification is not needed?

Carbon_Rod

Thu Mar 04 2010, 02:17AM

Registered Member #65 Joined: Thu Feb 09 2006, 06:43AM
Location:
Posts: 1155

Generally, the unique optical distortion and projection is calibrated using a known pattern.

The camera's intrinsic and extrinsic calibration is done with a chessboard:

OpenCV sample video and stereo vision source code:

Cheers,

Arkin

Thu Mar 04 2010, 04:29AM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

How necessary is the calibration really? It seems that if humans don't see the distortion, the main factor becomes resolution of the photo. Do modern cameras automatically compensate for distortion?

Anyway, i went ahead and did a simple test of image correlation using the SAD method. I used the fairly large chunks of 50x50 pixels. The top picture is the left picture and the bottom is the right. Since everything moves to the left, i only search to the left of the start. I also did 1D search as it was (and should) be aligned.

The top white square is the start patch. It searches for the best match in the bottom. White squares are what it searched, and the smaller squares are the found matching square.

This is working quite decently, but when i try to use smaller patches for higher resolution, it starts making alot more false matches. With so much white (walls and carpets), there is alot of possibilities.

In addition, the top left ceramic piece with the 2 right angle pieces caused problems. It started with the right right angle, and matched with the left one. How can i basically factor in pixel location? This is my code so far (yes, it's ugly, but only a test).

int bx = 700;
            int by = 100;
            ......
            int diff = 0;
            int min = 1000000000;
            int minx = 0;
            int miny = 0;
            for (int z = 0; z < bx; z+=50) {
                    for (int i = 0; i < 50; i++) {
                        for (int d = 0; d < 50; d++) {
                            Color co = new Color(fleft.getRGB(bx+i, by+d));
                            Color coo = new Color(fright.getRGB(i+z, d+by));

                            int r = coo.getRed() - co.getRed();
                            int g = coo.getGreen() - co.getGreen();
                            int b = coo.getBlue() - co.getBlue();

                            diff += Math.abs(r+g+b);

                        }
                    }
                    fright.getGraphics().drawRect(z, by, 50, 50);
                    
                    if(diff < min){
                        minx = z;
                        miny = by;
                        min = diff;
                    }
                    diff = 0;
            }

          
            fright.getGraphics().drawRect(minx+3,miny+3,44,44);
            fright.getGraphics().drawRect(minx+5,miny+5,40,40);

bx is start x patch
by is start y patch
min is the current "record"
minx and miny are current record position

Carbon_Rod

Thu Mar 04 2010, 05:19AM

Registered Member #65 Joined: Thu Feb 09 2006, 06:43AM
Location:
Posts: 1155

Arkin wrote ...

Do modern cameras automatically compensate for distortion?

Generally No... as most people won't notice unless an optical illusion is disclosed.

Cheers,

Bjørn

Thu Mar 04 2010, 01:00PM

Registered Member #27 Joined: Fri Feb 03 2006, 02:20AM
Location: Hyperborea
Posts: 2058

There is no easy solution to what you are trying to do, even humans from time to time walk on doors and fall down stairs.

One way to improve on your algorithm is to only search for blocks with sufficient complexity so they have less chance of getting false positives. Your current code averages the error from each colour channel, you might consider adding up the errors to reduce the amount of false positiveson coloured objects.

You might considering a recursive method starting with large blocks, then split them up if you find a good match. The smaller blocks you just search for in the local area, to better define your object and location.

There is no limit to the time you can spend on this, after 10 years you will still find cases that will cause your code to fail.

Arkin

Thu Mar 04 2010, 01:31PM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

So do i have to eliminate all the perspective distortion, so that road would become straight? I don't really see how that is a problem. Sure, to a human, they look different. But with computers, there is no thinking, but definte numbers, and numbers don't lie. I feel i am missing something here :)

I realize my code will never be perfect, especially since im not really good at programming either. However, if i get the code well enough, i just reasonably be able to eliminate strange blocks and/or spots on the disparity map. For example, i could eliminate all blocks smaller than x pixels squared. This is fine, as a agv doesn't really need to know if something small is ahead.

As for finding the complexity of the area, could i just find the maximum difference in colors in even smaller patches in the larger patch (1 pixel blocks are sure to return pixel anomolaies, i think)? What if i just avoid all colors that cover more than X% of the image? So many questions =p

I need to limit the image processing as much as i can, as i hope to eventually run this on an onboard computer with significantly less processing power.

Bjørn

Thu Mar 04 2010, 02:13PM

Registered Member #27 Joined: Fri Feb 03 2006, 02:20AM
Location: Hyperborea
Posts: 2058

You do not have to remove distortions, if you do you will get more accuracy. How much depends on the quality of your optics. The distortion will warp the shape of objects as they move around the frame. That will affect how well your code perform if the distortions are significant.

Finding the "complexity" of a block can range from finding the difference between the darkest and the brightest pixel in the block to something very complicated. For example a very high contrast block might be unsuitable if it contains a lot of high frequencies since it would be very sensitive to distiortions and small changes in parallax inside the block.

The simplest way of reducing the computing power required is to reduce the resolution of the image. If you halve the amount of resolution, you quarter the amount of data. You can also speed up the search dramatically by skipping blocks early when it is clear that it is a bad fit. In the same way you don't have to look at every position to find the best fit because moving the block one pixel over is not going to make it suddenly go from a bad fit to a good fit. So you could find the approximate position first, then refine it.

If this is going to run on something reasonable like an ARM chip you should be able to get good performance.

Arkin

Fri Mar 05 2010, 09:41PM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

Ok, so i integrated a minimum and maximum range, and it's working alot better.

For RGB comparison, right now it uses the absolute value of differences. Would ratios of R:G:B work as well, removing factor of shadows a bit more?

Arkin

Wed Mar 10 2010, 11:53PM

Registered Member #2140 Joined: Tue May 26 2009, 09:16PM
Location:
Posts: 53

I have been messing with different variations of the program, and i am running into many problems. The first is that pixels often match with totally unrelated pixels. My solution:
-A bigger resolution in effect takes the environment into consideration that a pixel is in, when using the SAD algorithm. Using bigger blocks is less resolution, but should be higher accuracy. If this is true, a big block could be calculated, and narrowed down, confined to the previouse matching block, searching only that block for matches. I am not sure, but i believe this will also give a speed increase. EDIT: This is basically what Bjorn said.

Also, when using different viewpoints, the closer objects block farther objects line of sight. This causes the matching algorithm to find something else. I have no idea how to avoid this. What might work?

This is an example of what the program currently does. This is a 10mp set of photos, at 1 pixel resolution.

This obviously isn't nearly close to what it actually is. However, you can notice that it seems to be workking better on closer objects. Also, on the glue bottle, you can make out a curve on which the right side looks blacker. This is the area covered by the closer object (a ceramic coconut pot btw) in one of the images. Obviously the text shouldn't even be visible, but should be fixed by a progressive scan.

This is a graph showing the resolution of the disparity map vs the time taken to finish. Right now, it is a bit slower (made a whole new class to make it easier for progressive scanning), but have an idea to make it multi-threaded.

EDIT: The originial picture doesn't have much reflections (just wood), this just brings it out alot.

Bjørn

Thu Mar 11 2010, 03:49AM

Registered Member #27 Joined: Fri Feb 03 2006, 02:20AM
Location: Hyperborea
Posts: 2058

It would be interesting if you could post your test pictures, then other people could have a try as well.

Also, when using different viewpoints, the closer objects block farther objects line of sight. This causes the matching algorithm to find something else. I have no idea how to avoid this. What might work?

You need to calculate a confidence value for your block and discard the ones that score too low. For example if your best fit is almost identical in score to other blocks far away you have low confidence in the correctness since one of the other ones might be the right one. Also if the best fit is not very good compared to the average of all blocks you should consider giving it a lower score.

1 2 next

Moderator(s): Chris Russell, Noelle, Alex, Tesladownunder, Dave Marshall, Dave Billington, Bjørn, Steve Conner, Wolfram, Kizmo, Mads Barnkob