NXTCam image capture

[gloomyandy]

Hi,
If you have an NXTCam you probably know that it is not possible to gain direct access to the video images. Well with the V3 version you can replace the firmware... So with a few tweaks and some leJOS/Java magic...
http://www.youtube.com/watch?v=9ppNCzDb7lU

Andy

PS It is really rather fun being able to build your own "kitchen" rover, and to try and pilot it with limited sensor/video data and no direct view/access. Makes you realize how good the Mars rover crew are...

[bbagnall]

That's really cool. How did you go about obtaining and modifying the firmware code?

[gloomyandy]

Hi Brian,
It's all open source and available from links on the Mindsensors site. The NXTCam code is based on an open source project call AVRCam (I actually used parts of the AVRCam viewer program, which is written in Java). With the V3 sensor Mindsensors have made it easy to update the AVR microcontroller on the board, and at the same time they released the source and tools.

The only closed source part now is the code that links the sensor with the NXT. Mindsensors provide two versions of the firmware for this (the NXTCam actually has three processors on it!), the default which provides the standard interface (similar to the V2 sensor), and a "generic" version that supports a simple read/write model. I use the generic version. It is a pity that this is not open source becauses it is actually rather awkward to use and this contributes a fair bit to the slow speed of the frame grab (but it will always be slow unless the NXT has much faster i2c).

It's a pretty cool sensor to play around with, and it's great that they have made it easier to do this sort of thing...

Andy

[gbh]

Hi Andy,

Regarding to the AVRCam open source project, i downloaded the source code, it is in C language. Means I have to write it in Java on my own?

How to write a firmware and upload it to NXTCam v3? is it in java?

Thanks

[gloomyandy]

The NXTCam firmware is always in Java. The details of the process are available from the mindsensors site:
http://www.mindsensors.com/index.php?mo ... AGE_id=129

You have to use the standard Avr toolchain which is basically a cross compiler. None of this is in Java. As I said before doing this is a pretty complex process and if you are not familiar with working with embedded devices at this level then you may want to find some other way of doing things.

[gbh]

Hi Andy,
Thanks for your reply. It's really quite complicated.

Is that any other possible way to transfer images captured by NXTCam to PC via bluetooth? I'm using lejos NXT API (NXTCam) but cannot find any method in it that can capture images.

Cause I saw the O&A section in mindsensors site mentioned that images from NXTCam cannot be transfer to NXT brick.

Thanks

[gloomyandy]

Hi,
There is no way with the standard NXTCam firmware of transferring images from the camera directly to the NXT. I guess you could hook up the camera via USB to a PC and then transfer the image to the NXT via Bluetooth, but that would require the camera to be attached to the PC, and would require you to write the transfer software...

Sorry

Andy

[gbh]

Hi Andy

If I wish to write the firmware, is it i need to start with AVR Studio 4 with AVR toolchain to view the open source AVR source code?

And where should i write the firmware in java? eclipse?

Thanks. I'm really appreciate for your help.

[gloomyandy]

Hi,
The details of how to go about writing your own firmware for the NXTCam V3 are provided on the Mindsensors web site (they also have a forum). The firmware is a mixture of C and AVR assembler (not Java).

The modified firmware I created is available here:
http://nxtcam.svn.sourceforge.net/viewv ... loomyandy/

You may want to take a look and see if you are happy working at this level before you flash or modify your NXTCam. Just to be clear it is possible to end up with a non functioning camera by doing this. I had to resort to using a hardware debug tool a couple of times to get mine working again...

Good luck...

Andy

[gbh]

Thanks for the quick reply and resources!!!
Thanks a lot.

[burti]

Just a question which comes to my mind:
What about compressing the image before sending it? (maybe huffmann-code (http://en.wikipedia.org/wiki/Huffman_coding) or something else (http://en.wikipedia.org/wiki/Image_compression))
Is it possible/imaginable (considering memory and cpu)?
I mean: can you use the resources which are wasted with object detection?

To sum it up:
You have 352x288=101,376[pixel].
If you use 8 bit/pixel, then you have an amount of 101,376B = 792KB.
If you use 16 bit/pixel, then you have an amount of 1,584 KB (which you can forget, right?).
You have a transfer rate of 125Kbps which are 15.625KB/s at HIGHSPEED mode.
Then you would have a transfer time of 50s? Why is it practically faster with 30 seconds?!
If yould half or quarter the size of the image to 396KB relatively 198KB, then you could transfer the image faster (maybe in 15 or 8 seconds).
Are my calcs wrong?

What do you think about that approach?
(It's not meant as a task for you but for me

Michael

[gloomyandy]

Hi Michael,
Just to clear a few things up. This project did not use 352x288 resolution or the high speed i2c interface. It used a lower resolution (176x144) and the standard 9.6Kbps i2c interface. My other, connect 4 project uses the higher resolution and the faster 125Kbps i2c interface.

But back to your questions. The bottom line is I'm not sure if compression would help or not. But things are a little more complex than your numbers...

1. The image format used by the device is not a simple RGB pixel format. Rather it uses a Bayer-Pattern image so the first line of pixels has alternating red and green values, while the next has alternating green and blue values and so on.
2. The NXTCam has only 4 bits per red green and blue value, which are packed into a byte.
3. A single "line" when grabbed from the image sensor actually contains two scan lines (one with RGRGRG and one with GBGBGBGB etc.). So a full 352x288 pixel image, actually consists of 144 lines each of which contains 352 bytes (because the 704 color values are actually packed 2 per byte). which gives a total image size of 144*352 bytes or approx 50Kb

Ok so now we get to the really complex bit! At 125Kbs this amount of data should only take 144*352*9/125000 seconds (the 9 is because over i2c each byte requires 9 bits), which is 3.64 seconds. So how come a frame takes over 20seconds to download? Firstly it is worth understanding the route that the data takes from the image sensor to the NXT...
1. The ATMega grabs the pixels with some very tricky synchronous code, there is no frame buffer chip. There is only sufficient memory in the ATMega to hold a single line of data (352 bytes long).
2. The AtMega transfers the data via an asynchronous UART interface to a PIC microcontroller. This link runs at 115Kbps
3. The PIC transfers the data via i2c to the NXT, using high speed mode this transfer runs at 125Kbps.
But things are not that simple. The generic PIC code supplied by Mindsensors only has a 75byte buffer available for transferring data from the NXTCam, so each line has to be read in sections (in my case I use 60bytes at a time), thus the 352byte line requires 6 transactions. Each transaction requires multiple i2c operations, you have to clear the state, issue the read command, then wait for data, the read the data. All in all this takes time, the minimum time period to transfer a single line of data is 352*9/125000 = 0.0253, but each new line is arriving every 0.0009 seconds (I have the camera setup to generate 7.5 frames per second). So there is no way to send the line data before the next line is available. This means that when reading a full frame if we simply read line 1 then line 2 etc. we will have to wait for the line to become available. On average this will mean a wait of 1/7.5/2 (0.066) seconds for every line. So in this case the total time to read a frame becomes:
time for data + wait time = 3.64 + 144*0.066 = 3.64 + 9.6 = 13.24
But in the real world we then have to add in the i2c transaction delays plus the time taken for the NXTCam to respond and send the data. Which is why we end up with a 20seconds plus time for a full frame. One way to reduce some of latency is to use an interleave when grabbing the lines in an attempt to reduce the per line latency and this is possible but it is tricky to get right.

So going back to the idea of using compression. It might help, but as you can see the actual data transmission time is only a small part of the overall cost. In addition there would be the additional memory requirement on the ATMega and the required processing.

Sorry about the long winded response. I hope it made some sort of sense. I happened to have a lot of the numbers available because I had been trying to work out what is going on and I thought they may be of interest to you and others...

Andy

[burti]

Thanks for your detailed reply (I don't mind reading much).
I see now why compression does not make so much sense.
I thought the transfer rate of 125Kbps combined with a "big" image size would be "the bottleneck" but I was wrong.
Thanks for that insight!

However I wonder how detection of objects can work with so little ressources and with so many delays and latencies

Michael

[gloomyandy]

Depends what you mean by detection of objects... In the case of the native NXTCam object detection that work is all done in the ATMega so many of the delays do not apply. In the case of things like my connect 4 project, well the board does not change that often. In that project I can detect a new move in under a second, which is easily fast enough. Remember that you don't have to always have a full frame for object detection...

Andy

[burti]

Hi.
Sorry to bother you again but why can NxtCamView capture images within 4 seconds?
How can it read all the lines so fast?

so each line has to be read in sections (in my case I use 60bytes at a time), thus the 352byte line requires 6 transactions.

why can't you use 75 bytes at a time instead of 60 bytes? (then you only have 5 sections instead of 6 ones)

One way to reduce some of latency is to use an interleave when grabbing the lines in an attempt to reduce the per line latency and this is possible but it is tricky to get right.

Do you mean changing the transfer order of the lines with "interleaving"?
Why is it so tricky? Do you mean you have to ensure that all lines are transferred finally?
(i.e. without repeating transferring lines which have already been transferred)
That's a mathematical problem, isn't it?
Given we always need 0.0253 seconds to transfer a line, then you miss about 29 lines (0.0253 / 0.0009 = 28.16) while you are transferring one line.
Having transferred line 1, then the next line arriving would be line 30 which you could transfer immediately.
Having transferred line 30, then the next line arriving would be line 59 which you could transfer immediately and so on.
You "only" need to find an easy algorithm with a fixed or flexible line distance (at least 29) so that all lines are transferred finally at all without transferring a line twice.
You get 5 lines per frame (144 / 29 = 4.8 ) so that you need to run over 29 frames (144 / 5 = 28.8 ). This can be done in 4 seconds.
Is that right or do I ignore one important detail by accident?

edit:
I discovered that you can traverse all lines with a line distance of 36 lines with an easy deterministic algorithm.
Have a look on that easy program.
It shows you a sorted set with all numbers between 1 and 144 as entries.
It loses its entries by counting up from 1 to 144 in steps of 36 several times.
So it is counting 1,37,73,109,2,38,74,110,3,39,... .
Every number (or line index) is traversed exactly once.

Code: Select all

import java.util.SortedSet;
import java.util.TreeSet;

/**
 *
 * @author Michael Mirwaldt
 *
 */
public class Main {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        SortedSet<Integer> set = new TreeSet<Integer>();
        for (int i = 1; i <= 144; i++) {
            set.add(i);
        }
        int index = 1;
        final int step = 36;
        System.out.println(set);
        for (int i = 1; i < 1000 && (set.size()>0); i++) {
            System.out.println(i + "->" + index);
            set.remove(index);
            index+=step;
            if(index>144) {
                index = index - 143;
            }           
        }

        System.out.println(set);
    }

}


(the i < 1000 condition is just a protection against an infinite loop.
As you can see, the loop terminates before.)

So you would only need to wait the time of 7 lines (36 - 29 = 7 ) in average instead of the time of 74 lines (0.066 / 0.0009 = 73.3).
This means that the wait time of 9.5 seconds (0.066 * 144) could be decreased to 0.91 seconds (144 * 7 * 0.0009) in total.
Therefore it could maybe be possible to capture an image within 5 seconds with a nxt which is 4 times faster than your first approach.

What a pity I cannot try it out myself unless I have got a nxtcam.
I ordered once some minutes ago.
Will take a week before I get it.

What do you think of that improvement?

Michael