Processing camera frames on Android

Recently I wanted to process and display camera frames using my Android G1. I’ve done similar things using Python on S60 and Windows Mobile 6 and expected it to be quite easy on the G1 as well. As first step I extended a SurfaceView that uses the camera and calls setPreviewCallback to register a onPreviewFrame callback and receive images from the camera as described in several tutorials. The camera frames are then displayed via my SurfaceView and I receive the according data as well.

However, I wanted to keep the processing of the frames and displaying the frames in sync. With the simple approach this is not possible because the onPreviewFrame is not synchronized with displaying the frames. My alternative was to not display the frames with the SurfaceView directly but convert the received image data to an OpenGL texture and render the camera viewfinder with OpenGL ES. This works surprisingly fast on my G1. In the video below I render the camera frames on an OpenGL rectangle to get some fancy effects.

My viewfinder is grayscale because I only copy the luminance part of the camera frames (which is encoded in a YUV colour space) to the OpenGL texture. Decomposing also the U and V part is probably a bit slower. Copying a 160×240 YUV frame to a 256×256 luminance array (which is used to create the texture) is very simple and looks as follows.

public static void yuvToLum160x240(byte[] yuv, byte[] lum) {
int lumCount=0;
int yuvCount=0;
for (int y=0;y<160;y++) {
System.arraycopy(yuv, yuvCount, lum, lumCount, 240);

12 thoughts on “Processing camera frames on Android

  1. Sebastien

    This is great stuff ! I have tried to achieve the same goal, but couldn’t open the camera without a valid surfaceview.
    Could you please give some hints on how you managed to do this ?

  2. admin Post author

    Thanks! I guess you have to create a surface view for the camera but you can put another view on top of it. That’s basically the way I do it. I posted a small example hopefully that is useful.

  3. Pingback: Camera image->NDK->OpenGL texture « Things about me

  4. Mark Inman

    Wow could you please zip your code and send it to me via email?! i am trying to implement a zoom function for a class and there are absolutely no tutorials on how to do this. i would be forever greatful!

  5. redrum008

    I get the frame in callbackprevie, convert it in black/white image, but i don’t know how to show this new image in black/white. How to make this without openGL?

  6. Jade Balmain

    Hi Niels,

    Where can I find a video tutorial on how to do this on my smartphone?

    Jade Balmain

Leave a Reply

Your email address will not be published. Required fields are marked *