Camera image->NDK->OpenGL texture

31 Replies

Since we are currently working on some augmented reality stuff for Android I need to show the camera image using OpenGL ES. It works great with pure Java if one uses only the grayscale image. However, I needed the color image. The G1’s camera delivers the image in a YUV format while OpenGL only understand RGB images. Unfortunately it is out of question to convert the YUV image to RGB in pure Java for images with 480×320 pixels. Thus, I used the NDK to implement the conversion. The code below does the job. It is based on code provided by Tom Gibara.

void toRGB565(unsigned short *yuvs, int widthIn, int heightIn, unsigned int *rgbs, int widthOut, int heightOut) {
  int half_widthIn = widthIn >> 1;

  //the end of the luminance data
  int lumEnd = (widthIn * heightIn) >> 1;
  //points to the next luminance value pair
  int lumPtr = 0;
  //points to the next chromiance value pair
  int chrPtr = lumEnd;
  //the end of the current luminance scanline
  int lineEnd = half_widthIn;

  int x,y;
  for (y=0;y> 1;
    for (x=0;x> 8) & 0xff;
      Y1 = Y1 & 0xff;
      int Cr = yuvs[chrPtr++];
      int Cb = ((Cr >> 8) & 0xff) - 128;
      Cr = (Cr & 0xff) - 128;

      int R, G, B;
      //generate first RGB components
      B = Y1 + ((454 * Cb) >> 8);
      if (B < 0) B = 0; if (B > 255) B = 255;
      G = Y1 - ((88 * Cb + 183 * Cr) >> 8);
      if (G < 0) G = 0; if (G > 255) G = 255;
      R = Y1 + ((359 * Cr) >> 8);
      if (R < 0) R = 0; if (R > 255) R = 255;
      int val = ((R & 0xf8) << 8) | ((G & 0xfc) << 3) | (B >> 3);

      //generate second RGB components
      B = Y1 + ((454 * Cb) >> 8);
      if (B < 0) B = 0; if (B > 255) B = 255;
      G = Y1 - ((88 * Cb + 183 * Cr) >> 8);
      if (G < 0) G = 0; if (G > 255) G = 255;
      R = Y1 + ((359 * Cr) >> 8);
      if (R < 0) R = 0; if (R > 255) R = 255;
      rgbs[yPosOut+x] = val | ((((R & 0xf8) << 8) | ((G & 0xfc) << 3) | (B >> 3)) << 16);
    }
    //skip back to the start of the chromiance values when necessary
    chrPtr = lumEnd + ((lumPtr  >> 1) / half_widthIn) * half_widthIn;
    lineEnd += half_widthIn;
  }
}

The code is not that optimized at the moment but can process a 480×320 image in ~25ms on my G1 (which is somewhat slow according to my student’s comments). In order to call this function from Java I needed a wrapper with a JNI signature:

/**
 * Converts the input image from YUV to a RGB 5_6_5 image.
 * The size of the output buffer must be at least the size of the input image.
 */
JNIEXPORT void JNICALL Java_de_offis_magic_core_NativeWrapper_image2TextureColor
  (JNIEnv *env, jclass clazz,
  jbyteArray imageIn, jint widthIn, jint heightIn,
  jobject imageOut, jint widthOut, jint heightOut,
  jint filter) {

	jbyte *cImageIn = (*env)->GetByteArrayElements(env, imageIn, NULL);
	jbyte *cImageOut = (jbyte*)(*env)->GetDirectBufferAddress(env, imageOut);


	toRGB565((unsigned short*)cImageIn, widthIn, heightIn, (unsigned int*)cImageOut, widthOut, heightOut);

	(*env)->ReleaseByteArrayElements(env, imageIn, cImageIn, JNI_ABORT);
}

To make it more interesting I added some filter to the camera image. There is a demo app in the market (direct link to the market). I tried to make the whole thing portable but would love to know if it works on other devices like the Motorola Milestone.
Sepia effectBlack & White effectFisheye effectInvert effect

31 thoughts on “Camera image->NDK->OpenGL texture”

  1. Ron

    I have the Moto Droid (android 2.0) and am in search of a reliable app that will allow me to use the Driod as a regular video camera and auto save the recordings to the SD card if possible with the ability to turn the led light on and off with in the app while recording.

  3. Casper van Wezel

    Hello,

    I have a question: is it possible for me to receive the code that belongs to this project/program. I have implemented it on the G1, but for some reason your application runs about (this is a guess) 2-3 fps faster. My goal is to do some image processing tasks on the android platform. There is no commercial interest, since this is part of my master thesis at the University of Utrecht.

    Thanks in advance,
    Greetings,
    Casper

  4. lee

    Moto droid verizon does not work, black, no trace. G1/adp fine. Send email, will forward you code for similar tests to play with and clean for demos.

  8. Jerry

    Hi

    I am a phd student at UK. I work on a project titled “image processing on mobile”. It would be greatly appreciated if you send me your source code.

    Best,

  9. kelvin

    Hi,
    You are work is great! I am also a research student in field of computer vision, would you kindly share the code for me as reference. I really appreciate the performance of you app.

    Thanks,

  10. Stefan Lederer

    Hi,

    great work. Is it also possible to extract the frames of a playing video and forward them to the post processing you use? This would be really important for me, maybe we can get in touch via email?

    Greetings
    Stefan Lederer
    University of Klagenfurt

  12. Slawek

    Hi all,

    I’m a little confused. I’ve checked this piece of code and I receive OutOfMem exception. I don’t know what is going on. Could you help me with it?

  14. Fred

    I’m a bit puzzled by the code:

    – You’re not using Y2
    – the 2 parts of the code that calculate R,G and B (one for val, one for rgbs[]) are identical

    Also, would there be a performance gain from declaring the temporary variables outside the loop?

    bets wishes,
    fred

  15. Komi

    Hello,
    Great work! I am trying to create a Computer Vision App for my bachelor thessis and i would like to use NDK, but i am not sure, how to use this JNI code in Java. Can you send me your java class, where you convert YUV byte array to bitmap by this native code?

    Thans a lot

  16. Mikael

    I can’t see that anyone answered Casper’s question above (#3), so I am going to ask it again in case it was overlooked:

    “is it possible to receive the code that belongs to this project/program?

    Thank you,
    Mikael

  17. Pep

    Hi all!

    nice work. It is always good to speed up the repetitive and slow pieces of code, like this tricky image conversion part.

    However, I was unable to make it work. How do you call your wrapper in the java file?

    Thanks!!

  18. btmejdi

    Hi all!

    nice work.
    I am a computer science student from Tunisia.
    I would like to know how toconnect the camera with NDK.
    Can anybody help me please?
    Could you please mail me the sources?

    Thank you!

  19. sunset

    hi,

    where should i do the conversion in NDK? and how should i call it from the java file? Is it possible for you to send this this source code? deeply appreciated!

  21. Rahul

    Hey Niels,Great post and a really helpful one too !

    Just wanted 2 point out that the program compiles fine ( with minor adjustments in the C++ routines ).However,while running the app on the google nexus s,it exits abnormally.

    I tried finding the source of the errror and found that the app exits when it reaches line 43 of the cpp file :rgbs[yPosOut+x] = val | ((((R & 0xf8) << 8) | ((G & 0xfc) <> 3)) << 16);

    I had passed a bytebuffer of the same size as the yuvs(both size:240×160) to the function and I am not really sure why the program exits here. The Log shows a "channel broken and program exits error".

    Hope you can help me out here!

    Cheers,
    Rahul

  23. Daniel Princ

    Hey,
    Is there any chance to get the source code? I could really use some example how to do filters like this on android 🙂

    Thanks!

  24. Dima

    Hello!
    I am a student in Ukraine. I work on a project named “Computer Vision”. It would be greatly appreciated if you send me your source code.
    Thank you very much!

  26. Nestor

    hello,

    I am completely new to ndk, and I have some questions regarding how to call this function.

    Could you comment more about how to call this function:
    1)In function header what is “jclass clazz”?
    2)What you pass for “jobject imageOut”?

  28. Pingback: Releasebytearrayelements Android | Gsm Android

  29. Pingback: Opengl Image Processing Android | world Best

  30. Pingback: Yuv To Rgb Android Ndk – Title

  31. Pingback: Differences between OpenCV JavaCV and OpenCV4Android – FIXBBS

Leave a Reply

Your email address will not be published. Required fields are marked *