Camera image as an OpenGL texture on top of the native camera viewfinder

41 Replies

I played a bit with the camera viewfinder on my G1 which is usually displayed directly by the camera driver. I hoped that I could synchronize the driver’s camera frame rendering with my own processing and visualization. After an hour or so later I now assume that this is not possible at the moment. However, while playing around I extended the example below as you can see in these screenshots.

OpenGL Camera ScreenshotOpenGL Camera Screenshot

An OpenGL cube textured with the camera frame is rendered on top of the standard camera viewfinder. Thus, the standard camera image in the background is colored while the the cube is only grayscale. I worry that I have to make the OpenGL texture colored as well soon. I also cleaned up the source code a bit by extending GLSurfaceView instead of doing most of the OpenGL stuff myself and using a SurfaceView. I uploaded an updated version to the android market (direct link to the android market). You find the sourceode here.

41 thoughts on “Camera image as an OpenGL texture on top of the native camera viewfinder”

  2. BiGAlex

    Really interesting project, thank you for the source code, I gave it a fast look, and it doesn’t looks to be really difficult to implement something useful… anyway, what’s your project about? 😛

    Thank you and… good job!

  4. Niels

    I’m not sure if I understand the thread but I’m pretty sure that you can draw a GLSurfaceView on top the camera’s SurfaceView on other devices. We tested it on Android 1.5 and 1.6. I assume that augmented reality apps, such as Layar and Wikitude, use the same approach and they work on all kinds of devices.

    However, the way I put the View’s on the screen seems a bit strange to me. I first call “setContentView(glView);” and then “addContentView(mPreview, …);”. I would expect that these calls put the glView below the other one but it’s the other way around…

  9. Henri

    Hi, It’s cool, but I get carbage collector running constantly. Is it possible to make it so that new memory wouldn’t get allocated while program is running? How can I check where the allocations are made?

  12. Henri

    Since I just started to learn developing for mobile devices: Might it be more reasonable to develop augmented reality (3D stuff on video) apps on JavaME or WindowsMobile rather than on Android? Or is it wiser to stick to Android?

  14. bauble

    does NOT work on droid motorola nor other 2.1 android. works fine on G1, adp. See market app “bauble” for a similar situation (code in resources) that had the same problem.

  17. Gabriela Vasselai

    It worked on my HTC Desire with Android 2.2 but I found an issue I don´t know how to solve.

    With your app open I press Home button (which only pauses the app) and if I enter the app again, the GL won´t draw anything. With your app open again, I have to press the Back button (which destroy the app) to make GL work again.

    Do you know any solution for this?

    Thanks and best regards,
    Gabriela

  18. Pingback: Sensor-based Augmented Reality made simple « Things about stuff

  19. gamijo

    I wondered how to set the background of the opengl view with the camera preview.. it was just a simple function. Thanks a lot for that.

  21. el-Grove

    It isn’t working properly on my HTC Desire with Android 2.2.
    After comment line:
    this.setEGLConfigChooser(5, 6, 5, 8, 16, 0);
    I get symmetrically, twice rendered the same cube with strange colors texture on. Commenting another line:
    this.getHolder().setFormat(PixelFormat.Translucent);
    or changing pixel format to Opaque is giving result with one cube rendered with gray scale texture captured from camera but with black background. Do you know any solution for this issue?

  22. D.Adams

    Hi,
    I really need your help.
    I want to draw the preview buffer on a simple 2D surface (not a cube..) And mirror the image. make it look like a twisted mirror.

    Can anyone throw in a tip? I know it’s suppose to be easy.

    Thanks for helping 🙂

  23. soumi

    Hello.

    I could fix problem with use this code

    “this.setEGLConfigChooser(8, 8, 8, 8, 16, 0);”

    instead of

    “this.setEGLConfigChooser(5, 6, 5, 8, 16, 0);”

    Hope this helps. Try it! 🙂

  24. Elecman

    Crashed on my Samsung Nexus S with Android 2.3.1 even when commenting:
    this.setEGLConfigChooser
    I upgraded Android to 2.3.3 and now it works, but only if commenting that line, otherwise it still crashes. However, the camera is only projected at the 3d object. The background is black.

  25. Cristian

    I tried to have a GL Surface View on top of camera surface, so that I can draw GLobjects with camera preview as background. I tried but didn’t work.

  26. Andromeda

    I gave it a fast look, and it doesn’t looks to be really difficult to implement something useful!Anyway, what’s your project about?Anyway, thanks for sharing.

  27. Shy

    It might be more reasonable to develop augmented reality (3D stuff on video) apps on JavaME or WindowsMobile rather than on Android? Or is it wiser to stick to Android?

  28. Moy2x

    Now my question is will all devices have this behavior? Can I use this behavior in my app to get openGL layered on top of camera preview? Please answer this query, its so cool to have openGL over the camera preview! It opens up for some amazing stuff!

  29. firstdeathmaker

    Also works on LG-V900 Tablet (Android 3.0) , but just so far that it shows the camera image. And some strange behaviour appeared: The camera image is shown with at good FPS, but delayed by 0.5 seconds. Like if there is an array that stores the images in between.

  30. Rahul

    This works on the google nexus-s as well ,.Thanks a ton ! .Its really helpful as I cudnt find any other code for this purpose till I stumbled upon this .

    Cheers

  31. Matthew Mitchell

    I’m looking at this now and it seems you can take a camera stream into a SurfaceTexture object and then render that.

  32. Matt J.

    It fails on a US T-Mobile HTC G2 running 2.3.4. It throws exceptions, once a RuntimeException on startPreview() but sometimes an IOException on setPreviewDisplay().

    I also get a lot of repetitions of the following line in logcat, which makes it even harder to debug:(

    W/CameraService( 1230): Overlay create failed – retrying

    The only change I made to the code is to add comments.

  35. Shreyash

    I want to give some vartex Shader effect in Inner camera preview. Will you please help me for it.
    I will be realy thankful to you if its done.

  36. McEvin

    I still had errors that were not mentioned yet.

    One: requestFeature() must be called before adding content …

    Fix: cut and paste the line
    requestWindowFeature(Window.FEATURE_NO_TITLE);
    directly under the call of the superclass in onResume()- method of the main activity

    Two: Method called after release() …

    Fix: insert the line
    mCamera.setPreviewCallback(null);
    before the call of stopPreview() in the OnSurfaceDestroyed()-Method of the CameraLayer-Class

    Thanks for the example!

    Kevin

  37. Amit

    I am trying to capture these video preview but not able capture video it will generate video file but show error message when i will open it like “Cannot play video”. Do you have any solution about it?

  38. jg

    Anyone figure out how to get the image in color? I tried using that decodeYUV420SP function that’s floating around the net & then doing

    this.bmp = Bitmap.createBitmap(this.rgbBuf, 512, 512, Config.ARGB_8888);

    followed by

    GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, this.bmp, 0);

    To use the bitmap as a texture, but no luck.

  40. jg

    I figured out that I need to convert from yuv to rgb565, but all the functions I’ve found don’t work :(. There’s two or three “toRGB565” floating around the net, but I get garbage with all of them :(.

  41. Geoffroy

    To get the colors, draw the image on a canvas (constructed from a blank bitmap) and it’ll be on your new bitmap in argb 🙂
    Not tested with this code though, as it’s not working for the moment

Leave a Reply

Your email address will not be published. Required fields are marked *