Goodbye garbage collector – patching Android to make real-time camera image processing feasible

If you want to process camera images on Android phones for real-time object recognition or content based Augmented Reality you probably heard about the Camera Preview Callback memory Issue. Each time your Java application gets a preview image from the system a new chunk of memory is allocated. When this memory chunk gets freed again by the Garbage Collector the system freezes for 100ms-200ms. This is especially bad if the system is under heavy load (I do object recognition on a phone – hooray it eats as much CPU power as possible). If you browse through Android’s 1.6 source code you realize that this is only because the wrapper (that protects us from the native stuff) allocates a new byte array each time a new frame is available. Build-in native code can, of course, avoid this issue.

I still hope someone will fix the Camera Preview Callback memory Issue but meanwhile I fixed it, at least for my phone, to build prototypes by patching the Donut’s (Android 1.6) source code. What you find below is just an ugly hack I did for myself! To reproduce it you should know how to compile Android from source.

Avoid memory allocation

Diving in the source code starts with the Java Wrapper of the Camera and its native counterpart android_hardware_Camera.cpp. A Java application calls setPreviewCallback, this method calls the native function android_hardware_Camera_setHasPreviewCallback, and the call is passed further down into the system. When the driver delivers a new frame towards the native wrapper in return it ends up in the function JNICameraContext::copyAndPost():

void JNICameraContext::copyAndPost(JNIEnv* env, const sp& dataPtr, int msgType)
{
    jbyteArray obj = NULL;

    // allocate Java byte array and copy data
    if (dataPtr != NULL) {
        ssize_t offset;
        size_t size;
        sp heap = dataPtr->getMemory(&offset, &size);
        LOGV("postData: off=%d, size=%d", offset, size);
        uint8_t *heapBase = (uint8_t*)heap->base();

        if (heapBase != NULL) {
            const jbyte* data = reinterpret_cast(heapBase + offset);
            obj = env->NewByteArray(size);
            if (obj == NULL) {
                LOGE("Couldn't allocate byte array for JPEG data");
                env->ExceptionClear();
            } else {
                env->SetByteArrayRegion(obj, 0, size, data);
            }
        } else {
            LOGE("image heap is NULL");
        }
    }

    // post image data to Java
    env->CallStaticVoidMethod(mCameraJClass, fields.post_event,
            mCameraJObjectWeak, msgType, 0, 0, obj);
    if (obj) {
        env->DeleteLocalRef(obj);
    }
}

The evil bouncer is the line obj = env->NewByteArray(size); which allocates a new Java byte array each time. For a frame with 480×320 pixels that means 230kb per call and that takes some time. Even worse this buffer must be freed later on by the Garbage Collector which takes even more time. Thus, the task is to avoid these allocations. I don’t care about compatibility with existing applications and want to keep the changes minimal. What I did is just a dirty hack but works for me quite well.

My approach is to allocate a Java byte array once and reuse it for every frame. First I added the following three variables to android_hardware_Camera.cpp:

static Mutex sPostDataLock; // A mutex that synchronizes calls to sCameraPreviewArrayGlobal
static jbyteArray sCameraPreviewArrayGlobal; // Buffer that is reused
static size_t sCameraPreviewArraySize=0; // Size of the buffer (or 0 if the buffer is not yet used)

To actually use the buffer I change the function copyAndPost by replacing it with the following code:

void JNICameraContext::copyAndPost(JNIEnv* env, const sp& dataPtr, int msgType) {
    if (dataPtr != NULL) {
        ssize_t offset;
        size_t size;
        sp heap = dataPtr->getMemory(&offset, &size);
        LOGV("postData: off=%d, size=%d", offset, size);
        uint8_t *heapBase = (uint8_t*)heap->base();

        if (heapBase != NULL) {
            const jbyte* data = reinterpret_cast(heapBase + offset);
            //HACK
            if ((sCameraPreviewArraySize==0) || (sCameraPreviewArraySize!=size)) {
                if (sCameraPreviewArraySize!=0) env->DeleteGlobalRef(sCameraPreviewArrayGlobal);
                sCameraPreviewArraySize=size;
                jbyteArray mCameraPreviewArray = env->NewByteArray(size);
                sCameraPreviewArrayGlobal=(jbyteArray)env->NewGlobalRef(mCameraPreviewArray);
                env->DeleteLocalRef(mCameraPreviewArray);
            }
            if (sCameraPreviewArrayGlobal == NULL) {
                LOGE("Couldn't allocate byte array for JPEG data");
                env->ExceptionClear();
            } else {
                env->SetByteArrayRegion(sCameraPreviewArrayGlobal, 0, size, data);
            }
        } else {
            LOGE("image heap is NULL");
        }
    }
    // post image data to Java
    env->CallStaticVoidMethod(mCameraJClass, fields.post_event, mCameraJObjectWeak, msgType, 0, 0, sCameraPreviewArrayGlobal);
}

If the buffer has the wrong size a new buffer is allocated. Otherwise the buffer is just reused. This hack has definitely some nasty side effects in common situations. However, to be nice we should delete the global refference to our buffer when the camera is released. Therefore, I add the following code to the end of android_hardware_Camera_release:

if (sCameraPreviewArraySize!=0) {
    Mutex::Autolock _l(sPostDataLock);
    env->DeleteGlobalRef(sCameraPreviewArrayGlobal);
    sCameraPreviewArraySize=0;
}

Finally, I have to change the mutex used in the function postData. The Java patch below avoids passing the camera image to another thread. Therefore, the thread that calls postData is the same thread that calls my Java code. To be able to call camera functions from that Java code I need another mutex for postData. Usually the mutex mLock is used through the line: Mutex::Autolock _l(mLock); and I replace this line with Mutex::Autolock _l(sPostDataLock);.

Outsmart Android’s message queue

Unfortunately this is only the first half our customization. Somewhere deep inside the system probably at the driver level (been there once – don’t want to go there again) is a thread which pumps the camera images into the system. This call ends up in the Java code of Camera.java. Thereby the frame is delivered to the postEventFromNative method inside Camera.java. However, afterwards the frame is not delivered directly to our application but takes a detour via Android’s message queue. This is pretty ugly if we reuse our frame buffer. The detour makes the process asynchronous. Since the buffer is permanently overwritten this leads to corrupted frames. If you want to avoid this detour this must be changed. The easiest solution (for me) is to take the code snippet that handles this callback from the method handleMessage:

            case CAMERA_MSG_PREVIEW_FRAME:
                if (mPreviewCallback != null) {
                    mPreviewCallback.onPreviewFrame((byte[])msg.obj, mCamera);
                    if (mOneShot) {
                        mPreviewCallback = null;
                    }
                }
                return;

and move it to the method postEventFromNative.

	if (what==CAMERA_MSG_PREVIEW_FRAME) {
                if (c.mPreviewCallback != null) {
                    c.mPreviewCallback.onPreviewFrame((byte[])obj, c);
                    if (c.mOneShot) {
                        c.mPreviewCallback = null;
                    }
                }
                return;
	}

This might have some nasty side effects in some not so specific situations. If you done all that you might want to join the discussion about Issue 2794 and propose an API change in the Camera API: Excessive GC caused by preview callbacks thread to find a proper solution for the Camera Preview Callback memory Issue (and leave a comment here if you have a better solution).

17 thoughts on “Goodbye garbage collector – patching Android to make real-time camera image processing feasible

  1. David Grimfors

    Nice article!
    I hope Google will provide a means of keeping the data in native domain for processing but that seems unlikely to happen in any near future.

  2. David Grimfors

    Is it really necessary to disable the message queue?

    First of all, the buffer should not (well, at least not very often) be overwritten during the time we use it unless our processing time exceeds the time between two frames, in which case we are in trouble anyway.

    And even if the frame is overwritten, it´s not a very big problem in the preview case due to
    1. The pixel data has a fixed bytes per pixel, meaning we will have no alignment issues.
    2. The framerate can be assumed to be high, meaning two consecutive frames are likely to be very similar.

    Just my thoughts, I haven´t tried it yet.

  3. Niels Post author

    Thanks a lot for your thoughts. You are completely right, avoiding the message queue is not necessary. You could, however, end up processing an image containing a mixture of two frames. Depends clearly on the application if this is a problem or not.

    Another glitch, if the message queue is not avoided, is that you probably have to pass the camera frames to another thread. If you process the frames in the message queue’s thread you delay other messages and might end with an unresponsive application or, even worse, provoke a message box that tells the user that your application does not respond

  4. Niels Post author

    @Peter thanks for your comment. I checked out your project before and it’s quite cool! I did some related stuff that tries to convey geographic information to blind users with sound or tactile feedback. However, we only tried to present abstract information. Would been cool to see the vOICe approach combined with a tactile output.

  5. Bo Brinkman

    Hi Niels (and everyone):
    This post has been incredibly valuable to me, as I am trying to get some mobile augmented-reality research off the ground.

    Since we are using Nexus Ones, I was a bit hesitant to void the warranty by unlocking them and installing a custom android. It turns out that this was not necessary for those of us that have eclair-based (Android 2.1) systems: The solution to the problem is already in android.hardware.Camera, but they have hidden the APIs for the time being.

    I have some code that shows how to work around this using reflection at: http://www.users.muohio.edu/brinkmwj/android/

  6. yaozr

    Hi Niels:

    I have adp1 with 1.6 and is trying mobile AR, the frame rate is simply not acceptable if doing anything (simple analysis or filter i.e.) on full preview frame data (480×320). I dont want to lose all other phone app, market etc, so custom build does not seems feasible for me, Bo Brinkman’s 2.1 finding seems great but then I have to upgrade my 1.6 adp1 to 2.1 taking the brick risk (if following the wellknown hacker: http://www.denraf.be/content/howto-android-21-g1adp1dream)… and I’m not sure if 2.1 camera sdk functionality is fully supported and tested in this risky road. Peter’s voIce is impressive but the “analysing” frame rate i a separate thread still seems a bit low for typical real time ar application even with sacrificing frame size.
    …. realy frustrating, would appreciate if anyone could shed some light for adp1 1.6 users ?

  7. Niels Post author

    Hey Bo,

    I just tested your code and it is fantastic. Even with the standard API and the Garbage Collector I get the frames quite fast on the Nexus One if using low resolution but with the hidden API it works fast even with high reslutions. I can process ~20 fps @ 480×800. Thanks a lot for sharing it!

    Niels

  8. bauble

    The problem though, is that if we code for hardware like this, we may as well just replace android completely, which appears to be the point of their problems… This example does NOT work on < 1.6? please msg.

    your prior demo glcam does NOT work on droid motorola nor other 2.1 android. works fine on G1, adp. See market app “bauble” for a similar situation (code in resources) that had the same problem.

  9. ivomania

    Hi, I’m Ivano Brogonzoli, a mobile software engineer. I’m Android and Augmented Reality excited and I found really useful this post when I was doing my preliminary state-of-the-art study on the android framework.

    It seems the reflection technique is no more needed on Froyo based systems. Please see few details at:

    http://superivomania.blogspot.com/2010/05/android-camera-performance-new-options.html

    My next step is to provide a wrapper camera class to use the reflection methods on 2.1 and the native ones in 2.2. I’ll advise when the wrapper will be ready… and where… :). In the meantime I wanna thank all of you to share your comment about this hot topic.

    Let’s keep in touch.

  10. Niels Post author

    Hi Ivano,

    sounds really great! I thought about doing the same thing and build a wrapper for a patched Android 1.6 and the approach using reflection for 2.1. If you have some code ready I would be highly interested 🙂

    Niels

  11. Pingback: Sensor-based Augmented Reality made simple « Things about stuff

  12. Edward Collins

    That’s a great solution. The garbage collection process was not able to handle the situation well and lead to bad performance but yours one is reliable. Thanks.

  13. md forte eye cream review

    The information that you share in us i very useful. I am studying with my Information Technology course and i can say that this information is very useful in my major subject, i will study this one in advance.

  14. Pingback: » Why Android is so Awesome – for Prototypes and Research Things about stuff

  15. Pingback: Why mobile web apps are slow | Sealed Abstract · Dented Reality

Leave a Reply

Your email address will not be published. Required fields are marked *