Gstreamer Camera 2 Source for Android

I'm very happy to send a new camera source element patch to support Android Camera 2 API.  This implementation is a part of R&D work which is sponsored by Collabora, Ltd.

Not much later since the previous element, ahcsrc, has been merged into 'gst-plugins-bad', Android N announced their public native level APIs. This news brought me another R&D opportunity to implement same featured, but more stable and little bit faster element to handle Camera Devices on Android.

I named the new element as 'ahc2src' because the previous one is just 'ahcsrc' and the fundamental API is Android Camera 2 NDK API. That means JNI is no longer required to access Camera devices. To understand why 'ahc2src' is better, here's comparison chart.

	ahcsrc	ahc2src
Supported Android version	> Lollipop (android-19)	> Nougat (android-24)
API	Camera API	Camera 2 NDK API
Device control	Java API through JNI	Native
Supported Image Format	YV12, YUY2, NV16, NV21, RGB16	NV12 (YUV_420_888)
Memory Copy	Java to GstBuffer	None (AImageReader inside)

Regarding memory copy, I'm not sure whether it doesn't perform inside of Android. However, ahc2src itself just passes image pointers to source pad and manage the reference counts.

How to get this new element?

Until merged and released gstreamer images which are built for Android Nougat, my personal repository will be continuously maintained to provide 'ahc2src' element. If you want to build from scratch, it might be annoying because of some issues which come from Android NDK.

See the details in the below links;

• BIONIC's  dlopen and dlsym (glib patch, gstreamer issue)
• Cebero build failure
• ahc2src working history

Build from Cerbero

In my personal repository, the patches are ready to build. It will fetch 'gst-plugins-bad' from here.

  $ git clone https://gitlab.collabora.com/joykim/cerbero.git
  $ git checkout wip/ahc2

Note that 'wip/ahc2' branch is potentially or frequently overwritten by forced push.

  $ ./cebero-uninstalled -c config/cross-android-nougat-arm64.cbc bootstrap
  $ ./cebero-uninstalled -c config/cross-android-nougat-arm64.cbc package gstreamer-1.0

Then, the generated SDK image should be extracted to a proper path. (e.g /Users/justin/Library/Android/gstreamer/devel/arm64)

Gstreamer Android Camera Example Application

From the point of Android application view, there's not much different from when using 'ahcsrc'. In theory, it might be enough to replace 'ahcsrc' with 'ahc2src'.

  $ git clone https://gitlab.collabora.com/joykim/gst-android-camera.git
  $ git checkout wip/ahc2

Then, you should modify 'gradle.properties' to set a proper gstreamer SDK path. (e.g, gstAndroidRoot=/Users/justin/Library/Android/gstreamer/devel)

  $ gradle installDebug

If there's no error, the example application will be installed successfully.

Enjoy your Gstreamer with Android Camera 2 source!

Gstreamer Camera Source for Android

Note: This article is outdated. The source element for android camera has been merged already. 


Using Android Camera via Gstreamer is not easy. There are two unofficial implementations; one is OpenWebRTC, another is android hardware camera source which is what I’d like to introduce here.

Actually, a source element for Android Hardware Camera has been developed already in Gstreamer 0.10. However, due to license problem and the other issues which I don’t know, the latest implementation was opened recently. Almost of Gstreamer developers might want to port this work from 0.10 to 1.0, but it’s not easy to fill the time gap between them. Perhaps, it’s not about level of difficulty, but rather it is about time consuming.

To use Android Camera with Gstreamer 1.0, we need to get few patches from my personal repository because review on my proposed patches are in progress.

gst-plugins-bad: https://git.collabora.com/cgit/user/joykim/gst-plugins-bad.git/log/?h=ahc

gst-camera-app: https://git.collabora.com/cgit/user/joykim/gst-camera-test.git/

Cerbero Patch: https://bugzilla.gnome.org/show_bug.cgi?id=755173

No one recommend building gst-plugins-bad for Android platform as it is. It requires a bit annoying tasks like setting environment variables, copying files into specific directories and typing hundreds of commands. Simply, we can use cerbero to get Gstreamer SDK image, but it’s not the scope of this article. :)

Here’s an url for pre-built SDK to make all thing simple.

SDK: https://git.collabora.com/~joykim/android/gst/SDK-ahcsrc/

This image should be extracted in a proper directory, in here, ~/Android/gstreamer I used.

$ mkdir ~/Android/gstreamer

$ cd ~/Android/gstreamer

$ tar jxvf /somewhere/gstreamer-1.0-android-armv7-1.5.90.tar.bz2

Then, GSTREAMER_ROOT_ANDROID is reuqired.

$ export GSTREAMER_ROOT_ANDROID=~/Android/gstreamer

Before compiling gst-camera-app, we need to set PATH variable with Android SDK as well as NDK. For SDK, tools and platform-tools directories should be in PATH. Regarding to NDK, the path as itself is sufficient to use ‘ndk-build’ command.

Now, we can clone gst-camera-app.

$ git clone git://git.collabora.com/git/user/joykim/gst-camera-test.git

In the directory of test application, Android project properties should be updated.

$ android update project -p .

Then, this application is ready to compile by just a command, ndk-build.

$ ndk-build 

However, if you want to get debug image, use this command instead.

$ ndk-build NDK_DEBUG=1

The test application uses glimagesink element. It causes a problem with android emulator so we need a real android device to test this application. Regarding to this issue, please refer to https://bugzilla.gnome.org/show_bug.cgi?id=753905

The command, ndk-build, generates all of c objects to connect gstreamer with Java. Now, we can build Java classes and install Android package into a real device.

$ ant debug && ant installd

Or

$ ant release && ant installr

Now, enjoy the camera.