Monday, October 9, 2017

Gstreamer Camera 2 Source for Android

I'm very happy to send a new camera source element patch to support Android Camera 2 API.  This implementation is a part of R&D work which is sponsored by Collabora, Ltd.

Not much later since the previous element, ahcsrc, has been merged into 'gst-plugins-bad', Android N announced their public native level APIs. This news brought me another R&D opportunity to implement same featured, but more stable and little bit faster element to handle Camera Devices on Android.

I named the new element as 'ahc2src' because the previous one is just 'ahcsrc' and the fundamental API is Android Camera 2 NDK API. That means JNI is no longer required to access Camera devices. To understand why 'ahc2src' is better, here's comparison chart.




ahcsrc
ahc2src
Supported Android version
> Lollipop (android-19)
> Nougat (android-24)
API
Camera API
Camera 2 NDK API
Device control
Java API through JNI
Native
Supported Image Format
YV12, YUY2, NV16, NV21, RGB16
NV12 (YUV_420_888)
Memory Copy
Java to GstBuffer
None (AImageReader inside)

Regarding memory copy, I'm not sure whether it doesn't perform inside of Android. However, ahc2src itself just passes image pointers to source pad and manage the reference counts.

How to get this new element?

Until merged and released gstreamer images which are built for Android Nougat, my personal repository will be continuously maintained to provide 'ahc2src' element. If you want to build from scratch, it might be annoying because of some issues which come from Android NDK.

See the details in the below links;

Build from Cerbero

In my personal repository, the patches are ready to build. It will fetch 'gst-plugins-bad' from here.

  $ git clone https://gitlab.collabora.com/joykim/cerbero.git
  $ git checkout wip/ahc2

Note that 'wip/ahc2' branch is potentially or frequently overwritten by forced push.

  $ ./cebero-uninstalled -c config/cross-android-nougat-arm64.cbc bootstrap
  $ ./cebero-uninstalled -c config/cross-android-nougat-arm64.cbc package gstreamer-1.0

Then, the generated SDK image should be extracted to a proper path. (e.g /Users/justin/Library/Android/gstreamer/devel/arm64)

Gstreamer Android Camera Example Application

From the point of Android application view, there's not much different from when using 'ahcsrc'. In theory, it might be enough to replace 'ahcsrc' with 'ahc2src'.

  $ git clone https://gitlab.collabora.com/joykim/gst-android-camera.git
  $ git checkout wip/ahc2

Then, you should modify 'gradle.properties' to set a proper gstreamer SDK path. (e.g, gstAndroidRoot=/Users/justin/Library/Android/gstreamer/devel)

  $ gradle installDebug

If there's no error, the example application will be installed successfully.

Enjoy your Gstreamer with Android Camera 2 source!

3 comments:

  1. Justin Kim,

    Thank you for this great improvement for gstreamer. Camera2 API is something we can work with.

    I am trying to capture 4k video from the camera and stream it as H265 video. Assuming that my android device has the HW ecnoder, what would be the right gstreamer pipeline for that purpose?

    Thanks,
    Yechiel

    ReplyDelete
  2. Unfortunately, Ahc2src is still in progress due to lack of interest from the community. Although it’s difficult to access Android camera 2 API via GStreamer source element, there are encoder and decoder elements in gst-plugins-bad. So IMHO, if you need to use Android camera api 2, capturing 4k video as I420 then streaming to pipeline by using appsrc which is connected to encoder element. It’s currently possible way but it’s a bit tricky.

    ReplyDelete
    Replies
    1. Hi Justin Kim,
      Thanks for your reply.
      Do you know of any other decent way of doing it from an android app without a gstreamer pipeline? Maybe using standard android apis such as Camera, MediaCodec and rtp lib for instance

      Delete