Examples for Banuba SDK on Android and Agora.io SDK integration via Agora Plugin Filters to enhance video calls with real-time face filters and virtual backgrounds.

Getting Started

Prerequisites

Visit agora.io to sign up and get the app id, client token and channel id. Please consult with Agora documentation about account management to find out exactly how mentioned IDs are created.
Activate the Banuba Face AR SDK extension to get the app key and app secret. Please check out the extension integration documentation if you have any questions regarding this step.

Dependencies

	Version	Description
io.agora.rtc:full-sdk	4.2.3	Agora RTC dependency
Banuba SDK	1.9.0	Banuba Face AR dependency for applying AR filters
banuba-agora-extension	1.3.0	Banuba Extension for Agora

Installation

Run the following command in the Terminal to clone the project

git clone https://github.com/Banuba/agora-plugin-filters-android.git

Add Banuba dependencies. Add repository to build.gradle to get Banuba dependencies

maven {
  name "GitHubPackages"
  url "https://maven.pkg.github.com/sdk-banuba/banuba-sdk-android"
  credentials {
    username = "sdk-banuba"
    password = "\u0067\u0068\u0070\u005f\u0033\u0057\u006a\u0059\u004a\u0067\u0071\u0054\u0058\u0058\u0068\u0074\u0051\u0033\u0075\u0038\u0051\u0046\u0036\u005a\u0067\u004f\u0041\u0053\u0064\u0046\u0032\u0045\u0046\u006a\u0030\u0036\u006d\u006e\u004a\u004a"
  }
}

Add required dependencies to app/build.gradle.

dependencies {
    // Banuba Face AR dependency for playing AR effects
    implementation "com.banuba.sdk:banuba_sdk:1.9.0"

    // Banuba extension for Agora
    implementation 'com.banuba.sdk.android:agora-extension:1.3.0'
    ...
}

Add Agora dependency implementation 'io.agora.rtc:full-sdk:x.x.x' in app/build.gradle
Add your keys to run the sample.

// Use Agora appId key created in Agora console
const val AGORA_APP_ID = ...

// Use Agora channel created in console
const val AGORA_CHANNEL = ...

// Use token created in Agora console with Agora by channel
// Important! This token can be omitted if your Agora project is in testing mode
const val AGORA_TOKEN = ""

// Use Banuba license token
const val BANUBA_LICENSE_TOKEN: String = ...

The sample includes a few basic AR effects, however you can download more. All effects must be added into effects folder in Android assets. Please modify the list of available effects in the sample.

Integration

The extension includes main BanubaExtensionManager class for interacting with AR effects.
Basically, AR effects should be placed to assets/effects folder.
Additionally, you can store AR effects on device internal storage and use File to load effect.

Enable extension

First, add Banuba extension BanubaExtensionManager.BANUBA_PLUGIN_NAME while initializing instance RtcEngineConfig

val rtcEngineConfig = RtcEngineConfig()
+rtcEngineConfig.addExtension(BANUBA_PLUGIN_NAME)

Next, you need to enable the extension in onCreate method by providing specific provider and extension names.

override fun onCreate(savedInstanceState: Bundle?) {
   super.onCreate(savedInstanceState)
   ...
+  agoraRtc.enableExtension(
+     BANUBA_PROVIDER_NAME,
+     BANUBA_EXTENSION_NAME,
+     true,
+     Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE
+  )
}

and disable the extension and Agora in onDestroy method

override fun onDestroy() {
   super.onDestroy()

   BanubaExtensionManager.destroy()

+  agoraRtc.enableExtension(
+     BANUBA_PROVIDER_NAME,
+     BANUBA_EXTENSION_NAME,
+     false,
+     Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE
+  )
   
   RtcEngine.destroy()
}

Manage effects

Initialize extension using BanubaExtensionManager.initialize method by providing a number of required params.
Please keep in mind that initialize should be invoked only after starting preview agoraRtc.startPreview().

 // Important - extension works only after start preview
addLocalVideo()
agoraRtc.startPreview()

+BanubaExtensionManager.initialize(
+   applicationContext,
+   BANUBA_LICENSE_TOKEN,
+   agoraRtc
+)

Load effects from assets

Use BanubaExtensionManager.loadEffect method for applying AR effects stored in Android assets/effects folder.

Load effects from path

Use agoraRtc.setExtensionProperty method for applying AR effects stored on device internal storage. Use empty string "" to cancel an effect.

agoraRtc.setExtensionProperty(
   BANUBA_PROVIDER_NAME,  // "Banuba"
   BANUBA_EXTENSION_NAME, // "BanubaFilter"
   "load_effect",
   path_to_effect
)

Finally, it is highly recommended to bind applying AR effects to Android lifecycle events by overriding onResume and onPause methods otherwise current effect might be applying even though the app is in the background and the user is not interacting with it.

     override fun onResume() {
        super.onResume()
        // Required to resume current Banuba Face AR effect
+       BanubaExtensionManager.resume()
     }

    override fun onPause() {
        super.onPause()

        // Required to pause current Banuba Face AR effect i.e. some AR effects might play audio or
        // continue processing something in background. ".pause" method helps to stop processing.
+       BanubaExtensionManager.pause()
    }

Interact with Effect API

Use BanubaExtensionManager.evalJs() method and pass String commands i.e. "Background = require('bnb_js/background'); Background.blur(0.6);" to interact with the effect API. Please visit Effect API guidelines to know more this feature.

Banuba / agora-plugin-filters-android