Processing

This guide explains how to add a custom Audio Processing component with MimiCore inside MimiSDK, providing your users with Mimified audio.

Introduction

The Android MimiSDK 5.0.0 release brings many changes, of particular significance is the redefining of the Processing APIs.

While some names might sound familiar, quite a lot has changed, so please read this guide carefully.

Tip: As with the MimiSDK in general, the Processing APIs make heavy use of Kotlin Coroutines. We recommend having a basic understanding of their concepts.

Goals

We started with a number of goals to improve the experience for partners using the Processing APIs:

• Provide easily accessible and transparent exposure of state, including for failures.
• Reduce overall parameter API complexity.
• Provide simpler high-level abstractions.
• Improve testability through distinct abstraction layers.

Architecture

The new Processing APIs can be divided into four main components:

• Processing Controller - The root entry point and context for Processing API access.
• Processing Session - The provider of access to processing Processing Parameters when activated.
• Processing Parameters - Individual parameter objects that provides state, data and mutation.
• Processing Applicators - Responsible for applying the Processing Parameter values to external systems.

System Overview

The following section describes the components in the Processing APIs.

Figure: Processing System Components

ProcessingController

Overview

The ProcessingController is a controller owned and initialized by MimiCore.

It responsible for the activation and deactivation of Processing Sessions, and therefore can be used as the source of truth for the active ProcessingSession instance.

Responsibilities

• Centralized location for accessing Processing APIs.
• Owner of active Processing Session
• Provides activation and deactivation of Processing Session.

Activation

As the ProcessingController owns the ProcessingSession status; it is the location where processing can be "activated" or "deactivated".

Note: Activation refers to the ability for parameters values to be transferred, not whether the physical processing effect is enabled or disabled.

Once there is an activated ProcessingSession, it is possible to access the ProcessingParameters and configure them appropriately for use.

Lifecycle

The ProcessingController is responsible for handling the ProcessingSession lifecycle in terms of activation and deactivation.

• Activating a session will set the activeSession property on the ProcessingController allowing the session to be easily referenced.
• Only a single session can be activated at one time.
• Deactivation clears the activeSession property. It does not directly affect the applied Mimi processing.
• Activating and deactivating a session triggers MimiObserver notifications to subscribed observers.

Tip: Most partner integrations will only need to use a single, long-lived ProcessingSession that roughly aligns with the application lifecycle.

ProcessingSession

Provides access to Processing Parameters.

Overview

As described in the previous section, a ProcessingSession is owned by a ProcessingController, which is responsible for handling the session lifecycle in terms of activation/deactivation.

The ProcessingSession itself owns the ProcessingParameters, and therefore is the gateway to parameter access.

Responsibilities

• Owner of all Processing Parameters, providing access to; isEnabled, intensity and preset.
• Source of truth for fitting which provides required Fitting information for Preset acquisition.
• Provides the ability for processing to be "interrupted" and automatically disabled due to external events (such as the hearing test launching).

Upon activation, the ProcessingSession will be initialized with all the parameters and properties it requires (such as fitting and data source objects). The previously applied parameter values are restored to their respective parameters. At this point the parameters are ready for use.

Processing Parameter

The focal point for state, application and data.

Overview

A ProcessingParameter represents an individual parameter that can be used to control Mimi processing functionality. It encompasses a value that can be mutated and applied asynchronously to subscribed components called Applicators.

Responsibilities

In general terms, a ProcessingParameter is responsible for:

• Providing access to the current parameter value.
• Applying values to a collection of subscribed Applicators using the value application sequence.
• Fetching values from remote data sources asynchronously (such as preset data).
• Providing state and value updates to a collection of subscribed observers.

Types of ProcessingParameters

To implement these behaviors on Android, there are two variants of ProcessingParameter:

• The MimiProcessingParameter whose value is updated via a synchronous value.
• The MimiFetchedProcessingParameter whose value is fetched from an asynchronously data source.

MimiSDK ProcessingParameter instances

The ProcessingSession defines the following ProcessingParameter instances to provide access and control of the Mimi processing:

• isEnabled - Whether Mimi processing is enabled or disabled. It is a MimiProcessingParameter.
• intensity - Intensity of Mimi processing. It is a MimiProcessingParameter.
• preset - Preset data model for use by a Mimi processor. It is a MimiFetchedProcessingParameter.

Parameter value application sequence summary

The internal process of updating a ProcessingParameter value or transferring the existing value to Applicators follows a specific sequence as executed by the ProcessingParameter:

1. The request is passed through the Delivery Mode handler.
2. The Applicators are iterated through, asked to verify that they are capable of applying the value via their canApply function.
3. If any Applicator fails canApply verification (by returning false), the value application sequence is abandoned with an error and subscribers are notified of the failure.
4. Applicators are requested to apply the new value.
5. If any Applicator fails to apply the value (by throwing an Exception), the application sequence is abandoned with an error and subscribers are notified of the failure.
6. If successful, value is updated and Observers are notified of the new value.
7. A return result value is given for the request.

Note: If no Applicators are present - then value is immediately updated and the result is successful.

Important: A given ProcessingParameter executes each value application sequence sequentially - it does not perform concurrent updates.

Delivery Mode

The Delivery Mode of a ProcessingParameter describes how it handles requests to update the parameter value.

There are currently two Delivery Mode strategies:

• Continuous performs each update sequentially, in the order they were requested.
• Discrete debounces requests so it only executes the value application sequence once the request value settles for the given interval without updates.

Note: They are defined per Processing Parameter. Previously, a similar concept was defined across the entire Processing system.

Applicator

Arguably the most relevant Processing API component for most integrators is the Applicator.

An Applicator is an accessory of a ProcessingParameter that is capable of performing the value application logic for its associated ProcessingParameter. It is roughly equivalent to the previous ProcessingHandler concept.

In simple terms, its role is apply ProcessingParameter values to an external system. When there is a request to update a ProcessingParameter value, its Applicators are requested to apply this value.

While an Applicator appears quite similar to an Observer, the two components have two distinct roles within the Processing API:

• An Applicator is responsible for the value application logic of a ProcessingParameter - the result of which influences the ProcessingParameter state.
• An Observer is only informed of events reflecting committed changes to the ProcessingParameter state; it cannot influence the ProcessingParameter state.

In the Android MimiSDK, the concept of the Applicator is represented by the MimiParameterApplicator type.

Responsibilities

• Provides a canApply function which can synchronously verify whether a requested value can be set on the Applicator.
• Provides an apply function which will allow the applicator to asynchronously apply a value to an external source.
• Optionally provides a value to be applied to other Applicators ("reverse synchronize").
• Ability to be removed from a ProcessingParameter.
• Ability to set a “Delivery Timeout” to specify a time duration which if exceeded is considered erroneous and will cause a value application sequence to fail.

Building an integration

This section highlights some of the key APIs used when integrating your processing system with the MimiSDK.

Initialization and Activation

Once you have already initialized MimiCore, you can access the Processing APIs from the ProcessingController and activate a ProcessingSession to add Applicators and modify ProcessingParameter values.

When activating a ProcessingSession, you need to provide a Fitting value.

The Fitting model provides data about the current processing environment and in turn how presets should be generated.

Example:

val processingController = MimiCore.processingController
val session = processingController.activateSession(Fitting(techLevel = 3)) // an example techLevel, yours will be defined by your Processing system

ProcessingParameter Operations

ProcessingParameter Value Operations

The following operations are provided on the ProcessingParameter and execute the value application sequence and return a ProcessingParameterResult indicating whether operation was successful.

Reading the ProcessingParameter state

Example value:

val activeSession : ProcessingSession = requireNotNull(MimiCore.processingController.activeSession.state)
val isEnabled : Boolean = activeSession.isEnabled.value 

Example observe:

// Acquire the active ProcessSession (assumes already activated!)
val activeSession : ProcessingSession = requireNotNull(MimiCore.processingController.activeSession.state)

// Observer for a MimiProcessingParameter, for example: isEnabled
activeSession.isEnabled.observe(
    applying = { current, applying -> 
        // add your code here to handle when a value application is in progress
    },
    failed = { current, failedToApply, exception -> 
        // add your code here to handle when a value application fails
    },
    ready = { current ->
        // add your code here to handle when value application succeeds
    }
)

// Observer for a MimiFetchedProcessingParameter, for example: preset
activeSession.preset.observe(
    fetching = { current ->
        // add your code here to handle when a value application is fetching from the data source
    },
    applying = { current, applying ->
        // add your code here to handle when a value application is in progress
    },
    failed = { current, failedToApply, exception ->
        // add your code here to handle when a value application fails
    },
    ready = { current ->
        // add your code here to handle when value application succeeds
    }
)

Adding an Applicator to a ProcessingParameter

There are two ways to define and add an Applicator. Each support different integration use cases depending on your needs.

The primary addApplicator function adds an "out-of-date" Applicator.

public fun addApplicator(
    canApply: (value: T) -> Boolean,
    apply: suspend (value: T) -> Unit,
    applyTimeoutMillis: Long
) : MimiParameterApplicator

Note: When adding an Applicator using this function, the Applicator will not automatically receive the current value and likewise; it has no influence on the current ProcessingParameter value.

The secondary addApplicator function adds an Applicator and automatically attempts to update the ProcessingParameter with the supplied value. The result of the update is returned as a ProcessingParameterResult.

public suspend fun addApplicator(
    value: T,
    canApply: (value: T) -> Boolean,
    apply: suspend (value: T) -> Unit,
    applyTimeoutMillis: Long
) : Pair<MimiParameterApplicator, ProcessingParameterResult>

In both cases, the Applicator is returned as a MimiParameterApplicator. This reference should be retained by the caller.

Defining your own Applicator

When creating an Applicator, we recommend delegating your canApply and apply functions to a class containing your custom processing logic. This custom processing logic depends entirely on your processing system. Generally, this approach helps make your code more modular and testable.

Note: This is simplified sample code to demonstrate the general sequence and may not reflect the best structure for your particular usecase.

Example: isEnabled Applicator

// Encapsulate your custom processing logic (example: isEnabled)
internal class CustomProcessingLogic {
    fun canApply(value: Boolean) : Boolean {
        TODO("Implement your logic to determine if your processing system is able to apply the given value")
    }

    suspend fun apply(value: Boolean) {
        TODO("Implement your logic to apply the given value in your processing system")
    }
}

// Declare an instance; depending on your usecase, you may want this to be a singleton.
val customProcessingLogic = CustomProcessingLogic()

// TODO - You should use an appropriate value for your integration.
val customApplyTimeoutMillis: Long = 10_000

// Access the relevant ProcessingParameter from the active ProcessingSession (assumes a ProcessingSession has been activated)
val isEnabledProcessingParameter =
    requireNotNull(MimiCore.processingController.activeSession.state).isEnabled

// Add the Applicator, delegating the calls to your custom processing logic instance.
val applicator = isEnabledProcessingParameter.addApplicator(
    canApply = customProcessingLogic::canApply,
    apply = customProcessingLogic::apply,
    applyTimeoutMillis = customApplyTimeoutMillis
)

// Cause the isEnabled ProcessingParameter to push its current value to the newly added Applicator
isEnabledProcessingParameter.synchronize()

Retain the applicator reference so that you can later remove it from the ProcessingParameter. Once an Applicator has been removed, it will no longer receive updates from the ProcessingParameter.

Example: Removing Applicator

// Removes the Applicator from its ProcessingParameter so it won't receive further updates.
applicator.remove()