Prerequisites
Audio 2 Face
Audio 2 face can be downloaded in the NVIDIAs Omniverse Launcher that can be found here: Windows Omniverse Launcher Download. Once installed just sign in and you will find the A2F install page in the “Exchange” tab.
Audio 2 Face .usda file
You will need acces to a .usda file to load in the application. You can use the examples given by NVIDIA but ideally you should create your own for each character you want to animate so the animation has the better posible look.
To use an NVIDIAs provided example, search for the one that better suits your needs, open it and save as it with the .usda file extension.
If you want to animate a metahuman with it search for mark_solved_arkit in the examples tab.
Disclaimer
Have in mind that Audio2Face application MUST be runing for this plugin to work. It can be running in headless mode.
Unreal Engine Blueprint Library
Initialize A2F Manager
The implementation is created arround a general manager that ensures the conection with the API stills always the same and stores almost any value you will need when developing to conecxt with this application.
Calling the Create Audio 2 Face Manager function in any blueprint will create this manager. You should store it in a veriable so it dont get garbage collected.
The manager initialization will request the A2F IP and port, by default it assumes you’re launching it in your own machine in the A2F default port. If it’s not the case, here is where you should change it.
Once the manager is initialized you should bind any events that you will want to use in your development, more usual ones are:
- Get Instances
- Get Player Instances
- Get Stream Live Link Nodes
- Get Blend Shape Solvers
Binding is done like usual:
Once binding is done, you have to load the .usda file you should have previously saved. I recomend to save it in the Unreals content folder so you can acces it from Bluepirnts Get Project Content Directory node and append whatever route you decided to save it.
The .usda file usually last a little longer to load after the On Complete notifies so I recomend to hardcode a few seconds delay after it before doing anything related to A2F.
After that, you can request for the instances that the application will need to call almost all the functions.
The events you previouly binded to those functions will be called once the instances are retrieved.
General usage
Accesing instances
Instances can be accesed in three diferent ways:
- Using the Blueprint Node Get Single Instance or Get Single Player Instance
This is the safest way and will only return the instance if there’s a single one of them, that should be the more usual scenario.
- Get the index 0 from the return value of the event binded to the called function
This way you are sure you requested the instances and A2F returned the instances so you can access them. Anyways is recomended to check if the value is valid to avoid accesing invalid memory adresses that would cause the application to crash.
- Get the index 0 of the A2FManager instance variable
Is the only way to acces them if you want to use just pure blueprint nodes and avoid using an axec pin. With this method is also recomended to check for the value validity.
In this example the three methods are showcased but you only have to use one of them
Instances validity will be always checked before trying to acces them from code but anyways is safer to check if them are valid or not before using them as input.
From here everything is pretty straight forward, please remember:
- You must use the A2F manager to call any function related to A2F as the target.
- Almost all the functions will require you to provide one of the instances to call them.
- Store the instances in variables to make them easily accesible and to avoid getting them garbage collected if not used.
Debugging
Tooltips
All the blueprint nodes have a tooltip that explains to you what that node does and what the inputs and outpus mean if you hover the mouse over the node. Check there for explanations about each node usage.
Logging
This library will notice few critical results and print any error to the Unreal Log under the category of Log_Audio2Face, if anything is not working check there for any message that may help you debug what’s happening.
1
Log_Audio2Face: Warning: USD file did not exist in path C:/Users/soram/Documents/GitHub/UE-A2F/Unreal/Content/ThirdParty/mark_solvedarkit.usda
All nodes list
General A2F
Get status
Will make the OnGetStatusCompleted delegate return true if the Audio 2 Face status is up and running.
Load USD
Loads an USD or USDA file to the active A2F stage.
It takes a few seconds and notifies on complete.
| Param | Description |
|---|---|
| FilePath | The absolute path of the file that will be loaded |
| OnComplete | Bind an event that will be called once the loading of the file is completed |
Get Instances
Retrieve all the FullFace Instances to the current A2F scene.
The instances will be returned by the OnGetInstancesCompleted delegate.
Set Frame
Sets the current frame.
| Param | Description |
|---|---|
| Instance | Fullface instance that will be updated |
| frame | The frame that will be set |
| as_timestamp | Set the timestamp (seconds) instead of the frame or not |
Get Frame
Retrieve the current frame.
The frame will be returned by the OnGetFrameCompleted delegate.
| Param | Description |
|---|---|
| Instance | Fullface instance that will be used |
| as_timestamp | Get the timestamp (in seconds) instead of the frame or not |
Player
Get Player Instances
Retrieve all player instances
Also refreshes the manager PlayerInstances property.
The player instances will be returned by the OnGetPlayerInstancesCompleted delegate.
Get Root Path
Retrieve current root path A2F is using to read the .wab files.
Also refreshes the manager RootPath property.
Root path will be returned by the OnGetRootPathCompleted delegate.
| Param | Description |
|---|---|
| PlayerInstance | Player instance from wich the root path will be retrieved |
Set Root Path
Overrides the current root path A2F is using to read the .wab files.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will have the root path overrided |
| NewPath | New path that will be used to read the .wab files from |
Get Tracks
Retrieves the names of all the .wab files in a player instance current root path.
The names are returned in form of string array by the OnGetTracksCompleted delegate.
| Param | Description |
|---|---|
| PlayerInstance | Player instance from wich the tracks in the root path will be retrieved |
Get Current Track
Retrieves the name of the .wab file currently active in a player instance.
| Param | Description |
|---|---|
| PlayerInstance | Player instance from wich the current track will be retrieved |
Set Track
Overrides the currently active track in a player instance.
Expects an audio filename within the root track folder.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will have the track overrided |
| NewTrack | New track name including extension (must be a name in the root path) |
| start_time_range | The starting rime of the track in seconds |
| end_time_range | The end time of the track in seconds (-1 for full lenght) |
Play
Starts playing the active track.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will start playing |
Pause
Pauses the active track.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will stop playing |
Rewind
Sets the current player to rewind mode.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will be set to rewind mode |
Set Looping
Sets the current player to loop or not from now on.
| Param | Description |
|---|---|
| PlayerInstance | Player instance that will loop or not |
| loop_audio | True or false value |
Audio 2 Emotion
Get Emotion Names
Retrieves the names if all available emotion names
The names will be returned by the OnGetEmotionNamesComplete delegate.
Get Emotion
Refreshes the values of the Emotions struct based on a specific frame.
The emotions struct will be returned by the OnGetEmotionComplete delegate.
It also can be accesed from the variable Emotions in the manager.
| Param | Description |
|---|---|
| Instance | Instance that will be used to gather the emotions value |
| frame | Frame that will be used to retrieve the Emotions value |
| as_timestamp | Determines if the values will be gathered using a frame o a value in seconds |
Set Emotion Value
Set a new value for a specific emotion.
Range is between 0 and 1.
| Param | Description |
|---|---|
| Instance | Instance that will have the emotion value overrided |
| EmotionToSet | Emotion that will have it’s value overrided |
| Value | The new motion value (Range between 0 and 1) |
Enable Emotion Streaming
Enables auto emotion when A2F recieves streaming audio.
| Param | Description |
|---|---|
| Instance | Instance that will have the emotion streaming enabled |
| enable | Enable or not the auto emotion while streaming |
Enable Emotion Auto Generate On Track Change
Enables Auto generate emotion values when a new track is selected using the SetNewTrack function
| Param | Description |
|---|---|
| Instance | Instance that will have the auto emotion generate on track change enabled |
| enable | Enable or not the auto emotion while changing track |
Get Emotion Values As Vectors
Returns a float array of the emotion values stored in the Emotions struct.
Use GetEmotion function to refresh it’s values.
Export
Get Blend Shape Solver
Retrieve all current scene blendshape solvers.
Also refreshes the manager BlendshapeSolvers property.
The blendshape solvers will be returned by the OnGetBlendshapeSolversCompleted delegate.
Get Stream Live Link Nodes
Retrieve all current scene Live link nodes.
Also refreshes the manager LivkLinkNodes property.
The live link nodes will be returned by the OnGetLiveLinkNodesCompleted delegate.
Activate Stream Live Link
Turn on or off the live link node streaming.
| Param | Description |
|---|---|
| LiveLinkNode | Live link node that will have the streaming turned on or off |
| activate | Determines if the streaming will be turned on or off |
Export Blendshapes
Export the animation in the selected format.
Existing output files will be overwritten.
| solverNode | Solver node that will be used for exporting |
| export_directory | Absolute directory where the file will be exported |
| file_name | The name the file will have |
| format | Select export format |
| batch | If true export all anims, else export only current anim |
| fps | The ammount of fps the animation will be exported in |
What’s next?
- Engine time module implementation for testing without having to launch the application.
- Latent implementations for GetInstance type functions, get service status and more.