Using with Unity
To install OpenVoiceSharp via NuGet, check here.
Once that's done, you will notice that you can now use OpenVoiceSharp. However, you will also notice upon running that your game/app will crash due to the fact the libraries are not in the folder.
First of, install the OpenVoiceSharp package on your MonoGame project.
To link the OpenVoiceSharp DLL files, create a dependency (linker) project and copy the required DLLs in the project's folder, in a folder names Plugins.
Then, for each required DLL file, make sure to set the compatibility to Intel-64 Windows x64.
This will be required to be done every time you update OpenVoiceSharp or for each module (OpusDotNet, RNNoise, WebRtcVad, etc...)
For more details on how to install it on your project, see the Setting Up page from Facepunch's website.
Note
For simplicity's sake, I used the Steamworks SDK for the networking via the Facepunch.Steamworks package. The following behavior and networking system can of course be customized according to your needs. To link the DLLs click here. Use the Steamworks 1.48a version of the SDK and the test steam app id 480 (Spacewar).
Now that you've linked the DLL files, lets start programming!
When it comes to recording microphone input, Unity makes it pretty difficult.
By default, Unity provides the Microphone class that allows you to natively record the microphone of the end user independently for all platforms. However, the API provides you with an unprompted buffer and relies on the main thread that gives an output of float32 array from -1.0f to 1.0f, at an unprompted rate meaning the audio microphone buffer will accumulate samples until GetData() is called meaning that if you have any kind of lag of overflow of samples from the input buffer, you will have to use something like a circular buffer in order to get the data properly & send the extra-buffers that accumulated over-time aswell. Aswell, resampling will be required, as the input sample rate depends on the sample rate of the input device.
As a reminder, if you decide to use Unity's native API, make sure to use the right format and to convert to 16-bit PCM and to be cautious of memory usage.
However, you can (natively to Windows) use the provided BasicMicrophoneRecorder class to do a better and stable job on another thread, meaning that if there is any kind of performance loss on the main thread, the data will not be corrupted (except if you send the data via the main thread). And so, for simplicity's sake, I've preferred (and recommend you) to use BasicMicrophoneRecorder.
Note
As mentioned before, for the networking, I've decided to use Facepunch.Steamworks (for Steam-based multiplayer to handle networking. Of course, not much has to be changed in your project as you really will have to change (depending on the way you send packets) is to send binary over the network, which should be supported by most networking protocols/libraries you can use on Unity.
The following samples are going to come from VoiceChatManager.cs.
// microphone rec
MicrophoneRecorder.DataAvailable += (pcmData, length) => {
// if not connected or not talking, ignore
if (!Connected) return;
if (!VoiceChatInterface.IsSpeaking(pcmData)) return;
// encode the audio data and apply noise suppression.
(byte[] encodedData, int encodedLength) = VoiceChatInterface.SubmitAudioData(pcmData, length);
// send packet to everyone (P2P)
foreach (SteamId steamId in VoiceBuffers.Keys)
SteamNetworking.SendP2PPacket(steamId, encodedData, encodedLength, 0, P2PSend.Reliable);
};
MicrophoneRecorder.StartRecording();You can replace SteamNetworking.SendP2PPacket by the method/way of your choice to send packets in the network and handle your way differently.
Tip
Like I say for any other engine or app, I highly advise you use another thread than the main thread to decode packets for performance. While OpenVoiceSharp overall is very light on the CPU, if you have optimization in mind or its crucial (like if your lobby has tons of players), it can be beneficial to look at decoding the voice data in another dedicated thread. But because this is meant to be a barebones boilerplate for you to implement, this should run relatively fine on the main thread and should be fast enough for all CPUs.
Because Unity doesn't support to natively read an unrestricted amount of PCM samples on demand (you need a filter for stuff like this and accumulate blank samples and it requires at least 4096 PCM samples sadly), you are going to need to use a circular buffer to accumulate the incoming data and once the buffer is full replay them. This adds latency sadly but if the latency is too low, it can cause audio cracking, since you have to constantly replay an audio clip, speaking of which will be used to play back our samples. Any extra samples are disregarded until the available samples are all read (consumed).
I create an AudioSource & AudioClip that will contain an empty clip and use a new circular buffer in which I will store the data later on at the recommended amount (Make sure to change the channels if you play back in mono, the sample rate should stay as 48000Hz) then, I store in a dictionary where I store the players and so on (you can handle that yourself with your own class or system).
// create audio clip
AudioSource voiceSource = GetComponentFromProfile<AudioSource>(profile, "VoiceSource");
// allows us to push voice data & read it when needed
CircularAudioBuffer<float> buffer = new(BufferSamples, RecommendedChunkAmount.Unity);
VoiceBuffers.Add(friend.Id, buffer);
AudioClip audioClip = AudioClip.Create(
"Voice",
buffer.BufferLength, // RECOMMENDED BUFFER AMOUNT
2,
48000,
false
// i do not use pcm read callback as it requires 4096 samples
// bad for latency and memory usage
);
voiceSource.clip = audioClip;Here, we create the audio playbacks to be ready to supply our incoming PCM data.
void HandleMessageFrom(SteamId steamid, byte[] data)
{
if (!AllowLoopback && steamid == SteamClient.SteamId) return;
if (!VoiceBuffers.ContainsKey(steamid))
return;
// decode and convert to float32
(byte[] decodedData, int decodedLength) = VoiceChatInterface.WhenDataReceived(data, data.Length);
float[] samples = new float[decodedLength / 2];
VoiceUtilities.Convert16BitToFloat(decodedData, samples);
// push to circular voice buffer
var circularBuffer = VoiceBuffers[steamid];
circularBuffer.PushChunk(samples);
VoiceBuffers[steamid] = circularBuffer;
}Unity requires float32 PCM data, meaning we will have to do a conversion when decoding our packets, then after converting from 16 bit PCM to float32, we will have to push that chunk data into our circular buffer, which will hold that data and wait for it to be read (consumed).
And then, in the main thread, we loop the players individually to read the samples and play them back.
// handle voice buffers reading (I recommend you thread this!!)
SteamId steamId;
for (int i = 0; i < keys.Length; i++)
{
steamId = keys[i];
// create all previous profiles
if (!VoiceBuffers.ContainsKey(steamId)) continue;
// check if full
CircularAudioBuffer<float> voiceBuffer = VoiceBuffers[steamId];
if (!voiceBuffer.BufferFull) continue;
// submit to audio clip and play
AudioSource voiceSource = GetComponentFromProfile<AudioSource>(GetProfile(steamId), "VoiceSource");
// read all data and play
voiceSource.clip.SetData(voiceBuffer.ReadAllBuffer(), 0);
voiceSource.Play();
VoiceBuffers[steamId] = voiceBuffer;
}Tip
Do not hesitate to look at the VoiceChatManager.cs file if you feel lost or you need to use the code.
And that's it!
soon
Demo using a soundboard, but voice can be used aswell.
There you go! You should now be able to have fully integrated OpenVoiceSharp in no time. Remember to check the barebone and example project in case you're lost here.