Audio denoising?

[Jawzper]

I see audio denoising is mentioned in the readme, but it's not clear to me how to actually do this.

Which tool should I be using? I just want to clean up voice clips with background noise so I can use it for cloning.

[0xShug0]

@Jawzper The denoisors rnnoise and zipenhancer are internal libs for model development, and currently are not exposed as runnable models.

[Jawzper]

Oh, ok. It sounds useful, any plans to provide an interface for this in the future?

[0xShug0]

Yes, there is a PR in the review queue (temporarily closed) for this.

[Disonantemus]

I also miss some denoise/enhance/restoration tools that can help to improve ASR from noisy audio recordings (as a previous pass); In the form of another CLI tool (as the PR) or maybe in another modes, like: s2s.

To get rid of noise, reverberation, improve voice, etc.

---

I think this is the PR: #367

The three models there are:

• deepfilternet2
• rnnoise
• zipenhancer

Audio.cpp (available from CLI)

There is AudioSR using s2s mode, but the model is so big (6GB) that I haven't used (too large 4 casual use), I think only upscale to 44/48kHz (no idea) without denoise (maybe I'm wrong).

Audio.cpp (not available from CLI)

In Audio Tools there is GTCRN that should do some denoise.

DeepFilterNet (CLI)

Abandoned 3 years ago, but still working.

• Rikorose/DeepFilterNet

FFmpeg

rnnoise is available to use in ffmpeg as an audio filter, example:

ffmpeg -i in.wav -af 'arnndn=model=./arnndn-models/std.rnnn:mix=0.8' out.wav

ZipEnhancer-rs

zipenhancer is available from this CLI.

More

CrispASR

Audio upscaling and restoration:

• Sidon
• VoxCPM2-vae

sukoon

URL: https://github.com/ruwadgroup/sukoon

has 3 denoise modes (fast and small):

• DeepFilterNet 3
• MDX-Net Kim Vocal 2
• MDX-Net UVR 9482

Example:

sukoon clean --engine hq in.wav

More

• https://github.com/Fovty/HushMic
• https://github.com/Allenmylath/Hush-vani
• https://github.com/DakeQQ/Audio-Denoiser-ONNX

[viebrix]

SAM-Audio from meta can do this and a lot more. But haven't had the time to test it locally. First try didn't work...
https://ai.meta.com/research/samaudio/
You need to register at meta to download it at huggingface:
https://huggingface.co/collections/facebook/sam-audio
On huggingface are also quant.-versions...

[0xShug0]

@viebrix I requested access to the SAM Audio repo. The ONNX version doesn’t have the licensing right, so I’d rather avoid any potential trouble by converting from those weights. It’s Meta, after all.

[0xShug0]

@Jawzper @Disonantemus @viebrix The built-in utilities are now exposed as a special model. https://github.com/0xShug0/audio.cpp/blob/main/docs/audio_tools.md#built-in-audio-utilities

[Disonantemus]

I tried this today, I was able to use the built-in tools, but, in the beginning give me some trouble, it wasn't mentioned in any place that safetensors files were needed, and they were in GitHub in folder assets/framework/audio_utilities.

I think this should be stated in some place, maybe:

• README.md
• docs/audio_tools.md
• another one

Because it's not ready or obvious, needs git clone or download files in a deep folder, for someone it's not a programmer.

[0xShug0]

Fixed. It allows to select model path instead of hardcoded "asset" path.

[0xShug0]

@viebrix Sam-Audio done!

[Jawzper]

It looks like there are a lot of tools there which can only be used from CLI, and only a few of them seem to be available in the GUI. I see voice conversion tools but not denoising tools for example, I guess they only appear in the model library if they fit in the existing studio categories?

Will all tools eventually be available for download and use via the GUI? Maybe there is room for an "other utilities" page in the studio? Or is it expected that the user learns the CLI interface if they want to get anything done with the tools not already shown in GUI?

[0xShug0]

@Jawzper They'll be in the UI. I usually do a UI pass near the release to add all the new models.

[viebrix]

@0xShug0 wow - many thanks for this brilliant enhancement and generally for your fabulous application!