FFT inestable on RXTX_MODE for Autotune project

[acien101]

Problem Description

Hi!

I have been testing the library with great results on ESP32S3 and the audio code SGTL5000. I am preparing a project for doing Autotune, that is correcting the pitch of a sound when the note is off.

The audio diagram would be something like:

                                                 -> FFT
I2S_INPUT -> SPLITTER :<
                                                -> PitchShift -> I2S_OUTPUT

The PitchShift I am using a Faust DSP block, that works great by itself. It gives more stable readings than the builtin library Pitch effect.

As a test I put a sin signal generator on the audio input, but the FFT starts being inestable, as is reading the processed signal and not the input signal. Causing a inestable tone at the output.

I have tested also AudioESP32FFT, that gives the same results. I tried to put some BufferedStream following a diagram similar to the next, but gives also the same results.

I have been trying to understand the problem, and seems that FFT is analyzing the output signal, and not the input signal, or a mix of the two of them, causing a inestable reading of the frequency.

Any thoughts about how to fix it?

I am pretty impressed with the library. With 100lines I can do FFT, Pitch, Mixing, etc. Its pretty impressive.

Device Description

ESP32S3 with SGTL5000 audio codec.

Sketch

#define USE_MEMORY_MANAGER

#include <Arduino.h>
#include "AudioTools.h"
#include "AudioControlSGTL5000.h"
#include "AudioTools/AudioLibs/AudioRealFFT.h"
#include "AudioTools/AudioLibs/AudioFaust.h"
#include "Pitcher.h"

#define I2S_BCLK   3   // Bit clock
#define I2S_LRC    6   // Word select / LR clock
#define I2S_DOUT   4   // Data out
#define I2S_DIN    5   // Data in (not used here)
#define I2S_MCLK   0   // Master clock (optional)

#define SAMPLE_RATE 44100
#define BIT_DEPTH   16

AudioControlSGTL5000 audio;

// Create i2S stream as audio sink and source
I2SStream i2s_stream;

MultiOutput splitter;

AudioRealFFT fft; // or AudioKissFFT
FaustStream<mydsp> faust(i2s_stream);

StreamCopy copier(splitter, i2s_stream);    // (To, From)

// Audio info: sample rate, channels, bit depth
AudioInfo info(SAMPLE_RATE, 2, BIT_DEPTH);

// display fft result
void fftResult(AudioFFTBase &fft){
    float diff;
    auto result = fft.result();
    if (result.magnitude>100){
        Serial.print(result.frequency);
        Serial.print(" ");
        Serial.print(result.magnitude);  
        Serial.print(" => ");
        Serial.print(result.frequencyAsNote(diff));
        Serial.print( " diff: ");
        Serial.println(diff);

        faust.setLabelValue("shift_semitones", - diff / 10);
    }
}

void setup() {
  Serial.begin(115200);

  // I2S config
  auto cfg_i2s_in = i2s_stream.defaultConfig(RXTX_MODE);
  cfg_i2s_in.sample_rate = SAMPLE_RATE;
  cfg_i2s_in.bits_per_sample = BIT_DEPTH;
  cfg_i2s_in.channels = 2;
  cfg_i2s_in.pin_bck = I2S_BCLK;
  cfg_i2s_in.pin_ws = I2S_LRC;
  cfg_i2s_in.pin_data = I2S_DOUT;
  cfg_i2s_in.pin_data_rx = I2S_DIN;
  cfg_i2s_in.pin_mck = I2S_MCLK;
  cfg_i2s_in.use_apll = true;

  i2s_stream.begin(cfg_i2s_in);

  // Setup Faust
  auto faust_cfg = faust.defaultConfig();
  faust.begin(faust_cfg);
  faust.setLabelValue("shift_semitones", 1);

  // setup multi output
  splitter.add(faust);
  splitter.add(fft);

  if(!audio.enable()){
    Serial.println("Something wrong happened");
  }
  audio.volume(0.5);

  // Setup FFT
  auto tcfg = fft.defaultConfig();
  tcfg.length = 8192;
  tcfg.channels = 2;
  tcfg.sample_rate = SAMPLE_RATE;
  tcfg.bits_per_sample = BIT_DEPTH;
  tcfg.callback = &fftResult;
  fft.begin(tcfg);
}

void loop() {
  // Generate and write audio frames to I2S
  copier.copy();
}

Other Steps to Reproduce

No response

What is your development environment (incl. core version info)

Platform IO 3.3.4

[env:esp32-s3]
platform = espressif32
board = esp32-s3-devkitc-1-n16r8v
framework = arduino
board_build.arduino.memory_type = qio_opi
board_build.flash_mode = qio
board_build.prsam_type = opi
board_upload.flash_size = 16MB
board_upload.maximum_size = 16777216
board_build.f_cpu = 240000000L 
build_flags =
    -DBOARD_HAS_PSRAM
    -mfix-esp32-psram-cache-issue
    -DCORE_DEBUG_LEVEL=5 
    -Wno-unused-variable
    -Wno-unused-but-set-variable
    -Wno-unused-function
    -Wno-format-extra-args 
    ;-Wl,-Map,output.map

I have checked existing issues, discussions and online documentation

• I confirm I have checked existing issues, discussions and online documentation

[pschatzmann]

Maybe the issue comes from the delays: so try to minimize the delays in the processing chain:

• minimize the i2s buffer
• try to define a stride to calculate the fft more often / decrease the fft length to speed up the calculation time
• check if the processing speed of AudioEspressifFFT gives faster calculation times
• remove the Serial prints
• try to reduce the sample rate