How to realize Chinese recognition of ESP-SR on ArduinoIDE
Sail-211010 opened this issue · comments
Board
ESP32-S3
Device Description
Only on the universal ESP32-S3, the pin is directly connected to the microphone, with no additional connections
Hardware Configuration
NO
Version
v3.0.0
IDE Name
ArduinoIDE
Operating System
Windows
Flash frequency
80
PSRAM enabled
yes
Upload speed
115200
Description
hello
-
I want to implement the use of ESP-SR in Arduino IDE, but I only see the script of generating English phonemes in the sample path. What should I do to generate Chinese corresponding data
Postscript: There has been a proposal to change the current default value from Multinet5 to Multinet7
Postscript: There has been a proposal to change the current default value from Multinet5 to Multinet7
-
At the same time, I would like to ask you how to modify the wake word. It seems that there is no function to directly modify the wake word in the ESP-SR library used on ArduinoIDE, right? What should I do if I want to modify it
Sketch
#include "ESP_I2S.h"
#include "ESP_SR.h"
#define I2S_PIN_BCK 15
#define I2S_PIN_WS 44
#define I2S_PIN_DIN 18
#define LIGHT_PIN 40
#define FAN_PIN 41
I2SClass i2s;
// Generated using the following command:
// python3 tools/gen_sr_commands.py "Turn on the light,Switch on the light;Turn off the light,Switch off the light,Go dark;Start fan;Stop fan"
enum {
SR_CMD_TURN_ON_THE_LIGHT,
SR_CMD_TURN_OFF_THE_LIGHT,
SR_CMD_START_FAN,
SR_CMD_STOP_FAN,
};
static const sr_cmd_t sr_commands[] = {
{0, "Turn on the light", "TkN nN jc LiT"},
{0, "Switch on the light", "SWgp nN jc LiT"},
{1, "Turn off the light", "TkN eF jc LiT"},
{1, "Switch off the light", "SWgp eF jc LiT"},
{1, "Go dark", "Gb DnRK"},
{2, "Start fan", "STnRT FaN"},
{3, "Stop fan", "STnP FaN"},
};
void onSrEvent(sr_event_t event, int command_id, int phrase_id) {
switch (event) {
case SR_EVENT_WAKEWORD: printf("WakeWord Detected!\r\n"); break;
case SR_EVENT_WAKEWORD_CHANNEL:
printf("WakeWord Channel %d Verified!\r\n", command_id);
ESP_SR.setMode(SR_MODE_COMMAND); // Switch to Command detection
break;
case SR_EVENT_TIMEOUT:
printf("Timeout Detected!\r\n");
ESP_SR.setMode(SR_MODE_WAKEWORD); // Switch back to WakeWord detection
break;
case SR_EVENT_COMMAND:
printf("Command %d Detected! %s\r\n", command_id, sr_commands[phrase_id].str);
switch (command_id) {
case SR_CMD_TURN_ON_THE_LIGHT: digitalWrite(LIGHT_PIN, HIGH); break;
case SR_CMD_TURN_OFF_THE_LIGHT: digitalWrite(LIGHT_PIN, LOW); break;
case SR_CMD_START_FAN: digitalWrite(FAN_PIN, HIGH); break;
case SR_CMD_STOP_FAN: digitalWrite(FAN_PIN, LOW); break;
default: printf("Unknown Command!\r\n"); break;
}
ESP_SR.setMode(SR_MODE_COMMAND); // Allow for more commands to be given, before timeout
// ESP_SR.setMode(SR_MODE_WAKEWORD); // Switch back to WakeWord detection
break;
default: printf("Unknown Event!\r\n"); break;
}
}
void setup() {
pinMode(LIGHT_PIN, OUTPUT);
digitalWrite(LIGHT_PIN, LOW);
pinMode(FAN_PIN, OUTPUT);
digitalWrite(FAN_PIN, LOW);
i2s.setPins(I2S_PIN_BCK, I2S_PIN_WS, -1, I2S_PIN_DIN);
i2s.setTimeout(1000);
i2s.begin(I2S_MODE_STD, 16000, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_STEREO);
ESP_SR.onEvent(onSrEvent);
ESP_SR.begin(i2s, sr_commands, sizeof(sr_commands) / sizeof(sr_cmd_t), SR_CHANNELS_STEREO, SR_MODE_WAKEWORD);
// ESP_SR.begin(i2s, sr_commands, sizeof(sr_commands) / sizeof(sr_cmd_t), SR_CHANNELS_MONO, SR_MODE_WAKEWORD);
}
void loop() {}Debug Message
NO
Other Steps to Reproduce
NO
I have checked existing issues, online documentation and the Troubleshooting Guide
- I confirm I have checked existing issues, online documentation and Troubleshooting guide.
In Arduino we support only the English variant. You can try the custom libs route or use Arduino as IDF component in order to gain the full functionality
Will there be further support in this regard
There is a second question, please help me reply, is how to switch the wake word
The wakeword is set here: https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP_SR/src/esp32-hal-sr.c#L344
Keep in mind that "hiesp" is the only enabled wakeword in Arduino
Do I have to modify the library files? Are there any formatting requirements here
You mean, I can't change the wake word in Arduino？
No. Only "Hi ESP" and English are currently supported
If I want to achieve Chinese recognition, do I just modify the transliteration data?（Refers to commands rather than wake-up words）
I am not sure if that would work. You can give it a shot. Your best bet is to recompile the libs to support Chinese instead of English https://docs.espressif.com/projects/arduino-esp32/en/latest/lib_builder.html
Looking at the link content, the operation needs a lot of steps. First, I will try to change the corresponding syllables of transliteration to try the effect, is it convenient to provide information in this respect, and can you list the conversion results of one or two Chinese characters (give one or two examples so that I can compare whether the results I modified are correct)?
Similar to
If I want to achieve Chinese recognition, do I just modify the transliteration data?（Refers to commands rather than wake-up words）
And I need to know how this relationship is derived, such as AE1/OW1, etc.
I suggest you ask all those questions here: https://github.com/espressif/esp-sr