An Auto-encoder for audio semantic communication, based on wav2vec at the platfrom of facebook's Fairseq. The Fairseq code is originally from https://github.com/pytorch/fairseq

中文描述

要新下载模型
需要新建数据集分割列表 manifest 文件
更改了 libri_label.py 数据库
注意安装fairseq 的版本，采用20201031的版本可以跑通
文件备注 -- Harlan_Thomas/wav2vec 文件用于修改代码并且调参数 -- semantic_commun 部分记录原先已经小修的代码 -- fairseq 已经git pull 代表最新 -- fairseq_20201031 10月31日版本代码
ASR系统参考 -- install OpenCL

Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks. We provide reference implementations of various sequence modeling papers:

Requirements and Installation

PyTorch version >= 1.4.0
Python version >= 3.6
For training new models, you'll also need an NVIDIA GPU and NCCL
To install fairseq and develop locally:

git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

# on MacOS:
# CFLAGS="-stdlib=libc++" pip install --editable ./

For faster training install NVIDIA's apex library:

git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \
  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \
  --global-option="--fast_multihead_attn" ./

For large datasets install PyArrow: pip install pyarrow
If you use Docker make sure to increase the shared memory size either with --ipc=host or --shm-size as command line options to nvidia-docker run.

Getting Started

The full documentation contains instructions for getting started, training new models and extending fairseq with new model types and tasks.

Pre-trained models and examples

wav2vec: Unsupervised Pre-training for Speech Recognition (Schneider et al., 2019)

Semantic communication Auto-encoder

Train a model

python train.py ./manifest/dev-other --save-dir ./w2v_thn/Rice_decoder/0623_tanh_tdiv2_feat8_6dB_Rice0_8_NR_redo --num-workers 1 --fp16 --max-update 40000 --save-interval 1 --no-epoch-checkpoints \
--arch w2vAutoEncoder --task decoder_training --lr 1e-05 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine \
--conv-feature-layers '[(8, 1, 1), (8, 2, 1), (8, 4, 2), (1, 1, 1)]' \
--conv-aggregator-layers '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--conv-feature-layers_en '[(8, 1, 1), (8, 2, 1), (8, 4, 2)]' \
--conv-aggregator-layers_en '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--skip-connections-agg --residual-scale 0.5  --warmup-updates 500 --warmup-init-lr 1e-03 --criterion wav2vecAuto_ecoder --num-negatives 10 \
--max-sample-size  51200 --max-tokens 1000000 --skip-invalid-size-inputs-valid-test > ./decoder_log_0623_tanh_tdiv2_feat8_6dB_Rice0_8_NR_redo 2>&1 &

Train a Federated Learning model, change into "--task FL_training"

python train.py ./manifest/FL_1_dev_other --save-dir ./w2v_thn/FL_AutoEncoder/0801_tdiv1_feat8_6dB_NR_10eFL1 --num-workers 1 --fp16 --max-update 20000 --save-interval 1 --no-epoch-checkpoints \
--arch w2vAutoEncoder --task FL_training --lr 1e-05 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine \
--conv-feature-layers '[(8, 1, 1), (8, 2, 1), (8, 4, 1), (1, 1, 1)]' \
--conv-aggregator-layers '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--conv-feature-layers_en '[(8, 1, 1), (8, 2, 1), (8, 4, 1)]' \
--conv-aggregator-layers_en '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--skip-connections-agg --residual-scale 0.5  --warmup-updates 500 --warmup-init-lr 1e-03 --criterion wav2vecAuto_ecoder_FL --num-negatives 10 \
--max-sample-size  51200 --max-tokens 1000000 --skip-invalid-size-inputs-valid-test > ./FL_log_0801_tdiv1_feat8_6dB_NR_10eFL1_2 2>&1 &

python train.py ./manifest/FL_2_dev_clean --save-dir ./w2v_thn/FL_AutoEncoder/0801_tdiv1_feat8_6dB_NR_10eFL2 --num-workers 1 --fp16 --max-update 20000 --save-interval 1 --no-epoch-checkpoints \
--arch w2vAutoEncoder --task FL_training --lr 1e-05 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine \
--conv-feature-layers '[(8, 1, 1), (8, 2, 1), (8, 4, 1), (1, 1, 1)]' \
--conv-aggregator-layers '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--conv-feature-layers_en '[(8, 1, 1), (8, 2, 1), (8, 4, 1)]' \
--conv-aggregator-layers_en '[(8, 2, 1), (8, 4, 1), (8, 8, 1), (8, 16, 1)]' \
--skip-connections-agg --residual-scale 0.5  --warmup-updates 500 --warmup-init-lr 1e-03 --criterion wav2vecAuto_ecoder_FL --num-negatives 10 \
--max-sample-size  51200 --max-tokens 1000000 --skip-invalid-size-inputs-valid-test > ./FL_log_0801_tdiv1_feat8_6dB_NR_10eFL2_2 2>&1 &

Inference the model: see "xxx_inference.ipynb"

Fairseq community

Twitter: https://twitter.com/fairseq
Facebook page: https://www.facebook.com/groups/fairseq.users
Google group: https://groups.google.com/forum/#!forum/fairseq-users

License

fairseq(-py) is MIT-licensed. The license applies to the pre-trained models as well.

Citation

Please cite as:

@inproceedings{ott2019fairseq,
  title = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  author = {Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},
  booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
  year = {2019},
}

HarlanThomas / wave2vec