开源软件名称：

开源软件地址：

开源软件介绍：

Lingvo

What is it?

Lingvo is a framework for building neural networks in Tensorflow, particularlysequence models.

A list of publications using Lingvo can be found here.

Table of Contents

• Releases
  • Major breaking changes
• Quick start
  • Installation
  • Running the MNIST image model
  • Running the machine translation model
  • Running the GShard transformer based giant language model
  • Running the 3d object detection model
• Models
  • Automatic Speech Recognition
  • Car
  • Image
  • Language Modelling
  • Machine Translation
• References
• License

Releases

Major breaking changes

NOTE: this is not a comprehensive list. Lingvo releases do not offer anyguarantees regarding backwards compatibility.

HEAD

Nothing here.

0.11.0

• General
  • Tensorflow 2.7 is now the required version.
  • Python 3.6 support has been removed.

0.10.0

• General
  • Tensorflow 2.6 is now the required version.
  • The theta_fn arg to CreateVariable() has been removed.

0.9.1

• General
  • Python 3.9 is now supported.
  • ops.beam_search_step now takes and returns an additional argbeam_done.
  • The namedtuple beam_search_helper.BeamSearchDecodeOutput now removes thefield done_hyps.

0.9.0

• General
  • Tensorflow 2.5 is now the required version.
  • Python 3.5 support has been removed.
  • py_utils.AddGlobalVN and py_utils.AddPerStepVN have been combined intopy_utils.AddVN.
  • BaseSchedule().Value() no longer takes a step arg.
  • Classes deriving from BaseSchedule should implement Value() not FProp().
  • theta.global_step has been removed in favor of py_utils.GetGlobalStep().
  • py_utils.GenerateStepSeedPair() no longer takes a global_step arg.
  • PostTrainingStepUpdate() no longer takes a global_step arg.
  • The fatal_errors argument to custom input ops now takes error messagesubstrings rather than integer error codes.

Quick start

Installation

There are two ways to set up Lingvo: installing a fixed version through pip, orcloning the repository and building it with bazel. Docker configurations areprovided for each case.

If you would just like to use the framework as-is, it is easiest to just installit through pip. This makes it possible to develop and train custom models usinga frozen version of the Lingvo framework. However, it is difficult to modify theframework code or implement new custom ops.

If you would like to develop the framework further and potentially contributepull requests, you should avoid using pip and clone the repository instead.

pip:

The Lingvo pip package can be installed withpip3 install lingvo.

See thecodelabfor how to get started with the pip package.

From sources:

The prerequisites are:

• a TensorFlow 2.7 installation,
• a C++ compiler (only g++ 7.3 is officially supported), and
• the bazel build system.

Refer to docker/dev.dockerfile for a set of workingrequirements.

git clone the repository, then use bazel to build and run targets directly.The python -m module commands in the codelab need to be mapped onto bazel run commands.

docker:

Docker configurations are available for both situations. Instructions can befound in the comments on the top of each file.

• lib.dockerfile has the Lingvo pip packagepreinstalled.
• dev.dockerfile can be used to build Lingvo fromsources.

How to install docker.

Running the MNIST image model

Preparing the input data

pip:

mkdir -p /tmp/mnistpython3 -m lingvo.tools.keras2ckpt --dataset=mnist

bazel:

mkdir -p /tmp/mnistbazel run -c opt //lingvo/tools:keras2ckpt -- --dataset=mnist

The following files will be created in /tmp/mnist:

• mnist.data-00000-of-00001: 53MB.
• mnist.index: 241 bytes.

Running the model

pip:

cd /tmp/mnistcurl -O https://raw.githubusercontent.com/tensorflow/lingvo/master/lingvo/tasks/image/params/mnist.pypython3 -m lingvo.trainer --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

(cpu) bazel build -c opt //lingvo:trainer(gpu) bazel build -c opt --config=cuda //lingvo:trainerbazel-bin/lingvo/trainer --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

After about 20 seconds, the loss should drop below 0.3 and a checkpoint will besaved, like below. Kill the trainer with Ctrl+C.

trainer.py:518] step:   205, steps/sec: 11.64 ... loss:0.25747201 ...checkpointer.py:115] Save checkpointcheckpointer.py:117] Save checkpoint done: /tmp/mnist/log/train/ckpt-00000205

Some artifacts will be produced in /tmp/mnist/log/control:

• params.txt: hyper-parameters.
• model_analysis.txt: model sizes for each layer.
• train.pbtxt: the training tf.GraphDef.
• events.*: a tensorboard events file.

As well as in /tmp/mnist/log/train:

• checkpoint: a text file containing information about the checkpoint files.
• ckpt-*: the checkpoint files.

Now, let's evaluate the model on the "Test" dataset. In the normal trainingsetup the trainer and evaler should be run at the same time as two separateprocesses.

pip:

python3 -m lingvo.trainer --job=evaler_test --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

bazel-bin/lingvo/trainer --job=evaler_test --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

Kill the job with Ctrl+C when it starts waiting for a new checkpoint.

base_runner.py:177] No new check point is found: /tmp/mnist/log/train/ckpt-00000205

The evaluation accuracy can be found slightly earlier in the logs.

base_runner.py:111] eval_test: step:   205, acc5: 0.99775392, accuracy: 0.94150388, ..., loss: 0.20770954, ...

Running the machine translation model

To run a more elaborate model, you'll need a cluster with GPUs. Please refer tothird_party/py/lingvo/tasks/mt/README.mdfor more information.

Running the GShard transformer based giant language model

To train a GShard language model with one trillion parameters on GCP usingCloudTPUs v3-512 using 512-way model parallelism, please refer tothird_party/py/lingvo/tasks/lm/README.mdfor more information.

Running the 3d object detection model

To run the StarNet model using CloudTPUs on GCP, please refer tothird_party/py/lingvo/tasks/car/README.md.

Models

Automatic Speech Recognition

• Listen, Attend and Spell.
  William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. ICASSP 2016.

  End-to-end Continuous Speech Recognition using Attention-based RecurrentNN: First Results.
  Jan Chorowski,Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. arXiv 2014.

  • asr.librispeech.Librispeech960Grapheme
  • asr.librispeech.Librispeech960Wpm

Car

• DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection.
  Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, DennyZhou, Quoc V. Le, Alan Yuille, Mingxing Tan. CVPR 2022.

  • Please stay tuned for code release in this repo!

• StarNet: Targeted Computation for Object Detection in Point Clouds.
  Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, YinZhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens,and Vijay Vasudevan. arXiv 2019.

  • car.kitti.StarNetCarModel0701
  • car.kitti.StarNetPedCycModel0704
  • car.waymo.StarNetVehicle
  • car.waymo.StarNetPed

Image

• Gradient-based learning applied to document recognition.
  Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. IEEE 1998.

  • image.mnist.LeNet5

Language Modelling

• Exploring the Limits of Language Modeling.
  Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and YonghuiWu. arXiv, 2016.

  • lm.one_billion_wds.WordLevelOneBwdsSimpleSampledSoftmax

• GShard: Scaling Giant Models with Conditional Computation and AutomaticSharding.
  Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun,Noam Shazeer and Zhifeng Chen arXiv, 2020.

  • lm.synthetic_packed_input.DenseLm1T16x16

Machine Translation

• The Best of Both Worlds: Combining Recent Advances in Neural MachineTranslation.
  Mia X. Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, LlionJones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, JakobUszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes.ACL 2018.

  • mt.wmt14_en_de.WmtEnDeTransformerBase
  • mt.wmt14_en_de.WmtEnDeRNMT
  • mt.wmtm16_en_de.WmtCaptionEnDeTransformer

• Self-supervised and Supervised Joint Training for Resource-rich NeuralMachine Translation.
  Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. ICML 2021.

  • mt.xendec.wmt14_en_de.WmtEnDeXEnDec

References

• API Docs
• Codelab

Please cite this paper when referencingLingvo.

@misc{shen2019lingvo,    title={Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling},    author={Jonathan Shen and Patrick Nguyen and Yonghui Wu and Zhifeng Chen and others},    year={2019},    eprint={1902.08295},    archivePrefix={arXiv},    primaryClass={cs.LG}}

License

Apache License 2.0