SriniVest / playground

This playground is intended for ease of prototyping real-time sign language models.

👋 Sign Translate

A bi-directional multilingual translation app for desktop and mobile, based on state-of-the-art real-time client side models.

Including:

• Desktop / Mobile ready UI
• Language selectors
• Translation direction selector
• Signed-to-spoken language translation
• Spoken-to-signed language translation

Signed-to-spoken language translation

• Camera / File upload video inputs
• SignWriting hand shape and orientation estimation
• SignWriting facial features estimation
• Language identification (Detect Language) - TODO
• Segmentation - TODO
• Tokenization - TODO
• SignWriting to spoken language translation - TODO
• Text-to-speech
• Copy / share / edit translation - TODO

Spoken-to-signed language translation

• Text input
• Microphone input - TODO
• Text-to-speech
• Spoken language text to SignWriting translation - TODO
• SignWriting to pose sequence - TODO
• Pose sequence to video - TODO (tensorflow/tfjs#5374)
• Text to pose sequence (server side)

Mobile support - TODOs

• Rethink app icon
• MediaPipe doesn't work on mobile (google-ai-edge/mediapipe#1427)

Sign Language Processing Playground

This playground is intended for ease of prototyping real-time sign language models.

Visit the playground!

It includes, as a basic first step, MediaPipe Holistic pose estimation, on top of which other predictions are performed:

• Sign language detection (Model based)
• SignWriting hand orientation (Rule based)
• SignWriting hand shape (Model based)
• Partial SignWriting non-manuals - eyebrows, eyes, mouthing (Model based)