A real-time ASL fingerspelling recognition system built with deep learning. ~7,000 training images, dual-layer CNN, Hunspell autocorrect — bridging the gap between sign language and text.
Sign2Text is an end-to-end deep learning pipeline that translates American Sign Language (ASL) fingerspelling into readable text in real time — making communication more accessible for the hearing-impaired community.
TensorFlow/Keras model with two convolutional layers trained on 7,000+ custom images for robust recognition across all 26 ASL letters.
OpenCV-powered video pipeline with Gaussian blur and adaptive thresholding delivers low-latency predictions at 30 frames per second.
Intelligent sentence formation using Hunspell dictionary integration to correct spelling errors in predicted character sequences.
Dedicated sub-classifiers resolve visually confusing sign pairs like D/R/U, boosting real-world accuracy to 98%.
Multi-frame consensus voting eliminates noisy single-frame errors and stabilizes letter predictions across video streams.
~7,000 images collected across all alphabets with variations in lighting, hand sizes, and orientations for maximum generalization.
Simulate the Sign2Text pipeline below. Click letters to build words, or let the auto-demo run through an example phrase.
A modular, multi-stage pipeline that goes from raw camera frames to autocorrected text output in milliseconds.
OpenCV reads real-time video frames from webcam at 30 FPS. Frames are normalized and resized to a fixed input dimension.
Gaussian blur removes noise. Adaptive thresholding creates clean binary hand masks. The ROI region is cropped and fed to the model.
A dual-layer TensorFlow/Keras CNN classifies each preprocessed frame into one of 26 letter classes with a softmax confidence score.
Predictions are buffered over N frames. The dominant class across the buffer is accepted as the stable letter prediction.
Hunspell dictionary checks formed words against known spellings and suggests corrections, enabling natural sentence assembly.
Letters like D, R, U share similar hand shapes. Sign2Text uses dedicated sub-classifiers that activate only when the primary model's confidence falls below a threshold, resolving ambiguity with higher precision.
Evaluated on a held-out test set. Results show consistent accuracy across all 26 ASL letters even in challenging lighting conditions.
Primary CNN classifier on full 26-class ASL alphabet
All 26 letters that Sign2Text can recognize in real time. Hover each card to see the hand description.