Student project demo showing a prototype that translates American Sign Language (ASL) hand signs to spoken words. The team collected a custom image dataset (≈1,200 images) covering 10 everyday ASL words, trained an SSD MobileNet v2 object-detection model in TensorFlow for ~20,000 epochs (reported ~88% accuracy), converted the model for Raspberry Pi deployment, used OpenCV for camera capture and inference, and Azure Cognitive Services for text-to-speech. The prototype demonstrates several translated words and notes future expansion to broader vocabularies.
Questions this video answers
More than 38 million
About 500,000
10 everyday words
Approximately 1,200 pictures
SSD MobileNet v2 (TensorFlow)
Chapters
Hello. We are first year student of the engineering cycle at ece. We are going to present our project of robotics created during our broad semester at Baruch College, CUNY. Today, more than 38 million of Americans suffer from a degree of hearing loss and about 500,000 use the same language. To eliminate these barriers of communication, we will attempt to develop a device capable of translating signs into speech. It will be based on the American Sign Language ASL that uses hand movements and facial expressions. For this project, we will only use the hand sign and no action recognition. The Raspberry PI will be equipped with the camera's record signs. Then the image will be analyzed by the neural network. Once the sign is organized, the system will translate it to the string. It will use a voice generator on that string to say it loud.
For this project, we create our own dataset which is a collection of images. For the moment, it is composed of 10 everyday words. Goodbye, I love you, friends. You're welcome. No. Good job. Yes, please.
Hello. Thank you.
In total, we have to take 1200 pictures. All these images have differences, right hand or left hand? Background zoom on the hand, lightning, etc. To finalize our dataset, we have labeled each of these images. To assign each sign the corresponding word.
To detect the signs, we had to create an artificial intelligence. For this we use TensorFlow, an artificial intelligence library and a pre trained model, the SSD MobileNet second version. We based our model on a neural neural network using the convolution method. It acts as a filter which allows us to keep only the important characteristics of the image. Then we train the model with 20,000 epochs to obtain an 88% accuracy. We transform the neural network into a dead network that is smaller and easily transferable to the Raspberry PI. We use the library OpenCV to lift the TensorFlow model with the webcam capturing the sign to detect. First we import the model and get the video flex. Then we applied the neural network to the framing process to have the label and the accuracy for this frame. To have a complete translator, we added a function that allows to say the decoded sign. For this we use the Connective Services, an API of Azure, the Microsoft Cloud. It allows to convert a text into voice. In our program. Once the sign is decoded by the neural network, we retrieve the label here, the wall corresponding to the sign. We then send it to the Azure cloud and we instantly retrieve the corresponding audio. And now let's do demonstration. Yes. I love you. No, you are welcome. Hello. You are welcome. Thank you. Good job. Yes, Goodbye. Please, friends.
To finish this project can evolute by increasing the number of signs to detect it could translate complete conversation instead of words. Thanks for listening to us.
