In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents.
Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
This was my first thought, too (after “which sign language?”).
That said, the video does show the model identifying facial and other body features, so it’d be interesting To know if this model _could_ be capable of working with a full sign language (ASL or otherwise).
It also got me thinking about how LLMs might handle signs as tokens, and it seems you did too! Would you just detect unique handshapes and hope the LLM could do the rest? Could you somehow encode orientation/movement/direction/locations and facial expressions? I presume current LLMs don’t have much training input on this kind of language - would we need to train new LLMs to even have a hope of them “understanding” 3-dimensional language?
I know this is well meaning, but you should know that in the disability community it's very common to have people invent tech for us without fully understanding our needs. (Smart canes, for instance.)
What your demo shows is a model understanding fingerspelling. Fingerspelling is ... a very small part of sign language.
To make the claim you are translating sign language you need to understand facial expressions, hand positioning/motion (help you vs help me, same hand shape, different positions/motions), classifiers, and a ton more stuff.
It's a really beautiful language, and I encourage everyone to learn it, but for your project I'd be very careful to not make a claim about translating sign language.
In the mean time, it sounds like you've got some passion for accessibility work? We're always looking for more skilled engineers to work in the a11y space. It does require some humility and willingness to listen to others needs.
It's possible that the page has changed since you read it, but it doesn't claim to translate all of sign language. Ultimately there hasn't been enough resources put into the problem for full translation. There isn't really anything that a single dev can do to change that. Models take money.
It says right at the top of the description "Turns sign language (fingerspelling) into..."
It should just say fingerspelling. Conflating these two is like writing "Translates Shakespeare (Sonnet XC) into ..."
It's wonderful to try and experiment with this stuff, but it's very common for people to think fingerspelling is sign language.
Even the claim of fingerspelling is perhaps questionable. Deaf fingerspelling is a) MUCH faster and b) doesn't always spell every letter. It's not uncommon to, for example merge two letters together (I+L+Y), be moving some fingers up while others move down as in JULY or even skipping letters as in JUNE (which may omit the N). Words like SOON will be signed S-OO-N where the O is dragged out to the right. The longer you drag the O the more o's are implied. SOOOON vs SOON is a matter of how far the O is slid. And non-manual markers are a part of fingerspelling. SOON? Vs SOON are different eyebrow and eye shapes.
This very much feels like a student project, which I applaud. I'm assuming good intent here. It also feels like a young person knowing enough to be dangerous and furthering stereotypes and misconceptions unintentionally. I love the effort, but I'd encourage them to qualify the intent of the project quite a bit.
I really don't want to discourage this person. Please don't misunderstand me. I'm critical because this is cool. If it was garbage, I'd move on and not say anything. I just think that when building for a disability community, you have a responsibility to not over promise.
As others have pointed out, it might be helpful to step back and learn more about sign language. More importantly though, you might want to learn more about Deaf culture and how this type of technology might be received
Assume good intent, imo. Most of us live in a community with a predominant single sl, whether that's ASL, BSL, etc. Learning one is how I interpreted their comment.
In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents. Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
That said, the video does show the model identifying facial and other body features, so it’d be interesting To know if this model _could_ be capable of working with a full sign language (ASL or otherwise).
It also got me thinking about how LLMs might handle signs as tokens, and it seems you did too! Would you just detect unique handshapes and hope the LLM could do the rest? Could you somehow encode orientation/movement/direction/locations and facial expressions? I presume current LLMs don’t have much training input on this kind of language - would we need to train new LLMs to even have a hope of them “understanding” 3-dimensional language?
What your demo shows is a model understanding fingerspelling. Fingerspelling is ... a very small part of sign language.
To make the claim you are translating sign language you need to understand facial expressions, hand positioning/motion (help you vs help me, same hand shape, different positions/motions), classifiers, and a ton more stuff.
It's a really beautiful language, and I encourage everyone to learn it, but for your project I'd be very careful to not make a claim about translating sign language.
In the mean time, it sounds like you've got some passion for accessibility work? We're always looking for more skilled engineers to work in the a11y space. It does require some humility and willingness to listen to others needs.
It should just say fingerspelling. Conflating these two is like writing "Translates Shakespeare (Sonnet XC) into ..."
It's wonderful to try and experiment with this stuff, but it's very common for people to think fingerspelling is sign language.
Even the claim of fingerspelling is perhaps questionable. Deaf fingerspelling is a) MUCH faster and b) doesn't always spell every letter. It's not uncommon to, for example merge two letters together (I+L+Y), be moving some fingers up while others move down as in JULY or even skipping letters as in JUNE (which may omit the N). Words like SOON will be signed S-OO-N where the O is dragged out to the right. The longer you drag the O the more o's are implied. SOOOON vs SOON is a matter of how far the O is slid. And non-manual markers are a part of fingerspelling. SOON? Vs SOON are different eyebrow and eye shapes.
This very much feels like a student project, which I applaud. I'm assuming good intent here. It also feels like a young person knowing enough to be dangerous and furthering stereotypes and misconceptions unintentionally. I love the effort, but I'd encourage them to qualify the intent of the project quite a bit.
I really don't want to discourage this person. Please don't misunderstand me. I'm critical because this is cool. If it was garbage, I'd move on and not say anything. I just think that when building for a disability community, you have a responsibility to not over promise.