Dear all,
This week I would like to present my thoughts on wearable technologies. We are all familiar with that subject and many articles were presented before, but it seems to me that so many possibilities of usage are still undiscovered.
In recent years there have been an increase in the number of researches on potential usage of wearable technologies in facilitating life of those facing challenges in everyday life. One of such groups are deaf people and their's communication with people who does not know sign language. They want to achieve it with the idea of developing glove-based communicating aid in mobile environment. Gloves using flex sensor, accelerometer and EMG sensors possibly could read the hand muscles movements and ascribe adequate word in english language. Scientists believe that they can use similar technology for development of innovative user interfaces between humans and computers. Below I present few links to scientific articles on developments made in last 2 years.
Sources:
Questions to answer:
- Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
- Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
- Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
Best Regards,
Cezary
Hmm I don't know if you know that in Multimedia departmen one of our collegue is doing something similar thing in terms of sign language recognition. He is doing this without using external device (in this examples gloves) but he uses Neural Networks for translation.
ReplyDelete1. Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
I think that gloves from this articles are only prototypes, so we can omit cables in final solution, maybe bluetooth? But if someone would like to leave those cables outside it would be impossible to use this gloves on daily basis.
But idea is quite nice.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Yes. Mobile phones and application for translating this kind of language using phones camera.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
Hmm I think that you may be thinkig about gloves from Minority Report :D. For me the most comfortable way with interacting with computer is by keyboard (my RSI problem is getting bigger so I should think about another one way :-)). But yes I think that this could be interesting when we would like to connect it with let say VR set, with this we could gain more possibilities for architects or engineers.
Thank You for response. I did not know that someone in our school is working on similar solution, but it is good to know that. As we can see these are just some solutions by students in university, so obviously it is not the final solution. Now new solutions comes to my mind - maybe something like google glasses that will reed sign language and write the translation would be a good solution? There are multiple solutions for solving problem of connection. Of course I prefer interaction with computer by keyboard, as you mentioned, but people are looking for more futuristic solutions.
Delete1. I think that such technology could be used on daily basis. A friend of mine created a similar glove (with flex sensors and stuff) much time ago, to control a mechanical, 3d-printed arm. It was for his thesis. All the things were home-made, but wearing that glove wasn't uncomfortable or something.
ReplyDeleteFrom month to month, electronic components become cheaper and smaller, so in my opinion, wearing a glove prepared by a professional company, would be a really nice experience :)
And when it comes to a mobile app, I'd rather consider something like Google Pixel Buds (https://www.blog.google/products/pixel/pixel-buds/), which use voice as a way of communication.
2. I think technologies like in Xbox Kinect device can be used for translating the gestures. Especially connecting Kinect-like sensors with smart glasses and so on...
3. As we know, a picture is worth a thousand words, so let me sum up my 1000-word essay like that: https://9gag.com/gag/aRjyyX7
That is great to hear that there is another person working in our country on similar solution - probably in many other universities there are similar attempts. That is true electronics become cheaper and cheaper, today's innovation in few months will be an old crap. Of course Xbox device is one of the possibilities, but definitely not the cheapest one... And not portable one. Of course there was few attempts to create user interfaces between humans and computers, but still I hope for something great to be created.
DeleteI think that the final product would be much cheaper and user-friendly than the prototype. I'm pretty sure that the problem of cables can be solved somehow and in the end they wouldn't be a problem.
ReplyDeleteI agree with Tomasz that mobile phones and apps might be a solution, but one has to bear in mind that there is no ideal product and that each and every concept has its pros and cons. Nothing can substitute hearing and nothing can substitute ordinary communication basing on speaking-hearing.We can only try to find the best possible option and develop the idea to help people cope with such obstacles.
Of course when there will be mass production the final product will be cheaper then the use completely handmade. Of course the best solution should be chosen by someone who really uses it on a daily basis. We can have our own opinion but ease of use should be best for deaf people. Probably many user experience testing will be involved before the production will start. I hope that the designers and programmers will consider this factor.
DeleteIn my opinion this is a very simplistic approach to the communication between the hearing and the deaf. In other words, this may be a heroic shot at the wrong problem. But first, the questions, and then more arguments.
ReplyDelete1. (convenience): It may be cumbersome and expensive, but for this purpose it shouldn't be a deal-breaker. The problem is, the idea would not work at all, so it's not relevant if running around with gloves (and cables, and a QR Code linking to the app) is too much of a hassle or not.
2. (alternatives): The communication between the hearing and the deaf is so far possible through human interpreters. Hearing relatives of deaf people know both some natural language and sign language. There are pidgin sign languages like the "Signed Polish" (not to be confused with Polish Sign Language). I think the answer could be some system of visual communication.
3. (computer interface?) I think such systems exist already. Maybe they can't replace a keyboard for coding (although coupled with something like https://fossdroid.com/a/dasher.html - who knows?), but a game controller for entertainment and rehab is definitely on the table. A company (https://egzotech.com/) from Gliwice is making EMG controllers for playing games on Android devices. And these are not just games written for that purpose, but common titles. Such a controller can have up to 6 degrees of freedom, allowing for practicing multiple muscle groups / neural connections.
So much for the questions, and now for the obstacles in the communication with the deaf. We need first and foremost to disabuse ourselves from the mental model, that a deaf person is just like a normal person wearing earplugs. There is a cultural difference, a different way of processing information.
The proposed system with a glove and a tablet misses one important component: the communication channel in the other direction. How does the hearing person respond? By typing on the tablet, and showing it to the deaf person? Or - just like now - they should be expected to read our lips?
Textual communication works best with people who acquired deafness, but used to speak and read before. English and French notwithstanding, text is just a phonetic record of the spoken word. It can be learned (like Hanzi "alphabet" by normal people), but is not convenient and natural for deaf people to use and process.
In my opinion the only reasonable way to include the deaf in our audio-based communication scheme is simply to... give them access to the sound. As soon as possible, when their hearing impairment is discovered. Probably right now it's science-fiction, but mind-machine interface is the way to go. The next best thing are effective hearing aids.
Of course, the examples given by me are just one of many ways to solve this problem. They are not the best of all existing, they are only the examples of the various techniques used. I see that you have a lot of experience in this topic. I decided to introduce this topic because I did not know that they were working on finding a way to communicate with deaf people. Of course, thank you for sharing your knowledge. I am surprised that the Polish company has created such an interesting product and I have never heard about it yet, I am sure I will know more about it soon.
DeleteYou are absolutely right, I commented in the comments above - it is the most important to understand these people and find a solution that is the best in their's opinion, and not according to us - programmers or engineers.
ReplyDelete1 Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
I thing this technology won't be commonly used because it is inconvienient. I've seen solution of recognition of hand signs with neural networks as Tomasz mentioned.
2 Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Probably I don't get the idea ,but if I were in such situation it would be more convenient to use SMS.
3 Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
In professional application there is no way. For developers keyboard would be better. On the other hand it can has interesting capabilities in entertainment.
Thank You for sharing your thoughts. I think that using just text messages would be hard. It is hard to completely understand how it is to not hear, but I tried to communicate with someone like that just by writing on the paper. But it is hard to reflect regular conversation with writing, especially to express emotions. Usually writing is just way of short communication without deeper meaning, but those people for sure want to have normal conversations like anyone else. As You can read above out collages already heard about human computer communication interpreters so it is absolutely possible.
DeleteDo you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think that this solution could be used on a daily basis. Presented ones can be improved and there will be no problem with huge amount of cables which could really disrupt hand gestures. There may be used only mobile phone with camera and a computer vison algorithm recognisign gestures as people do.
Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
One way may be by writing but it is not very comfortable. In a future they will probably develope some glasses converting sign gesters into words.
Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I thnik that it may be another, additional interface. It may be used in games or some special applications. I don't think that it can replace keyboard.
Of course, these are just prototypes. It would be great to combine the knowledge of all the people working on the same solution. I would reject some solutions that have been tried out by others and focus on the most promising ones. I also came to the solution with the glasses, but on the other hand would all the people have to carry such glasses? It also does not seem optimal, maybe for the family members or for friends of that person it is possible, but for strangers not so much.
Delete1. As Tomasz said, in the final version they probably would use a wireless communication. Currently, indeed it looks uncomfortable.
ReplyDelete2. It was already mentioned, but just typing on any electronic device with text to speech synthesizer seems like a much more convenient solution and even more expressive way of communication.
As for technological aids in communication in sign language, computer vision sounds like a good match, but I'm not sure if it doesn't require a stereo camera. It seems that depths of movements also plays an important role in sign languages.
3. For me, a much better interface of this type is the Leap Motion. It also recognizes gestures and doesn't require wearing any gloves or other devices.
I also consider wireless communication as the best solution. Of course, there are many external factors, such as lighting, which may make it impossible to read the characters and the whole device will not fulfill its task. System based on a hand tracking device called Leap Motion, used for signs recognition seems more than promising. Such system uses a Support Vector Machine for sign classification.
Delete1. Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI might see some uncomfortables in wearing such device but I see many advantages. Let see robot or avatar seen in glaces or other virtual reality that translates verbal language into the deaf sign language and vice versa.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
I belive I propose an idea in previouse answer.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I believe some day such devices appears on the market.
Thank You for the short answer. In my opinion another problem may be to make people aware that there is such kind of communication and they should learn it. It will be a problem for the elderly, just as any other new technology in the market. Some of the proposals that you have come with are a bit too complicated. Remember that it is about communication on the street with random people, without complicated and expensive equipment.
DeleteNo, of course wearing this kind of device is too problematic and people wouldn't bother after few days/weeks of excitement caused by the first impression. But I am sure they're going to become less and less problematic and more ergonomic. This stage is temporal and crucial for usability tests.
ReplyDeleteText synthesizer seems like a good idea. Maybe a google-glass-like device might be useful? Only properly conducted usability tests can prove best solutions for practical tasks like this one.
I don't believe in modern human computer interfaces :) There were so many of them developed in past 10-15 years and still nothing's better than an old, good mouse + keyboard (or an Apple touchpad ;)).
Sometime ago people did not believed in the possibility of any interaction with the computer, and than someone created the devices that you mentioned and suddenly people saw such possibility. So I would not say that such day will not come, because probably one day we will not be using the mouse and keyboard.
DeleteAs I have said few times, I do not think technology is as important as teaching ordinary people to take advantage of some new solution. This may prove to be the most difficult.
Thank you for the interesting topic. This is an interesting and good solution. But wearing anything on your hands can be embarrassing. I'm not worried about cables, I think it's an improvement. Communication must be two-sided. There are applications, but I would rather write something to the deaf person;)
ReplyDeleteI would not call it embarrassing but maybe uncomfortable. As I mentioned before just writing on paper or text message is rather for short meaningless conversations and those people want to have a normal ones, and that requires better solutions.
Delete1. Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteTo be honest - even if it is uncomfortable - it could be the only way to communicate with world for the part of people. Taking into account this kind of situation - my answer is no it is not :). Nevertheless I am pretty sure that a design will be optimized on the last steps of development, when other tricky problems will be solved.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Camera in mobile device and dedicated smartphone app. I believe in a couple years it will be standard and it will improve the quality of deaf people life.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I believe it is some kind of interface and it works. :) Kinect is some kind of interface that translates our moves. We become a controller/ a gamepad.
Maybe it will not be the only way to communicate, but surely these people would be willing to try some new solution if it can help them to live easier. Even if the solution you propose seems quite difficult, suddenly the person on the street need to have some application on phone and thanks to the camera read these signs ... as for me it is unlikely, we need something that works as fast as speaking.
Delete1. Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI’m not sure but I think that it wouldn’t be so commonly used as it seems to be. I find it really uncomfortable to deal with cables or to use mobile application. On the other hand I agree with Katarzyna and in my opinion some issues could be improved for the convenience of using this product.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
I think about a mobile application which allows the recognition of gestures recorded by camera, translation of gestures into words and word playback through the speaker. Maybe such application exists – I don’t know. However the idea of smart gloves seems to be promising.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
In my opinion this invention has the potential to exist in many areas related to communication between human and computer. For example commanding devices in industry.
At this moment, a crazy solution came to my head. If the glove or phone connected to this glove had a speaker ... and somehow when person starts showing signs, the words would be spoken? Such a solution could be quite fast, but it still does not solve the problem of understanding what the other person says. At least half of the problem would be solved. I believe in creating such an interface. As we know various strange solutions are shown in the movies, but some of them are actually implemented and used in real life.
DeleteYou are right. I forgot about the second half of the problem: understanding what other person says. Despite this there is little to do to create a good tool for deaf people.
DeleteThank you for the interesting topic and rich materials. I don’t know about this topic, but I'm wondering if it was enough to have a bracelet on each hand and a ring on each finger. Ring - a micro transmitter and a bracelet would read the position of each finger and transmit information about them and their location to a device that is equipped with a speech synthesizer, and based on the received information which generates sound. Such a device would provide the possibility of transmitting from the deaf to the rest of the population. The other way would be to use the glasses of expanded reality. Another problem is that the Deaf World is hermetically sealed. The effectiveness of medical treatment for deafness is very high - the problem is that deaf people are afraid that as they are cured they will be rejected by their society and often have no good relationships with people who can hear well. A very positive thing is that the presented solution could bring together these two worlds.
ReplyDeleteThank You for comments. I am not an expert I electronics by any means. But Your solution seems logical and potentially can work. Our colleagues came with great simplifying solutions to use just application that can read signs with smartphone camera and write text. I think you thought about augmented reality and yes that is another solution. I don’t know personally such people but whatever the disabilities person posses I don’t really focus on that. I am always trying to comfort and make person feel absolute best. Many people never had an occasion to even meet such people and probably at the first sight it will be hard to cooperate but we should came with open attitude.
Delete- Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI would answer not straight-forward. I think that until other methods, that I've described below became more popular, any other available method should be used to provide a way to communicate with the deaf person in specific environments (such as government institutions).
- Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
I think that better idea would be the usage of something that everybody already has in their pockets. Smartphones and their cameras, in my opinion, should be sufficient way to grab and process real-time video of a person who is using sign gestures language and translate it into written word in the real time. Such project even exists, and it has Polish roots; it's called http://migam.org (but I don't know how advanced it is). Another implementation takes place at the Rzeszow University of Technology (http://sykomi.kia.prz.edu.pl/). I also think that large upgrade of such systems would occur when smartphones would be armed with depth field cameras. I think that after iPhone X has it's premiere and introduced such technology (earlier was Google with it's Tango project with implementation in Asus and Lenovo phones), it would be used more commonly. That gives the developers new, native way to create apps that could really sense the environment.
- Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I think- no, in my opinion, there should be more elegant way to achieve that goal. However, I see here an opportunity for other applications, such as med-tech. Haptic interfaces are still in the upgrade process so I think that there is a large field of development there.
Thank a lot for your comment, only You brought really important situation like visiting government institutions. Again we can see that Polish developers also work on similar projects. As our colleagues mentioned there were prototypes of such interfaces and I really believe that in 10 years we will use our computers in completely different ways. After using mouses we started using touch screens. Maybe there is another milestone to overcome. This will be very useful in medical field, especially in rehabilitation. I guess in few years we will be surprised with new developments.
DeleteDo you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think gloves with wires can be considered as a proof of concept. Main success factor for such equipment will be always great user experience
Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Unfortunately no, It;s not my area of interest.
Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
Definitely yes. There is large space for improvement
Of course, as I have mentioned many times, these are only examples of the initial phase of the solution, but without them it would be hard to imagine the next steps. Unfortunately, despite the lack of interest in the topic, you did not decide to look on any other solution. Maybe this will encourage You to look for more information.
DeleteDo you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think it won't be used on the daily basis. Looking at the state of Augmented Reality capabilities of our mobile devices it's possible we won't need any glove to interpret the signs.
Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
As I mentioned above I think AR is capable of reading the signs so the only device needed is listener mobile phone.
Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I don't think so. We will rather go into the direction of no additional devices needed for Human Computer interaction so most likely it will be voice like in Science Fiction movies.
I still think the psychological barrier is more important than the device itself. These people are not so open, they do not want to publicly admit to their exasperation. Therefore, convincing them to use such a device as well as public awareness is key here. I would rather focus on the interface and social campaigns.
DeleteI think you are wrong about the interface, because there are already prototypes, many powerful corporations are working on solutions, and I can not imagine that in 20 years still operating the computer will look just like it looks today.
1. Definitely, such technologies give us many opportunities that people will be very interested in using them in spite of the disadvantages. All the more so I guess very soon those cables and applications shall be replaced with more user-friendly solutions.
ReplyDelete2. I don’t know better idea at the moment. Bust observing the scientists concentrated on looking for possibilities which can make our life easier and help people in real need, I suppose such solutions shall be appearing in following years. On the other hand we should appreciate the possibilities gained so far.
3. It doesn’t lead to such development, because it has already become kind of such interface. This is only the way of communication. Very innovate one, because of its intelligence. To communicate this way may be a real adventure. It can surprise us with its knowledge about the signals we send. And it can give us information about the data we would like to keep instead of spreading out – as a result it can teach us to do that. Very inspirational.
I think the best solution could have been found long ago and implemented if the flow of information between scientists was better and easier. The cooperation of scientists from different parts of the world and outstanding specialists, both programmers and electronics, would result in the least annoying device and easy operation system as well as lack of discomfort for the users.
Delete1.Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think that in such form this won't be used on a daily basis. If the technology becomes more affordable and approachable it will succeed. People usually do not get used to something which isn't comfortable.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Maybe some kind of a display? Translation with a camera in a real-time seems to be a good idea.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
I am quite sure there is a chance that such glove one day may lead to development of user interfaces between humans and computers. However, implants may have a greater impact.
Naturally, implants seem to be the most real and tangible way to interact with a computer, although I personally would not like to use it. I am against chips and other devices that would be implanted in our body. I have seen prototypes on kickstarter or other crowdfunding portal, and I am even more disgusted with such inventions. I think it can be created in less invasive way.
DeleteDo you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteNow, it's a glove with cables. Tomorrow it will be just a glove. Technology is changing rapidly, so let's wait and see what's coming.
Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
Telepathy :D
Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
Scientists across the world are trying to develop new interface for this particular case for a long time. From what I know, we are getting closer. One day it won't be a problem. However, it's hard to predict when.
I know that Your answer is humorous, but the problem is evident, and for these people it is not really funny. It is probably as persistent as being blind or physically disabled. The sense of inferiority and barriers in everyday life appear at every step.
DeleteAs for the interface I bet that within 10 years our usage of the computer will look completely different. We may have cables connected to head or implanted chips under the skin, so we can control the computer.
Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think it could be used on a daily basis. Of course using a glove with so many cables is uncomfortable, but it is kind of technical issue. The technology is developing so fast nowadays, that probably in some time the glove will be reduced to the one looking like a normal, textile glove, or even something smaller, like a bracelet.
Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
I think the direction presented in an article is good. The device reading gestures and translating them seems to be an interesting idea. Maybe the device itself might be changed to something more convenient (kind of smartwatch, or as I mentioned in previous answer a bracelet).
Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
It might be. Especially in 3D design or for virtual or augumented reality. It also can be a way of the communictaion between seniors or disabled people with their computer assistants. But from the other side, the moves and gestures can be read without the glove, just only with a camera (or a set of cameras).
On the other hand, it is true that even the interaction between the deaf and the rest of people is a problem. There is a barrier - a shame, a sense of being different - the addition of a glove, which will expose them in the crowds even more is not a good solution. The idea of using wearables is very good idea, actually a smartwatch or glasses could solve the problem. There are probably even better solutions to this problem already found, but with our knowledge we still see a huge amount of solutions, which gives me hope than soon great solution will be shown.
Delete1. Do you think such technology could be used on daily basis? Isn’t it too uncomfortable to wear glove with many cables and tell people to use mobile app to translate their’s gestures?
ReplyDeleteI think it can be used on daily basis. In the past we thought that cellular phones must be as small as possible, no one wanted to have a brick in pocket and now we have phablets that have screen that have 7 inches. So this is just a matter of usage. We can adapt to everything as long as it is useful but in my opinion we should go in different direction. Translating sign language based on AI and view from camera combine with AR glasses will have much bigger inpact.
2. Do you have any better idea of solving problem of communication between deaf people and those who does not know sign language?
As mentioned before. Couple years ago I was working on a project to create an automatic translator based on Kinect platform. It was great idea but technology had too many limitations. Now I think that the concept was wrong, the translation should be done based on camera. If we combine it with AR glasses than everyone could have such translator on his face.
3. Is there any chance that such glove one day may lead to development of user interfaces between humans and computers?
This question has no sense cause we already have interfaces based on such devices 😊
As usual, we all comment on the last moments, and it is a pity because we could have been discussing and finding cool solutions for all week. Of course you are right, the first version is a simple working version, which should inspire to find a better solution, less expensive and easier to use. There are many solutions, but remember that this should be the cheapest and easiest solution, these people will have technological barriers and not great financial means. As for the last question, I meant implementing such an interface for everyday use of computers by the most ordinary people in everyday life, and yet it has not come.
Delete