Today I would like to present you the next "big thing" in Artificial Neural Network. Paper was released in October 2017 and after this date everybody involved in ANNs field are talking about it.
Paper was made by "grandfather" of deep learning Geoffrey E Hinton in colaboration with two grad students.
https://arxiv.org/abs/1710.09829
More on Youtube (about what is wrong with today's approach)
Questions:
1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Questions:
ReplyDelete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
As I see the Capsule Networks are particularly good at handling different types of visual stimulus and encoding things like pose (position, size, orientation), deformation, velocity, albedo, hue, texture etc. The brain must have a mechanism for “routing” low level visual information to what it believes is the best capsule for handling it. I am not an expert in AI but it seems that capsule is a nested set of neural layers. So in a regular neural network you keep on adding more layers. In CapsNet you would add more layers inside a single layer.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
-High computational cost.
- If you don't have a good GPU they are quite slow to train (for complex tasks).
-They use to need a lot of training data.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I don’t know, but probably he faced some problems with current Convolutional neural network.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
In deep learning, Convolutional Neural Networks are found to give the most accurate results in solving real world problems. CNN is used in computer vision, mainly in face recognition, scene labelling, image classification, action recognition, human pose estimation and document analysis.
Hmm, for the most points you are right. The most important thing in capsule network is how they are handling changes in pixels. For example in CNN everything works fine until you change something in one pixel whole solution can crash. When it comes to complexity of training this kind of networks usually one GPU is not enough. In PJA we have one computer with 6 GPUs (Nvidia Tesla K40) and sometimes it is not enough, sometimes we need to wait whole weekend to finish one experiment.
Delete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteIt sounds like it is something closer to the way humans perceive the world. Finally ;)
3d perception is something missing in the current image recognition solutions. In my opinion any improvement in this area can be the next "big thing".
I agree with Roland Memisevic that this tech will help extracting more "understanding" from the same data.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
The article itself pointed out some disadvantages, like problem with recognizing cat from a new angle. My son (who is 7 months) performs better here ;)
Image recognition systems strongly rely on a training data and those systems can be still tricked. Noticeable examples from the top of my head are:
- Chihuahua or muffin; short comic strip here: http://www.commitstrip.com/en/2017/11/03/vision-api-fail/, explanation here: https://medium.freecodecamp.org/chihuahua-or-muffin-my-search-for-the-best-computer-vision-api-cbda4d6b425d
- little distortion can trick CNN: https://arxiv.org/pdf/1412.6572.pdf, page 3
Moreover, as stated in the article, the image recognition mechanisms may help progressing with self-driving cars or AI doctors. Let's remember that currently we are far away from that. Take a look at this simple demonstration: https://techcrunch.com/2017/03/17/laying-a-trap-for-self-driving-cars/.
Will Capsules help here? Perhaps.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I think he refers to the way computer vision works in general. As mentioned earlier, we don't take an advantage of 3d.
I feel like people just accepted that computer vision "has to work this way" and there are no many researches on how to improve, how to get closer to the human way of seeing the world.
Every time someone has to teach a computer to recognize cats, he starts with looking for cat pictures. Preferably from different angles. And he doesn't think how to change, improve the algorithm.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
In all the services from the already linked article: https://medium.freecodecamp.org/chihuahua-or-muffin-my-search-for-the-best-computer-vision-api-cbda4d6b425d
The cloasest solution is one called "One Shot Learning", in this solution scienctist are trying to show that you can do categorisation of an image with just one or few training examples (this solution is inspired by how human brain can do categorization of an object). This link with chihuahua is great I will use it in my presetation :D. And it shows one of the problem with CNNs. When it comes to getting closer with CNN to how human brain works I don't that this will be possible because we don't in 100% how human brain works.
Delete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteYes I think that Capsule Networks can be the next "big thing" in NN field because they are built to learn to recognize things more like humans do. Also they seems to require less training data. That could make them work faster and cheaper because annotating training data is very expensive.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
Todays approach like CNN requires a lot of training data. It is very time consuming to train them properly.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I think that he thought about the way we train neural networks. We are expanding training datasets and layers of netwrks instead of think about new solutions in non standard way.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
CNN are usead very often in computer vision or natural language processing. They may be used in sentiment analysis or document classification. They are also used for feature generation for instance in named entity recognition.
Yes with annotation you are right in 100%. We have medical dataset with tumors and we are waiting ages for annotation this dataset by doctors when it comes to sonograms (or something similar) we need to remember that doctor experience in this matter is very very important and this adds additional cost.
ReplyDeleteHmmm do you have any good paper about CNN in NLP? I know that sometimes people are using CNN to speach problem (sonographs) but I have not hear anything about CNN with text. If you have something interesting in this matter please let me know I would relly love to read this.
I don't remember everything at the moment but here is one example for ner and bi-lstm cnn https://arxiv.org/pdf/1511.08308.pdf
Deleteand here for cnn for sentiment analysis http://www.aclweb.org/anthology/C14-1008
Thanks for the papers :-)
DeleteHello,
ReplyDeletethank you for that article.
1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
It is indeed interesting approach, especially in the field of in the field of image recognition because it allows generalizing the data to a greater degree from different points of view with less training input.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
CNN requires lots of data to be trained and validated. For example, in the field of image recognition, if we would like to recognize an object, it is required to "show" it in multiple poses so CNN could generalize it, because CNN has too few levels of the structure according to Hinton.
3. In this article Hinton said "I think the way we’re doing
computer vision is just wrong.". You know what he had on his mind?
I think that it's because CNN's are simply a bad representation of the human vision system. CNN's are "translation invariant", so we have to train them with thousands of various samples to get accurate results. By the example of the cat: we do not have to show thousands of pictures of a cat to children so it could know how a cat looks like. People have embedded mechanism to generalize data based on key factors, that we are able to determine intuitively.
4. Can you give some examples where CNN's (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Right now they are being used in, for example, computer vision (image recognition, scene labelling), OCR applications or natural language processing.
Excluding what everyone had said, I agree with the second point completely, also argumentation in point three looks cool. I don't have anything more to add.
Delete1. It's an interesting approach but I'm still not convinced that it may be a revolution in the field. First of all, it sounds like it was overrated just because it's the Hinton's work. Secondly, the achieved results don't show any improvements. I wish the evaluation was performed on more challenging dataset rather than MNIST. However, it definitely can be a source of many interesting research problems.
ReplyDelete2. CNNs have a lot of trainable parameters and require many training examples to work. They also arent rotational invariant. Also, they don't keep a spatial information.
3. Maybe he meant that we don't try to make computers "understand" for example that face is an object where eyes, nose and mouth are located in particular relation to each other and not just randomly scattered over the scene.
4. The most prominent use case for CNNs is image classification for medical purposes, such solutions outperform experts from the field. They are also used in military, for example in satellite photographs analysis. CNNs can be used also for NLP but probably RNNs are the more common approach here.
Why do you think this is overrated? What kind of improvement you had in your mind? I am not a big fan of Hinton's work but we need to give him a credit for showing how to use backpropagation (also we need to add that he wasn't alone in this article) and he was co-creator of Boltzmann machine, so I think we can trust him. When it comes to MNIST it is very common dataset in papers (second one will be CIFAR) because it is available for public and you can develop someones approach and check results.
Delete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteDefinitely. Capsule Networks can be the next "big thing" in the Neural Network field. I agree with my previous speakers, they do remind me of the way humans perceive the world.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
The amount of data that needs to be trained and validated is really vast, so the process is extremely time consuming.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I do agree with my previous speakers. He thought of the way computer visioning works in general, but it's all related to the fact that we don't actually get how human brain really works and so we are not able to create computers, machines that would be ideal representation of the human brain. As long as researches are not able to overcome this obstacle, a perfect solution won't be found. Each and every computer, technique, method, programme, whether it's related to this field of science or other will have flaws.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Computer vision.
Thanks for your answer.
DeleteThank you for a very interesting article. I have not met before with this technology. But I think after reading this article that this can be a great thing. and change the approach to Artificial Intelligence. Most impressed me on the example of the cat: "It teaches a computer to recognize a cat from many angles, for example, could require thousands of photos covering a variety of perspectives. recognize a household pet. " I keep my fingers crossed for this work. The topic caught my attention and I will follow.
ReplyDeleteHi Maciej, yes this topic is very intresting and most important is that there is still something todo in this area. You should definetly look at some examples from popular toolkits or/and watch some tutorials online (ie Youtube).
Delete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteWell it's sounds promising but how I can tell if even Geoffrey Hinton said it's very intuitive and he was thinking about this for long time but he is not quite sure how far this technology go and what it will cause.
Maybe it will revolutionise Computer Vision but maybe we will find the limitation which is not letting us use this technology in production.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
We need a lot of input data to learn simple concepts. For example to teach the network about the cats one has to show thousands of cats from different angels to make network capable of recognising the cat. We cannot show just one cat and than change the environment and show network another cat because it will fail to recognise the object. So CNN are very bad at conceptualisation of the topic.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
Yes beside teaching shapes he wants additional knowledge to be attached to the network. That's what capsule networks are all about.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Handwriting recognition, image transcribing this is the most used area but I bet there are many more to find.
Thanks for interesting article.
M
Question1
DeleteYou are the first person which draw attention to this :D.
When we heard with my colegue about Capsule Network we were joking that Hinton saw this solution in his dreams :-). But after article and other interviews we realise that he was thinking about it for around 40 years. So we can compare his 40 years old theory to a dream :-).
The rest of the answers is the same as above. I have nothing more to write: D.
1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteI am sure they could be the next 'big thing' in NN field, because, as I understood from the article they work more similarly to human brain than currently used NNs. Even if the Capsule Networks will not succed somehow this way of NN development is promising.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
As author of the article mentioned (and it was already described in previous comments) that CNNs needs a lot of training data and they are not very efficient (e.g. time-consuming).
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I think that he talks about the genaral approach for the computer vision. We are creating structures and employ algorithms that do not follow or imitate human senses, but we expect that their results will be similar to human reasoning and perception. But if we really would like to build Neural Network we should be inspired by the human brain.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
I usually associate CNNs with image recognition. So maybe some medical use, as recognizing the results of CT scans?
Yes you are right in all your answers, I don't need to add anything more to that :-). All was said before.
DeleteThank you for your interesting article. I do not know anything about this topic but I understand it as a new style of Neural Network learning - perhaps based on some definition of abstraction classes. This approach can lower the cost of learning neural networks and can have a positive effect on learning time. If I think about it, there is a new problem - how to define these classes - maybe it's banal but I it is not my area of interests, so maybe my arguments are wrong.
ReplyDeleteWe will see in near future what this solution brings to us. You have got good hunch.
DeleteHi, thanks for this interesting article.
ReplyDelete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
As far as I understood, this approach is quite promising when it comes to process images, which is a demanding task. It overcame the limitations of Convolutional neural networks. Capsule Networks works similarly to a human's brain.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
Convolutional neural networks perform excellent in image classification when the test set is very close to the training set. The performance is lost when the images have e.g. other different orientation. It is possible to solved by adding different variations of the same image in training stage.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
I think we have perceived it in a wrong way bearing in mind wrong approach which had too many limitations.
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
I think of image recognition (obviously) and face recognition.
1. I guess Capsule Networks are very important step in Neural Network field, because there is a chance that it may remedy weaknesses of previous machine-learning systems. And that is crucial for practical usages, for example in self-driving cars, where elimination of threats can guarantee total security. That may help to achieve faster success of such projects, because Capsule Networks introduce to software data much more closer to reality. We gain something more “human”, more true, more authentic. We can trust it more as better reality coping solution.
ReplyDelete2. Today’s approach assumes – in spite of huge algorithms’ development made through last years – very artificial reflection of reality. This is enough to feel insufficiency. What’s more and what’s really important, in such terms the solutions used so far don’t fit to professional usages.
3. I guess he mentioned this artificiality I indicated above. He mention the intuition aspect, feeling that something is wrong without defining it. Such feelings can be only connected to the huge difference between real life and virtual life. While the ideal purpose is to lose the border between them.
4. CNN is used for example in:
- drug discovery,
- games, for example checkers,
- (facial recognition) criminology,
- education – I was using such tool to learn Italian language.
I think that generally subject of ANNs is very overrated, so I wouldn't be so optimistic about this new approach. However, it do sound promising, because it's something that at least try to fight the biggest limitation of "classical" ANN techniques - tones of data needed for training. ANNs at the beginning were supposed to imitate the brain (either human of animal brain), but it soon became obvious that we can learn very quickly on little data, and ANNs need enormous datasets to generalize well. This is what is wrong with CNNs. I think this is what Hinton had on his mind while saying those words - that we try to mimic human behavior with algorithms that surely work differently than our brains.
ReplyDelete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteI have no idea. This is far from being close to my research field. However, I can guess and say ye, why not :)
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
Harder questions please...
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
Even better one...
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Image recognition. Let's find people across the world without a problem.
1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
ReplyDeleteIn my opinion it may be a “big thing” because of its approximation to the human perception of the reality. Greater efficiency: less trainings and better results.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
I think that it is too complex process. Its complexity entails great demands such as high hardware requirements or multiple steps of learning.
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
In my point of view Hinton has come to the conclusion that it is likely to be a better approach to this subject that could radically change the results in this research area. In addition, he suspects that his research unfortunately does not follow this path...
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
It’s not my cup of tea but I think that CNN can be used where we have an image or video analysis. First of all two examples mentioned in the article: self-driving cars and software that plays doctor. Moreover it would be nice to have it in video surveillance or handwriting recognition.
hello, I am sorry that I was not able to join this conversation earlier but you can find my answers posted below:
ReplyDelete1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
yes, it can probably. Example with one different pixel in image (dataset) was not an good example in my opinion. current solutions are able to solve this kind of problem. For instance by generating multiple image transformations.
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
And here is a source of problem, which valuable training sets are. Without properly labelled data we have a problem. Training CNN network is expensive (time and resources)
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
Nope, but I guess in his opinion Capsule Networks are better (maybe proper way)
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?1. Do you think that Capsule Networks can be the next "big thing" in Neural Networks field? Why?
2. What is wrong with today's approach (with CNN - Convolutional neural network)?
3. In this article Hinton said "I think the way we’re doing computer vision is just wrong.". You know what he had on his mind?
4. Can you give some examples where CNNs (Convolutional neural network) are used? If not maybe you can think of some examples in which you could use them?
Visual inspection during production process is very good use case