In order to go beyond the black-box deep neural network, this paper introduces a novel learning framework: Embedded Prototype Subspace Classification (EPSC), which is based on subspaces (or manifolds) for image classification (especially handwritten text). The proposed EPSC framework is intended to be both easy to comprehend and visualise, with mathematically well-defined components. This is inspired from the idea that subspaces can be regarded as neural nets.
The idea is to instead use t-SNE to generate compact clusters of prototypes that are found by applying kernel density estimation (KDE) and the watershed transform if the inverse image (IWS). This is basically a way to visualise similarity among images presented in this work and an extension is presented here.
Furthermore, each compact cluster can be represented by a subspace, which are generated by PCA (i.e. the weights i the net are produced by PCA). Hence, the subspace and the corresponding neural net is completely mathematically defined.
The figure above show the neural net based on EPSC that is used for classification. In the first stage, features x of length m are computed from the input data I of length s. Neurons compute the response ρ from weights w, using the quadratic response function, which follows from the fact that the net is derived from computing the norm of a projected vector x into the sub space. In the second layer, the weights ω and the linear response function are used. Finally, the argmax is computed from the k similar nets with the same input features x, but different weights ω. These weights are set to reflect that fact that initial vectors in the PCA are more important than the subsequent ones.
The above illustration show how the EPSC learning process can handle sub-classes within a class. Three different forms of how to write a Japanese Hiragana character appear as three main clusters. The IWS finds each area and the heatmap shows how the different characters look like that are used to form each subspace. The number suggests how many images are forming each heatmap/subspace.
The above illustration shows how the EPSC learning framework takes labeled images I be- longing to different classes and compute features x. Subsequently, t-SNE is applied on each class by choosing the two first values from PCA instead of a random guess. KDE and IWS are used to find k clusters within each class, for which subspaces are constructed.
With this rather simple approach that requires no back propagation, just one t-SNE for each individual label, we were able to achieve 99.2% accuracy for MNIST and 96.92% accuracy for K-MNIST, just to mention two. More results are found in the paper. This is method do not beat the state of the art when it comes to accuracy, but beats most of the methods with comparable accuracy when it comes to speed, both for learning and for classification.
- Embedded Prototype Subspace Classification: A Subspace Learning Framework.
A. Hast, M. Lind, E. Vats.
The 18th International Conference on Computer Analysis of Images and Patterns (CAIP), Salerno, 2-6 September, 2019. pp. 581-592.
@inproceedings{Hast1352303,
author = {Hast, Anders and Lind, Mats and Vats, Ekta},
booktitle = {In: Vento M., Percannella G. (eds) Computer Analysis of Images and Patterns. CAIP 2019.},
publisher = {Springer, Cham},
pages = {581--592},
title = {Embedded Prototype Subspace Classification : A subspace learning framework},
series = {Lecture Notes in Computer Science},
URL = {https://caip2019.unisa.it/},
year = {2019}
}
Anders Hast 
One thought on “Embedded Prototype Subspace Classification: A subspace learning framework”