{"id":454,"date":"2021-12-31T00:05:00","date_gmt":"2021-12-30T23:05:00","guid":{"rendered":"https:\/\/raulbalanza.me\/?p=454"},"modified":"2024-01-18T23:18:51","modified_gmt":"2024-01-18T22:18:51","slug":"recognizing-handwritten-digits-efficiently","status":"publish","type":"post","link":"https:\/\/raulbalanza.me\/es\/2021\/12\/31\/recognizing-handwritten-digits-efficiently\/","title":{"rendered":"Recognizing handwritten digits efficiently"},"content":{"rendered":"


Este art\u00edculo fue escrito originalmente en Ingl\u00e9s. Sin embargo, una versi\u00f3n traducida autom\u00e1ticamente se encuentra aqu\u00ed<\/a>.<\/em><\/p>\n\n\n\n

The complete scientific paper with a detailed description of all the performed experiments and their solutions is available aqu\u00ed<\/a>.<\/em><\/p>\n\n\n\n

In September of this year, I moved to Prague (Czech Republic \ud83c\udde8\ud83c\uddff) to spend five months studying at the Czech Technical University in Prague<\/a> (\u010cVUT) as part of an international exchange program during the last semester of my Bachelor’s degree.<\/p>\n\n\n\n

For the semestral work of one of the courses that I am taking, I have had the opportunity of doing some research to compare several Artificial Intelligence<\/em> approaches for the same purpose: recognizing and classifying handwritten digits.<\/p>\n\n\n\n

The task of recognizing handwritten digits<\/strong> consists of observing an image of a written number and determining to which digit the image corresponds. This task, which is trivial for humans, has been deeply studied in the AI field, obtaining almost perfect results. However, the objective of this research was not (only) to find the most accurate approach to take: it was to find a balance between high accuracy and high ‘efficiency’ (i.e., a model that required the least amount of computing resources).<\/p>\n\n\n\n

The dataset<\/h2>\n\n\n\n

MNIST<\/a><\/em>, a very well-known dataset of computer vision, was the employed dataset for all the experiments that were carried out. This dataset has served since its release as the basis for benchmarking classification algorithms. It consists of a training set of 60K examples and a test set of 10K examples.<\/p>\n\n\n\n

\"\"
Sample digits from the MNIST dataset<\/figcaption><\/figure><\/div>\n\n\n\n

Although this dataset is effectively solved, it is still widely used for estimating the performance of some models, and deciding how to explore possible improvements. There also exist several variants of this dataset with different types of data, designed for similar purposes.<\/p>\n\n\n\n

Experiments<\/h2>\n\n\n\n

During the whole research, a total of four computational intelligence techniques were used, with at least three variations for each of them in order to optimize as far as possible the accuracy of each model. A brief description of each of them follows.<\/p>\n\n\n\n

Convolutional Neural Networks<\/h5>\n\n\n\n

CNNs are Deep Learning models that assign importance to various aspects of the input samples and are able to differentiate them from each other. The preprocessing required using this approach is much lower when compared to other classification algorithms, and the architecture is analogous to that of the connectivity pattern of neurons in the human brain.<\/p>\n\n\n

\n
\"\"
Example scheme of a CNN designed for the MNIST dataset<\/figcaption><\/figure><\/div>\n\n\n

After designing a model and applying certain modifications to try to improve the final results, an accuracy of 98.85%<\/strong> was reached during training<\/span>; and 99.31%<\/strong> during testing<\/span>.<\/p>\n\n\n\n

Zero-shot learning<\/h5>\n\n\n\n

ZSL is a problem setup in machine learning where, at test time, a learner observes samples from classes that were not observed during training, and needs to predict the class they belong to. These methods generally work by associating observed and non-observed classes through some form of auxiliary information, which encodes distinguishing properties of objects.<\/p>\n\n\n

\n
\"\"
Example scheme of a ZSL model<\/figcaption><\/figure><\/div>\n\n\n

A model was designed in which some digits where provided in the training phase, and some only in the testing (inference) phase, an accuracy of 72.42%<\/strong> was reached during training<\/span>; and 48.06%<\/strong> during testing<\/span>.<\/p>\n\n\n\n

However, it must be taken into account that this is a predictive approach with a limited amount of information in training time, an aspect that makes the results still considerably good.<\/p>\n\n\n\n

k<\/em>-Nearest Neighbors classifier<\/h5>\n\n\n\n

k<\/em>-NN is a non-parametric classification method used for classification, among others. The input consists of the k<\/em> closest training examples in a dataset, and the output is a class membership. An object is classified to the most common class among its nearest neighbors.<\/p>\n\n\n

\n
\"\"
k-NN method classification example<\/figcaption><\/figure><\/div>\n\n\n

In this case, data was preprocessed using the PCA<\/a> dimensionality reduction technique (to remove noisy data in each sample), and then classified using the k<\/em>-NN method, using different values for k<\/em>. An accuracy of 97.60%<\/strong> was reached during training<\/span>; and 97.16%<\/strong> during testing<\/span>.<\/p>\n\n\n\n

Probability distributions<\/h5>\n\n\n\n

Finally, the last tested approach was to assume that the data followed different probability distributions, and to try estimating its parameters. In total, three distributions were explored.<\/p>\n\n\n\n

Some smoothing techniques were applied on the data to avoid overfitting and, because each of these probability distributions are suitable for different kinds of data, it was required to preprocess the samples in some cases.<\/p>\n\n\n\n

The results obtained for each of the distributions were:<\/p>\n\n\n\n