This article presents the problem of improving the classifier of handwritten letters from historical alphabets, using letter classification algorithms and transliterating them to Latin. We apply it on Palmyrene alphabet, which is a complex alphabet with letters, some of which are very similar to each other. We created a mobile application for Palmyrene alphabet that is able to transliterate hand-written letters or letters that are given as photograph images. At first, the core of the application was based on MobileNet, but the classification results were not suitable enough. In this article, we suggest an improved, better performing convolutional neural network architecture for hand-written letter classifier used in our mobile application. Our suggested new convolutional neural network architecture shows an improvement in accuracy from 0.6893 to 0.9821 by 142% for hand-written model in comparison with the original MobileNet. Future plans are to improve the photographic model as well.