A visual nervous system inspired approach to optical character recognition is proposed in this paper with the hope to touch human performance in a limited extent. Particularly, the application of features motivated by the hierarchical structure of the visual ventral stream for recognition of both English and Persian handwritten digits is investigated. Features are derived by combining position and scale invariant edge detectors in a hierarchy over neighboring positions and multiple orientations. The extracted features are then used to train and test a classifier. We examine three types of classifiers: ANN, SVM and kNN to show that features are not dependent on a specific classifier which is in support of these features. The evaluation of the proposed method over standard Persian and English handwritten digit datasets shows high recognition rates of 99.63% and 98.9%, respectively. A stability analysis is also performed to demonstrate the robustness of this method to orientation, scale, and translation distortions.