In this paper the definition of Hermite-Hermite matrix polynomials is introduced starting from the Hermite matrix polynomials. An explicit representation, a matrix recurrence relation for the Hermite-Hermite matrix polynomials are given and differential equations satisfied by them is presented. A new expansion of the matrix exponential for a wide class of matrices in terms of Hermite-Hermite matrix polynomials is proposed.