This paper presents advanced methodology for the analysis of the electroencephalographic activity (EEG) of the brain aimed to monitor the cognitive states of an operator. The methodology of EEG analysis is based on two main approaches: linear methods based on Fourier transform, Linear Stochastic Models, Multi-covariance analysis, and nonlinear methods based on estimation of state space attractor, state space dimension, D2 dimension and the Largest Lyapunov Exponent (LLE). The correct application of these methods is supported by the study of stability, dynamics and space distribution of EEG signal. The uncertainty of adopting a new methodology, such as presented chaos theory, for EEG signal analysis is minimized by the adequate setup of experiments and by evaluation of results against well adopted power spectral estimates calculated by Fourier transform. For better understanding of the underlying processes behind EEG, the basic mental states such as relaxation, single and complex number count, and Raven test are analyzed and compared with the vigilance states. The averaged behavior of the computed markers of the EEG signal is studied with respect to a reaction time scale by the evaluation of a set of experiments. Because of this complex approach, the presented methodology is able to track the ongoing changes in EEG activity during the process of falling asleep. The automatic detection of vigilance changes is a consequent step to this work. Usability of such device in various fields of everyday life is of the high importance.