The affective state of car drivers has a great impact on their driving behavior. In case of a high stress level, car drivers take more risk in turn over, car following and in neglecting traffic warning signs. In the paper we present first a list of factors which have an impact on the psychological state of the car driver with a focus on the affective state of car drivers. Next we give an outline of our multimodal system to assess the affective state of car drivers by analysis of facial expressions and speech signals. The output of our system is the basic input of a stress-o-meter providing the amount of the stress in a graphical way. We assume that this leads to a reduction of physical arousal and mental reappraisal of the situation. We present some experimental results of our assessment modules. So far we performed only lab experiments. At this moment we research noise canceling techniques in the car environment, which enable video and speech analysis in the car in real traffic environments.