This paper presents a novel robust optimal control approach for attitude stabilization of a flexible spacecraft in the presence of external disturbances. An optimal control law is formulated by using concepts of inverse optimal control, proportional-integral-derivative control and a control Lyapunov function. A modified extended state observer is used to compensate for the total disturbances. High-gain and second order sliding mode algorithms are merged to obtain the proposed modified extended state observer. The second method of Lyapunov is used to demonstrate its properties including the convergence rate and ultimate boundedness of the estimation error. The proposed controller can stabilize the attitude control system and minimize a cost functional. Moreover, this controller achieves robustness against bounded external disturbances and the disturbances caused by the elastic vibration of flexible appendages. Numerical simulations are provided to demonstrate the performance of the developed controller.