Autonomous robotic wheelchairs based on visual guidance have been devoted to road edge detection. However, the after-detection process, especially the physical interpretation of what had been detected, needs more investigation. There is a wide gap between the scene model based on image processing algorithms and the physical model of the environment where the robotic wheelchair progresses. The aim of this paper is to investigate the interaction between the scene model and the world model; and also a visual control scheme for robot guidance that minimizes the model error induced by processing raw image data is proposed. This solution is developed based on a fuzzy control system, which uses the knowledge base information and the scene model to control the robot motion. On the other hand, the fuzzy control system is finely tuned by feed-backing the mean square errors between the scene model parameters and the knowledge-base data. Finally, the fuzzy controller uses results of these calculations to home the robot on the planned path. This paper also shows the principle of this system and the simulation results confirming the feasibility of the approach.