Abstract: In this paper, an autonomous navigation algorithm is based on Line-of-Sight (LOS) vector in order to solve the orbit determination problem of deep space probe during the interplanetary phase. Considering the characters of the deep space mission, the Sun LOS vector and Earth LOS vector are served as the measurement of the navigation system. Following the measurement principle of the Sun senor, the corresponding measurement model and error expression of Sun LOS vector are derived. The measurement model of the pixel and line in the camera coordinates is given through the analysis of the navigation camera, and the measurement error of Earth LOS vector is deduced. The position vector and its error expression are derived in detail based on the geometric relation of the LOS vectors in the inertial space. Finally, the autonomous navigation algorithm is established utilizing extended Kalman filter, and is demonstrated by the practical data of Deep Impact mission.