Abstract: The magnetic properties of the fluxgate sensor core determine that signals at multiple harmonic frequencies can be generated under single-frequency excitation, all of which can accurately reflect the magnitude of the measured magnetic field. Based on the noise characterization of each harmonic signal of the fluxgate in the open-loop configuration, this paper investigates the optimization method for signal demodulation of fluxgate magnetometers to achieve the optimal system noise performance. First, a theoretical analysis of the noise characteristics of fluxgate sensors is performed. Subsequently, experiments including probe harmonic characterization, open-loop and closed-loop magnetometer performance tests are designed to measure the performance parameters of each even harmonic, compare the demodulation performance of sinusoidal and square-wave signals, and explore the influence of phase deviation on demodulation performance. Finally, the optimal demodulation ratio for multi-harmonic signals is derived based on the Lagrange multiplier method. The experimental results demonstrate that square-wave demodulation has significant advantages over sinusoidal demodulation: the magnetic noise is reduced by 14.6% in the open-loop configuration and by 27% in the closed-loop configuration. In addition, the phase deviation must be controlled within 25° to preserve the noise advantage of square-wave demodulation. The optimal demodulation ratio of the 2nd, 6th, and 10th harmonics is finally derived to be approximately 1:2:0.5, which achieves the optimal trade-off between voltage noise suppression and gain retention.