Aiming at the problem that the traditional modeling methods cannot describe the real characteristics of flyback converters more accurately, the average state model of the fractional-order flyback converter operating in the intermittent inductance current mode was established by using fractional-order calculus and state space average method. The DC equilibrium point and small signal transfer function were obtained, and the difference between the model and the integer order mathematical model was compared. The state space model and circuit simulation model of fractional-order flyback converter were built in Simulink, and the experimental circuit of fractional-order flyback converter was constructed. The state space model, circuit model simulation and experimental results were compared to prove the correctness of the fractional-order model. The results show that the fractional order mathematical model can reflect the change of the order of inductance and capacitance, and the theoretical analysis is better consistent with the actual operation of the converter. A more accurate mathematical model of flyback converter can be established by combining fractional calculus theory with state space average method, which provides a reference for the modeling of other converters.