Abstract: With the development of new power systems in the direction of "double high"(high proportion of new energy and high proportion of power electronic equipment), the problem of broadband oscillation caused by this is becoming more and more serious. In order to better grasp the dynamic characteristics of broadband oscillation, a multi-mode identification of broadband oscillation based on adaptive bandpass filter and GA-VMD-ESPRIT is proposed. Firstly, based on the modal frequency and amplitude information provided by the FFT modal detection grading link, the designed adaptive band-pass filter is used to adaptively set the center frequency, bandwidth and other parameters to realize the extraction or separation of signals in each frequency band. Then the filtered signals are corrected and compensated, and then the variable modal decomposition(GA-VMD) based on genetic algorithm to optimize the parameters is used to decompose the signals in each frequency band, and at the same time, ESPRIT is combined to identify the parameters, so as to obtain the waveforms of each mode as well as the parameter information. Finally, based on Matlab/Simulink, the proposed method is verified by self-synthetic analog signal and dual VSC grid-connected system simulation. The results show that the proposed method is able to accurately decompose and recognize the signals of broadband and multimodal overlapping, which is expected to provide certain technical support for real-time early warning and auxiliary decision-making of broadband oscillation in the future.