In order to solve the problem of imbalance of data samples in multi-modal data, It use resource-rich text modal knowledge to help model resource-poor audio modalities, and construct a supervised training method that uses similarity between auxiliary modalities. Emotion recognition neural network. The network can simultaneously recognize emotions and generate feature vectors for emotion recognition tasks of another modality. It consists of two independent text and audio modal representation modules and a cross-modal information interaction module. In the learning process, the emotion recognition features of the text modal are used as labels, and the similarity task with the audio modal is used to Supervise the representation model of the audio modal, which helps the audio modal to achieve better results in its own emotion recognition task. Experimental results show that this neural network can perform four emotion classifications on the IEMOCAP data set to achieve a weighted accuracy of 82.6% and an unweighted accuracy of 81.3%. It provides a reference and method basis for emotion recognition and auxiliary modeling in the multi-modal field of artificial intelligence.