Aiming at the problems of low recognition accuracy and easy to be affected by illumination conditions in gesture recognition based on yolov3 algorithm, a gesture recognition algorithm based on improved yolov3 is proposed. Firstly, a smaller detection scale is added to the original three detection scales to improve the detection ability of small targets. Secondly, the mean square error loss was replaced by loss to improve the accuracy of prediction frame and regression speed. At the same time, the adaptive balance parameter and focus parameter are used to improve the focal loss function and add it to the total loss function to alleviate the imbalance problem of training samples. The experimental results show that the map index reaches 90.38%, which is 6.62% higher than that before the improvement, and FPS is improved nearly twice.