In order to improve the accuracy of weed recognition in cane fields under natural environment, a weed detection algorithm based on improved Faster Regionbased Convolutional Neural Network (Faster RCNN) was proposed. Firstly, in the feature extraction stage, the balanced feature pyramid module was used to balance the semantic features at all levels to strengthen the extraction of deep features of weed images. Secondly, the dynamic label assignment was used to dynamically adjust the label prediction threshold of the network to solve the problem of scarcity of positive samples in the early stage of training. Finally, soft nonmaximum suppression was used to optimize the model, which was able to reduce the missed detection of singletype targets and improve the positioning accuracy of targets by improving the nonmaximum suppression of the original model.The experimental results showed that the mean average precision of the optimized algorithm reached 81.3%, which compared with the original Faster RCNN algorithm, the precision was improved by 6.2%, and the average test time for each image was 0.132 s. There were 6.5%, 4.7%, and 5.1% improvements in the average precision of the intersection over union of 0.5 and the across scale of small and medium, respectively. The proposed algorithm has high detection precision and stability, which can meet the needs of sugarcane field weed detection in complex natural environment.