With the development of deep learning network model applications in the field of computer vision, the performance of object detection in various agricultural scenarios has been greatly boosted. Unlike large-scale deep learning networks deployed in cloud servers, lightweight deep learning networks, due to their smaller number of parameters and computing power, show potential in agricultural scenarios with limited hardware resources and higher real-time requirements, such as the object detection of fruit and vegetable picking robots, object detection of crop pests and weeds, and crop phenotyping, among other tasks. We provide an overview of the model structure, key technology modules and model performance of the current mainstream lightweight deep learning networks, and conduct a comparative analysis; summarize the research progress of lightweight deep learning networks in three major application scenarios, namely, fruit object detection, grain spike detection, and crop pest and disease detection; and analyze the scarcity of universal datasets, the weakness of the model generalization ability, the accuracy and efficiency of model detection, and the lack of model generalization ability in the application of lightweight deep learning networks in the detection of crop targets. It also analyses the scarcity of universal datasets, weak model generalization ability, and difficulty in balancing model detection accuracy and detection efficiency in crop object detection applications, and looks forward to further improving the object detection performance through the enhancement of agricultural datasets in terms of quantity, quality, and diversity, the optimization of the structure of the lightweight deep learning network, the application of migration learning, and the hardware acceleration technology of edge devices.