Abstract: To solve the problem of low real-time detection and lack of defect images data in fabric defect detection. The fabric defect detection algorithm based on two-stage deep transfer learning is proposed, realizing the real-time detection and efficient training of four types of defects in plaid fabric. First stage transfer, fabric defect prior knowledge transfer algorithm which takes the feature of defects as the transfer object is designed and four kinds of defect pre-boxes parameters with optimal intersection ratios are obtained by clustering algorithm. The pre-boxes with prior knowledge are used to replace the method based on fabric features in defects location, realizing the transfer of the fabric defect features' prior knowledge and improving the location speed of fabric defects. Second stage transfer, a fabric feature extraction ability transfer algorithm is designed to take advantage of the universal feature among different fabrics, the universal feature extraction abilities of convolutional neural network are realized by transfering parameters of linen detection model to patterned plaid detection model, so as to reduce the number of samples that model training need and improve the effectiveness of model training. The experimental results show that, in terms of detection performance, the detection algorithm we proposed has a detection accuracy of 95%, which is better than traditional algorithm and can satisfy the requirements of detection. In terms of model training, the number of fabric defects are reduced at least 50%, and the speed of model training is increased by more than 3 times.