深度学习网络fine-tune原理研究 - 以卷积神经网络为例 - 郑瀚Andrew
2023-4-26 17:51:0 Author: www.cnblogs.com(查看原文) 阅读量:21 收藏
2023-4-26 17:51:0 Author: www.cnblogs.com(查看原文) 阅读量:21 收藏
预训练模型就是已经用数据集训练好了的模型，这里的数据集一般指大型数据集。比如
- VGG16/19
- Resnet
- Imagenet
- COCO
正常情况下，在图像识别任务中常用的VGG16/19等网络是他人调试好的优秀网络，我们无需再修改其网络结构。
参考资料：
https://zhuanlan.zhihu.com/p/35890660 https://github.com/szagoruyko/loadcaffe
用一个单神经元网络解释模型微调的基本原理，
- Step1：假设我们的神经网络符合下面的形式：Y = W * X
- Step2：现在我们要找到一个W，使得当输入X=2时，输出Y=1，也就是希望W=0.5：1 = W * 2
- Step3：按照神经网络的基本训练过程，首先要对W进行初始化，初始化的值符合均值为0，方差为1的分布，假设W初始化为0.1：Y = 0.1 * X
- Step4：现在开始训练FP过程，当输入X=2时，W=0.1，输出Y=0.2，这个时候实际值和目标值1的误差为0.8：1 <====== 0.2 = 0.1 * 2
- Step5：开始BP反向传导，0.8的误差经过反向传播去更新权值W，假如这次更新为W=0.2，输出位0.4，与目标值的误差为0.6：1 <====== 0.4 = 0.2 * 2
- Step6：可能经过10次或20次BP反向传导，W终于得到了我们想要的0.5：Y = 0.5 * X
- Step7：如果最开始初始化的时候有人告诉你，W的值应该在0.47附近
- Step8：那么从最开始训练，你与目标值的误差就只有0.06了，那么可能只要一步两步BP，就能将W调整到0.5：1 <====== 0.94 = 0.47 * 2
Step7就相当于给你一个预训练模型（pre-trained model），Step8就是基于这个预训练模型去微调（fine-tune）。
可以看到，相对于从头开始训练，微调省去了大量计算资源和计算时间，提高了计算效率，甚至提高了准确率（因为在超大规模训练过程中，模型可能陷入局部次优空间中无法跳出，预训练相当于已经探好了最难的一部分路，后面的路下游模型走起来就轻松了）。
细心的读者可能会注意到，预训练模型对下游fine-tune任务效果的好坏，和以下几个因素有关：
- 预训练模型训练所用的语料和下游fine-tune任务的重合度：本质上，预训练模型的模型权重参数，代表的是喂入预训练模型的语料。如果预训练任何和下游fine-tune任务领域相差太大，则预训练模型的参数几乎不能起到提效的帮助，甚至可能帮倒忙。
- 预训练模型自身的容量：理论上，如果预训练模型足够大，能够包含下游任务的一部分核心部分，则预训练模型可以通过权重重调整，在fine-tune的过程中，激活一部分神经元以及关闭一部分神经元，以此使预训练模型朝着下游任务的方向去“生长”。
- 预训练模型使用的语料库是否足够大和种类丰富，因为这决定了预训练模型是否完成了足够的预训练，否则如果上游预训练模型没有完成收敛，接入下游fine-tune的时候，预训练模型也依然需要进行大量的微调，这对极大拖慢整体模型的收敛。反之，如果预训练模型已经基本完成了收敛，则对下游fine-tune训练的数据集要求就很小，fine-tune就可以基于一个小数据集依然可以得到较好的效果，同时也仅需要较少的训练时间。
- 预训练模型输入层的向量化方式、张量维度、嵌入方式、编码方位、shape维度等等，和下游fine-tune任务的这些参数结构是否完全一致（或者是否具备一定的迁移性），理论上说，输入层的结构是一种特征工程的经验形式，它本身也代表了模型对目标任务的某种抽象。打个比方，用于文本生成任务的模型，如果将一个像素图片“强行转换适配”输入进去，最终训练和预测的效果都不会好。
卷积神经网络的核心是：
- 浅层卷积层提取基础特征，比如边缘，轮廓等基础特征
- 深层卷积层提取抽象特征，比如整个脸型
- 全连接层根据特征组合进行评分分类
使用大型数据集训练的预训练模型，已经具备了提取浅层基础特征和深层抽象特征的能力。相比不做微调，这种方法具备以下优势：
- 避免了从头开始训练，减少了训练时间，节省了计算资源
- 避免了模型不收敛、参数不够优化、准确率低、模型泛化能力低、容易过拟合等问题
数据集1：数据量少，但数据相似度非常高
在这种情况下，我们所做的只是修改最后几层或最终的softmax图层的输出类别。
数据集2：数据量少，数据相似度低
在这种情况下，我们可以冻结预训练模型的初始层（比如k层），并再次训练剩余的（n-k）层。由于新数据集的相似度较低，因此根据新数据集对较高层进行重新训练具有重要意义。
数据集3 - 数据量大，数据相似度低
在这种情况下，由于我们有一个大的数据集，我们的神经网络训练将会很有效。但是，由于我们的数据与用于训练我们的预训练模型的数据相比有很大不同。使用预训练模型进行的预测不会有效。因此，最好根据你的数据从头开始训练神经网络（Training from scatch）。
数据集4：数据量大，数据相似度高
这是理想情况。在这种情况下，预训练模型应该是最有效的。使用模型的最好方法是保留模型的体系结构和模型的初始权重。然后，我们可以使用在预先训练的模型中的权重来重新训练该模型。
- 通常的做法是截断预先训练好的网络的最后一层（softmax层），并用与我们自己的问题相关的新的softmax层替换它。例如，ImageNet上预先训练好的网络带有1000个类别的softmax图层。如果我们的任务是对10个类别的分类，则网络的新softmax层将由10个类别组成，而不是1000个类别。然后，我们在网络上运行预先训练的权重。确保执行交叉验证，以便网络能够很好地推广。
- 使用较小的学习率来训练网络。由于我们预计预先训练的权重相对于随机初始化的权重已经相当不错，我们不想过快地扭曲它们太多。通常的做法是使初始学习率比用于从头开始训练（Training from scratch）的初始学习率小10倍。
- 如果数据集数量过少，我们进来只训练最后一层，如果数据集数量中等，冻结预训练网络的前几层的权重也是一种常见做法。这是因为前几个图层捕捉了与我们的新问题相关的通用特征，如曲线和边。我们希望保持这些权重不变。相反，我们会让网络专注于学习后续深层中特定于数据集的特征。
常见的预训练分类网络有牛津的VGG模型、谷歌的Inception模型、微软的ResNet模型等，他们都是预训练的用于分类和检测的卷积神经网络（CNN）。
本次选用的是VGG16模型，是一个在ImageNet数据集上预训练的模型，分类性能优秀，对其他数据集适应能力优秀。
0x1：直接基于VGG16进行手写数字预测
from tensorflow.keras.applications.vgg16 import VGG16 from tensorflow.keras.preprocessing import image from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions import numpy as np model = VGG16(weights='imagenet') img_path = '6.webp' img = image.load_img(img_path, target_size=(224, 224)) x = image.img_to_array(img) x = np.expand_dims(x, axis=0) x = preprocess_input(x) preds = model.predict(x) # decode the results into a list of tuples (class, description, probability) # (one such list for each sample in the batch) print('Predicted:', decode_predictions(preds, top=3)[0])
6.webp
输出结果：
Predicted: [('n03532672', 'hook', 0.4591384), ('n02910353', 'buckle', 0.032941677), ('n01930112', 'nematode', 0.032439113)]
可以看到，VGG16输出的最高概率预测结果是hook，很明显，VGG16的训练集并没有关于数字图片的样本。
0x2：通过手写数字，可视化VGG16各个层参数
from keras.models import Model from tensorflow.keras.applications.vgg16 import VGG16 from tensorflow.keras.preprocessing import image from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions import numpy as np import cv2 import matplotlib.pyplot as plt def vis_conv(images, n, name, t): """visualize conv output and conv filter. Args: img: original image. n: number of col and row. t: vis type. name: save name. """ size = 64 margin = 5 if t == 'filter': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin, 3)) if t == 'conv': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin)) for i in range(n): for j in range(n): if t == 'filter': filter_img = images[i + (j * n)] if t == 'conv': filter_img = images[..., i + (j * n)] filter_img = cv2.resize(filter_img, (size, size)) # Put the result in the square `(i, j)` of the results grid horizontal_start = i * size + i * margin horizontal_end = horizontal_start + size vertical_start = j * size + j * margin vertical_end = vertical_start + size if t == 'filter': results[horizontal_start: horizontal_end, vertical_start: vertical_end, :] = filter_img if t == 'conv': results[horizontal_start: horizontal_end, vertical_start: vertical_end] = filter_img # Display the results grid plt.imshow(results) plt.savefig('images/{}_{}.jpg'.format(t, name), dpi=600) plt.show() def conv_output(model, layer_name, img): """Get the output of conv layer. Args: model: keras model. layer_name: name of layer in the model. img: processed input image. Returns: intermediate_output: feature map. """ # this is the placeholder for the input images input_img = model.input try: # this is the placeholder for the conv output out_conv = model.get_layer(layer_name).output except: raise Exception('Not layer named {}!'.format(layer_name)) # get the intermediate layer model intermediate_layer_model = Model(inputs=input_img, outputs=out_conv) # get the output of intermediate layer model intermediate_output = intermediate_layer_model.predict(img) return intermediate_output[0] if __name__ == '__main__': model = VGG16(weights='imagenet') img_path = '6.webp' img = image.load_img(img_path, target_size=(224, 224)) x = image.img_to_array(img) x = np.expand_dims(x, axis=0) x = preprocess_input(x) preds = model.predict(x) # decode the results into a list of tuples (class, description, probability) # (one such list for each sample in the batch) print('Predicted:', decode_predictions(preds, top=3)[0]) conv_output_block1_conv1 = conv_output(model, "block1_conv1", x) print("block1_conv1: ", conv_output_block1_conv1) vis_conv(conv_output_block1_conv1, 8, "block1_conv1", 'conv') conv_output_block1_conv2 = conv_output(model, "block1_conv2", x) print("block1_conv2: ", conv_output_block1_conv2) vis_conv(conv_output_block1_conv2, 8, "block1_conv2", 'conv') conv_output_block2_conv1 = conv_output(model, "block2_conv1", x) print("block2_conv1: ", conv_output_block2_conv1) vis_conv(conv_output_block2_conv1, 8, "block2_conv1", 'conv') conv_output_block2_conv2 = conv_output(model, "block2_conv2", x) print("block2_conv2: ", conv_output_block2_conv2) vis_conv(conv_output_block2_conv2, 8, "block2_conv2", 'conv') conv_output_block3_conv1 = conv_output(model, "block3_conv1", x) print("block3_conv1: ", conv_output_block3_conv1) vis_conv(conv_output_block3_conv1, 8, "block3_conv1", 'conv') conv_output_block3_conv2 = conv_output(model, "block3_conv2", x) print("block3_conv2: ", conv_output_block3_conv2) vis_conv(conv_output_block3_conv2, 8, "block3_conv2", 'conv') conv_output_block5_conv3 = conv_output(model, "block5_conv3", x) print("block5_conv3: ", conv_output_block5_conv3) vis_conv(conv_output_block5_conv3, 8, "block5_conv3", 'conv') print("fc1: ", conv_output(model, "fc1", x)) print("fc2: ", conv_output(model, "fc2", x)) print("predictions: ", conv_output(model, "predictions", x))
输出结果：
1/1 [==============================] - 2s 2s/step Predicted: [('n03532672', 'hook', 0.4591384), ('n02910353', 'buckle', 0.032941677), ('n01930112', 'nematode', 0.032439113)] 1/1 [==============================] - 0s 53ms/step block1_conv1: [[[ 0. 42.11969 0. ... 0. 32.04823 0. ] [ 0. 46.303555 82.50592 ... 0. 324.38284 164.56157 ] [ 0. 46.303555 82.50592 ... 0. 324.38284 164.56157 ] ... [ 0. 46.303555 82.50592 ... 0. 324.38284 164.56157 ] [ 0. 46.303555 82.50592 ... 0. 324.38284 164.56157 ] [ 2.61003 32.20762 173.75212 ... 0. 517.4678 391.77734 ]] [[ 0. 56.784718 0. ... 0. 0. 0. ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] ... [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [377.4901 38.781555 204.19121 ... 0. 382.94656 378.29724 ]] [[ 0. 56.784718 0. ... 0. 0. 0. ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] ... [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [377.4901 38.781555 204.19121 ... 0. 382.94656 378.29724 ]] ... [[ 0. 56.784718 0. ... 0. 0. 0. ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] ... [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [377.4901 38.781555 204.19121 ... 0. 382.94656 378.29724 ]] [[ 0. 56.784718 0. ... 0. 0. 0. ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] ... [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [ 2.401019 58.89123 63.275116 ... 0. 2.2158926 10.105784 ] [377.4901 38.781555 204.19121 ... 0. 382.94656 378.29724 ]] [[ 0. 39.011864 0. ... 0. 0. 0. ] [323.5314 39.00029 97.09346 ... 0. 0. 67.22728 ] [323.5314 39.00029 97.09346 ... 0. 0. 67.22728 ] ... [323.5314 39.00029 97.09346 ... 0. 0. 67.22728 ] [323.5314 39.00029 97.09346 ... 0. 0. 67.22728 ] [523.7337 25.070164 184.00014 ... 0. 144.83621 315.41928 ]]] 1/1 [==============================] - 0s 84ms/step block1_conv2: [[[982.48444 65.59724 0. ... 81.02978 698.99084 172.65338 ] [256.9937 101.16306 8.7225065 ... 203.38603 340.56735 0. ] [314.77548 126.94779 0. ... 159.34764 175.0137 0. ] ... [314.77548 126.94779 0. ... 159.34764 175.0137 0. ] [ 63.2487 0. 0. ... 125.09357 413.46884 33.402287 ] [ 0. 0. 0. ... 32.059208 0. 7.143284 ]] [[401.39062 97.3492 0. ... 134.1313 454.73416 0. ] [ 0. 97.926704 136.89134 ... 259.61768 632.9747 0. ] [ 0. 125.44156 95.91204 ... 174.20306 390.24847 0. ] ... [ 0. 125.44156 95.91204 ... 174.20306 390.24847 0. ] [ 0. 0. 109.98622 ... 103.348114 854.354 0. ] [ 0. 0. 0. ... 0. 394.38068 0. ]] [[396.95483 167.3767 0. ... 69.25613 207.11255 4.1853294] [ 0. 174.81584 76.58766 ... 161.11617 339.40433 0. ] [151.61284 87.23442 16.130083 ... 6.742235 1.1302795 0. ] ... [151.61284 87.23442 16.130083 ... 6.742235 1.1302795 0. ] [ 0. 0. 70.19446 ... 0. 479.9812 254.07501 ] [ 0. 0. 0. ... 0. 199.8518 50.87436 ]] ... [[396.95483 167.3767 0. ... 69.25613 207.11255 4.1853294] [ 0. 174.81584 76.58766 ... 161.11617 339.40433 0. ] [151.61284 87.23442 16.130083 ... 6.742235 1.1302795 0. ] ... [151.61284 87.23442 16.130083 ... 6.742235 1.1302795 0. ] [ 0. 0. 70.19446 ... 0. 479.9812 254.07501 ] [ 0. 0. 0. ... 0. 199.8518 50.87436 ]] [[196.74297 0. 0. ... 76.20704 371.12302 239.03537 ] [ 0. 0. 54.11582 ... 132.80391 642.51025 472.34528 ] [ 0. 0. 4.422485 ... 7.28855 283.40457 706.94666 ] ... [ 0. 0. 4.422485 ... 7.28855 283.40457 706.94666 ] [ 0. 0. 54.947617 ... 0. 688.73157 731.2318 ] [ 0. 0. 0. ... 0. 364.4021 284.65625 ]] [[ 0. 0. 0. ... 0. 0. 0. ] [ 0. 0. 0. ... 0. 407.869 0. ] [ 0. 0. 0. ... 0. 198.98882 101.46747 ] ... [ 0. 0. 0. ... 0. 198.98882 101.46747 ] [ 0. 0. 0. ... 0. 534.15466 69.81046 ] [287.62454 0. 0. ... 0. 764.0485 0. ]]] 1/1 [==============================] - 0s 76ms/step block2_conv1: [[[ 0. 0. 146.08685 ... 1138.9917 0. 1914.1439 ] [ 0. 0. 617.18994 ... 630.32166 0. 0. ] [ 0. 0. 479.59012 ... 803.52374 0. 281.59882 ] ... [ 0. 0. 479.59012 ... 803.52374 0. 281.59882 ] [ 0. 0. 583.4128 ... 895.7679 0. 715.7333 ] [ 0. 0. 1087.817 ... 2163.6226 0. 0. ]] [[ 0. 657.53296 0. ... 660.99 461.2479 1719.0864 ] [ 0. 823.556 349.60562 ... 0. 542.6992 0. ] [ 0. 748.83795 131.92645 ... 30.981398 517.1108 82.481895] ... [ 0. 748.83795 131.92645 ... 30.981398 517.1108 82.481895] [ 0. 826.5497 252.64777 ... 64.045074 392.9257 619.41876 ] [ 0. 693.9135 1073.2073 ... 1989.0895 697.90814 0. ]] [[ 0. 239.73143 0. ... 901.56885 274.7921 1343.2406 ] [ 0. 214.44774 181.45721 ... 0. 279.94656 0. ] [ 0. 130.28665 0. ... 90.52182 205.50911 130.00967 ] ... [ 0. 130.28665 0. ... 90.52182 205.50911 130.00967 ] [ 0. 230.28584 60.274647 ... 54.528107 35.845345 758.34717 ] [ 0. 283.4764 837.31805 ... 1669.6423 417.16782 390.9171 ]] ... [[ 0. 239.73143 0. ... 901.56885 274.7921 1343.2406 ] [ 0. 214.44774 181.45721 ... 0. 279.94656 0. ] [ 0. 130.28665 0. ... 90.52182 205.50911 130.00967 ] ... [ 0. 130.28665 0. ... 90.52182 205.50911 130.00967 ] [ 0. 230.28584 60.274647 ... 54.528107 35.845345 758.34717 ] [ 0. 283.4764 837.31805 ... 1669.6423 417.16782 390.9171 ]] [[ 0. 149.2003 0. ... 467.1346 130.91127 1713.3496 ] [ 0. 89.11 283.70944 ... 0. 236.00652 0. ] [ 0. 21.128517 52.216312 ... 0. 233.49413 93.75622 ] ... [ 0. 21.128517 52.216312 ... 0. 233.49413 93.75622 ] [ 0. 120.84711 171.13362 ... 0. 73.68687 632.3945 ] [ 0. 207.82211 976.44196 ... 1907.8083 525.08185 29.64562 ]] [[ 0. 296.92758 171.61426 ... 975.3303 292.51434 1616.5455 ] [ 0. 235.07794 710.6981 ... 276.39038 0. 0. ] [ 0. 116.03024 512.0845 ... 650.45764 53.27237 331.76382 ] ... [ 0. 116.03024 512.0845 ... 650.45764 53.27237 331.76382 ] [ 0. 247.85234 603.1937 ... 753.06476 57.02111 653.146 ] [ 0. 435.59036 1229.345 ... 2149.0642 365.4059 0. ]]] WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7ff36c074790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. 1/1 [==============================] - 0s 96ms/step block2_conv2: [[[ 19.134865 65.2908 388.85107 ... 77.567345 0. 0. ] [ 385.78787 0. 83.92136 ... 823.738 0. 0. ] [ 362.76718 0. 0. ... 770.1545 0. 0. ] ... [ 370.19595 0. 0. ... 693.7316 0. 0. ] [ 395.07098 1163.4445 0. ... 685.89105 0. 0. ] [ 393.64594 221.8914 0. ... 779.5206 0. 0. ]] [[ 0. 0. 658.96985 ... 266.29254 1334.6693 0. ] [ 175.15945 0. 0. ... 927.1358 410.14014 0. ] [ 113.65867 0. 0. ... 705.73663 115.82475 341.95673 ] ... [ 89.81759 278.56213 0. ... 651.8543 775.20416 502.7654 ] [ 136.82233 1937.8406 0. ... 647.9445 302.8629 525.4279 ] [ 262.19644 357.42938 0. ... 750.1874 0. 489.33453 ]] [[ 0. 0. 418.21606 ... 12.688118 795.45483 0. ] [ 234.67218 0. 0. ... 426.10312 0. 0. ] [ 145.08507 0. 0. ... 287.3707 0. 296.64294 ] ... [ 103.087685 305.11697 62.120567 ... 267.3017 545.9968 524.84625 ] [ 235.22937 2067.736 239.66722 ... 172.1788 407.2032 489.35236 ] [ 323.7679 407.43408 319.0578 ... 341.47412 0. 345.82104 ]] ... [[ 0. 0. 580.24994 ... 68.54731 589.51636 0. ] [ 201.64163 0. 157.14062 ... 501.0832 0. 0. ] [ 133.07848 0. 0. ... 351.53003 0. 415.4161 ] ... [ 86.24023 465.5442 22.741163 ... 337.74213 215.66536 622.05804 ] [ 174.42499 2174.4937 46.142918 ... 286.23798 212.43034 572.5916 ] [ 282.7715 504.28677 132.34572 ... 501.6414 0. 371.98062 ]] [[ 0. 0. 247.89134 ... 337.7562 870.8283 0. ] [ 129.28552 0. 0. ... 976.0519 0. 0. ] [ 0. 107.290855 0. ... 696.99493 0. 248.08282 ] ... [ 0. 545.71716 0. ... 687.88995 175.53624 456.3958 ] [ 40.394768 2056.4695 0. ... 716.48956 157.10045 438.98425 ] [ 169.84534 324.61182 357.57187 ... 724.79034 0. 279.55737 ]] [[ 0. 0. 0. ... 108.35586 1594.9191 0. ] [ 0. 0. 0. ... 641.5959 631.3734 0. ] [ 0. 0. 0. ... 476.7445 236.77658 0. ] ... [ 0. 0. 0. ... 514.81213 659.1744 0. ] [ 0. 558.51337 0. ... 529.3481 646.179 0. ] [ 0. 0. 318.3686 ... 567.25116 0. 85.41164 ]]] WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7ff36c0250d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. 1/1 [==============================] - 0s 86ms/step block3_conv1: [[[ 104.03467 0. 7676.7437 ... 284.6595 104.21471 495.0637 ] [ 0. 313.47745 5637.235 ... 773.39124 312.671 710.7272 ] [ 0. 626.0542 4799.9775 ... 797.72 329.52908 588.2553 ] ... [ 0. 646.8998 4819.6846 ... 770.5025 316.77924 555.9198 ] [ 0. 247.11465 5635.4976 ... 528.78986 281.3929 570.0344 ] [ 30.971907 0. 7807.489 ... 149.22829 247.03853 569.7642 ]] [[ 0. 871.3891 5385.873 ... 138.57967 142.74121 983.03674 ] [ 0. 1012.0134 499.58597 ... 162.09428 256.54013 1158.3336 ] [ 0. 1021.0573 28.230726 ... 184.61717 219.79193 785.92285 ] ... [ 0. 1050.2477 0. ... 146.40399 266.05975 744.28723 ] [ 0. 998.98596 374.99014 ... 64.251434 274.85852 940.72205 ] [ 0. 715.7788 5695.999 ... 181.9697 113.495964 998.7362 ]] [[ 0. 715.5003 4931.604 ... 174.02486 218.0967 733.6579 ] [ 0. 782.053 98.879425 ... 201.88213 215.22943 785.8126 ] [ 0. 754.37915 0. ... 156.65364 84.32829 489.68857 ] ... [ 0. 784.71075 0. ... 119.22443 137.86731 454.97656 ] [ 0. 741.68567 0. ... 42.16644 243.78513 592.8224 ] [ 0. 564.874 5148.9604 ... 128.61302 147.20853 733.89886 ]] ... [[ 0. 496.68298 4885.435 ... 318.65524 245.03665 575.7172 ] [ 0. 486.5161 27.83389 ... 512.1368 232.01933 566.13635 ] [ 0. 477.45157 0. ... 499.04877 68.24934 263.7914 ] ... [ 0. 499.77722 0. ... 459.50702 132.83049 226.18076 ] [ 0. 439.5999 0. ... 320.2604 207.68942 371.76605 ] [ 0. 339.14404 5100.4336 ... 253.7242 112.67809 590.3231 ]] [[ 0. 347.25443 5573.7017 ... 627.8705 275.148 631.8805 ] [ 0. 358.82916 292.3079 ... 979.4485 303.31757 662.5002 ] [ 0. 478.66336 0. ... 1011.04913 144.6257 358.16284 ] ... [ 0. 500.74857 0. ... 972.3128 223.55475 336.5134 ] [ 0. 355.48328 104.18472 ... 832.22375 270.79025 496.9038 ] [ 0. 219.11375 5712.4497 ... 539.98773 84.06546 667.78613 ]] [[ 0. 604.2773 7762.388 ... 492.06854 294.44586 373.23422 ] [ 0. 660.0235 5493.3257 ... 210.03978 176.89102 304.05936 ] [ 0. 675.077 4603.5874 ... 169.29701 125.09003 53.69849 ] ... [ 0. 701.2141 4594.911 ... 142.22992 227.38722 59.698753] [ 0. 718.4968 5527.42 ... 161.2458 129.69702 249.47922 ] [ 0. 586.12274 8277.507 ... 435.10352 0. 348.29013 ]]] 1/1 [==============================] - 0s 105ms/step block3_conv2: [[[ 0. 971.66376 794.8841 ... 172.1506 10.597431 794.8708 ] [ 0. 291.7925 826.4213 ... 39.319454 0. 718.3281 ] [ 0. 156.54356 802.0568 ... 0. 0. 503.39447 ] ... [ 0. 401.88135 1241.3585 ... 0. 0. 362.15497 ] [ 0. 675.3719 1448.097 ... 0. 9.820769 410.58932 ] [ 0. 10.890532 953.4981 ... 233.22906 0. 579.7396 ]] [[ 575.767 1863.7603 592.8948 ... 245.05453 0. 1068.8091 ] [ 514.8801 844.0041 222.19751 ... 0. 0. 788.1397 ] [ 19.14704 444.27817 111.57798 ... 0. 0. 409.57492 ] ... [ 252.99167 848.908 513.1679 ... 0. 0. 312.90305 ] [ 591.92786 1448.2924 630.19824 ... 0. 0. 504.8597 ] [ 0. 379.8196 763.054 ... 0. 72.78092 733.65424 ]] [[ 287.43423 1910.7128 349.80966 ... 387.3527 0. 1265.1278 ] [ 0. 740.3088 124.85873 ... 0. 0. 918.3699 ] [ 0. 286.83832 118.424774 ... 0. 177.10791 486.00412 ] ... [ 0. 735.8566 558.8175 ... 0. 193.26689 449.90454 ] [ 53.59411 1525.2466 651.7935 ... 0. 103.276146 716.995 ] [ 0. 603.8922 836.88104 ... 50.30762 191.5637 884.57367 ]] ... [[ 292.4923 1834.398 444.55945 ... 540.1754 14.972595 1457.0437 ] [ 0. 642.0181 319.91138 ... 44.719204 156.22743 1106.5459 ] [ 0. 170.11359 338.21768 ... 0. 376.42972 603.82666 ] ... [ 0. 581.471 737.77203 ... 0. 400.47705 579.5313 ] [ 0. 1367.3385 798.4122 ... 0. 260.49323 826.0262 ] [ 0. 544.79816 826.0728 ... 77.14375 283.54224 990.2182 ]] [[ 584.30676 1950.905 596.8577 ... 740.97327 81.50432 1820.6097 ] [ 116.4952 835.781 588.2435 ... 225.01852 196.70117 1720.8013 ] [ 0. 356.6451 615.6922 ... 77.022446 354.97284 1198.3191 ] ... [ 0. 733.50824 1012.90985 ... 0. 296.32776 1099.1088 ] [ 186.0945 1339.9901 1179.6779 ... 0. 191.2773 1315.7777 ] [ 0. 384.91098 1044.8905 ... 228.41646 209.99303 1404.8423 ]] [[ 608.40894 1603.2566 899.59283 ... 999.1029 64.82636 1448.8973 ] [ 733.8801 1092.808 762.7826 ... 444.4963 137.76027 1666.6692 ] [ 293.81265 823.8305 784.1011 ... 267.9691 135.08733 1363.0045 ] ... [ 359.85425 1058.6151 1013.9297 ... 163.37076 159.4037 1266.5629 ] [ 682.56195 1274.0765 1125.9093 ... 177.28194 135.8132 1424.4539 ] [ 148.46483 454.86966 954.3874 ... 199.56137 320.5976 1351.9453 ]]] 1/1 [==============================] - 0s 146ms/step block5_conv3: [[[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]] [[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]] [[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]] ... [[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 1.2440066 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]] [[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]] [[0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0.5987776 0. ] ... [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ] [0. 0. 0. ... 0. 0. 0. ]]] 1/1 [==============================] - 0s 138ms/step fc1: [2.9445024 0. 0. ... 3.411133 0. 0.92348397] 1/1 [==============================] - 0s 143ms/step fc2: [0. 0. 0. ... 0.01487154 0. 0. ] 1/1 [==============================] - 0s 153ms/step predictions: [8.86679481e-06 3.92886477e-06 1.90436788e-06 1.05222316e-05 2.95820100e-05 1.05888921e-05 8.18475996e-07 2.97174847e-05 1.72565160e-05 2.33364801e-04 4.08327742e-06 7.54527573e-05 2.13582698e-05 5.62608966e-06 2.66997467e-05 6.33711488e-06 5.91164498e-05 2.96048438e-05 1.54325113e-04 1.61606149e-04 4.36949313e-06 2.27579949e-04 3.22062464e-04 3.87774286e-04 1.82932072e-05 1.78626142e-04 1.06591207e-04 1.24504077e-04 7.32575209e-05 1.67771868e-05 1.42938734e-06 5.67790994e-06 4.40411623e-06 3.51705899e-06 4.54849214e-06 8.11457880e-07 1.23381051e-06 9.66434072e-07 1.74248067e-04 4.33074547e-06 3.25646602e-06 1.54293630e-05 1.26219347e-05 1.96861256e-05 5.53511854e-05 5.12993356e-05 5.80043570e-06 9.02399624e-05 7.22741834e-06 1.27374151e-05 2.59617846e-05 3.38299797e-05 2.39712815e-03 1.74616615e-03 4.85557830e-04 3.29024158e-04 3.13818571e-04 2.12321938e-05 5.26380085e-04 2.88680475e-03 1.84028875e-03 8.09462945e-05 6.80478770e-05 2.14936007e-02 3.19925428e-04 1.66888710e-03 2.04587798e-03 3.49455629e-04 1.27097068e-03 2.72739620e-04 2.91247284e-06 4.48031962e-04 1.57972545e-06 7.40459245e-06 1.13488693e-06 9.91967318e-06 1.07233291e-05 1.95666644e-06 1.85278812e-04 3.24987195e-04 5.19032947e-05 2.71407112e-06 1.49551488e-05 4.88938567e-05 5.17146982e-05 1.10298810e-04 1.06869438e-05 2.46440832e-05 4.66025049e-05 3.63443614e-05 1.12969128e-05 2.55341893e-05 6.81859092e-05 1.14550072e-04 5.32956028e-06 7.00814735e-06 1.15207583e-03 1.13513424e-05 1.45131880e-05 9.87223932e-04 1.41433545e-03 1.42524106e-04 1.87630758e-05 1.45034219e-05 7.02911166e-06 4.97291239e-07 1.44432697e-05 2.14918982e-05 2.49657751e-04 1.75241279e-04 7.07811036e-04 3.24391127e-02 2.39375167e-05 4.87557236e-06 1.68786224e-04 1.72599885e-05 3.57204262e-04 3.65287306e-05 9.62294143e-06 6.66444794e-06 6.23000451e-05 4.13392627e-05 1.25281904e-05 2.46514765e-06 1.16787705e-05 4.36370010e-06 1.15267576e-05 1.03567043e-04 1.90633407e-04 5.81696622e-06 5.24300151e-04 4.73948821e-05 6.00396706e-05 2.62725348e-06 9.41229882e-06 4.48861829e-05 3.18245611e-06 1.09500215e-05 3.04656010e-06 2.81243956e-06 2.78029938e-06 2.36011829e-06 1.19211859e-06 1.40344800e-05 6.92425092e-05 2.19969384e-04 2.38212277e-04 5.18192837e-06 6.66403794e-05 1.19804699e-05 1.12324997e-05 7.82153511e-05 1.48655672e-05 7.67800702e-06 3.79271878e-05 9.57871907e-06 1.36488925e-05 9.52548271e-06 1.79516901e-05 3.11920776e-05 1.07268534e-04 2.04860935e-05 1.13185033e-05 4.38859715e-05 6.85368195e-06 3.27570451e-05 4.21883669e-06 9.13747954e-06 1.15643152e-05 7.98587553e-06 9.62191461e-06 2.23533661e-05 1.18041371e-05 4.75581110e-05 6.63245373e-06 3.38082427e-05 9.82034999e-06 2.01295570e-05 8.89091098e-06 2.23101542e-05 2.10599119e-05 1.95221619e-05 2.93983067e-05 1.35727038e-04 4.10272987e-05 9.92941568e-05 8.58638596e-05 4.45206533e-05 7.41288459e-05 4.27207560e-05 7.12208493e-05 1.87339421e-04 5.40639439e-06 6.58450299e-05 7.53286349e-06 1.91383544e-04 1.07185342e-05 3.62894643e-05 1.38193327e-05 3.58770776e-05 9.85885981e-06 8.50519336e-06 1.47193816e-04 6.64993204e-05 7.43968712e-06 2.07755093e-05 3.51842573e-05 4.39709947e-06 3.83616753e-05 2.99786516e-05 1.62636991e-06 5.47050422e-06 9.95857590e-07 8.05376112e-06 1.96713572e-05 1.18257765e-06 1.11786721e-05 3.49282709e-05 4.67216933e-06 1.05762056e-05 5.35382169e-05 1.22479163e-04 8.24888684e-06 2.67953932e-04 3.17708400e-05 1.71653865e-05 1.05027771e-04 2.14162956e-05 4.88646037e-06 4.61531381e-05 7.45789384e-06 2.91185825e-05 5.80204323e-05 8.73349563e-05 3.94712624e-05 4.85797500e-05 6.84601901e-06 1.49850293e-05 4.85138225e-05 7.45706493e-05 1.98496113e-04 3.00224547e-05 8.45372233e-06 6.48311516e-06 6.54547603e-06 3.71917267e-05 2.83854206e-06 1.78560749e-05 3.07140799e-05 2.26468183e-05 5.00164570e-05 4.60664432e-06 5.20592039e-05 3.10437244e-05 7.79263937e-05 5.62111791e-06 1.49219180e-04 6.47040315e-06 5.18403431e-06 2.83422069e-05 1.08114955e-05 1.53456713e-05 2.45812495e-04 1.05807967e-05 4.69596816e-05 1.61335429e-05 1.00145635e-05 5.69761096e-06 1.74532361e-05 1.20673076e-05 9.43993200e-06 5.01738941e-06 3.85100338e-06 1.40547309e-05 7.89373280e-06 4.30665978e-06 9.39401434e-06 8.81400138e-06 2.69927250e-06 2.62271114e-05 8.21756657e-06 1.31640641e-04 1.97637601e-05 6.78912620e-05 1.72004147e-04 2.91035598e-04 3.54252334e-05 2.54558254e-05 1.38019350e-05 1.91044728e-06 7.22885125e-06 1.33819249e-05 7.12421388e-06 7.87766548e-05 1.78281352e-05 3.34753531e-05 6.08029450e-06 2.98858026e-06 1.37939816e-04 3.45666740e-05 6.52200970e-06 3.16649130e-05 3.49477432e-06 1.01652977e-05 8.41250403e-06 7.48465573e-06 1.35816648e-04 7.22609548e-06 4.39557334e-06 1.19831084e-05 3.40422557e-05 1.52454516e-06 1.69746852e-06 1.34438051e-06 1.76554167e-05 2.88769229e-06 4.23087977e-06 1.05430786e-06 5.98303768e-06 4.44874831e-06 7.20610979e-06 9.38479934e-06 6.35911192e-07 5.10396058e-06 6.53882182e-07 1.40259897e-06 4.55490772e-06 7.53509375e-05 9.45165266e-06 4.56607668e-04 3.46355228e-06 3.41798623e-05 3.84768509e-06 1.31142251e-05 6.59345415e-06 1.28755937e-05 9.35764911e-06 7.91293678e-06 2.35082607e-05 2.26178645e-06 3.31025512e-05 2.76681226e-06 1.68231236e-05 6.80708763e-06 1.29108651e-06 6.85924388e-05 3.70900016e-05 1.71985685e-05 1.25700643e-03 1.33214565e-03 3.10425255e-02 5.59107903e-05 3.51523668e-05 4.40640397e-05 1.89676175e-05 5.17027183e-05 9.10625458e-05 1.45803888e-05 1.62041426e-04 5.17400040e-05 4.09077838e-05 4.03765211e-04 1.52316759e-04 4.66719284e-05 2.34392573e-04 1.60122636e-05 4.58906061e-06 6.39632344e-05 9.06162240e-05 7.67958554e-05 1.55225789e-05 2.62458780e-05 4.54723631e-05 2.71644458e-05 1.16712208e-05 6.18937993e-05 4.40446502e-06 1.69388259e-05 4.64936107e-04 1.75527806e-04 2.73151163e-05 6.96121060e-05 3.32106974e-05 8.41600195e-06 2.08298861e-05 1.21705219e-04 6.25848115e-05 5.26691438e-05 3.41659279e-06 1.30620274e-05 7.36525923e-04 4.74398075e-06 3.45263470e-05 1.00253281e-04 9.23935477e-06 2.03607378e-05 1.13465694e-05 2.19904769e-06 5.09470337e-05 4.19838540e-03 1.00290286e-03 2.63983256e-05 2.80405875e-05 2.27962232e-07 9.34973650e-06 2.28096338e-04 4.37624931e-06 4.99454563e-06 3.53755640e-05 9.63599712e-04 4.64696450e-06 4.22794583e-05 2.49279110e-04 1.11948924e-04 4.00889257e-04 2.80806180e-05 2.20467977e-04 7.32972927e-04 4.86506411e-04 2.13944048e-04 2.51623533e-05 1.58264316e-04 1.89990387e-04 5.65126655e-04 1.82046060e-05 1.41215526e-06 5.97492181e-05 2.10429396e-04 1.14815513e-04 2.95700811e-05 2.83271838e-05 5.36805019e-04 3.18742881e-04 5.33307139e-05 3.37226847e-05 1.48667343e-04 7.55067822e-06 1.52780412e-04 2.95972204e-05 1.19778932e-04 3.52832176e-05 4.95642707e-05 2.11865432e-03 4.00052872e-03 2.43429913e-05 1.71246738e-05 4.72480082e-04 1.61542965e-04 1.42520032e-04 3.93152914e-06 2.28453027e-05 5.02332638e-04 5.61465931e-05 4.19722019e-05 1.03473103e-05 9.32566982e-05 2.48103228e-04 3.92103073e-04 2.74504127e-05 1.31670722e-05 8.29012133e-05 2.35334755e-05 4.90546154e-05 6.12018048e-04 3.29416767e-02 7.38703005e-04 1.45032809e-05 2.26052930e-06 5.55469996e-05 1.41960825e-03 1.75519352e-04 1.39583615e-04 5.32880076e-05 1.64087061e-02 9.01359745e-05 3.83946863e-05 1.97320719e-06 3.78321100e-04 6.72588721e-05 3.71041562e-04 4.72625870e-05 1.61895136e-04 2.04839933e-04 3.22288433e-05 3.52817528e-06 7.15582646e-05 4.79896989e-05 3.53601732e-04 4.54594474e-03 3.57284152e-05 3.91601556e-04 4.97426256e-04 1.83074051e-04 1.46165185e-05 1.81997917e-03 8.16113879e-06 3.32378513e-05 1.41442579e-05 6.49202193e-05 1.11072080e-03 3.96973446e-05 3.17696031e-05 3.51422088e-04 1.33094509e-04 1.45075168e-03 5.18648769e-04 3.23256850e-02 2.24043634e-02 8.97353857e-06 1.05607351e-05 1.93923479e-05 1.62865545e-05 2.40424965e-02 4.11161134e-04 1.48271674e-05 2.35818443e-05 9.94408154e-04 1.43786694e-03 1.77713620e-04 6.38488700e-06 2.69750108e-05 2.89386335e-05 7.80405207e-06 6.41705119e-04 9.40548416e-05 1.12757407e-05 2.28892022e-05 5.97430590e-05 8.32233782e-05 9.89095061e-05 2.82501249e-04 3.17303883e-03 3.17591184e-05 2.72919406e-05 3.76993694e-06 7.63166972e-05 2.03596119e-05 2.04267471e-05 7.24468118e-05 1.95511733e-03 1.77471829e-05 9.32528783e-05 4.18644668e-05 2.39925605e-04 7.61114425e-05 1.34542322e-04 1.36341987e-04 2.72285729e-06 1.63320874e-06 3.19918210e-04 3.58488120e-04 3.70486436e-04 2.89479376e-05 1.04429608e-04 9.23851803e-06 4.99161706e-06 4.57598726e-05 4.37971874e-04 1.42190562e-04 7.56013542e-05 3.04936093e-05 8.39943314e-05 1.95028661e-05 3.22055821e-05 8.87363876e-06 4.10715120e-06 1.06259424e-04 1.45254788e-04 3.37117890e-05 5.98966608e-06 7.07039202e-04 2.28137978e-05 2.17670658e-05 6.64460094e-05 2.68764183e-04 3.21332118e-05 2.31042814e-05 2.60967878e-04 1.32772921e-05 4.10596476e-06 5.84332611e-06 6.55371468e-06 1.23988102e-05 6.55802956e-04 7.23824138e-03 9.32764597e-05 4.86513818e-05 9.33450181e-04 3.47442023e-04 2.15923501e-04 6.65367479e-05 2.31268095e-05 1.44284004e-05 1.65621004e-05 8.02202194e-05 4.12447916e-05 1.73158958e-04 1.83570213e-04 6.93245465e-06 5.82744105e-05 4.59138393e-01 2.13877211e-05 5.71083569e-04 8.65393588e-07 1.98091984e-05 5.98172264e-05 1.43234164e-03 2.18751738e-04 5.04269119e-05 7.31506816e-06 9.18616934e-05 6.62255989e-05 4.11376823e-06 8.62064189e-05 1.26205377e-05 4.61055140e-04 1.08992020e-02 2.66485131e-05 2.64866627e-04 6.62679813e-05 3.60291087e-05 2.66545121e-05 1.77872658e-04 2.01556808e-03 2.79729593e-05 1.23751124e-05 9.58843448e-04 4.60301017e-05 3.57670524e-06 1.80370233e-04 4.55380687e-05 7.21158649e-05 1.30548215e-04 1.50785688e-03 2.28181725e-05 3.10816872e-03 3.36440653e-03 1.46413358e-05 1.88198217e-04 1.48446697e-05 5.23523013e-05 2.66233925e-04 7.40830819e-06 1.24792755e-03 2.70143355e-04 1.57155337e-05 1.91499304e-04 1.47366300e-04 3.55853881e-05 1.17728428e-04 4.92268661e-03 4.48991996e-05 1.09024140e-05 3.84566956e-05 6.10373390e-05 1.22622978e-05 3.02930621e-05 8.43525595e-06 5.89327174e-05 1.77384354e-05 9.33787192e-07 1.97890895e-05 3.96184361e-04 3.92160400e-05 2.23727948e-05 1.97188201e-05 1.45821277e-05 8.40431021e-04 9.53494819e-05 1.51549818e-06 4.29444408e-05 1.63255812e-04 1.67631064e-04 4.67124803e-04 1.54450056e-04 7.67227630e-06 1.39268965e-03 1.28351869e-02 5.00910636e-03 2.14553881e-03 1.01173273e-03 5.63595968e-04 4.91360843e-05 7.19250529e-05 1.75622830e-04 9.74295926e-06 3.01298278e-04 5.54160670e-06 1.24025473e-05 1.86115030e-05 5.24135203e-06 4.18825774e-04 8.15189014e-06 2.72685011e-05 3.91247977e-06 9.30270925e-03 1.53549627e-04 1.02977538e-05 1.25478473e-05 6.06908216e-05 1.17585540e-03 1.44420778e-02 5.44897193e-05 1.53933608e-04 3.76078897e-05 5.28023884e-06 3.16303522e-05 7.72568455e-06 2.10181301e-04 1.01022335e-04 1.40602220e-04 1.49609783e-04 1.14452605e-05 2.89457548e-05 3.71720322e-04 2.38283264e-05 1.56697679e-05 1.03104067e-04 1.83217016e-05 2.14195767e-04 2.84243783e-04 5.28251330e-05 2.34640265e-05 7.24710208e-06 1.14483064e-05 3.84075614e-03 7.89254773e-05 3.62368992e-05 1.83144530e-05 1.27833104e-04 4.90006569e-05 2.73585611e-04 3.29049872e-05 3.17845872e-04 4.15099430e-06 3.84936793e-05 2.37875720e-05 2.39650180e-05 5.25766482e-05 3.92098336e-05 1.74029192e-04 9.73390524e-06 4.05609608e-05 2.88089959e-05 1.40124266e-05 7.07016588e-05 2.67811352e-04 1.82182499e-04 1.04057754e-03 2.22881761e-04 2.29549150e-05 1.89316197e-05 8.92643220e-06 4.58891445e-05 3.33298551e-04 1.64505072e-05 1.24487444e-04 5.65690698e-06 1.05001331e-04 9.23672560e-05 1.13163114e-05 3.78826895e-04 2.27822075e-05 1.01369282e-04 6.74335679e-05 3.10279633e-04 9.25418772e-06 1.27698237e-04 3.26955749e-04 1.14762376e-03 2.19624781e-04 6.66490960e-05 2.14133486e-02 3.51987910e-05 8.94156867e-04 2.21527134e-05 4.55056172e-04 3.78276734e-03 2.22083996e-04 1.75435252e-06 6.83424514e-06 5.30135003e-06 9.82568872e-06 7.31593987e-04 5.02913608e-04 1.05920284e-04 1.37225297e-05 4.70397354e-05 3.52310999e-05 1.94082645e-06 1.88696606e-03 9.90767949e-05 2.96163809e-04 5.25712385e-05 2.51091842e-04 3.75009346e-04 1.63949630e-03 4.90943727e-04 1.16265301e-05 5.67344978e-06 2.31106173e-06 3.50021315e-03 4.65841076e-05 1.78817288e-06 8.85085377e-04 9.18242076e-05 7.04456979e-05 3.29272407e-05 2.82556066e-05 4.24005484e-05 7.68456357e-06 3.95997573e-04 1.53113469e-05 8.22980510e-05 1.13864508e-05 5.75939293e-06 1.57799313e-05 4.28730937e-06 1.95369706e-03 2.18281384e-05 2.45123647e-06 2.31460072e-05 3.70571047e-06 4.15719463e-04 1.83777098e-04 8.83020984e-05 6.35228772e-03 2.97277846e-04 4.78334114e-04 2.54291444e-06 7.86322809e-04 1.68983519e-04 4.10227221e-05 1.39408348e-05 1.27657084e-04 1.54425681e-04 6.44958694e-04 4.67360951e-04 3.12464399e-04 1.91629195e-04 1.59293544e-04 9.02580359e-05 4.43688259e-05 4.75798175e-03 3.29969975e-04 5.27197990e-05 1.94561470e-03 5.67512780e-06 5.76647471e-05 2.69354228e-03 6.31902512e-05 2.22443996e-05 1.21067016e-04 1.64704434e-05 1.82369724e-04 6.47963898e-04 2.28299050e-05 3.77393553e-05 5.06583950e-04 4.50800035e-05 1.00449382e-04 3.34154814e-03 3.99357203e-04 2.91576784e-04 1.50415999e-05 3.74069023e-05 5.49084434e-05 4.58612112e-06 1.48610940e-04 4.78453649e-06 5.50092373e-04 7.97398843e-06 1.57916365e-04 2.02938754e-04 5.12932615e-07 7.93720974e-05 1.12120520e-04 5.25517389e-05 3.85814092e-05 2.26931297e-03 7.04336446e-04 6.22067500e-06 3.24391597e-03 3.80431273e-04 3.58487770e-04 1.87326194e-04 9.27425208e-06 2.41902526e-05 1.49540312e-04 1.11114350e-05 1.33219073e-04 3.20076477e-04 4.06427571e-04 1.01031001e-04 1.21471225e-04 1.39722342e-05 1.44775596e-03 9.68599925e-05 3.95861082e-03 4.69980342e-03 9.22366689e-06 1.76984744e-04 1.27497464e-02 7.51280750e-05 2.43124009e-06 2.66393145e-05 1.12654387e-04 6.42826344e-05 1.49543357e-05 1.02014004e-04 1.28567963e-05 4.39918629e-04 1.48356130e-05 2.16726930e-05 5.48292974e-06 6.84269753e-06 3.99081706e-04 7.80194241e-05 4.16754920e-05 2.64994364e-04 5.74018704e-05 1.37182778e-05 1.14159811e-05 1.43100833e-05 8.88659633e-06 1.06520629e-05 8.85260033e-06 5.77346009e-06 8.25636380e-06 2.16832796e-05 8.95236644e-06 2.41253983e-05 1.22884922e-02 4.22514995e-06 1.92215273e-04 1.14893555e-05 9.26747089e-06 3.80918646e-05 2.01568528e-05 1.82601270e-05 1.66271129e-05 1.34015800e-05 2.14585361e-05 4.06647559e-05 4.95586664e-06 9.35641292e-05 1.71769386e-06 2.04159005e-05 1.20855322e-04 5.78344843e-05 1.86209247e-04 9.47380249e-05 4.63605829e-06 2.66953939e-05 3.25924228e-03 1.41969331e-05 5.21058428e-05 4.36145774e-06 2.74305257e-05 5.58478814e-06 1.86247416e-04 3.99841110e-06 1.27266567e-05 6.94881010e-06 1.22096226e-05 2.83595796e-06 1.18330085e-04 1.95743614e-05 4.30598017e-03 4.46661434e-05 3.86449283e-05 5.68860676e-04 4.67448954e-05 4.63131000e-05 9.05528850e-07 9.58958262e-05 2.17774905e-05 6.33123418e-05 1.37754439e-04 1.22215988e-05 1.86073139e-05 3.51532503e-06 6.26382916e-06 2.33038532e-04 2.20870024e-05 2.65913677e-05 2.15374494e-05 7.98078545e-05 3.53792457e-05 1.82980682e-06 8.29796772e-05 2.25770145e-05 5.53840528e-06 1.28692284e-06 2.41941045e-04 3.47754917e-06 3.37785932e-05 7.95326923e-06 3.53732721e-05 8.06681346e-04]
View Code
conv_block1_conv1
conv_block1_conv2
conv_block2_conv1
conv_block2_conv2
conv_block3_conv1
conv_block3_conv2
conv_block5_conv3
0x3：使用VGG16作为预训练模型，fine-tune一个手写数字识别卷积网络，再观察基模型和fine-tune模型的可视化
通过前面的章节我们知道，VGG16预训练模型不太具备手写数字识别能力，也就是说，预训练模型的样本领域和下游任务的样本领域存在较大差异。
在这种情况下，我们通过下面几个实验来更加本质地理解fine-tune在模型层面具体调整了哪些东西。
1、直接设计一个CNN卷积网络进行手写数字识别
# In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> Flatten -> Dense -> Dropout -> Out model = Sequential() model.add(Conv2D(filters=32, kernel_size=(5, 5), padding='Same', activation='relu', input_shape = (28, 28, 1))) model.add(Conv2D(filters=32, kernel_size=(5, 5), padding='Same', activation='relu')) model.add(MaxPool2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(filters=64, kernel_size=(3, 3), padding='Same', activation='relu')) model.add(Conv2D(filters=64, kernel_size=(3, 3), padding='Same', activation='relu')) model.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(256, activation="relu")) model.add(Dropout(0.5))
上图是一个比较标准、主流的CNN卷积网络，在一个相对不大的训练集上，通过普通的设备进行训练就可以得到较好的效果。
2、冻结预训练基模型，调整下游fine-tune层参数：VGG16 without toplayer + 全连接层 迁移学习
之所以选择这种模型设计结构，出于以下几点考虑：
- 尽量复用预训练基模型中可以迁移到下游任务的模型部分：理论上，VGG16已经基本完成了局部->整体视野的卷积核训练，这种能力在手写数字识别中也是可以复用的（存在迁移学习的前提条件），这点通过前面章节可视化卷积核的实验也可以看出。
- VGG16的顶层结构设计初衷是进行1000种生活常见图形识别，并不适配10类数字识别的任务，所以toplayer需要舍弃，同时在下游接上一个新的神经网络结构，专门用于10类手写数字识别。
- 由于下游任何和预训练基模型存在任务领域的差异，所以下游fine-tune对训练样本集有一定的要求，因此我们需要对数据进行增强，在原数据集的基础上随机旋转、平移、缩放、产生噪音，从而更好地聚焦于数字特征的提取，而不是数据集本身。
- VGG16的输入层是RGB 224*224*3彩色像素图片，MNIST手写数字是GRAY 28*28*1灰度像素图片，虽然存在一定差异，但总体上都属于像素图片领域范畴，具有一定的迁移学习的理论基础。
- 训练过程实际上是在训练下游fine-tune的全连接网络，前面的卷积模型参数是”固化（不可训练的）。
下面代码使用了 keras.applications.vgg16 中的 VGG16，在线获取已有的 VGG16 模型及参数，获取后冻结 VGG16 中的所有参数进行训练。
在这之后添加几层 relu 全连接以及用于多分类的 softmax 全连接，原则上基本参考了通用CNN手写数字识别卷积网络的设计。
stacking之后的新的模型结构如下：
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 48, 48, 3)] 0 block1_conv1 (Conv2D) (None, 48, 48, 64) 1792 block1_conv2 (Conv2D) (None, 48, 48, 64) 36928 block1_pool (MaxPooling2D) (None, 24, 24, 64) 0 block2_conv1 (Conv2D) (None, 24, 24, 128) 73856 block2_conv2 (Conv2D) (None, 24, 24, 128) 147584 block2_pool (MaxPooling2D) (None, 12, 12, 128) 0 block3_conv1 (Conv2D) (None, 12, 12, 256) 295168 block3_conv2 (Conv2D) (None, 12, 12, 256) 590080 block3_conv3 (Conv2D) (None, 12, 12, 256) 590080 block3_pool (MaxPooling2D) (None, 6, 6, 256) 0 block4_conv1 (Conv2D) (None, 6, 6, 512) 1180160 block4_conv2 (Conv2D) (None, 6, 6, 512) 2359808 block4_conv3 (Conv2D) (None, 6, 6, 512) 2359808 block4_pool (MaxPooling2D) (None, 3, 3, 512) 0 block5_conv1 (Conv2D) (None, 3, 3, 512) 2359808 block5_conv2 (Conv2D) (None, 3, 3, 512) 2359808 block5_conv3 (Conv2D) (None, 3, 3, 512) 2359808 block5_pool (MaxPooling2D) (None, 1, 1, 512) 0 flatten (Flatten) (None, 512) 0 dense (Dense) (None, 4096) 2101248 dropout (Dropout) (None, 4096) 0 dense_1 (Dense) (None, 4096) 16781312 dropout_1 (Dropout) (None, 4096) 0 dense_2 (Dense) (None, 10) 40970 ================================================================= Total params: 33,638,218 Trainable params: 33,638,218 Non-trainable params: 0
训练代码：
from keras.models import Model, load_model from tensorflow.keras.applications.vgg16 import VGG16 from tensorflow.keras.preprocessing import image from tensorflow.keras.backend import reshape, expand_dims, spatial_2d_padding, spatial_3d_padding from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions from tensorflow.keras.layers import Dense, GlobalAveragePooling1D, Flatten, Dropout, Conv1D from keras.optimizers import SGD from keras.datasets import mnist from keras.utils import to_categorical import tensorflow as tf import numpy as np import cv2 import matplotlib.pyplot as plt import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" def vis_conv(images, n, name, t): """visualize conv output and conv filter. Args: img: original image. n: number of col and row. t: vis type. name: save name. """ size = 64 margin = 5 if t == 'filter': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin, 3)) if t == 'conv': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin)) for i in range(n): for j in range(n): if t == 'filter': filter_img = images[i + (j * n)] if t == 'conv': filter_img = images[..., i + (j * n)] filter_img = cv2.resize(filter_img, (size, size)) # Put the result in the square `(i, j)` of the results grid horizontal_start = i * size + i * margin horizontal_end = horizontal_start + size vertical_start = j * size + j * margin vertical_end = vertical_start + size if t == 'filter': results[horizontal_start: horizontal_end, vertical_start: vertical_end, :] = filter_img if t == 'conv': results[horizontal_start: horizontal_end, vertical_start: vertical_end] = filter_img # Display the results grid plt.imshow(results) plt.savefig('images/{}_{}.jpg'.format(t, name), dpi=600) plt.show() def conv_output(model, layer_name, img): """Get the output of conv layer. Args: model: keras model. layer_name: name of layer in the model. img: processed input image. Returns: intermediate_output: feature map. """ # this is the placeholder for the input images input_img = model.input try: # this is the placeholder for the conv output out_conv = model.get_layer(layer_name).output except: raise Exception('Not layer named {}!'.format(layer_name)) # get the intermediate layer model intermediate_layer_model = Model(inputs=input_img, outputs=out_conv) # get the output of intermediate layer model intermediate_output = intermediate_layer_model.predict(img) return intermediate_output[0] def get_mnist_data(): (X_train_data, Y_train_data), (X_test_data, Y_test_data) = mnist.load_data() # convert Y label into one-hot Y_train_data = to_categorical(Y_train_data) Y_test_data = to_categorical(Y_test_data) X_train_data = X_train_data.astype('float32') / 255.0 X_test_data = X_test_data.astype('float32') / 255.0 # reshape the mnist data in 48*48*3 X_train_data = expand_dims(X_train_data, axis=-1) X_test_data = expand_dims(X_test_data, axis=-1) X_train_data = tf.pad(X_train_data, [[0, 0], [2, 18], [2, 18], [1, 1]]) X_test_data = tf.pad(X_test_data, [[0, 0], [2, 18], [2, 18], [1, 1]]) # prepare validate/train date X_train_val = X_train_data[-2000:, ...] X_train_data = X_train_data[:-2000, ...] Y_train_val = Y_train_data[-2000:] Y_train_data = Y_train_data[:-2000] print("np.shape(X_train_data): ", np.shape(X_train_data)) print("np.shape(X_test_data): ", np.shape(X_test_data)) print("np.shape(X_train_val): ", np.shape(X_train_val)) print("np.shape(Y_train_data): ", np.shape(Y_train_data)) print("np.shape(Y_train_val): ", np.shape(Y_train_val)) print("Y_train_data[0]: ", Y_train_data[0]) return (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) def train_fine_tune(): # create the base pre-trained model base_model = VGG16(weights='imagenet', include_top=False, input_shape=(48, 48, 3)) # create a fine-tune model x = base_model.output print("base_model.input_shape: ", base_model.input_shape) print("base_model.input_shape[1:]: ", base_model.input_shape[1:]) print("base_model.output_shape: ", base_model.output_shape) print("base_model.output_shape[1:]: ", base_model.output_shape[1:]) # let's add a fully-connected layer x = Flatten()(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.5)(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.5)(x) # and a logistic layer -- let's say we have 10 classes predictions = Dense(10, activation='softmax')(x) # this is the new model(vgg16+fine-tune model) we will train model = Model(inputs=base_model.input, outputs=predictions) print("model.input_shape: ", model.input_shape) print("model.input_shape[1:]: ", model.input_shape[1:]) print("model.output_shape: ", model.output_shape) print("model.output_shape[1:]: ", model.output_shape[1:]) model.summary() # i.e. freeze all convolutional VGG16 layers for layer in base_model.layers: layer.trainable = False # compile the model (should be done *after* setting layers to non-trainable) sgd = SGD(lr=1e-5, decay=1e-6, momentum=0.5, nesterov=True) # 优化函数，设定学习率（lr）等参数，注意，fine-tune的学习率一般要小于预训练基模型的10倍以下 # model.compile(loss='categorical_crossentropy', optimizer="rmsprop") model.compile(loss='categorical_crossentropy', optimizer=sgd) # load the mnist data, and fine-tune the new model (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) = get_mnist_data() # train the model on the new data for a few epochs history = model.fit( X_train_data, Y_train_data, batch_size=32, epochs=20, validation_data=(X_train_val, Y_train_val) ) model.save('vgg16_plus_dnn_for_mnist.h5') def predict(): model = load_model("./vgg16_plus_dnn_for_mnist.h5") (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) = get_mnist_data() # 查看第一张图片 plt.imshow(X_test_data[0]) plt.show() print("前十个图片对应的标签： \n", np.argmax(Y_test_data[:10], axis=1)) print("取前十张图片测试集预测：\n", np.argmax(model.predict(X_test_data[:10]), axis=1)) if __name__ == '__main__': train_fine_tune() # predict()
加载fine-tune好的模型，并对测试集进行测试，
前十个图片对应的标签： [7 2 1 0 4 1 4 9 5 9] 取前十张图片测试集预测： [1 2 1 0 4 1 4 9 3 3]
预测错误的图片（索引：0、8、9）如下：
模型预测错误成了：1
模型预测错误成了：1
模型预测错误成了：3
可以看到，在预训练基模型参数不变的情况下进行fine-tune，如果预训练基模型的编码方式、张量维度、张量大小等因为和下游任务不完全一样，则最终迁移学习的效果并不能达到最好。
3、允许预训练基模型参数微调，同时调整下游fine-tune层参数：VGG16 without toplayer + 全连接层 迁移学习
现在我们调整一下策略：
- VGG16预训练基模型虽然也是卷积神经网络，训练过程也充分拟合了，但是训练样本库并不包含手写数字图片，因此可以推断VGG16并不包含手写数字识别的“知识”
- 基于VGG16+fine-tune进行手写数字识别，本质上是一个迁移学习，因此基模型本身也需要进行一定的学习调整
from keras.models import Model, load_model from tensorflow.keras.applications.vgg16 import VGG16 from tensorflow.keras.preprocessing import image from tensorflow.keras.backend import reshape, expand_dims, spatial_2d_padding, spatial_3d_padding from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions from tensorflow.keras.layers import Dense, GlobalAveragePooling1D, Flatten, Dropout, Conv1D from keras.optimizers import SGD from keras.datasets import mnist from keras.utils import to_categorical import tensorflow as tf import numpy as np import cv2 import matplotlib.pyplot as plt import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" def vis_conv(images, n, name, t): """visualize conv output and conv filter. Args: img: original image. n: number of col and row. t: vis type. name: save name. """ size = 64 margin = 5 if t == 'filter': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin, 3)) if t == 'conv': results = np.zeros((n * size + 7 * margin, n * size + 7 * margin)) for i in range(n): for j in range(n): if t == 'filter': filter_img = images[i + (j * n)] if t == 'conv': filter_img = images[..., i + (j * n)] filter_img = cv2.resize(filter_img, (size, size)) # Put the result in the square `(i, j)` of the results grid horizontal_start = i * size + i * margin horizontal_end = horizontal_start + size vertical_start = j * size + j * margin vertical_end = vertical_start + size if t == 'filter': results[horizontal_start: horizontal_end, vertical_start: vertical_end, :] = filter_img if t == 'conv': results[horizontal_start: horizontal_end, vertical_start: vertical_end] = filter_img # Display the results grid plt.imshow(results) plt.savefig('images/{}_{}.jpg'.format(t, name), dpi=600) plt.show() def conv_output(model, layer_name, img): """Get the output of conv layer. Args: model: keras model. layer_name: name of layer in the model. img: processed input image. Returns: intermediate_output: feature map. """ # this is the placeholder for the input images input_img = model.input try: # this is the placeholder for the conv output out_conv = model.get_layer(layer_name).output except: raise Exception('Not layer named {}!'.format(layer_name)) # get the intermediate layer model intermediate_layer_model = Model(inputs=input_img, outputs=out_conv) # get the output of intermediate layer model intermediate_output = intermediate_layer_model.predict(img) return intermediate_output[0] def get_mnist_data(): (X_train_data, Y_train_data), (X_test_data, Y_test_data) = mnist.load_data() # convert Y label into one-hot Y_train_data = to_categorical(Y_train_data) Y_test_data = to_categorical(Y_test_data) X_train_data = X_train_data.astype('float32') / 255.0 X_test_data = X_test_data.astype('float32') / 255.0 # reshape the mnist data in 48*48*3 X_train_data = expand_dims(X_train_data, axis=-1) X_test_data = expand_dims(X_test_data, axis=-1) X_train_data = tf.pad(X_train_data, [[0, 0], [2, 18], [2, 18], [1, 1]]) X_test_data = tf.pad(X_test_data, [[0, 0], [2, 18], [2, 18], [1, 1]]) # prepare validate/train date X_train_val = X_train_data[-2000:, ...] X_train_data = X_train_data[:-2000, ...] Y_train_val = Y_train_data[-2000:] Y_train_data = Y_train_data[:-2000] print("np.shape(X_train_data): ", np.shape(X_train_data)) print("np.shape(X_test_data): ", np.shape(X_test_data)) print("np.shape(X_train_val): ", np.shape(X_train_val)) print("np.shape(Y_train_data): ", np.shape(Y_train_data)) print("np.shape(Y_train_val): ", np.shape(Y_train_val)) print("Y_train_data[0]: ", Y_train_data[0]) return (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) def train_fine_tune(base_model_freeze=True): # create the base pre-trained model base_model = VGG16(weights='imagenet', include_top=False, input_shape=(48, 48, 3)) # create a fine-tune model x = base_model.output print("base_model.input_shape: ", base_model.input_shape) print("base_model.input_shape[1:]: ", base_model.input_shape[1:]) print("base_model.output_shape: ", base_model.output_shape) print("base_model.output_shape[1:]: ", base_model.output_shape[1:]) # let's add a fully-connected layer x = Flatten()(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.5)(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.5)(x) # and a logistic layer -- let's say we have 10 classes predictions = Dense(10, activation='softmax')(x) # this is the new model(vgg16+fine-tune model) we will train model = Model(inputs=base_model.input, outputs=predictions) print("model.input_shape: ", model.input_shape) print("model.input_shape[1:]: ", model.input_shape[1:]) print("model.output_shape: ", model.output_shape) print("model.output_shape[1:]: ", model.output_shape[1:]) model.summary() if base_model_freeze: # i.e. freeze all convolutional VGG16 layers for layer in base_model.layers: layer.trainable = False else: for layer in base_model.layers: layer.trainable = True # compile the model (should be done *after* setting layers to non-trainable) sgd = SGD(lr=1e-5, decay=1e-6, momentum=0.5, nesterov=True) # 优化函数，设定学习率（lr）等参数，注意，fine-tune的学习率一般要小于预训练基模型的10倍以下 # model.compile(loss='categorical_crossentropy', optimizer="rmsprop") model.compile(loss='categorical_crossentropy', optimizer=sgd) # load the mnist data, and fine-tune the new model (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) = get_mnist_data() # train the model on the new data for a few epochs history = model.fit( X_train_data, Y_train_data, batch_size=32, epochs=20, validation_data=(X_train_val, Y_train_val) ) if base_model_freeze: model.save('vgg16_plus_dnn_for_mnist_base_model_freeze.h5') else: model.save('vgg16_plus_dnn_for_mnist_base_model_train.h5') def predict(): model = load_model("./vgg16_plus_dnn_for_mnist.h5") (X_train_data, Y_train_data), (X_test_data, Y_test_data), (X_train_val, Y_train_val) = get_mnist_data() # 查看图片 plt.imshow(X_test_data[0]) plt.show() plt.imshow(X_test_data[8]) plt.show() plt.imshow(X_test_data[9]) plt.show() print("前十个图片对应的标签： \n", np.argmax(Y_test_data[:10], axis=1)) print("取前十张图片测试集预测：\n", np.argmax(model.predict(X_test_data[:10]), axis=1)) def visual_cnnkernel_with_number_6(): img_path = '6.webp' img = image.load_img(img_path, target_size=(224, 224)) plt.imshow(img) plt.show() x = image.img_to_array(img) x = np.expand_dims(x, axis=0) x = preprocess_input(x) print("np.shape(x): ", np.shape(x)) if __name__ == '__main__': train_fine_tune(base_model_freeze=False) # predict() # visual_cnnkernel_with_number_6()
可以看到，相比freeze预训练基模型，允许预训练基模型参数微调后，训练过程的val_loss和loss都开始了稳步下降，这说明了预训练基模型+fine-tune模型整体都在不断朝着你和手写数字识别的方向进行优化，同时预训练基模型自身也需要针对手写数字识别这个下游场景进行再学习和微调。
反过来讲，读者可以注意上之前的章节中freeze了基模型的参数，val_loss和loss始终下不来，模型呈现出了一种欠拟合的状态。
加载fine-tune好的模型，并对测试集进行测试，
前十个图片对应的标签： [7 2 1 0 4 1 4 9 5 9] 取前十张图片测试集预测： [7 2 1 0 4 1 4 9 5 9]
可以看到，允许了预训练基模型的参数调整后，在mnist数据集上进行fine-tune，得到的最终模型，很好地适配了新的识别任务。
可视化基模型卷积核，观察基模型卷积核在fine-tune前后的变化。
conv_block1_conv1
conv_block1_conv2
conv_block2_conv1
conv_block2_conv2
conv_block3_conv1
conv_block3_conv2
conv_block5_conv3
可以看到，经过fine-tune之后，基模型的卷积核的感知野和感知强度变大了，这也是为什么fine-tune之后识别mnist手写数字识别能力变强的原因。
0x4：用于下游任务的样本量大小对fine-tune效果的影响
参考链接：
https://arxiv.org/pdf/1409.1556.pdf https://www.cnblogs.com/LittleHann/p/6792511.html https://github.com/xiaochus/VisualizationCNN/blob/master/vis.py https://www.quora.com/What-is-the-VGG-neural-network https://blog.csdn.net/wphkadn/article/details/86772708 https://zhuanlan.zhihu.com/p/483559442 https://keras.io/api/applications/ https://blog.csdn.net/newlw/article/details/126127251
文章来源: https://www.cnblogs.com/LittleHann/p/17354069.html
如有侵权请联系:admin#unsafe.sh
如有侵权请联系:admin#unsafe.sh