2022-09-27 22:32:35 [INFO] ------------Environment Information------------- platform: Linux-4.15.0-140-generic-x86_64-with-debian-stretch-sid Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Build cuda_11.2.r11.2/compiler.29618528_0 cudnn: 8.2 GPUs used: 1 CUDA_VISIBLE_DEVICES: None GPU: ['GPU 0: Tesla V100-SXM2-32GB'] GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0 PaddlePaddle: 2.3.2 ------------------------------------------------ /home/aistudio/SwinUNet/medicalseg/cvlibs/config.py:452: UserWarning: Warning: The data dir now is /home/aistudio/SwinUNet/data/, you should change the data_root in the global.yml if this directory didn't have enough space .format(absolute_data_dir)) 2022-09-27 22:32:35 [INFO] ---------------Config Information--------------- batch_size: 24 data_root: data/ export: inference_helper: type: SwinUNetInferenceHelper transforms: - size: - 1 - 224 - 224 type: Resize3D iters: 14000 loss: coef: - 1 types: - coef: - 0.4 - 0.6 losses: - type: CrossEntropyLoss weight: null - type: DiceLoss type: MixedLoss lr_scheduler: end_lr: 0 learning_rate: 0.05 power: 0.9 type: PolynomialDecay model: backbone: type: SwinTransformer_tinyer_patch4_window7_224 num_classes: 9 pretrained: pretrained/pretrained.pdparams type: SwinUNet optimizer: momentum: 0.9 type: sgd weight_decay: 0.0001 test_dataset: dataset_root: ./Synapse_npy mode: test num_classes: 9 result_dir: ./output transforms: - size: - 1 - 224 - 224 type: Resize3D type: Synapse train_dataset: dataset_root: ./Synapse_npy mode: train num_classes: 9 result_dir: ./output transforms: - flip_axis: - 1 - 2 rotate_planes: - - 1 - 2 type: RandomFlipRotation3D - degrees: 20 prob: 0.5 rotate_planes: - - 1 - 2 type: RandomRotation3D - size: - 1 - 224 - 224 type: Resize3D type: Synapse val_dataset: dataset_root: ./Synapse_npy mode: test num_classes: 9 result_dir: ./output transforms: - size: - 1 - 224 - 224 type: Resize3D type: Synapse ------------------------------------------------ W0927 22:32:35.547045 646 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2 W0927 22:32:35.547094 646 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2. ---final upsample expand_first--- 2022-09-27 22:32:36 [INFO] Loading pretrained model from pretrained/pretrained.pdparams 2022-09-27 22:32:37 [WARNING] layers_up.0.expand.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.0.norm.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.0.norm.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.1.upsample.expand.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.1.upsample.norm.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.1.upsample.norm.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.2.upsample.expand.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.2.upsample.norm.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] layers_up.2.upsample.norm.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.1.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.1.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.2.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.2.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.3.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] concat_back_dim.3.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] norm.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] norm.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] norm_up.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] norm_up.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] up.expand.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] up.norm.weight is not in pretrained model 2022-09-27 22:32:37 [WARNING] up.norm.bias is not in pretrained model 2022-09-27 22:32:37 [WARNING] output.weight is not in pretrained model 2022-09-27 22:32:37 [INFO] There are 217/240 variables loaded into SwinUNet. /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.bool, the right dtype will convert to paddle.float32 format(lhs_dtype, rhs_dtype, lhs_dtype)) 2022-09-27 22:33:09 [INFO] [TRAIN] epoch: 1, iter: 100/14000, loss: 0.6122, DSC: 16.7551, lr: 0.049682, batch_cost: 0.3201, reader_cost: 0.07667, ips: 74.9723 samples/sec | ETA 01:14:09 2022-09-27 22:33:37 [INFO] [TRAIN] epoch: 2, iter: 200/14000, loss: 0.5132, DSC: 25.2525, lr: 0.049360, batch_cost: 0.2867, reader_cost: 0.06706, ips: 83.7120 samples/sec | ETA 01:05:56 2022-09-27 22:34:07 [INFO] [TRAIN] epoch: 3, iter: 300/14000, loss: 0.4842, DSC: 28.8506, lr: 0.049038, batch_cost: 0.2932, reader_cost: 0.06873, ips: 81.8536 samples/sec | ETA 01:06:56 2022-09-27 22:34:36 [INFO] [TRAIN] epoch: 4, iter: 400/14000, loss: 0.4449, DSC: 34.0134, lr: 0.048716, batch_cost: 0.2923, reader_cost: 0.06653, ips: 82.1180 samples/sec | ETA 01:06:14 2022-09-27 22:35:05 [INFO] [TRAIN] epoch: 5, iter: 500/14000, loss: 0.3908, DSC: 42.0202, lr: 0.048393, batch_cost: 0.2906, reader_cost: 0.06478, ips: 82.5956 samples/sec | ETA 01:05:22 2022-09-27 22:35:34 [INFO] [TRAIN] epoch: 6, iter: 600/14000, loss: 0.3351, DSC: 50.1796, lr: 0.048070, batch_cost: 0.2890, reader_cost: 0.06385, ips: 83.0563 samples/sec | ETA 01:04:32 2022-09-27 22:36:03 [INFO] [TRAIN] epoch: 7, iter: 700/14000, loss: 0.2944, DSC: 56.4564, lr: 0.047747, batch_cost: 0.2927, reader_cost: 0.06809, ips: 81.9900 samples/sec | ETA 01:04:53 2022-09-27 22:36:32 [INFO] [TRAIN] epoch: 8, iter: 800/14000, loss: 0.2446, DSC: 63.6051, lr: 0.047424, batch_cost: 0.2863, reader_cost: 0.06161, ips: 83.8140 samples/sec | ETA 01:02:59 2022-09-27 22:37:00 [INFO] [TRAIN] epoch: 9, iter: 900/14000, loss: 0.2088, DSC: 69.1388, lr: 0.047101, batch_cost: 0.2791, reader_cost: 0.05345, ips: 85.9908 samples/sec | ETA 01:00:56 2022-09-27 22:37:29 [INFO] [TRAIN] epoch: 10, iter: 1000/14000, loss: 0.1865, DSC: 72.4751, lr: 0.046777, batch_cost: 0.2896, reader_cost: 0.06307, ips: 82.8669 samples/sec | ETA 01:02:45 2022-09-27 22:37:29 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 948ms/step - batch_cost: 0.9481 - reader cost: 0.6078 2022-09-27 22:37:40 [INFO] [EVAL] #Images: 12, Dice: 0.7165, Loss: 0.189221 2022-09-27 22:37:40 [INFO] [EVAL] Class dice: [0.9886 0.7106 0.4716 0.7761 0.7329 0.8862 0.4361 0.8559 0.5906] 2022-09-27 22:37:41 [INFO] [EVAL] The model with the best validation mDice (0.7165) was saved at iter 1000. 2022-09-27 22:38:09 [INFO] [TRAIN] epoch: 11, iter: 1100/14000, loss: 0.1768, DSC: 73.7447, lr: 0.046453, batch_cost: 0.2778, reader_cost: 0.05686, ips: 86.4016 samples/sec | ETA 00:59:43 2022-09-27 22:38:38 [INFO] [TRAIN] epoch: 12, iter: 1200/14000, loss: 0.1687, DSC: 75.0158, lr: 0.046129, batch_cost: 0.2869, reader_cost: 0.06100, ips: 83.6520 samples/sec | ETA 01:01:12 2022-09-27 22:39:06 [INFO] [TRAIN] epoch: 13, iter: 1300/14000, loss: 0.1544, DSC: 77.0789, lr: 0.045805, batch_cost: 0.2772, reader_cost: 0.05544, ips: 86.5755 samples/sec | ETA 00:58:40 2022-09-27 22:39:36 [INFO] [TRAIN] epoch: 15, iter: 1400/14000, loss: 0.1492, DSC: 77.8505, lr: 0.045480, batch_cost: 0.3038, reader_cost: 0.07520, ips: 79.0089 samples/sec | ETA 01:03:47 2022-09-27 22:40:04 [INFO] [TRAIN] epoch: 16, iter: 1500/14000, loss: 0.1442, DSC: 78.4964, lr: 0.045155, batch_cost: 0.2839, reader_cost: 0.06032, ips: 84.5398 samples/sec | ETA 00:59:08 2022-09-27 22:40:33 [INFO] [TRAIN] epoch: 17, iter: 1600/14000, loss: 0.1431, DSC: 78.7183, lr: 0.044830, batch_cost: 0.2859, reader_cost: 0.05928, ips: 83.9424 samples/sec | ETA 00:59:05 2022-09-27 22:41:01 [INFO] [TRAIN] epoch: 18, iter: 1700/14000, loss: 0.1374, DSC: 79.6121, lr: 0.044504, batch_cost: 0.2823, reader_cost: 0.05867, ips: 85.0217 samples/sec | ETA 00:57:52 2022-09-27 22:41:31 [INFO] [TRAIN] epoch: 19, iter: 1800/14000, loss: 0.1332, DSC: 80.1034, lr: 0.044178, batch_cost: 0.3009, reader_cost: 0.07429, ips: 79.7548 samples/sec | ETA 01:01:11 2022-09-27 22:42:00 [INFO] [TRAIN] epoch: 20, iter: 1900/14000, loss: 0.1225, DSC: 81.7241, lr: 0.043852, batch_cost: 0.2827, reader_cost: 0.05835, ips: 84.9087 samples/sec | ETA 00:57:00 2022-09-27 22:42:29 [INFO] [TRAIN] epoch: 21, iter: 2000/14000, loss: 0.1221, DSC: 81.7073, lr: 0.043526, batch_cost: 0.2978, reader_cost: 0.07141, ips: 80.5781 samples/sec | ETA 00:59:34 2022-09-27 22:42:29 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 951ms/step - batch_cost: 0.9507 - reader cost: 0.6347 2022-09-27 22:42:41 [INFO] [EVAL] #Images: 12, Dice: 0.7757, Loss: 0.149105 2022-09-27 22:42:41 [INFO] [EVAL] Class dice: [0.9924 0.8023 0.539 0.8539 0.7911 0.9238 0.5361 0.8723 0.6709] 2022-09-27 22:42:42 [INFO] [EVAL] The model with the best validation mDice (0.7757) was saved at iter 2000. 2022-09-27 22:43:09 [INFO] [TRAIN] epoch: 22, iter: 2100/14000, loss: 0.1189, DSC: 82.2875, lr: 0.043200, batch_cost: 0.2712, reader_cost: 0.04804, ips: 88.4951 samples/sec | ETA 00:53:47 2022-09-27 22:43:39 [INFO] [TRAIN] epoch: 23, iter: 2200/14000, loss: 0.1172, DSC: 82.5023, lr: 0.042873, batch_cost: 0.2932, reader_cost: 0.06682, ips: 81.8503 samples/sec | ETA 00:57:39 2022-09-27 22:44:07 [INFO] [TRAIN] epoch: 24, iter: 2300/14000, loss: 0.1156, DSC: 82.7168, lr: 0.042546, batch_cost: 0.2810, reader_cost: 0.05510, ips: 85.3972 samples/sec | ETA 00:54:48 2022-09-27 22:44:35 [INFO] [TRAIN] epoch: 25, iter: 2400/14000, loss: 0.1117, DSC: 83.2587, lr: 0.042218, batch_cost: 0.2853, reader_cost: 0.06301, ips: 84.1285 samples/sec | ETA 00:55:09 2022-09-27 22:45:04 [INFO] [TRAIN] epoch: 26, iter: 2500/14000, loss: 0.1097, DSC: 83.6294, lr: 0.041891, batch_cost: 0.2881, reader_cost: 0.06059, ips: 83.3028 samples/sec | ETA 00:55:13 2022-09-27 22:45:33 [INFO] [TRAIN] epoch: 27, iter: 2600/14000, loss: 0.1125, DSC: 83.0406, lr: 0.041563, batch_cost: 0.2845, reader_cost: 0.05762, ips: 84.3541 samples/sec | ETA 00:54:03 2022-09-27 22:46:02 [INFO] [TRAIN] epoch: 29, iter: 2700/14000, loss: 0.1084, DSC: 83.7854, lr: 0.041234, batch_cost: 0.2957, reader_cost: 0.07457, ips: 81.1554 samples/sec | ETA 00:55:41 2022-09-27 22:46:31 [INFO] [TRAIN] epoch: 30, iter: 2800/14000, loss: 0.1085, DSC: 83.7931, lr: 0.040906, batch_cost: 0.2926, reader_cost: 0.06523, ips: 82.0316 samples/sec | ETA 00:54:36 2022-09-27 22:47:00 [INFO] [TRAIN] epoch: 31, iter: 2900/14000, loss: 0.1142, DSC: 82.8782, lr: 0.040577, batch_cost: 0.2897, reader_cost: 0.06464, ips: 82.8555 samples/sec | ETA 00:53:35 2022-09-27 22:47:29 [INFO] [TRAIN] epoch: 32, iter: 3000/14000, loss: 0.1056, DSC: 84.2043, lr: 0.040248, batch_cost: 0.2837, reader_cost: 0.05708, ips: 84.6110 samples/sec | ETA 00:52:00 2022-09-27 22:47:29 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 957ms/step - batch_cost: 0.9563 - reader cost: 0.6412 2022-09-27 22:47:40 [INFO] [EVAL] #Images: 12, Dice: 0.7895, Loss: 0.140233 2022-09-27 22:47:40 [INFO] [EVAL] Class dice: [0.9943 0.8139 0.5071 0.8788 0.8086 0.9347 0.5622 0.8921 0.7136] 2022-09-27 22:47:42 [INFO] [EVAL] The model with the best validation mDice (0.7895) was saved at iter 3000. 2022-09-27 22:48:10 [INFO] [TRAIN] epoch: 33, iter: 3100/14000, loss: 0.1071, DSC: 83.9401, lr: 0.039919, batch_cost: 0.2820, reader_cost: 0.05580, ips: 85.1030 samples/sec | ETA 00:51:13 2022-09-27 22:48:39 [INFO] [TRAIN] epoch: 34, iter: 3200/14000, loss: 0.0996, DSC: 85.0784, lr: 0.039589, batch_cost: 0.2931, reader_cost: 0.06698, ips: 81.8951 samples/sec | ETA 00:52:45 2022-09-27 22:49:08 [INFO] [TRAIN] epoch: 35, iter: 3300/14000, loss: 0.0982, DSC: 85.3232, lr: 0.039259, batch_cost: 0.2848, reader_cost: 0.06058, ips: 84.2692 samples/sec | ETA 00:50:47 2022-09-27 22:49:36 [INFO] [TRAIN] epoch: 36, iter: 3400/14000, loss: 0.0974, DSC: 85.4209, lr: 0.038928, batch_cost: 0.2875, reader_cost: 0.06181, ips: 83.4915 samples/sec | ETA 00:50:47 2022-09-27 22:50:06 [INFO] [TRAIN] epoch: 37, iter: 3500/14000, loss: 0.1051, DSC: 84.1926, lr: 0.038598, batch_cost: 0.3006, reader_cost: 0.07288, ips: 79.8301 samples/sec | ETA 00:52:36 2022-09-27 22:50:35 [INFO] [TRAIN] epoch: 38, iter: 3600/14000, loss: 0.0974, DSC: 85.4394, lr: 0.038267, batch_cost: 0.2830, reader_cost: 0.05914, ips: 84.8097 samples/sec | ETA 00:49:03 2022-09-27 22:51:04 [INFO] [TRAIN] epoch: 39, iter: 3700/14000, loss: 0.0986, DSC: 85.2534, lr: 0.037936, batch_cost: 0.2942, reader_cost: 0.06622, ips: 81.5872 samples/sec | ETA 00:50:29 2022-09-27 22:51:32 [INFO] [TRAIN] epoch: 40, iter: 3800/14000, loss: 0.1007, DSC: 84.7745, lr: 0.037604, batch_cost: 0.2810, reader_cost: 0.05931, ips: 85.4015 samples/sec | ETA 00:47:46 2022-09-27 22:52:02 [INFO] [TRAIN] epoch: 41, iter: 3900/14000, loss: 0.0970, DSC: 85.3773, lr: 0.037272, batch_cost: 0.2944, reader_cost: 0.06835, ips: 81.5144 samples/sec | ETA 00:49:33 2022-09-27 22:52:31 [INFO] [TRAIN] epoch: 43, iter: 4000/14000, loss: 0.0981, DSC: 85.3137, lr: 0.036940, batch_cost: 0.2944, reader_cost: 0.07255, ips: 81.5286 samples/sec | ETA 00:49:03 2022-09-27 22:52:31 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 944ms/step - batch_cost: 0.9438 - reader cost: 0.6295 2022-09-27 22:52:42 [INFO] [EVAL] #Images: 12, Dice: 0.7838, Loss: 0.144942 2022-09-27 22:52:42 [INFO] [EVAL] Class dice: [0.9945 0.842 0.5433 0.8717 0.7701 0.9402 0.5761 0.8937 0.6225] 2022-09-27 22:52:43 [INFO] [EVAL] The model with the best validation mDice (0.7895) was saved at iter 3000. 2022-09-27 22:53:12 [INFO] [TRAIN] epoch: 44, iter: 4100/14000, loss: 0.1002, DSC: 84.8892, lr: 0.036607, batch_cost: 0.2855, reader_cost: 0.05773, ips: 84.0620 samples/sec | ETA 00:47:06 2022-09-27 22:53:40 [INFO] [TRAIN] epoch: 45, iter: 4200/14000, loss: 0.0995, DSC: 85.1162, lr: 0.036274, batch_cost: 0.2814, reader_cost: 0.06055, ips: 85.2875 samples/sec | ETA 00:45:57 2022-09-27 22:54:10 [INFO] [TRAIN] epoch: 46, iter: 4300/14000, loss: 0.1006, DSC: 84.8869, lr: 0.035941, batch_cost: 0.2987, reader_cost: 0.07148, ips: 80.3483 samples/sec | ETA 00:48:17 2022-09-27 22:54:38 [INFO] [TRAIN] epoch: 47, iter: 4400/14000, loss: 0.0990, DSC: 85.0928, lr: 0.035607, batch_cost: 0.2832, reader_cost: 0.06068, ips: 84.7381 samples/sec | ETA 00:45:18 2022-09-27 22:55:06 [INFO] [TRAIN] epoch: 48, iter: 4500/14000, loss: 0.0929, DSC: 86.0756, lr: 0.035273, batch_cost: 0.2858, reader_cost: 0.06095, ips: 83.9712 samples/sec | ETA 00:45:15 2022-09-27 22:55:36 [INFO] [TRAIN] epoch: 49, iter: 4600/14000, loss: 0.1024, DSC: 84.5761, lr: 0.034939, batch_cost: 0.2946, reader_cost: 0.06826, ips: 81.4679 samples/sec | ETA 00:46:09 2022-09-27 22:56:05 [INFO] [TRAIN] epoch: 50, iter: 4700/14000, loss: 0.0943, DSC: 85.7276, lr: 0.034604, batch_cost: 0.2891, reader_cost: 0.06684, ips: 83.0159 samples/sec | ETA 00:44:48 2022-09-27 22:56:35 [INFO] [TRAIN] epoch: 51, iter: 4800/14000, loss: 0.0922, DSC: 86.0841, lr: 0.034269, batch_cost: 0.2964, reader_cost: 0.07341, ips: 80.9680 samples/sec | ETA 00:45:27 2022-09-27 22:57:03 [INFO] [TRAIN] epoch: 52, iter: 4900/14000, loss: 0.0932, DSC: 85.9933, lr: 0.033934, batch_cost: 0.2850, reader_cost: 0.05943, ips: 84.2070 samples/sec | ETA 00:43:13 2022-09-27 22:57:33 [INFO] [TRAIN] epoch: 53, iter: 5000/14000, loss: 0.0935, DSC: 85.8793, lr: 0.033598, batch_cost: 0.2986, reader_cost: 0.07240, ips: 80.3732 samples/sec | ETA 00:44:47 2022-09-27 22:57:33 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 938ms/step - batch_cost: 0.9374 - reader cost: 0.6225 2022-09-27 22:57:44 [INFO] [EVAL] #Images: 12, Dice: 0.8025, Loss: 0.131825 2022-09-27 22:57:44 [INFO] [EVAL] Class dice: [0.9951 0.8529 0.5614 0.8783 0.809 0.9477 0.6064 0.9014 0.6698] 2022-09-27 22:57:46 [INFO] [EVAL] The model with the best validation mDice (0.8025) was saved at iter 5000. 2022-09-27 22:58:14 [INFO] [TRAIN] epoch: 54, iter: 5100/14000, loss: 0.0923, DSC: 86.0647, lr: 0.033262, batch_cost: 0.2864, reader_cost: 0.06172, ips: 83.8121 samples/sec | ETA 00:42:28 2022-09-27 22:58:43 [INFO] [TRAIN] epoch: 55, iter: 5200/14000, loss: 0.0912, DSC: 86.2771, lr: 0.032926, batch_cost: 0.2909, reader_cost: 0.06704, ips: 82.5014 samples/sec | ETA 00:42:39 2022-09-27 22:59:12 [INFO] [TRAIN] epoch: 56, iter: 5300/14000, loss: 0.0899, DSC: 86.4819, lr: 0.032589, batch_cost: 0.2828, reader_cost: 0.06069, ips: 84.8717 samples/sec | ETA 00:41:00 2022-09-27 22:59:41 [INFO] [TRAIN] epoch: 58, iter: 5400/14000, loss: 0.0941, DSC: 85.7004, lr: 0.032251, batch_cost: 0.2932, reader_cost: 0.06891, ips: 81.8422 samples/sec | ETA 00:42:01 2022-09-27 23:00:10 [INFO] [TRAIN] epoch: 59, iter: 5500/14000, loss: 0.0897, DSC: 86.4526, lr: 0.031914, batch_cost: 0.2910, reader_cost: 0.06582, ips: 82.4733 samples/sec | ETA 00:41:13 2022-09-27 23:00:39 [INFO] [TRAIN] epoch: 60, iter: 5600/14000, loss: 0.0948, DSC: 85.6694, lr: 0.031576, batch_cost: 0.2859, reader_cost: 0.05934, ips: 83.9455 samples/sec | ETA 00:40:01 2022-09-27 23:01:07 [INFO] [TRAIN] epoch: 61, iter: 5700/14000, loss: 0.0894, DSC: 86.6363, lr: 0.031237, batch_cost: 0.2833, reader_cost: 0.06181, ips: 84.7157 samples/sec | ETA 00:39:11 2022-09-27 23:01:37 [INFO] [TRAIN] epoch: 62, iter: 5800/14000, loss: 0.0865, DSC: 87.0234, lr: 0.030898, batch_cost: 0.3007, reader_cost: 0.07443, ips: 79.8241 samples/sec | ETA 00:41:05 2022-09-27 23:02:06 [INFO] [TRAIN] epoch: 63, iter: 5900/14000, loss: 0.0860, DSC: 87.0207, lr: 0.030559, batch_cost: 0.2952, reader_cost: 0.07269, ips: 81.3088 samples/sec | ETA 00:39:50 2022-09-27 23:02:37 [INFO] [TRAIN] epoch: 64, iter: 6000/14000, loss: 0.0848, DSC: 87.2353, lr: 0.030219, batch_cost: 0.3026, reader_cost: 0.07274, ips: 79.3227 samples/sec | ETA 00:40:20 2022-09-27 23:02:37 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 951ms/step - batch_cost: 0.9508 - reader cost: 0.6355 2022-09-27 23:02:48 [INFO] [EVAL] #Images: 12, Dice: 0.8115, Loss: 0.126609 2022-09-27 23:02:48 [INFO] [EVAL] Class dice: [0.995 0.8581 0.5971 0.8888 0.8154 0.9455 0.6112 0.9083 0.6845] 2022-09-27 23:02:50 [INFO] [EVAL] The model with the best validation mDice (0.8115) was saved at iter 6000. 2022-09-27 23:03:19 [INFO] [TRAIN] epoch: 65, iter: 6100/14000, loss: 0.0820, DSC: 87.6742, lr: 0.029879, batch_cost: 0.2907, reader_cost: 0.06538, ips: 82.5500 samples/sec | ETA 00:38:16 2022-09-27 23:03:48 [INFO] [TRAIN] epoch: 66, iter: 6200/14000, loss: 0.0830, DSC: 87.4390, lr: 0.029539, batch_cost: 0.2887, reader_cost: 0.06166, ips: 83.1349 samples/sec | ETA 00:37:31 2022-09-27 23:04:17 [INFO] [TRAIN] epoch: 67, iter: 6300/14000, loss: 0.0807, DSC: 87.9230, lr: 0.029198, batch_cost: 0.2976, reader_cost: 0.07182, ips: 80.6406 samples/sec | ETA 00:38:11 2022-09-27 23:04:48 [INFO] [TRAIN] epoch: 68, iter: 6400/14000, loss: 0.0851, DSC: 87.1429, lr: 0.028856, batch_cost: 0.3036, reader_cost: 0.07674, ips: 79.0611 samples/sec | ETA 00:38:27 2022-09-27 23:05:16 [INFO] [TRAIN] epoch: 69, iter: 6500/14000, loss: 0.0830, DSC: 87.5024, lr: 0.028514, batch_cost: 0.2847, reader_cost: 0.06056, ips: 84.2881 samples/sec | ETA 00:35:35 2022-09-27 23:05:46 [INFO] [TRAIN] epoch: 70, iter: 6600/14000, loss: 0.0828, DSC: 87.5420, lr: 0.028172, batch_cost: 0.2958, reader_cost: 0.06746, ips: 81.1434 samples/sec | ETA 00:36:28 2022-09-27 23:06:15 [INFO] [TRAIN] epoch: 72, iter: 6700/14000, loss: 0.0898, DSC: 86.3793, lr: 0.027829, batch_cost: 0.2937, reader_cost: 0.06831, ips: 81.7166 samples/sec | ETA 00:35:43 2022-09-27 23:06:45 [INFO] [TRAIN] epoch: 73, iter: 6800/14000, loss: 0.0826, DSC: 87.5347, lr: 0.027486, batch_cost: 0.2955, reader_cost: 0.06713, ips: 81.2057 samples/sec | ETA 00:35:27 2022-09-27 23:07:13 [INFO] [TRAIN] epoch: 74, iter: 6900/14000, loss: 0.0842, DSC: 87.2589, lr: 0.027142, batch_cost: 0.2842, reader_cost: 0.06135, ips: 84.4514 samples/sec | ETA 00:33:37 2022-09-27 23:07:42 [INFO] [TRAIN] epoch: 75, iter: 7000/14000, loss: 0.0882, DSC: 86.6553, lr: 0.026798, batch_cost: 0.2930, reader_cost: 0.06931, ips: 81.9160 samples/sec | ETA 00:34:10 2022-09-27 23:07:42 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 957ms/step - batch_cost: 0.9567 - reader cost: 0.6421 2022-09-27 23:07:54 [INFO] [EVAL] #Images: 12, Dice: 0.8106, Loss: 0.127450 2022-09-27 23:07:54 [INFO] [EVAL] Class dice: [0.9955 0.8624 0.5998 0.8704 0.8123 0.9473 0.6 0.9067 0.7009] 2022-09-27 23:07:55 [INFO] [EVAL] The model with the best validation mDice (0.8115) was saved at iter 6000. 2022-09-27 23:08:22 [INFO] [TRAIN] epoch: 76, iter: 7100/14000, loss: 0.0819, DSC: 87.6376, lr: 0.026453, batch_cost: 0.2764, reader_cost: 0.05180, ips: 86.8284 samples/sec | ETA 00:31:47 2022-09-27 23:08:51 [INFO] [TRAIN] epoch: 77, iter: 7200/14000, loss: 0.0818, DSC: 87.6726, lr: 0.026108, batch_cost: 0.2880, reader_cost: 0.05965, ips: 83.3351 samples/sec | ETA 00:32:38 2022-09-27 23:09:19 [INFO] [TRAIN] epoch: 78, iter: 7300/14000, loss: 0.0801, DSC: 87.9555, lr: 0.025762, batch_cost: 0.2837, reader_cost: 0.05818, ips: 84.5889 samples/sec | ETA 00:31:40 2022-09-27 23:09:48 [INFO] [TRAIN] epoch: 79, iter: 7400/14000, loss: 0.0806, DSC: 87.8422, lr: 0.025416, batch_cost: 0.2903, reader_cost: 0.06523, ips: 82.6741 samples/sec | ETA 00:31:55 2022-09-27 23:10:17 [INFO] [TRAIN] epoch: 80, iter: 7500/14000, loss: 0.0782, DSC: 88.2361, lr: 0.025069, batch_cost: 0.2870, reader_cost: 0.06091, ips: 83.6246 samples/sec | ETA 00:31:05 2022-09-27 23:10:46 [INFO] [TRAIN] epoch: 81, iter: 7600/14000, loss: 0.0822, DSC: 87.5998, lr: 0.024722, batch_cost: 0.2857, reader_cost: 0.06172, ips: 84.0077 samples/sec | ETA 00:30:28 2022-09-27 23:11:14 [INFO] [TRAIN] epoch: 82, iter: 7700/14000, loss: 0.0789, DSC: 88.1274, lr: 0.024374, batch_cost: 0.2844, reader_cost: 0.05696, ips: 84.3808 samples/sec | ETA 00:29:51 2022-09-27 23:11:44 [INFO] [TRAIN] epoch: 83, iter: 7800/14000, loss: 0.0858, DSC: 86.9893, lr: 0.024025, batch_cost: 0.2977, reader_cost: 0.07336, ips: 80.6228 samples/sec | ETA 00:30:45 2022-09-27 23:12:13 [INFO] [TRAIN] epoch: 84, iter: 7900/14000, loss: 0.0823, DSC: 87.5464, lr: 0.023676, batch_cost: 0.2867, reader_cost: 0.06091, ips: 83.7052 samples/sec | ETA 00:29:08 2022-09-27 23:12:42 [INFO] [TRAIN] epoch: 86, iter: 8000/14000, loss: 0.0813, DSC: 87.6940, lr: 0.023327, batch_cost: 0.2940, reader_cost: 0.07128, ips: 81.6241 samples/sec | ETA 00:29:24 2022-09-27 23:12:42 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 12s 964ms/step - batch_cost: 0.9641 - reader cost: 0.6500 2022-09-27 23:12:54 [INFO] [EVAL] #Images: 12, Dice: 0.8085, Loss: 0.127570 2022-09-27 23:12:54 [INFO] [EVAL] Class dice: [0.9953 0.859 0.6037 0.849 0.8167 0.9479 0.5841 0.9083 0.7121] 2022-09-27 23:12:54 [INFO] [EVAL] The model with the best validation mDice (0.8115) was saved at iter 6000. 2022-09-27 23:13:23 [INFO] [TRAIN] epoch: 87, iter: 8100/14000, loss: 0.0813, DSC: 87.7059, lr: 0.022977, batch_cost: 0.2867, reader_cost: 0.06418, ips: 83.7219 samples/sec | ETA 00:28:11 2022-09-27 23:13:52 [INFO] [TRAIN] epoch: 88, iter: 8200/14000, loss: 0.0783, DSC: 88.2358, lr: 0.022626, batch_cost: 0.2906, reader_cost: 0.06604, ips: 82.5955 samples/sec | ETA 00:28:05 2022-09-27 23:14:21 [INFO] [TRAIN] epoch: 89, iter: 8300/14000, loss: 0.0830, DSC: 87.4123, lr: 0.022275, batch_cost: 0.2958, reader_cost: 0.07013, ips: 81.1366 samples/sec | ETA 00:28:06 2022-09-27 23:14:51 [INFO] [TRAIN] epoch: 90, iter: 8400/14000, loss: 0.0773, DSC: 88.3500, lr: 0.021923, batch_cost: 0.2948, reader_cost: 0.07209, ips: 81.4098 samples/sec | ETA 00:27:30 2022-09-27 23:15:20 [INFO] [TRAIN] epoch: 91, iter: 8500/14000, loss: 0.0864, DSC: 86.9026, lr: 0.021570, batch_cost: 0.2883, reader_cost: 0.05729, ips: 83.2394 samples/sec | ETA 00:26:25 2022-09-27 23:15:49 [INFO] [TRAIN] epoch: 92, iter: 8600/14000, loss: 0.0787, DSC: 88.1577, lr: 0.021217, batch_cost: 0.2873, reader_cost: 0.06438, ips: 83.5497 samples/sec | ETA 00:25:51 2022-09-27 23:16:18 [INFO] [TRAIN] epoch: 93, iter: 8700/14000, loss: 0.0818, DSC: 87.5836, lr: 0.020863, batch_cost: 0.2977, reader_cost: 0.07101, ips: 80.6170 samples/sec | ETA 00:26:17 2022-09-27 23:16:47 [INFO] [TRAIN] epoch: 94, iter: 8800/14000, loss: 0.0760, DSC: 88.5867, lr: 0.020508, batch_cost: 0.2864, reader_cost: 0.06222, ips: 83.7949 samples/sec | ETA 00:24:49 2022-09-27 23:17:16 [INFO] [TRAIN] epoch: 95, iter: 8900/14000, loss: 0.0779, DSC: 88.2380, lr: 0.020153, batch_cost: 0.2950, reader_cost: 0.06755, ips: 81.3640 samples/sec | ETA 00:25:04 2022-09-27 23:17:45 [INFO] [TRAIN] epoch: 96, iter: 9000/14000, loss: 0.0779, DSC: 88.2319, lr: 0.019797, batch_cost: 0.2888, reader_cost: 0.06435, ips: 83.0959 samples/sec | ETA 00:24:04 2022-09-27 23:17:45 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 955ms/step - batch_cost: 0.9553 - reader cost: 0.6407 2022-09-27 23:17:57 [INFO] [EVAL] #Images: 12, Dice: 0.8162, Loss: 0.122939 2022-09-27 23:17:57 [INFO] [EVAL] Class dice: [0.9959 0.8589 0.6034 0.8888 0.8044 0.9535 0.5951 0.9119 0.7335] 2022-09-27 23:17:58 [INFO] [EVAL] The model with the best validation mDice (0.8162) was saved at iter 9000. 2022-09-27 23:18:27 [INFO] [TRAIN] epoch: 97, iter: 9100/14000, loss: 0.0743, DSC: 88.8206, lr: 0.019441, batch_cost: 0.2904, reader_cost: 0.05982, ips: 82.6553 samples/sec | ETA 00:23:42 2022-09-27 23:18:56 [INFO] [TRAIN] epoch: 98, iter: 9200/14000, loss: 0.0761, DSC: 88.5175, lr: 0.019083, batch_cost: 0.2850, reader_cost: 0.06143, ips: 84.2201 samples/sec | ETA 00:22:47 2022-09-27 23:19:24 [INFO] [TRAIN] epoch: 100, iter: 9300/14000, loss: 0.0813, DSC: 87.6231, lr: 0.018725, batch_cost: 0.2859, reader_cost: 0.05999, ips: 83.9437 samples/sec | ETA 00:22:23 2022-09-27 23:19:54 [INFO] [TRAIN] epoch: 101, iter: 9400/14000, loss: 0.0787, DSC: 88.0743, lr: 0.018366, batch_cost: 0.2931, reader_cost: 0.07118, ips: 81.8796 samples/sec | ETA 00:22:28 2022-09-27 23:20:23 [INFO] [TRAIN] epoch: 102, iter: 9500/14000, loss: 0.0819, DSC: 87.5425, lr: 0.018007, batch_cost: 0.2938, reader_cost: 0.06649, ips: 81.6899 samples/sec | ETA 00:22:02 2022-09-27 23:20:52 [INFO] [TRAIN] epoch: 103, iter: 9600/14000, loss: 0.0827, DSC: 87.4250, lr: 0.017646, batch_cost: 0.2883, reader_cost: 0.06441, ips: 83.2493 samples/sec | ETA 00:21:08 2022-09-27 23:21:21 [INFO] [TRAIN] epoch: 104, iter: 9700/14000, loss: 0.0786, DSC: 88.0794, lr: 0.017285, batch_cost: 0.2868, reader_cost: 0.06103, ips: 83.6882 samples/sec | ETA 00:20:33 2022-09-27 23:21:50 [INFO] [TRAIN] epoch: 105, iter: 9800/14000, loss: 0.0757, DSC: 88.5958, lr: 0.016923, batch_cost: 0.2956, reader_cost: 0.06729, ips: 81.1896 samples/sec | ETA 00:20:41 2022-09-27 23:22:20 [INFO] [TRAIN] epoch: 106, iter: 9900/14000, loss: 0.0803, DSC: 87.8233, lr: 0.016560, batch_cost: 0.2965, reader_cost: 0.07048, ips: 80.9492 samples/sec | ETA 00:20:15 2022-09-27 23:22:48 [INFO] [TRAIN] epoch: 107, iter: 10000/14000, loss: 0.0781, DSC: 88.1857, lr: 0.016196, batch_cost: 0.2867, reader_cost: 0.05868, ips: 83.7248 samples/sec | ETA 00:19:06 2022-09-27 23:22:48 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 12s 958ms/step - batch_cost: 0.9583 - reader cost: 0.6433 2022-09-27 23:23:00 [INFO] [EVAL] #Images: 12, Dice: 0.8229, Loss: 0.118284 2022-09-27 23:23:00 [INFO] [EVAL] Class dice: [0.996 0.8704 0.6037 0.893 0.8011 0.9493 0.6033 0.9141 0.7748] 2022-09-27 23:23:02 [INFO] [EVAL] The model with the best validation mDice (0.8229) was saved at iter 10000. 2022-09-27 23:23:30 [INFO] [TRAIN] epoch: 108, iter: 10100/14000, loss: 0.0755, DSC: 88.5851, lr: 0.015831, batch_cost: 0.2812, reader_cost: 0.05571, ips: 85.3424 samples/sec | ETA 00:18:16 2022-09-27 23:23:59 [INFO] [TRAIN] epoch: 109, iter: 10200/14000, loss: 0.0799, DSC: 87.8771, lr: 0.015465, batch_cost: 0.2899, reader_cost: 0.06490, ips: 82.7949 samples/sec | ETA 00:18:21 2022-09-27 23:24:28 [INFO] [TRAIN] epoch: 110, iter: 10300/14000, loss: 0.0746, DSC: 88.7639, lr: 0.015099, batch_cost: 0.2903, reader_cost: 0.06371, ips: 82.6750 samples/sec | ETA 00:17:54 2022-09-27 23:24:57 [INFO] [TRAIN] epoch: 111, iter: 10400/14000, loss: 0.0763, DSC: 88.4674, lr: 0.014731, batch_cost: 0.2844, reader_cost: 0.06012, ips: 84.3832 samples/sec | ETA 00:17:03 2022-09-27 23:25:26 [INFO] [TRAIN] epoch: 112, iter: 10500/14000, loss: 0.0752, DSC: 88.6502, lr: 0.014362, batch_cost: 0.2894, reader_cost: 0.06496, ips: 82.9248 samples/sec | ETA 00:16:52 2022-09-27 23:25:55 [INFO] [TRAIN] epoch: 113, iter: 10600/14000, loss: 0.0758, DSC: 88.5158, lr: 0.013993, batch_cost: 0.2902, reader_cost: 0.06358, ips: 82.7072 samples/sec | ETA 00:16:26 2022-09-27 23:26:24 [INFO] [TRAIN] epoch: 115, iter: 10700/14000, loss: 0.0735, DSC: 88.9274, lr: 0.013622, batch_cost: 0.2961, reader_cost: 0.07176, ips: 81.0511 samples/sec | ETA 00:16:17 2022-09-27 23:26:54 [INFO] [TRAIN] epoch: 116, iter: 10800/14000, loss: 0.0794, DSC: 87.9333, lr: 0.013250, batch_cost: 0.2994, reader_cost: 0.07088, ips: 80.1589 samples/sec | ETA 00:15:58 2022-09-27 23:27:23 [INFO] [TRAIN] epoch: 117, iter: 10900/14000, loss: 0.0773, DSC: 88.2611, lr: 0.012877, batch_cost: 0.2821, reader_cost: 0.05941, ips: 85.0819 samples/sec | ETA 00:14:34 2022-09-27 23:27:53 [INFO] [TRAIN] epoch: 118, iter: 11000/14000, loss: 0.0772, DSC: 88.2760, lr: 0.012502, batch_cost: 0.2997, reader_cost: 0.07137, ips: 80.0671 samples/sec | ETA 00:14:59 2022-09-27 23:27:53 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 944ms/step - batch_cost: 0.9439 - reader cost: 0.6292 2022-09-27 23:28:04 [INFO] [EVAL] #Images: 12, Dice: 0.8176, Loss: 0.121719 2022-09-27 23:28:04 [INFO] [EVAL] Class dice: [0.9959 0.8571 0.5954 0.9 0.7994 0.9528 0.5938 0.9108 0.7533] 2022-09-27 23:28:05 [INFO] [EVAL] The model with the best validation mDice (0.8229) was saved at iter 10000. 2022-09-27 23:28:34 [INFO] [TRAIN] epoch: 119, iter: 11100/14000, loss: 0.0773, DSC: 88.2950, lr: 0.012127, batch_cost: 0.2814, reader_cost: 0.05701, ips: 85.2921 samples/sec | ETA 00:13:36 2022-09-27 23:29:03 [INFO] [TRAIN] epoch: 120, iter: 11200/14000, loss: 0.0733, DSC: 88.9416, lr: 0.011750, batch_cost: 0.2938, reader_cost: 0.06785, ips: 81.6779 samples/sec | ETA 00:13:42 2022-09-27 23:29:32 [INFO] [TRAIN] epoch: 121, iter: 11300/14000, loss: 0.0749, DSC: 88.6797, lr: 0.011372, batch_cost: 0.2881, reader_cost: 0.06292, ips: 83.3037 samples/sec | ETA 00:12:57 2022-09-27 23:30:01 [INFO] [TRAIN] epoch: 122, iter: 11400/14000, loss: 0.0738, DSC: 88.8430, lr: 0.010992, batch_cost: 0.2910, reader_cost: 0.06410, ips: 82.4636 samples/sec | ETA 00:12:36 2022-09-27 23:30:28 [INFO] [TRAIN] epoch: 123, iter: 11500/14000, loss: 0.0792, DSC: 87.9536, lr: 0.010611, batch_cost: 0.2740, reader_cost: 0.05007, ips: 87.5875 samples/sec | ETA 00:11:25 2022-09-27 23:30:58 [INFO] [TRAIN] epoch: 124, iter: 11600/14000, loss: 0.0745, DSC: 88.7226, lr: 0.010228, batch_cost: 0.2986, reader_cost: 0.07227, ips: 80.3754 samples/sec | ETA 00:11:56 2022-09-27 23:31:26 [INFO] [TRAIN] epoch: 125, iter: 11700/14000, loss: 0.0737, DSC: 88.8944, lr: 0.009844, batch_cost: 0.2825, reader_cost: 0.05996, ips: 84.9427 samples/sec | ETA 00:10:49 2022-09-27 23:31:55 [INFO] [TRAIN] epoch: 126, iter: 11800/14000, loss: 0.0724, DSC: 89.0683, lr: 0.009458, batch_cost: 0.2895, reader_cost: 0.06052, ips: 82.8919 samples/sec | ETA 00:10:36 2022-09-27 23:32:25 [INFO] [TRAIN] epoch: 127, iter: 11900/14000, loss: 0.0749, DSC: 88.6619, lr: 0.009071, batch_cost: 0.2976, reader_cost: 0.07167, ips: 80.6427 samples/sec | ETA 00:10:24 2022-09-27 23:32:55 [INFO] [TRAIN] epoch: 129, iter: 12000/14000, loss: 0.0744, DSC: 88.7293, lr: 0.008681, batch_cost: 0.3023, reader_cost: 0.07786, ips: 79.3786 samples/sec | ETA 00:10:04 2022-09-27 23:32:55 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 11s 950ms/step - batch_cost: 0.9496 - reader cost: 0.6347 2022-09-27 23:33:07 [INFO] [EVAL] #Images: 12, Dice: 0.8170, Loss: 0.122663 2022-09-27 23:33:07 [INFO] [EVAL] Class dice: [0.9959 0.8602 0.5862 0.9026 0.8161 0.9532 0.5814 0.913 0.7448] 2022-09-27 23:33:08 [INFO] [EVAL] The model with the best validation mDice (0.8229) was saved at iter 10000. 2022-09-27 23:33:37 [INFO] [TRAIN] epoch: 130, iter: 12100/14000, loss: 0.0745, DSC: 88.7466, lr: 0.008290, batch_cost: 0.2872, reader_cost: 0.06356, ips: 83.5679 samples/sec | ETA 00:09:05 2022-09-27 23:34:07 [INFO] [TRAIN] epoch: 131, iter: 12200/14000, loss: 0.0747, DSC: 88.6304, lr: 0.007896, batch_cost: 0.2974, reader_cost: 0.07098, ips: 80.6984 samples/sec | ETA 00:08:55 2022-09-27 23:34:36 [INFO] [TRAIN] epoch: 132, iter: 12300/14000, loss: 0.0719, DSC: 89.1490, lr: 0.007500, batch_cost: 0.2900, reader_cost: 0.06410, ips: 82.7467 samples/sec | ETA 00:08:13 2022-09-27 23:35:05 [INFO] [TRAIN] epoch: 133, iter: 12400/14000, loss: 0.0790, DSC: 87.9619, lr: 0.007102, batch_cost: 0.2966, reader_cost: 0.06806, ips: 80.9161 samples/sec | ETA 00:07:54 2022-09-27 23:35:34 [INFO] [TRAIN] epoch: 134, iter: 12500/14000, loss: 0.0702, DSC: 89.4331, lr: 0.006702, batch_cost: 0.2880, reader_cost: 0.06210, ips: 83.3478 samples/sec | ETA 00:07:11 2022-09-27 23:36:04 [INFO] [TRAIN] epoch: 135, iter: 12600/14000, loss: 0.0708, DSC: 89.3249, lr: 0.006299, batch_cost: 0.2940, reader_cost: 0.06839, ips: 81.6454 samples/sec | ETA 00:06:51 2022-09-27 23:36:33 [INFO] [TRAIN] epoch: 136, iter: 12700/14000, loss: 0.0749, DSC: 88.6544, lr: 0.005893, batch_cost: 0.2953, reader_cost: 0.06879, ips: 81.2677 samples/sec | ETA 00:06:23 2022-09-27 23:37:02 [INFO] [TRAIN] epoch: 137, iter: 12800/14000, loss: 0.0756, DSC: 88.5279, lr: 0.005483, batch_cost: 0.2922, reader_cost: 0.06597, ips: 82.1396 samples/sec | ETA 00:05:50 2022-09-27 23:37:31 [INFO] [TRAIN] epoch: 138, iter: 12900/14000, loss: 0.0734, DSC: 88.9147, lr: 0.005071, batch_cost: 0.2888, reader_cost: 0.06371, ips: 83.1086 samples/sec | ETA 00:05:17 2022-09-27 23:38:00 [INFO] [TRAIN] epoch: 139, iter: 13000/14000, loss: 0.0720, DSC: 89.0665, lr: 0.004654, batch_cost: 0.2901, reader_cost: 0.06462, ips: 82.7164 samples/sec | ETA 00:04:50 2022-09-27 23:38:00 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 12s 973ms/step - batch_cost: 0.9723 - reader cost: 0.6578 2022-09-27 23:38:12 [INFO] [EVAL] #Images: 12, Dice: 0.8189, Loss: 0.120966 2022-09-27 23:38:12 [INFO] [EVAL] Class dice: [0.9959 0.8686 0.5791 0.8994 0.812 0.9519 0.5969 0.9105 0.7556] 2022-09-27 23:38:14 [INFO] [EVAL] The model with the best validation mDice (0.8229) was saved at iter 10000. 2022-09-27 23:38:42 [INFO] [TRAIN] epoch: 140, iter: 13100/14000, loss: 0.0717, DSC: 89.2136, lr: 0.004234, batch_cost: 0.2879, reader_cost: 0.06676, ips: 83.3632 samples/sec | ETA 00:04:19 2022-09-27 23:39:11 [INFO] [TRAIN] epoch: 141, iter: 13200/14000, loss: 0.0730, DSC: 88.9068, lr: 0.003808, batch_cost: 0.2860, reader_cost: 0.05985, ips: 83.9207 samples/sec | ETA 00:03:48 2022-09-27 23:39:41 [INFO] [TRAIN] epoch: 143, iter: 13300/14000, loss: 0.0784, DSC: 88.0366, lr: 0.003378, batch_cost: 0.3001, reader_cost: 0.07590, ips: 79.9765 samples/sec | ETA 00:03:30 2022-09-27 23:40:10 [INFO] [TRAIN] epoch: 144, iter: 13400/14000, loss: 0.0708, DSC: 89.3090, lr: 0.002941, batch_cost: 0.2938, reader_cost: 0.07135, ips: 81.6853 samples/sec | ETA 00:02:56 2022-09-27 23:40:39 [INFO] [TRAIN] epoch: 145, iter: 13500/14000, loss: 0.0708, DSC: 89.3075, lr: 0.002496, batch_cost: 0.2857, reader_cost: 0.06027, ips: 84.0155 samples/sec | ETA 00:02:22 2022-09-27 23:41:07 [INFO] [TRAIN] epoch: 146, iter: 13600/14000, loss: 0.0715, DSC: 89.1609, lr: 0.002043, batch_cost: 0.2845, reader_cost: 0.05936, ips: 84.3670 samples/sec | ETA 00:01:53 2022-09-27 23:41:37 [INFO] [TRAIN] epoch: 147, iter: 13700/14000, loss: 0.0746, DSC: 88.6361, lr: 0.001578, batch_cost: 0.2964, reader_cost: 0.06735, ips: 80.9774 samples/sec | ETA 00:01:28 2022-09-27 23:42:06 [INFO] [TRAIN] epoch: 148, iter: 13800/14000, loss: 0.0714, DSC: 89.2363, lr: 0.001097, batch_cost: 0.2859, reader_cost: 0.06225, ips: 83.9426 samples/sec | ETA 00:00:57 2022-09-27 23:42:36 [INFO] [TRAIN] epoch: 149, iter: 13900/14000, loss: 0.0725, DSC: 89.0133, lr: 0.000591, batch_cost: 0.3011, reader_cost: 0.07146, ips: 79.7134 samples/sec | ETA 00:00:30 2022-09-27 23:43:04 [INFO] [TRAIN] epoch: 150, iter: 14000/14000, loss: 0.0699, DSC: 89.4464, lr: 0.000009, batch_cost: 0.2810, reader_cost: 0.05721, ips: 85.4059 samples/sec | ETA 00:00:00 2022-09-27 23:43:04 [INFO] Start evaluating (total_samples: 12, total_iters: 12)... 12/12 [==============================] - 12s 961ms/step - batch_cost: 0.9609 - reader cost: 0.6462 2022-09-27 23:43:15 [INFO] [EVAL] #Images: 12, Dice: 0.8209, Loss: 0.119803 2022-09-27 23:43:15 [INFO] [EVAL] Class dice: [0.996 0.8672 0.5958 0.9037 0.8136 0.9524 0.5932 0.9106 0.7557] 2022-09-27 23:43:17 [INFO] [EVAL] The model with the best validation mDice (0.8229) was saved at iter 10000. 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. 's flops has been counted 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. Cannot find suitable count function for . Treat it as zero FLOPs. Cannot find suitable count function for . Treat it as zero FLOPs. Cannot find suitable count function for . Treat it as zero FLOPs. 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. Total Flops: 5896181760 Total Params: 27146592