----------- Configuration Arguments ----------- gpus: None heter_worker_num: None heter_workers: http_port: None ips: 127.0.0.1 log_dir: log nproc_per_node: None server_num: None servers: training_script: train.py training_script_args: ['--do_eval', '--use_vdl', '--config', 'configs/dnlnet/dnlnet_resnet101_os8_voc12aug_512x512_40.yml'] worker_num: None workers: ------------------------------------------------ launch train in GPU mode 2021-01-18 09:45:15 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.6.12 |Anaconda, Inc.| (default, Sep 8 2020, 23:10:56) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-32GB', 'GPU 1: Tesla V100-SXM2-32GB', 'GPU 2: Tesla V100-SXM2-32GB', 'GPU 3: Tesla V100-SXM2-32GB', 'GPU 4: Tesla V100-SXM2-32GB', 'GPU 5: Tesla V100-SXM2-32GB', 'GPU 6: Tesla V100-SXM2-32GB', 'GPU 7: Tesla V100-SXM2-32GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc1 OpenCV: 4.5.1 ------------------------------------------------ 2021-01-18 09:45:15 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss model: backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd type: DNLNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ W0118 09:45:15.903084 39387 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2 W0118 09:45:15.903177 39387 device_context.cc:330] device: 0, cuDNN Version: 7.6. 2021-01-18 09:45:19 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2021-01-18 09:45:21 [INFO] There are 530/530 variables loaded into ResNet_vd. I0118 09:45:21.353114 39387 nccl_context.cc:179] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0 2021-01-18 09:45:33 [INFO] [TRAIN] epoch=1, iter=10/40000, loss=2.6460, lr=0.009998, batch_cost=0.8598, reader_cost=0.0403 | ETA 09:33:04 2021-01-18 09:45:40 [INFO] [TRAIN] epoch=1, iter=20/40000, loss=1.5266, lr=0.009996, batch_cost=0.7832, reader_cost=0.0001 | ETA 08:41:50 2021-01-18 09:45:48 [INFO] [TRAIN] epoch=1, iter=30/40000, loss=1.6505, lr=0.009993, batch_cost=0.7903, reader_cost=0.0001 | ETA 08:46:29 2021-01-18 09:45:56 [INFO] [TRAIN] epoch=1, iter=40/40000, loss=1.5362, lr=0.009991, batch_cost=0.7949, reader_cost=0.0001 | ETA 08:49:22 2021-01-18 09:46:04 [INFO] [TRAIN] epoch=1, iter=50/40000, loss=1.2492, lr=0.009989, batch_cost=0.7873, reader_cost=0.0001 | ETA 08:44:11 2021-01-18 09:46:12 [INFO] [TRAIN] epoch=1, iter=60/40000, loss=1.2971, lr=0.009987, batch_cost=0.7879, reader_cost=0.0001 | ETA 08:44:30 2021-01-18 09:46:20 [INFO] [TRAIN] epoch=1, iter=70/40000, loss=1.4915, lr=0.009984, batch_cost=0.7919, reader_cost=0.0001 | ETA 08:47:02 2021-01-18 09:46:28 [INFO] [TRAIN] epoch=1, iter=80/40000, loss=1.2779, lr=0.009982, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:45:43 2021-01-18 09:46:36 [INFO] [TRAIN] epoch=1, iter=90/40000, loss=1.4596, lr=0.009980, batch_cost=0.7913, reader_cost=0.0001 | ETA 08:46:21 2021-01-18 09:46:44 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.2946, lr=0.009978, batch_cost=0.7927, reader_cost=0.0001 | ETA 08:47:09 2021-01-18 09:46:52 [INFO] [TRAIN] epoch=1, iter=110/40000, loss=1.4040, lr=0.009975, batch_cost=0.7930, reader_cost=0.0001 | ETA 08:47:13 2021-01-18 09:47:00 [INFO] [TRAIN] epoch=1, iter=120/40000, loss=1.0142, lr=0.009973, batch_cost=0.7884, reader_cost=0.0001 | ETA 08:44:00 2021-01-18 09:47:07 [INFO] [TRAIN] epoch=1, iter=130/40000, loss=1.1730, lr=0.009971, batch_cost=0.7917, reader_cost=0.0001 | ETA 08:46:03 2021-01-18 09:47:15 [INFO] [TRAIN] epoch=1, iter=140/40000, loss=1.3080, lr=0.009969, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:45:32 2021-01-18 09:47:23 [INFO] [TRAIN] epoch=1, iter=150/40000, loss=1.6487, lr=0.009966, batch_cost=0.7961, reader_cost=0.0001 | ETA 08:48:43 2021-01-18 09:47:31 [INFO] [TRAIN] epoch=1, iter=160/40000, loss=1.8108, lr=0.009964, batch_cost=0.7876, reader_cost=0.0001 | ETA 08:42:57 2021-01-18 09:47:39 [INFO] [TRAIN] epoch=1, iter=170/40000, loss=1.0484, lr=0.009962, batch_cost=0.7884, reader_cost=0.0001 | ETA 08:43:23 2021-01-18 09:47:47 [INFO] [TRAIN] epoch=1, iter=180/40000, loss=1.5431, lr=0.009960, batch_cost=0.7913, reader_cost=0.0001 | ETA 08:45:09 2021-01-18 09:47:55 [INFO] [TRAIN] epoch=1, iter=190/40000, loss=1.3266, lr=0.009957, batch_cost=0.7849, reader_cost=0.0001 | ETA 08:40:47 2021-01-18 09:48:03 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=1.0607, lr=0.009955, batch_cost=0.7939, reader_cost=0.0001 | ETA 08:46:38 2021-01-18 09:48:11 [INFO] [TRAIN] epoch=1, iter=210/40000, loss=1.3011, lr=0.009953, batch_cost=0.7835, reader_cost=0.0001 | ETA 08:39:36 2021-01-18 09:48:19 [INFO] [TRAIN] epoch=1, iter=220/40000, loss=0.9400, lr=0.009951, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:42:47 2021-01-18 09:48:26 [INFO] [TRAIN] epoch=1, iter=230/40000, loss=0.9734, lr=0.009948, batch_cost=0.7927, reader_cost=0.0001 | ETA 08:45:27 2021-01-18 09:48:34 [INFO] [TRAIN] epoch=1, iter=240/40000, loss=1.2534, lr=0.009946, batch_cost=0.7890, reader_cost=0.0001 | ETA 08:42:49 2021-01-18 09:48:42 [INFO] [TRAIN] epoch=1, iter=250/40000, loss=1.1385, lr=0.009944, batch_cost=0.7883, reader_cost=0.0001 | ETA 08:42:13 2021-01-18 09:48:50 [INFO] [TRAIN] epoch=1, iter=260/40000, loss=1.2609, lr=0.009942, batch_cost=0.7890, reader_cost=0.0001 | ETA 08:42:34 2021-01-18 09:48:58 [INFO] [TRAIN] epoch=1, iter=270/40000, loss=1.1829, lr=0.009939, batch_cost=0.7924, reader_cost=0.0001 | ETA 08:44:40 2021-01-18 09:49:06 [INFO] [TRAIN] epoch=1, iter=280/40000, loss=1.0009, lr=0.009937, batch_cost=0.7849, reader_cost=0.0001 | ETA 08:39:37 2021-01-18 09:49:14 [INFO] [TRAIN] epoch=1, iter=290/40000, loss=1.0539, lr=0.009935, batch_cost=0.7957, reader_cost=0.0001 | ETA 08:46:39 2021-01-18 09:49:22 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=1.0645, lr=0.009933, batch_cost=0.7872, reader_cost=0.0001 | ETA 08:40:49 2021-01-18 09:49:30 [INFO] [TRAIN] epoch=1, iter=310/40000, loss=1.3911, lr=0.009930, batch_cost=0.7941, reader_cost=0.0001 | ETA 08:45:18 2021-01-18 09:49:38 [INFO] [TRAIN] epoch=1, iter=320/40000, loss=1.3980, lr=0.009928, batch_cost=0.7916, reader_cost=0.0001 | ETA 08:43:30 2021-01-18 09:49:45 [INFO] [TRAIN] epoch=1, iter=330/40000, loss=0.7945, lr=0.009926, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:42:11 2021-01-18 09:49:53 [INFO] [TRAIN] epoch=1, iter=340/40000, loss=0.8395, lr=0.009924, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:40:51 2021-01-18 09:50:01 [INFO] [TRAIN] epoch=1, iter=350/40000, loss=0.9789, lr=0.009921, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:42:47 2021-01-18 09:50:09 [INFO] [TRAIN] epoch=1, iter=360/40000, loss=0.9798, lr=0.009919, batch_cost=0.7864, reader_cost=0.0001 | ETA 08:39:34 2021-01-18 09:50:17 [INFO] [TRAIN] epoch=1, iter=370/40000, loss=0.9814, lr=0.009917, batch_cost=0.7943, reader_cost=0.0001 | ETA 08:44:38 2021-01-18 09:50:25 [INFO] [TRAIN] epoch=1, iter=380/40000, loss=1.0829, lr=0.009915, batch_cost=0.7893, reader_cost=0.0001 | ETA 08:41:12 2021-01-18 09:50:33 [INFO] [TRAIN] epoch=1, iter=390/40000, loss=0.8459, lr=0.009912, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:43:57 2021-01-18 09:50:41 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.5742, lr=0.009910, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:42:06 2021-01-18 09:50:49 [INFO] [TRAIN] epoch=1, iter=410/40000, loss=0.8177, lr=0.009908, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:41:58 2021-01-18 09:50:57 [INFO] [TRAIN] epoch=1, iter=420/40000, loss=0.7646, lr=0.009906, batch_cost=0.7904, reader_cost=0.0001 | ETA 08:41:23 2021-01-18 09:51:05 [INFO] [TRAIN] epoch=1, iter=430/40000, loss=0.9355, lr=0.009903, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:43:25 2021-01-18 09:51:12 [INFO] [TRAIN] epoch=1, iter=440/40000, loss=0.9822, lr=0.009901, batch_cost=0.7912, reader_cost=0.0001 | ETA 08:41:39 2021-01-18 09:51:20 [INFO] [TRAIN] epoch=1, iter=450/40000, loss=0.7471, lr=0.009899, batch_cost=0.7893, reader_cost=0.0001 | ETA 08:40:16 2021-01-18 09:51:28 [INFO] [TRAIN] epoch=1, iter=460/40000, loss=1.3499, lr=0.009897, batch_cost=0.7870, reader_cost=0.0001 | ETA 08:38:38 2021-01-18 09:51:36 [INFO] [TRAIN] epoch=1, iter=470/40000, loss=1.0667, lr=0.009894, batch_cost=0.7848, reader_cost=0.0001 | ETA 08:37:02 2021-01-18 09:51:44 [INFO] [TRAIN] epoch=1, iter=480/40000, loss=0.7918, lr=0.009892, batch_cost=0.7894, reader_cost=0.0001 | ETA 08:39:56 2021-01-18 09:51:52 [INFO] [TRAIN] epoch=1, iter=490/40000, loss=0.9879, lr=0.009890, batch_cost=0.7872, reader_cost=0.0001 | ETA 08:38:23 2021-01-18 09:52:00 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=1.3547, lr=0.009888, batch_cost=0.7854, reader_cost=0.0001 | ETA 08:37:03 2021-01-18 09:52:08 [INFO] [TRAIN] epoch=1, iter=510/40000, loss=1.3539, lr=0.009885, batch_cost=0.7861, reader_cost=0.0001 | ETA 08:37:23 2021-01-18 09:52:15 [INFO] [TRAIN] epoch=1, iter=520/40000, loss=1.1384, lr=0.009883, batch_cost=0.7897, reader_cost=0.0001 | ETA 08:39:37 2021-01-18 09:52:23 [INFO] [TRAIN] epoch=1, iter=530/40000, loss=0.8778, lr=0.009881, batch_cost=0.7869, reader_cost=0.0001 | ETA 08:37:37 2021-01-18 09:52:31 [INFO] [TRAIN] epoch=1, iter=540/40000, loss=0.8491, lr=0.009879, batch_cost=0.7860, reader_cost=0.0001 | ETA 08:36:57 2021-01-18 09:52:39 [INFO] [TRAIN] epoch=1, iter=550/40000, loss=1.2561, lr=0.009876, batch_cost=0.7888, reader_cost=0.0001 | ETA 08:38:38 2021-01-18 09:52:47 [INFO] [TRAIN] epoch=1, iter=560/40000, loss=0.9603, lr=0.009874, batch_cost=0.7887, reader_cost=0.0001 | ETA 08:38:26 2021-01-18 09:52:55 [INFO] [TRAIN] epoch=1, iter=570/40000, loss=0.9529, lr=0.009872, batch_cost=0.7875, reader_cost=0.0001 | ETA 08:37:30 2021-01-18 09:53:03 [INFO] [TRAIN] epoch=1, iter=580/40000, loss=0.7652, lr=0.009870, batch_cost=0.7924, reader_cost=0.0001 | ETA 08:40:36 2021-01-18 09:53:11 [INFO] [TRAIN] epoch=1, iter=590/40000, loss=1.1903, lr=0.009867, batch_cost=0.7907, reader_cost=0.0001 | ETA 08:39:19 2021-01-18 09:53:19 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=1.1714, lr=0.009865, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:38:38 2021-01-18 09:53:26 [INFO] [TRAIN] epoch=1, iter=610/40000, loss=1.7056, lr=0.009863, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:38:46 2021-01-18 09:53:34 [INFO] [TRAIN] epoch=1, iter=620/40000, loss=1.1641, lr=0.009861, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:40:08 2021-01-18 09:53:42 [INFO] [TRAIN] epoch=1, iter=630/40000, loss=0.7543, lr=0.009858, batch_cost=0.7930, reader_cost=0.0001 | ETA 08:40:20 2021-01-18 09:53:50 [INFO] [TRAIN] epoch=1, iter=640/40000, loss=1.1561, lr=0.009856, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:40:41 2021-01-18 09:53:58 [INFO] [TRAIN] epoch=1, iter=650/40000, loss=0.6645, lr=0.009854, batch_cost=0.7851, reader_cost=0.0001 | ETA 08:34:55 2021-01-18 09:54:06 [INFO] [TRAIN] epoch=1, iter=660/40000, loss=1.1677, lr=0.009852, batch_cost=0.7580, reader_cost=0.0001 | ETA 08:17:01 2021-01-18 09:54:14 [INFO] [TRAIN] epoch=2, iter=670/40000, loss=0.7856, lr=0.009849, batch_cost=0.8025, reader_cost=0.0179 | ETA 08:46:01 2021-01-18 09:54:22 [INFO] [TRAIN] epoch=2, iter=680/40000, loss=0.5930, lr=0.009847, batch_cost=0.7927, reader_cost=0.0001 | ETA 08:39:29 2021-01-18 09:54:30 [INFO] [TRAIN] epoch=2, iter=690/40000, loss=0.8299, lr=0.009845, batch_cost=0.7855, reader_cost=0.0001 | ETA 08:34:37 2021-01-18 09:54:37 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=1.2595, lr=0.009843, batch_cost=0.7930, reader_cost=0.0001 | ETA 08:39:25 2021-01-18 09:54:45 [INFO] [TRAIN] epoch=2, iter=710/40000, loss=1.2456, lr=0.009840, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:37:28 2021-01-18 09:54:53 [INFO] [TRAIN] epoch=2, iter=720/40000, loss=1.0701, lr=0.009838, batch_cost=0.7820, reader_cost=0.0001 | ETA 08:31:56 2021-01-18 09:55:01 [INFO] [TRAIN] epoch=2, iter=730/40000, loss=0.8114, lr=0.009836, batch_cost=0.7985, reader_cost=0.0001 | ETA 08:42:36 2021-01-18 09:55:09 [INFO] [TRAIN] epoch=2, iter=740/40000, loss=0.7156, lr=0.009834, batch_cost=0.7924, reader_cost=0.0001 | ETA 08:38:31 2021-01-18 09:55:17 [INFO] [TRAIN] epoch=2, iter=750/40000, loss=1.1516, lr=0.009831, batch_cost=0.7918, reader_cost=0.0001 | ETA 08:37:58 2021-01-18 09:55:25 [INFO] [TRAIN] epoch=2, iter=760/40000, loss=1.2135, lr=0.009829, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:36:30 2021-01-18 09:55:33 [INFO] [TRAIN] epoch=2, iter=770/40000, loss=1.3188, lr=0.009827, batch_cost=0.7875, reader_cost=0.0001 | ETA 08:34:52 2021-01-18 09:55:41 [INFO] [TRAIN] epoch=2, iter=780/40000, loss=0.6444, lr=0.009825, batch_cost=0.7963, reader_cost=0.0001 | ETA 08:40:31 2021-01-18 09:55:49 [INFO] [TRAIN] epoch=2, iter=790/40000, loss=0.7302, lr=0.009822, batch_cost=0.7888, reader_cost=0.0001 | ETA 08:35:30 2021-01-18 09:55:56 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.7526, lr=0.009820, batch_cost=0.7833, reader_cost=0.0001 | ETA 08:31:44 2021-01-18 09:56:04 [INFO] [TRAIN] epoch=2, iter=810/40000, loss=1.2476, lr=0.009818, batch_cost=0.7931, reader_cost=0.0001 | ETA 08:38:01 2021-01-18 09:56:12 [INFO] [TRAIN] epoch=2, iter=820/40000, loss=0.7408, lr=0.009816, batch_cost=0.7903, reader_cost=0.0001 | ETA 08:36:05 2021-01-18 09:56:20 [INFO] [TRAIN] epoch=2, iter=830/40000, loss=0.9516, lr=0.009813, batch_cost=0.7932, reader_cost=0.0001 | ETA 08:37:51 2021-01-18 09:56:28 [INFO] [TRAIN] epoch=2, iter=840/40000, loss=0.6377, lr=0.009811, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:36:53 2021-01-18 09:56:36 [INFO] [TRAIN] epoch=2, iter=850/40000, loss=0.7450, lr=0.009809, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:36:12 2021-01-18 09:56:44 [INFO] [TRAIN] epoch=2, iter=860/40000, loss=0.7533, lr=0.009807, batch_cost=0.7865, reader_cost=0.0001 | ETA 08:33:03 2021-01-18 09:56:52 [INFO] [TRAIN] epoch=2, iter=870/40000, loss=0.9856, lr=0.009804, batch_cost=0.7903, reader_cost=0.0001 | ETA 08:35:22 2021-01-18 09:57:00 [INFO] [TRAIN] epoch=2, iter=880/40000, loss=0.7258, lr=0.009802, batch_cost=0.7868, reader_cost=0.0001 | ETA 08:33:00 2021-01-18 09:57:08 [INFO] [TRAIN] epoch=2, iter=890/40000, loss=0.8669, lr=0.009800, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:35:51 2021-01-18 09:57:16 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.8919, lr=0.009797, batch_cost=0.7962, reader_cost=0.0001 | ETA 08:38:51 2021-01-18 09:57:23 [INFO] [TRAIN] epoch=2, iter=910/40000, loss=0.9411, lr=0.009795, batch_cost=0.7884, reader_cost=0.0001 | ETA 08:33:38 2021-01-18 09:57:31 [INFO] [TRAIN] epoch=2, iter=920/40000, loss=0.8917, lr=0.009793, batch_cost=0.7912, reader_cost=0.0001 | ETA 08:35:21 2021-01-18 09:57:39 [INFO] [TRAIN] epoch=2, iter=930/40000, loss=0.5814, lr=0.009791, batch_cost=0.7872, reader_cost=0.0001 | ETA 08:32:35 2021-01-18 09:57:47 [INFO] [TRAIN] epoch=2, iter=940/40000, loss=0.9076, lr=0.009788, batch_cost=0.7877, reader_cost=0.0001 | ETA 08:32:48 2021-01-18 09:57:55 [INFO] [TRAIN] epoch=2, iter=950/40000, loss=0.9766, lr=0.009786, batch_cost=0.7926, reader_cost=0.0001 | ETA 08:35:49 2021-01-18 09:58:03 [INFO] [TRAIN] epoch=2, iter=960/40000, loss=0.9227, lr=0.009784, batch_cost=0.7889, reader_cost=0.0001 | ETA 08:33:17 2021-01-18 09:58:11 [INFO] [TRAIN] epoch=2, iter=970/40000, loss=0.9287, lr=0.009782, batch_cost=0.7864, reader_cost=0.0001 | ETA 08:31:34 2021-01-18 09:58:19 [INFO] [TRAIN] epoch=2, iter=980/40000, loss=0.6430, lr=0.009779, batch_cost=0.7908, reader_cost=0.0001 | ETA 08:34:15 2021-01-18 09:58:27 [INFO] [TRAIN] epoch=2, iter=990/40000, loss=1.1977, lr=0.009777, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:32:41 2021-01-18 09:58:34 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.9352, lr=0.009775, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:32:31 2021-01-18 09:58:35 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 09:59:03 [INFO] [EVAL] #Images=1449 mIoU=0.4955 Acc=0.8750 Kappa=0.7215 2021-01-18 09:59:03 [INFO] [EVAL] Class IoU: [0.8897 0.73 0.2548 0.5714 0.3896 0.4497 0.7565 0.6002 0.628 0.1283 0.2708 0.0278 0.5488 0.4582 0.5517 0.6303 0.3506 0.6217 0.3395 0.6712 0.5368] 2021-01-18 09:59:03 [INFO] [EVAL] Class Acc: [0.9353 0.8597 0.2695 0.8308 0.6394 0.797 0.946 0.7869 0.9388 0.6624 0.6103 0.9274 0.6827 0.4946 0.593 0.6787 0.702 0.8868 0.5505 0.8111 0.768 ] 2021-01-18 09:59:12 [INFO] [EVAL] The model with the best validation mIoU (0.4955) was saved at iter 1000. 2021-01-18 09:59:20 [INFO] [TRAIN] epoch=2, iter=1010/40000, loss=0.6977, lr=0.009773, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:32:03 2021-01-18 09:59:28 [INFO] [TRAIN] epoch=2, iter=1020/40000, loss=0.9659, lr=0.009770, batch_cost=0.7907, reader_cost=0.0001 | ETA 08:33:39 2021-01-18 09:59:36 [INFO] [TRAIN] epoch=2, iter=1030/40000, loss=0.9494, lr=0.009768, batch_cost=0.7872, reader_cost=0.0001 | ETA 08:31:17 2021-01-18 09:59:44 [INFO] [TRAIN] epoch=2, iter=1040/40000, loss=0.7351, lr=0.009766, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:33:42 2021-01-18 09:59:51 [INFO] [TRAIN] epoch=2, iter=1050/40000, loss=0.6934, lr=0.009764, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:34:39 2021-01-18 09:59:59 [INFO] [TRAIN] epoch=2, iter=1060/40000, loss=0.6774, lr=0.009761, batch_cost=0.7918, reader_cost=0.0001 | ETA 08:33:52 2021-01-18 10:00:07 [INFO] [TRAIN] epoch=2, iter=1070/40000, loss=0.7309, lr=0.009759, batch_cost=0.7951, reader_cost=0.0001 | ETA 08:35:54 2021-01-18 10:00:15 [INFO] [TRAIN] epoch=2, iter=1080/40000, loss=0.7979, lr=0.009757, batch_cost=0.7877, reader_cost=0.0001 | ETA 08:30:56 2021-01-18 10:00:23 [INFO] [TRAIN] epoch=2, iter=1090/40000, loss=0.9394, lr=0.009755, batch_cost=0.7958, reader_cost=0.0001 | ETA 08:36:04 2021-01-18 10:00:31 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=1.0140, lr=0.009752, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:34:00 2021-01-18 10:00:39 [INFO] [TRAIN] epoch=2, iter=1110/40000, loss=1.2224, lr=0.009750, batch_cost=0.7822, reader_cost=0.0001 | ETA 08:26:58 2021-01-18 10:00:47 [INFO] [TRAIN] epoch=2, iter=1120/40000, loss=0.8830, lr=0.009748, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:33:45 2021-01-18 10:00:55 [INFO] [TRAIN] epoch=2, iter=1130/40000, loss=0.9015, lr=0.009746, batch_cost=0.7939, reader_cost=0.0001 | ETA 08:34:18 2021-01-18 10:01:03 [INFO] [TRAIN] epoch=2, iter=1140/40000, loss=1.0554, lr=0.009743, batch_cost=0.7919, reader_cost=0.0001 | ETA 08:32:54 2021-01-18 10:01:11 [INFO] [TRAIN] epoch=2, iter=1150/40000, loss=1.0978, lr=0.009741, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:32:26 2021-01-18 10:01:19 [INFO] [TRAIN] epoch=2, iter=1160/40000, loss=0.7488, lr=0.009739, batch_cost=0.7922, reader_cost=0.0001 | ETA 08:32:49 2021-01-18 10:01:26 [INFO] [TRAIN] epoch=2, iter=1170/40000, loss=0.7414, lr=0.009737, batch_cost=0.7877, reader_cost=0.0001 | ETA 08:29:46 2021-01-18 10:01:34 [INFO] [TRAIN] epoch=2, iter=1180/40000, loss=0.7429, lr=0.009734, batch_cost=0.7915, reader_cost=0.0001 | ETA 08:32:05 2021-01-18 10:01:42 [INFO] [TRAIN] epoch=2, iter=1190/40000, loss=0.9920, lr=0.009732, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:30:01 2021-01-18 10:01:50 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.7754, lr=0.009730, batch_cost=0.7910, reader_cost=0.0001 | ETA 08:31:30 2021-01-18 10:01:58 [INFO] [TRAIN] epoch=2, iter=1210/40000, loss=0.7313, lr=0.009728, batch_cost=0.7891, reader_cost=0.0001 | ETA 08:30:09 2021-01-18 10:02:06 [INFO] [TRAIN] epoch=2, iter=1220/40000, loss=0.7055, lr=0.009725, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:29:37 2021-01-18 10:02:14 [INFO] [TRAIN] epoch=2, iter=1230/40000, loss=0.8610, lr=0.009723, batch_cost=0.7857, reader_cost=0.0001 | ETA 08:27:42 2021-01-18 10:02:22 [INFO] [TRAIN] epoch=2, iter=1240/40000, loss=0.8208, lr=0.009721, batch_cost=0.7906, reader_cost=0.0001 | ETA 08:30:42 2021-01-18 10:02:30 [INFO] [TRAIN] epoch=2, iter=1250/40000, loss=0.6910, lr=0.009719, batch_cost=0.7918, reader_cost=0.0001 | ETA 08:31:22 2021-01-18 10:02:37 [INFO] [TRAIN] epoch=2, iter=1260/40000, loss=0.9261, lr=0.009716, batch_cost=0.7881, reader_cost=0.0001 | ETA 08:28:50 2021-01-18 10:02:45 [INFO] [TRAIN] epoch=2, iter=1270/40000, loss=1.1104, lr=0.009714, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:30:03 2021-01-18 10:02:53 [INFO] [TRAIN] epoch=2, iter=1280/40000, loss=0.8702, lr=0.009712, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:32:10 2021-01-18 10:03:01 [INFO] [TRAIN] epoch=2, iter=1290/40000, loss=0.6743, lr=0.009710, batch_cost=0.7900, reader_cost=0.0001 | ETA 08:29:41 2021-01-18 10:03:09 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.7646, lr=0.009707, batch_cost=0.7905, reader_cost=0.0001 | ETA 08:29:50 2021-01-18 10:03:17 [INFO] [TRAIN] epoch=2, iter=1310/40000, loss=0.7307, lr=0.009705, batch_cost=0.7873, reader_cost=0.0001 | ETA 08:27:38 2021-01-18 10:03:25 [INFO] [TRAIN] epoch=2, iter=1320/40000, loss=0.6505, lr=0.009703, batch_cost=0.7711, reader_cost=0.0001 | ETA 08:17:05 2021-01-18 10:03:33 [INFO] [TRAIN] epoch=3, iter=1330/40000, loss=0.8619, lr=0.009700, batch_cost=0.8030, reader_cost=0.0154 | ETA 08:37:31 2021-01-18 10:03:41 [INFO] [TRAIN] epoch=3, iter=1340/40000, loss=0.9594, lr=0.009698, batch_cost=0.7901, reader_cost=0.0001 | ETA 08:29:06 2021-01-18 10:03:49 [INFO] [TRAIN] epoch=3, iter=1350/40000, loss=0.6593, lr=0.009696, batch_cost=0.7951, reader_cost=0.0001 | ETA 08:32:12 2021-01-18 10:03:56 [INFO] [TRAIN] epoch=3, iter=1360/40000, loss=0.5537, lr=0.009694, batch_cost=0.7849, reader_cost=0.0001 | ETA 08:25:27 2021-01-18 10:04:04 [INFO] [TRAIN] epoch=3, iter=1370/40000, loss=0.8565, lr=0.009691, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:29:54 2021-01-18 10:04:12 [INFO] [TRAIN] epoch=3, iter=1380/40000, loss=0.6199, lr=0.009689, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:29:46 2021-01-18 10:04:20 [INFO] [TRAIN] epoch=3, iter=1390/40000, loss=0.8331, lr=0.009687, batch_cost=0.7890, reader_cost=0.0001 | ETA 08:27:45 2021-01-18 10:04:28 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.9899, lr=0.009685, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:30:01 2021-01-18 10:04:36 [INFO] [TRAIN] epoch=3, iter=1410/40000, loss=0.6697, lr=0.009682, batch_cost=0.7950, reader_cost=0.0001 | ETA 08:31:20 2021-01-18 10:04:44 [INFO] [TRAIN] epoch=3, iter=1420/40000, loss=0.8223, lr=0.009680, batch_cost=0.7909, reader_cost=0.0001 | ETA 08:28:31 2021-01-18 10:04:52 [INFO] [TRAIN] epoch=3, iter=1430/40000, loss=0.6122, lr=0.009678, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:30:13 2021-01-18 10:05:00 [INFO] [TRAIN] epoch=3, iter=1440/40000, loss=0.6505, lr=0.009676, batch_cost=0.7899, reader_cost=0.0001 | ETA 08:27:39 2021-01-18 10:05:08 [INFO] [TRAIN] epoch=3, iter=1450/40000, loss=0.6143, lr=0.009673, batch_cost=0.7917, reader_cost=0.0001 | ETA 08:28:39 2021-01-18 10:05:16 [INFO] [TRAIN] epoch=3, iter=1460/40000, loss=0.6887, lr=0.009671, batch_cost=0.7878, reader_cost=0.0001 | ETA 08:26:02 2021-01-18 10:05:23 [INFO] [TRAIN] epoch=3, iter=1470/40000, loss=0.6045, lr=0.009669, batch_cost=0.7879, reader_cost=0.0001 | ETA 08:25:57 2021-01-18 10:05:31 [INFO] [TRAIN] epoch=3, iter=1480/40000, loss=0.8470, lr=0.009667, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:28:47 2021-01-18 10:05:39 [INFO] [TRAIN] epoch=3, iter=1490/40000, loss=0.7769, lr=0.009664, batch_cost=0.7876, reader_cost=0.0001 | ETA 08:25:31 2021-01-18 10:05:47 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.9791, lr=0.009662, batch_cost=0.7921, reader_cost=0.0001 | ETA 08:28:15 2021-01-18 10:05:55 [INFO] [TRAIN] epoch=3, iter=1510/40000, loss=0.7970, lr=0.009660, batch_cost=0.7946, reader_cost=0.0001 | ETA 08:29:44 2021-01-18 10:06:03 [INFO] [TRAIN] epoch=3, iter=1520/40000, loss=0.7020, lr=0.009658, batch_cost=0.7929, reader_cost=0.0001 | ETA 08:28:29 2021-01-18 10:06:11 [INFO] [TRAIN] epoch=3, iter=1530/40000, loss=0.7745, lr=0.009655, batch_cost=0.7935, reader_cost=0.0001 | ETA 08:28:46 2021-01-18 10:06:19 [INFO] [TRAIN] epoch=3, iter=1540/40000, loss=1.3032, lr=0.009653, batch_cost=0.7915, reader_cost=0.0001 | ETA 08:27:22 2021-01-18 10:06:27 [INFO] [TRAIN] epoch=3, iter=1550/40000, loss=0.8155, lr=0.009651, batch_cost=0.7910, reader_cost=0.0001 | ETA 08:26:54 2021-01-18 10:06:35 [INFO] [TRAIN] epoch=3, iter=1560/40000, loss=0.7634, lr=0.009649, batch_cost=0.7899, reader_cost=0.0001 | ETA 08:26:02 2021-01-18 10:06:43 [INFO] [TRAIN] epoch=3, iter=1570/40000, loss=0.7284, lr=0.009646, batch_cost=0.7883, reader_cost=0.0001 | ETA 08:24:53 2021-01-18 10:06:51 [INFO] [TRAIN] epoch=3, iter=1580/40000, loss=0.9961, lr=0.009644, batch_cost=0.7912, reader_cost=0.0001 | ETA 08:26:38 2021-01-18 10:06:58 [INFO] [TRAIN] epoch=3, iter=1590/40000, loss=0.5914, lr=0.009642, batch_cost=0.7830, reader_cost=0.0001 | ETA 08:21:16 2021-01-18 10:07:06 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.7227, lr=0.009639, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:26:51 2021-01-18 10:07:14 [INFO] [TRAIN] epoch=3, iter=1610/40000, loss=0.6959, lr=0.009637, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:25:35 2021-01-18 10:07:22 [INFO] [TRAIN] epoch=3, iter=1620/40000, loss=0.7954, lr=0.009635, batch_cost=0.7883, reader_cost=0.0001 | ETA 08:24:13 2021-01-18 10:07:30 [INFO] [TRAIN] epoch=3, iter=1630/40000, loss=0.7689, lr=0.009633, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:25:06 2021-01-18 10:07:38 [INFO] [TRAIN] epoch=3, iter=1640/40000, loss=0.8627, lr=0.009630, batch_cost=0.7923, reader_cost=0.0001 | ETA 08:26:31 2021-01-18 10:07:46 [INFO] [TRAIN] epoch=3, iter=1650/40000, loss=0.8513, lr=0.009628, batch_cost=0.7938, reader_cost=0.0001 | ETA 08:27:21 2021-01-18 10:07:54 [INFO] [TRAIN] epoch=3, iter=1660/40000, loss=0.6669, lr=0.009626, batch_cost=0.7934, reader_cost=0.0001 | ETA 08:26:58 2021-01-18 10:08:02 [INFO] [TRAIN] epoch=3, iter=1670/40000, loss=0.7161, lr=0.009624, batch_cost=0.7860, reader_cost=0.0001 | ETA 08:22:08 2021-01-18 10:08:10 [INFO] [TRAIN] epoch=3, iter=1680/40000, loss=0.9345, lr=0.009621, batch_cost=0.7975, reader_cost=0.0001 | ETA 08:29:21 2021-01-18 10:08:17 [INFO] [TRAIN] epoch=3, iter=1690/40000, loss=0.8524, lr=0.009619, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:25:58 2021-01-18 10:08:25 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.7616, lr=0.009617, batch_cost=0.7875, reader_cost=0.0001 | ETA 08:22:41 2021-01-18 10:08:33 [INFO] [TRAIN] epoch=3, iter=1710/40000, loss=0.8256, lr=0.009615, batch_cost=0.7901, reader_cost=0.0001 | ETA 08:24:11 2021-01-18 10:08:41 [INFO] [TRAIN] epoch=3, iter=1720/40000, loss=0.7558, lr=0.009612, batch_cost=0.7939, reader_cost=0.0001 | ETA 08:26:28 2021-01-18 10:08:49 [INFO] [TRAIN] epoch=3, iter=1730/40000, loss=0.7625, lr=0.009610, batch_cost=0.7910, reader_cost=0.0001 | ETA 08:24:32 2021-01-18 10:08:57 [INFO] [TRAIN] epoch=3, iter=1740/40000, loss=0.8514, lr=0.009608, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:23:52 2021-01-18 10:09:05 [INFO] [TRAIN] epoch=3, iter=1750/40000, loss=0.6879, lr=0.009606, batch_cost=0.7939, reader_cost=0.0001 | ETA 08:26:07 2021-01-18 10:09:13 [INFO] [TRAIN] epoch=3, iter=1760/40000, loss=0.7276, lr=0.009603, batch_cost=0.7949, reader_cost=0.0001 | ETA 08:26:35 2021-01-18 10:09:21 [INFO] [TRAIN] epoch=3, iter=1770/40000, loss=0.6679, lr=0.009601, batch_cost=0.7923, reader_cost=0.0001 | ETA 08:24:49 2021-01-18 10:09:29 [INFO] [TRAIN] epoch=3, iter=1780/40000, loss=0.8578, lr=0.009599, batch_cost=0.7935, reader_cost=0.0001 | ETA 08:25:28 2021-01-18 10:09:37 [INFO] [TRAIN] epoch=3, iter=1790/40000, loss=0.7499, lr=0.009597, batch_cost=0.7897, reader_cost=0.0001 | ETA 08:22:55 2021-01-18 10:09:45 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.4942, lr=0.009594, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:22:50 2021-01-18 10:09:52 [INFO] [TRAIN] epoch=3, iter=1810/40000, loss=0.6386, lr=0.009592, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:23:42 2021-01-18 10:10:00 [INFO] [TRAIN] epoch=3, iter=1820/40000, loss=0.9947, lr=0.009590, batch_cost=0.7864, reader_cost=0.0001 | ETA 08:20:23 2021-01-18 10:10:08 [INFO] [TRAIN] epoch=3, iter=1830/40000, loss=0.6883, lr=0.009588, batch_cost=0.7927, reader_cost=0.0001 | ETA 08:24:17 2021-01-18 10:10:16 [INFO] [TRAIN] epoch=3, iter=1840/40000, loss=0.7105, lr=0.009585, batch_cost=0.7865, reader_cost=0.0001 | ETA 08:20:14 2021-01-18 10:10:24 [INFO] [TRAIN] epoch=3, iter=1850/40000, loss=0.7746, lr=0.009583, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:23:12 2021-01-18 10:10:32 [INFO] [TRAIN] epoch=3, iter=1860/40000, loss=0.8433, lr=0.009581, batch_cost=0.7926, reader_cost=0.0001 | ETA 08:23:51 2021-01-18 10:10:40 [INFO] [TRAIN] epoch=3, iter=1870/40000, loss=0.6504, lr=0.009578, batch_cost=0.7912, reader_cost=0.0001 | ETA 08:22:48 2021-01-18 10:10:48 [INFO] [TRAIN] epoch=3, iter=1880/40000, loss=0.6792, lr=0.009576, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:24:16 2021-01-18 10:10:56 [INFO] [TRAIN] epoch=3, iter=1890/40000, loss=0.4485, lr=0.009574, batch_cost=0.7868, reader_cost=0.0001 | ETA 08:19:45 2021-01-18 10:11:04 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.9820, lr=0.009572, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:23:24 2021-01-18 10:11:12 [INFO] [TRAIN] epoch=3, iter=1910/40000, loss=0.7374, lr=0.009569, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:20:14 2021-01-18 10:11:19 [INFO] [TRAIN] epoch=3, iter=1920/40000, loss=0.7942, lr=0.009567, batch_cost=0.7897, reader_cost=0.0001 | ETA 08:21:12 2021-01-18 10:11:27 [INFO] [TRAIN] epoch=3, iter=1930/40000, loss=0.6656, lr=0.009565, batch_cost=0.7947, reader_cost=0.0001 | ETA 08:24:15 2021-01-18 10:11:35 [INFO] [TRAIN] epoch=3, iter=1940/40000, loss=0.8440, lr=0.009563, batch_cost=0.7922, reader_cost=0.0001 | ETA 08:22:29 2021-01-18 10:11:43 [INFO] [TRAIN] epoch=3, iter=1950/40000, loss=0.6010, lr=0.009560, batch_cost=0.7895, reader_cost=0.0001 | ETA 08:20:39 2021-01-18 10:11:51 [INFO] [TRAIN] epoch=3, iter=1960/40000, loss=0.5741, lr=0.009558, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:20:58 2021-01-18 10:11:59 [INFO] [TRAIN] epoch=3, iter=1970/40000, loss=0.8624, lr=0.009556, batch_cost=0.7950, reader_cost=0.0001 | ETA 08:23:54 2021-01-18 10:12:07 [INFO] [TRAIN] epoch=3, iter=1980/40000, loss=0.4165, lr=0.009554, batch_cost=0.7722, reader_cost=0.0001 | ETA 08:09:20 2021-01-18 10:12:15 [INFO] [TRAIN] epoch=4, iter=1990/40000, loss=0.7319, lr=0.009551, batch_cost=0.7918, reader_cost=0.0169 | ETA 08:21:35 2021-01-18 10:12:23 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.7703, lr=0.009549, batch_cost=0.7939, reader_cost=0.0001 | ETA 08:22:46 2021-01-18 10:12:23 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 10:12:51 [INFO] [EVAL] #Images=1449 mIoU=0.5967 Acc=0.9024 Kappa=0.7915 2021-01-18 10:12:51 [INFO] [EVAL] Class IoU: [0.9042 0.7015 0.3211 0.6571 0.4926 0.6239 0.8299 0.6749 0.8215 0.2995 0.5759 0.4953 0.7302 0.5388 0.5628 0.7768 0.3924 0.6043 0.3629 0.6465 0.5196] 2021-01-18 10:12:51 [INFO] [EVAL] Class Acc: [0.9602 0.7359 0.3344 0.6835 0.668 0.8166 0.9199 0.7213 0.9391 0.4953 0.7807 0.7401 0.8344 0.7224 0.5876 0.8464 0.7702 0.7463 0.7534 0.6996 0.577 ] 2021-01-18 10:13:00 [INFO] [EVAL] The model with the best validation mIoU (0.5967) was saved at iter 2000. 2021-01-18 10:13:08 [INFO] [TRAIN] epoch=4, iter=2010/40000, loss=0.4858, lr=0.009547, batch_cost=0.7839, reader_cost=0.0001 | ETA 08:16:20 2021-01-18 10:13:16 [INFO] [TRAIN] epoch=4, iter=2020/40000, loss=0.7593, lr=0.009545, batch_cost=0.7881, reader_cost=0.0001 | ETA 08:18:53 2021-01-18 10:13:24 [INFO] [TRAIN] epoch=4, iter=2030/40000, loss=0.5601, lr=0.009542, batch_cost=0.7860, reader_cost=0.0001 | ETA 08:17:25 2021-01-18 10:13:32 [INFO] [TRAIN] epoch=4, iter=2040/40000, loss=0.8078, lr=0.009540, batch_cost=0.7934, reader_cost=0.0001 | ETA 08:21:56 2021-01-18 10:13:40 [INFO] [TRAIN] epoch=4, iter=2050/40000, loss=0.6656, lr=0.009538, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:21:16 2021-01-18 10:13:48 [INFO] [TRAIN] epoch=4, iter=2060/40000, loss=0.7412, lr=0.009536, batch_cost=0.7977, reader_cost=0.0001 | ETA 08:24:24 2021-01-18 10:13:56 [INFO] [TRAIN] epoch=4, iter=2070/40000, loss=0.8239, lr=0.009533, batch_cost=0.7954, reader_cost=0.0001 | ETA 08:22:48 2021-01-18 10:14:04 [INFO] [TRAIN] epoch=4, iter=2080/40000, loss=0.6457, lr=0.009531, batch_cost=0.7963, reader_cost=0.0001 | ETA 08:23:16 2021-01-18 10:14:11 [INFO] [TRAIN] epoch=4, iter=2090/40000, loss=0.5591, lr=0.009529, batch_cost=0.7924, reader_cost=0.0001 | ETA 08:20:39 2021-01-18 10:14:19 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.6731, lr=0.009526, batch_cost=0.7975, reader_cost=0.0001 | ETA 08:23:43 2021-01-18 10:14:27 [INFO] [TRAIN] epoch=4, iter=2110/40000, loss=0.6829, lr=0.009524, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:17:38 2021-01-18 10:14:35 [INFO] [TRAIN] epoch=4, iter=2120/40000, loss=0.7957, lr=0.009522, batch_cost=0.7886, reader_cost=0.0001 | ETA 08:17:52 2021-01-18 10:14:43 [INFO] [TRAIN] epoch=4, iter=2130/40000, loss=0.5451, lr=0.009520, batch_cost=0.7922, reader_cost=0.0001 | ETA 08:20:00 2021-01-18 10:14:51 [INFO] [TRAIN] epoch=4, iter=2140/40000, loss=0.6859, lr=0.009517, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:20:05 2021-01-18 10:14:59 [INFO] [TRAIN] epoch=4, iter=2150/40000, loss=0.6432, lr=0.009515, batch_cost=0.7900, reader_cost=0.0001 | ETA 08:18:22 2021-01-18 10:15:07 [INFO] [TRAIN] epoch=4, iter=2160/40000, loss=0.7010, lr=0.009513, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:19:05 2021-01-18 10:15:15 [INFO] [TRAIN] epoch=4, iter=2170/40000, loss=0.5520, lr=0.009511, batch_cost=0.7941, reader_cost=0.0001 | ETA 08:20:41 2021-01-18 10:15:23 [INFO] [TRAIN] epoch=4, iter=2180/40000, loss=0.6976, lr=0.009508, batch_cost=0.7910, reader_cost=0.0001 | ETA 08:18:35 2021-01-18 10:15:31 [INFO] [TRAIN] epoch=4, iter=2190/40000, loss=0.7477, lr=0.009506, batch_cost=0.7877, reader_cost=0.0001 | ETA 08:16:21 2021-01-18 10:15:38 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.5787, lr=0.009504, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:16:24 2021-01-18 10:15:46 [INFO] [TRAIN] epoch=4, iter=2210/40000, loss=0.5864, lr=0.009502, batch_cost=0.7969, reader_cost=0.0001 | ETA 08:21:55 2021-01-18 10:15:54 [INFO] [TRAIN] epoch=4, iter=2220/40000, loss=0.6975, lr=0.009499, batch_cost=0.7900, reader_cost=0.0001 | ETA 08:17:26 2021-01-18 10:16:02 [INFO] [TRAIN] epoch=4, iter=2230/40000, loss=0.7015, lr=0.009497, batch_cost=0.7916, reader_cost=0.0001 | ETA 08:18:19 2021-01-18 10:16:10 [INFO] [TRAIN] epoch=4, iter=2240/40000, loss=0.7104, lr=0.009495, batch_cost=0.7891, reader_cost=0.0001 | ETA 08:16:34 2021-01-18 10:16:18 [INFO] [TRAIN] epoch=4, iter=2250/40000, loss=0.6424, lr=0.009493, batch_cost=0.7993, reader_cost=0.0001 | ETA 08:22:54 2021-01-18 10:16:26 [INFO] [TRAIN] epoch=4, iter=2260/40000, loss=0.4883, lr=0.009490, batch_cost=0.7945, reader_cost=0.0001 | ETA 08:19:45 2021-01-18 10:16:34 [INFO] [TRAIN] epoch=4, iter=2270/40000, loss=0.6121, lr=0.009488, batch_cost=0.7938, reader_cost=0.0001 | ETA 08:19:11 2021-01-18 10:16:42 [INFO] [TRAIN] epoch=4, iter=2280/40000, loss=0.9542, lr=0.009486, batch_cost=0.7886, reader_cost=0.0001 | ETA 08:15:44 2021-01-18 10:16:50 [INFO] [TRAIN] epoch=4, iter=2290/40000, loss=0.5885, lr=0.009483, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:18:05 2021-01-18 10:16:58 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.6807, lr=0.009481, batch_cost=0.7922, reader_cost=0.0001 | ETA 08:17:46 2021-01-18 10:17:06 [INFO] [TRAIN] epoch=4, iter=2310/40000, loss=0.6470, lr=0.009479, batch_cost=0.7871, reader_cost=0.0001 | ETA 08:14:24 2021-01-18 10:17:14 [INFO] [TRAIN] epoch=4, iter=2320/40000, loss=0.7377, lr=0.009477, batch_cost=0.7943, reader_cost=0.0001 | ETA 08:18:48 2021-01-18 10:17:22 [INFO] [TRAIN] epoch=4, iter=2330/40000, loss=0.8298, lr=0.009474, batch_cost=0.7943, reader_cost=0.0001 | ETA 08:18:42 2021-01-18 10:17:29 [INFO] [TRAIN] epoch=4, iter=2340/40000, loss=1.0527, lr=0.009472, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:16:32 2021-01-18 10:17:37 [INFO] [TRAIN] epoch=4, iter=2350/40000, loss=0.8452, lr=0.009470, batch_cost=0.7917, reader_cost=0.0001 | ETA 08:16:47 2021-01-18 10:17:45 [INFO] [TRAIN] epoch=4, iter=2360/40000, loss=0.4795, lr=0.009468, batch_cost=0.7908, reader_cost=0.0001 | ETA 08:16:04 2021-01-18 10:17:53 [INFO] [TRAIN] epoch=4, iter=2370/40000, loss=0.6673, lr=0.009465, batch_cost=0.7886, reader_cost=0.0001 | ETA 08:14:36 2021-01-18 10:18:01 [INFO] [TRAIN] epoch=4, iter=2380/40000, loss=0.6664, lr=0.009463, batch_cost=0.7956, reader_cost=0.0001 | ETA 08:18:52 2021-01-18 10:18:09 [INFO] [TRAIN] epoch=4, iter=2390/40000, loss=0.6290, lr=0.009461, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:17:31 2021-01-18 10:18:17 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.7328, lr=0.009459, batch_cost=0.7912, reader_cost=0.0001 | ETA 08:15:48 2021-01-18 10:18:25 [INFO] [TRAIN] epoch=4, iter=2410/40000, loss=0.6292, lr=0.009456, batch_cost=0.7868, reader_cost=0.0001 | ETA 08:12:55 2021-01-18 10:18:33 [INFO] [TRAIN] epoch=4, iter=2420/40000, loss=0.7955, lr=0.009454, batch_cost=0.7907, reader_cost=0.0001 | ETA 08:15:15 2021-01-18 10:18:41 [INFO] [TRAIN] epoch=4, iter=2430/40000, loss=0.6970, lr=0.009452, batch_cost=0.7905, reader_cost=0.0001 | ETA 08:14:57 2021-01-18 10:18:49 [INFO] [TRAIN] epoch=4, iter=2440/40000, loss=0.4886, lr=0.009450, batch_cost=0.7951, reader_cost=0.0001 | ETA 08:17:45 2021-01-18 10:18:57 [INFO] [TRAIN] epoch=4, iter=2450/40000, loss=0.6316, lr=0.009447, batch_cost=0.7968, reader_cost=0.0001 | ETA 08:18:39 2021-01-18 10:19:04 [INFO] [TRAIN] epoch=4, iter=2460/40000, loss=0.7723, lr=0.009445, batch_cost=0.7882, reader_cost=0.0001 | ETA 08:13:07 2021-01-18 10:19:12 [INFO] [TRAIN] epoch=4, iter=2470/40000, loss=0.6147, lr=0.009443, batch_cost=0.7931, reader_cost=0.0001 | ETA 08:16:04 2021-01-18 10:19:20 [INFO] [TRAIN] epoch=4, iter=2480/40000, loss=1.0086, lr=0.009440, batch_cost=0.7880, reader_cost=0.0001 | ETA 08:12:46 2021-01-18 10:19:28 [INFO] [TRAIN] epoch=4, iter=2490/40000, loss=0.8841, lr=0.009438, batch_cost=0.7877, reader_cost=0.0001 | ETA 08:12:24 2021-01-18 10:19:36 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.7920, lr=0.009436, batch_cost=0.7927, reader_cost=0.0001 | ETA 08:15:27 2021-01-18 10:19:44 [INFO] [TRAIN] epoch=4, iter=2510/40000, loss=0.6065, lr=0.009434, batch_cost=0.7918, reader_cost=0.0001 | ETA 08:14:45 2021-01-18 10:19:52 [INFO] [TRAIN] epoch=4, iter=2520/40000, loss=0.7224, lr=0.009431, batch_cost=0.7942, reader_cost=0.0001 | ETA 08:16:04 2021-01-18 10:20:00 [INFO] [TRAIN] epoch=4, iter=2530/40000, loss=0.7701, lr=0.009429, batch_cost=0.7883, reader_cost=0.0001 | ETA 08:12:17 2021-01-18 10:20:08 [INFO] [TRAIN] epoch=4, iter=2540/40000, loss=0.7329, lr=0.009427, batch_cost=0.7899, reader_cost=0.0001 | ETA 08:13:09 2021-01-18 10:20:16 [INFO] [TRAIN] epoch=4, iter=2550/40000, loss=0.6830, lr=0.009425, batch_cost=0.7848, reader_cost=0.0001 | ETA 08:09:49 2021-01-18 10:20:23 [INFO] [TRAIN] epoch=4, iter=2560/40000, loss=0.4814, lr=0.009422, batch_cost=0.7895, reader_cost=0.0001 | ETA 08:12:39 2021-01-18 10:20:31 [INFO] [TRAIN] epoch=4, iter=2570/40000, loss=0.7066, lr=0.009420, batch_cost=0.7884, reader_cost=0.0001 | ETA 08:11:49 2021-01-18 10:20:39 [INFO] [TRAIN] epoch=4, iter=2580/40000, loss=0.8120, lr=0.009418, batch_cost=0.7910, reader_cost=0.0001 | ETA 08:13:19 2021-01-18 10:20:47 [INFO] [TRAIN] epoch=4, iter=2590/40000, loss=0.9958, lr=0.009416, batch_cost=0.7914, reader_cost=0.0001 | ETA 08:13:27 2021-01-18 10:20:55 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=1.0509, lr=0.009413, batch_cost=0.7936, reader_cost=0.0001 | ETA 08:14:41 2021-01-18 10:21:03 [INFO] [TRAIN] epoch=4, iter=2610/40000, loss=0.5533, lr=0.009411, batch_cost=0.7899, reader_cost=0.0001 | ETA 08:12:15 2021-01-18 10:21:11 [INFO] [TRAIN] epoch=4, iter=2620/40000, loss=0.6208, lr=0.009409, batch_cost=0.7854, reader_cost=0.0001 | ETA 08:09:19 2021-01-18 10:21:19 [INFO] [TRAIN] epoch=4, iter=2630/40000, loss=0.5184, lr=0.009406, batch_cost=0.7945, reader_cost=0.0001 | ETA 08:14:50 2021-01-18 10:21:27 [INFO] [TRAIN] epoch=4, iter=2640/40000, loss=0.6017, lr=0.009404, batch_cost=0.7874, reader_cost=0.0001 | ETA 08:10:17 2021-01-18 10:21:35 [INFO] [TRAIN] epoch=5, iter=2650/40000, loss=0.5766, lr=0.009402, batch_cost=0.7855, reader_cost=0.0115 | ETA 08:08:58 2021-01-18 10:21:42 [INFO] [TRAIN] epoch=5, iter=2660/40000, loss=0.6453, lr=0.009400, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:10:43 2021-01-18 10:21:50 [INFO] [TRAIN] epoch=5, iter=2670/40000, loss=0.7041, lr=0.009397, batch_cost=0.7898, reader_cost=0.0001 | ETA 08:11:24 2021-01-18 10:21:58 [INFO] [TRAIN] epoch=5, iter=2680/40000, loss=0.5628, lr=0.009395, batch_cost=0.7955, reader_cost=0.0001 | ETA 08:14:49 2021-01-18 10:22:06 [INFO] [TRAIN] epoch=5, iter=2690/40000, loss=0.7371, lr=0.009393, batch_cost=0.7871, reader_cost=0.0001 | ETA 08:09:25 2021-01-18 10:22:14 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.8013, lr=0.009391, batch_cost=0.7902, reader_cost=0.0001 | ETA 08:11:14 2021-01-18 10:22:22 [INFO] [TRAIN] epoch=5, iter=2710/40000, loss=0.8002, lr=0.009388, batch_cost=0.7886, reader_cost=0.0001 | ETA 08:10:06 2021-01-18 10:22:30 [INFO] [TRAIN] epoch=5, iter=2720/40000, loss=0.6923, lr=0.009386, batch_cost=0.7916, reader_cost=0.0001 | ETA 08:11:51 2021-01-18 10:22:38 [INFO] [TRAIN] epoch=5, iter=2730/40000, loss=0.7605, lr=0.009384, batch_cost=0.7878, reader_cost=0.0001 | ETA 08:09:21 2021-01-18 10:22:46 [INFO] [TRAIN] epoch=5, iter=2740/40000, loss=0.5318, lr=0.009382, batch_cost=0.7946, reader_cost=0.0001 | ETA 08:13:27 2021-01-18 10:22:54 [INFO] [TRAIN] epoch=5, iter=2750/40000, loss=0.6066, lr=0.009379, batch_cost=0.7864, reader_cost=0.0001 | ETA 08:08:15 2021-01-18 10:23:02 [INFO] [TRAIN] epoch=5, iter=2760/40000, loss=0.6303, lr=0.009377, batch_cost=0.7956, reader_cost=0.0001 | ETA 08:13:47 2021-01-18 10:23:09 [INFO] [TRAIN] epoch=5, iter=2770/40000, loss=0.4700, lr=0.009375, batch_cost=0.7896, reader_cost=0.0001 | ETA 08:09:58 2021-01-18 10:23:17 [INFO] [TRAIN] epoch=5, iter=2780/40000, loss=0.5203, lr=0.009372, batch_cost=0.7960, reader_cost=0.0001 | ETA 08:13:48 2021-01-18 10:23:25 [INFO] [TRAIN] epoch=5, iter=2790/40000, loss=0.6149, lr=0.009370, batch_cost=0.7883, reader_cost=0.0001 | ETA 08:08:54 2021-01-18 10:23:33 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.7333, lr=0.009368, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:12:03 2021-01-18 10:23:41 [INFO] [TRAIN] epoch=5, iter=2810/40000, loss=0.6364, lr=0.009366, batch_cost=0.7882, reader_cost=0.0001 | ETA 08:08:34 2021-01-18 10:23:49 [INFO] [TRAIN] epoch=5, iter=2820/40000, loss=0.5993, lr=0.009363, batch_cost=0.7904, reader_cost=0.0001 | ETA 08:09:48 2021-01-18 10:23:57 [INFO] [TRAIN] epoch=5, iter=2830/40000, loss=0.7840, lr=0.009361, batch_cost=0.7950, reader_cost=0.0001 | ETA 08:12:31 2021-01-18 10:24:05 [INFO] [TRAIN] epoch=5, iter=2840/40000, loss=0.4683, lr=0.009359, batch_cost=0.7908, reader_cost=0.0001 | ETA 08:09:46 2021-01-18 10:24:13 [INFO] [TRAIN] epoch=5, iter=2850/40000, loss=0.6822, lr=0.009357, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:11:25 2021-01-18 10:24:21 [INFO] [TRAIN] epoch=5, iter=2860/40000, loss=0.6687, lr=0.009354, batch_cost=0.7911, reader_cost=0.0001 | ETA 08:09:42 2021-01-18 10:24:29 [INFO] [TRAIN] epoch=5, iter=2870/40000, loss=0.8636, lr=0.009352, batch_cost=0.7924, reader_cost=0.0001 | ETA 08:10:20 2021-01-18 10:24:37 [INFO] [TRAIN] epoch=5, iter=2880/40000, loss=0.6531, lr=0.009350, batch_cost=0.7951, reader_cost=0.0001 | ETA 08:11:55 2021-01-18 10:24:44 [INFO] [TRAIN] epoch=5, iter=2890/40000, loss=0.4815, lr=0.009348, batch_cost=0.7921, reader_cost=0.0001 | ETA 08:09:54 2021-01-18 10:24:52 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.5560, lr=0.009345, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:09:42 2021-01-18 10:25:00 [INFO] [TRAIN] epoch=5, iter=2910/40000, loss=0.4690, lr=0.009343, batch_cost=0.7846, reader_cost=0.0001 | ETA 08:05:00 2021-01-18 10:25:08 [INFO] [TRAIN] epoch=5, iter=2920/40000, loss=0.4928, lr=0.009341, batch_cost=0.7888, reader_cost=0.0001 | ETA 08:07:27 2021-01-18 10:25:16 [INFO] [TRAIN] epoch=5, iter=2930/40000, loss=0.5187, lr=0.009338, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:09:50 2021-01-18 10:25:24 [INFO] [TRAIN] epoch=5, iter=2940/40000, loss=0.5953, lr=0.009336, batch_cost=0.7937, reader_cost=0.0001 | ETA 08:10:16 2021-01-18 10:25:32 [INFO] [TRAIN] epoch=5, iter=2950/40000, loss=0.5862, lr=0.009334, batch_cost=0.7917, reader_cost=0.0001 | ETA 08:08:51 2021-01-18 10:25:40 [INFO] [TRAIN] epoch=5, iter=2960/40000, loss=0.5714, lr=0.009332, batch_cost=0.7920, reader_cost=0.0001 | ETA 08:08:57 2021-01-18 10:25:48 [INFO] [TRAIN] epoch=5, iter=2970/40000, loss=0.6059, lr=0.009329, batch_cost=0.7885, reader_cost=0.0001 | ETA 08:06:36 2021-01-18 10:25:56 [INFO] [TRAIN] epoch=5, iter=2980/40000, loss=0.5918, lr=0.009327, batch_cost=0.7899, reader_cost=0.0001 | ETA 08:07:22 2021-01-18 10:26:04 [INFO] [TRAIN] epoch=5, iter=2990/40000, loss=0.5635, lr=0.009325, batch_cost=0.7928, reader_cost=0.0001 | ETA 08:09:02 2021-01-18 10:26:11 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.7366, lr=0.009323, batch_cost=0.7839, reader_cost=0.0001 | ETA 08:03:22 2021-01-18 10:26:11 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 10:26:40 [INFO] [EVAL] #Images=1449 mIoU=0.5932 Acc=0.9082 Kappa=0.7906 2021-01-18 10:26:40 [INFO] [EVAL] Class IoU: [0.9115 0.8505 0.3781 0.759 0.5342 0.4009 0.7463 0.6951 0.7367 0.2738 0.6868 0.1113 0.6719 0.6953 0.6404 0.7759 0.4061 0.6376 0.2776 0.7006 0.5681] 2021-01-18 10:26:40 [INFO] [EVAL] Class Acc: [0.9376 0.9088 0.4309 0.8395 0.6148 0.9285 0.8347 0.8329 0.7656 0.5499 0.7653 0.9929 0.8394 0.852 0.7694 0.9168 0.6031 0.7964 0.8173 0.872 0.8676] 2021-01-18 10:26:46 [INFO] [EVAL] The model with the best validation mIoU (0.5967) was saved at iter 2000. 2021-01-18 10:26:54 [INFO] [TRAIN] epoch=5, iter=3010/40000, loss=0.4802, lr=0.009320, batch_cost=0.7832, reader_cost=0.0001 | ETA 08:02:48 2021-01-18 10:27:02 [INFO] [TRAIN] epoch=5, iter=3020/40000, loss=0.5780, lr=0.009318, batch_cost=0.7826, reader_cost=0.0001 | ETA 08:02:18 2021-01-18 10:27:09 [INFO] [TRAIN] epoch=5, iter=3030/40000, loss=0.4165, lr=0.009316, batch_cost=0.7878, reader_cost=0.0001 | ETA 08:05:25 2021-01-18 10:27:17 [INFO] [TRAIN] epoch=5, iter=3040/40000, loss=1.0786, lr=0.009314, batch_cost=0.7932, reader_cost=0.0001 | ETA 08:08:37 2021-01-18 10:27:25 [INFO] [TRAIN] epoch=5, iter=3050/40000, loss=0.4831, lr=0.009311, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:08:02 2021-01-18 10:27:33 [INFO] [TRAIN] epoch=5, iter=3060/40000, loss=0.7322, lr=0.009309, batch_cost=0.7876, reader_cost=0.0001 | ETA 08:04:52 2021-01-18 10:27:41 [INFO] [TRAIN] epoch=5, iter=3070/40000, loss=0.6176, lr=0.009307, batch_cost=0.7931, reader_cost=0.0001 | ETA 08:08:08 2021-01-18 10:27:49 [INFO] [TRAIN] epoch=5, iter=3080/40000, loss=0.4572, lr=0.009304, batch_cost=0.7888, reader_cost=0.0001 | ETA 08:05:21 2021-01-18 10:27:57 [INFO] [TRAIN] epoch=5, iter=3090/40000, loss=0.4985, lr=0.009302, batch_cost=0.7862, reader_cost=0.0001 | ETA 08:03:39 2021-01-18 10:28:05 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.7612, lr=0.009300, batch_cost=0.7945, reader_cost=0.0001 | ETA 08:08:36 2021-01-18 10:28:13 [INFO] [TRAIN] epoch=5, iter=3110/40000, loss=0.5329, lr=0.009298, batch_cost=0.7886, reader_cost=0.0001 | ETA 08:04:52 2021-01-18 10:28:21 [INFO] [TRAIN] epoch=5, iter=3120/40000, loss=0.5216, lr=0.009295, batch_cost=0.7917, reader_cost=0.0001 | ETA 08:06:36 2021-01-18 10:28:29 [INFO] [TRAIN] epoch=5, iter=3130/40000, loss=0.7712, lr=0.009293, batch_cost=0.7950, reader_cost=0.0001 | ETA 08:08:31 2021-01-18 10:28:36 [INFO] [TRAIN] epoch=5, iter=3140/40000, loss=0.7265, lr=0.009291, batch_cost=0.7879, reader_cost=0.0001 | ETA 08:04:00 2021-01-18 10:28:44 [INFO] [TRAIN] epoch=5, iter=3150/40000, loss=0.6623, lr=0.009289, batch_cost=0.7889, reader_cost=0.0001 | ETA 08:04:30 2021-01-18 10:28:52 [INFO] [TRAIN] epoch=5, iter=3160/40000, loss=0.6645, lr=0.009286, batch_cost=0.7889, reader_cost=0.0001 | ETA 08:04:23 2021-01-18 10:29:00 [INFO] [TRAIN] epoch=5, iter=3170/40000, loss=0.6203, lr=0.009284, batch_cost=0.7896, reader_cost=0.0001 | ETA 08:04:40 2021-01-18 10:29:08 [INFO] [TRAIN] epoch=5, iter=3180/40000, loss=0.6186, lr=0.009282, batch_cost=0.7881, reader_cost=0.0001 | ETA 08:03:38 2021-01-18 10:29:16 [INFO] [TRAIN] epoch=5, iter=3190/40000, loss=0.7065, lr=0.009280, batch_cost=0.7923, reader_cost=0.0001 | ETA 08:06:02 2021-01-18 10:29:24 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.5961, lr=0.009277, batch_cost=0.7863, reader_cost=0.0001 | ETA 08:02:14 2021-01-18 10:29:32 [INFO] [TRAIN] epoch=5, iter=3210/40000, loss=0.7619, lr=0.009275, batch_cost=0.7882, reader_cost=0.0001 | ETA 08:03:19 2021-01-18 10:29:40 [INFO] [TRAIN] epoch=5, iter=3220/40000, loss=0.5417, lr=0.009273, batch_cost=0.7908, reader_cost=0.0001 | ETA 08:04:45 2021-01-18 10:29:47 [INFO] [TRAIN] epoch=5, iter=3230/40000, loss=0.6931, lr=0.009270, batch_cost=0.7897, reader_cost=0.0001 | ETA 08:03:58 2021-01-18 10:29:55 [INFO] [TRAIN] epoch=5, iter=3240/40000, loss=0.4653, lr=0.009268, batch_cost=0.7941, reader_cost=0.0001 | ETA 08:06:29 2021-01-18 10:30:03 [INFO] [TRAIN] epoch=5, iter=3250/40000, loss=0.6469, lr=0.009266, batch_cost=0.7888, reader_cost=0.0001 | ETA 08:03:09 2021-01-18 10:30:11 [INFO] [TRAIN] epoch=5, iter=3260/40000, loss=0.6902, lr=0.009264, batch_cost=0.7882, reader_cost=0.0001 | ETA 08:02:38 2021-01-18 10:30:19 [INFO] [TRAIN] epoch=5, iter=3270/40000, loss=0.4708, lr=0.009261, batch_cost=0.7907, reader_cost=0.0001 | ETA 08:04:03 2021-01-18 10:30:27 [INFO] [TRAIN] epoch=5, iter=3280/40000, loss=0.6638, lr=0.009259, batch_cost=0.7893, reader_cost=0.0001 | ETA 08:03:03 2021-01-18 10:30:35 [INFO] [TRAIN] epoch=5, iter=3290/40000, loss=0.5739, lr=0.009257, batch_cost=0.7893, reader_cost=0.0001 | ETA 08:02:56 2021-01-18 10:30:43 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.5622, lr=0.009255, batch_cost=0.7922, reader_cost=0.0001 | ETA 08:04:32 2021-01-18 10:30:51 [INFO] [TRAIN] epoch=6, iter=3310/40000, loss=0.5866, lr=0.009252, batch_cost=0.7765, reader_cost=0.0197 | ETA 07:54:48 2021-01-18 10:30:58 [INFO] [TRAIN] epoch=6, iter=3320/40000, loss=0.4393, lr=0.009250, batch_cost=0.7931, reader_cost=0.0001 | ETA 08:04:52 2021-01-18 10:31:06 [INFO] [TRAIN] epoch=6, iter=3330/40000, loss=0.4926, lr=0.009248, batch_cost=0.7923, reader_cost=0.0001 | ETA 08:04:12 2021-01-18 10:31:14 [INFO] [TRAIN] epoch=6, iter=3340/40000, loss=0.4717, lr=0.009245, batch_cost=0.7921, reader_cost=0.0001 | ETA 08:03:58 2021-01-18 10:31:22 [INFO] [TRAIN] epoch=6, iter=3350/40000, loss=0.5431, lr=0.009243, batch_cost=0.7906, reader_cost=0.0001 | ETA 08:02:56 2021-01-18 10:31:30 [INFO] [TRAIN] epoch=6, iter=3360/40000, loss=0.5288, lr=0.009241, batch_cost=0.7921, reader_cost=0.0001 | ETA 08:03:43 2021-01-18 10:31:38 [INFO] [TRAIN] epoch=6, iter=3370/40000, loss=0.4340, lr=0.009239, batch_cost=0.7950, reader_cost=0.0001 | ETA 08:05:22 2021-01-18 10:31:46 [INFO] [TRAIN] epoch=6, iter=3380/40000, loss=0.5313, lr=0.009236, batch_cost=0.8022, reader_cost=0.0001 | ETA 08:09:36 2021-01-18 10:31:54 [INFO] [TRAIN] epoch=6, iter=3390/40000, loss=0.4720, lr=0.009234, batch_cost=0.7868, reader_cost=0.0001 | ETA 08:00:04 2021-01-18 10:32:02 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.6072, lr=0.009232, batch_cost=0.7833, reader_cost=0.0001 | ETA 07:57:50 2021-01-18 10:32:10 [INFO] [TRAIN] epoch=6, iter=3410/40000, loss=0.5552, lr=0.009230, batch_cost=0.7874, reader_cost=0.0001 | ETA 08:00:12 2021-01-18 10:32:18 [INFO] [TRAIN] epoch=6, iter=3420/40000, loss=0.6673, lr=0.009227, batch_cost=0.7893, reader_cost=0.0001 | ETA 08:01:12 2021-01-18 10:32:25 [INFO] [TRAIN] epoch=6, iter=3430/40000, loss=0.6286, lr=0.009225, batch_cost=0.7840, reader_cost=0.0001 | ETA 07:57:50 2021-01-18 10:32:33 [INFO] [TRAIN] epoch=6, iter=3440/40000, loss=0.6937, lr=0.009223, batch_cost=0.7926, reader_cost=0.0001 | ETA 08:02:55 2021-01-18 10:32:41 [INFO] [TRAIN] epoch=6, iter=3450/40000, loss=0.5452, lr=0.009221, batch_cost=0.7891, reader_cost=0.0001 | ETA 08:00:42 2021-01-18 10:32:49 [INFO] [TRAIN] epoch=6, iter=3460/40000, loss=0.5387, lr=0.009218, batch_cost=0.7953, reader_cost=0.0001 | ETA 08:04:21 2021-01-18 10:32:57 [INFO] [TRAIN] epoch=6, iter=3470/40000, loss=0.8568, lr=0.009216, batch_cost=0.7929, reader_cost=0.0001 | ETA 08:02:43 2021-01-18 10:33:05 [INFO] [TRAIN] epoch=6, iter=3480/40000, loss=0.5323, lr=0.009214, batch_cost=0.7941, reader_cost=0.0001 | ETA 08:03:21 2021-01-18 10:33:13 [INFO] [TRAIN] epoch=6, iter=3490/40000, loss=0.7026, lr=0.009211, batch_cost=0.7889, reader_cost=0.0001 | ETA 08:00:03 2021-01-18 10:33:21 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.8395, lr=0.009209, batch_cost=0.7948, reader_cost=0.0001 | ETA 08:03:30 2021-01-18 10:33:29 [INFO] [TRAIN] epoch=6, iter=3510/40000, loss=0.6676, lr=0.009207, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:59:17 2021-01-18 10:33:37 [INFO] [TRAIN] epoch=6, iter=3520/40000, loss=0.5568, lr=0.009205, batch_cost=0.7931, reader_cost=0.0001 | ETA 08:02:11 2021-01-18 10:33:45 [INFO] [TRAIN] epoch=6, iter=3530/40000, loss=0.3977, lr=0.009202, batch_cost=0.7929, reader_cost=0.0001 | ETA 08:01:56 2021-01-18 10:33:53 [INFO] [TRAIN] epoch=6, iter=3540/40000, loss=0.6041, lr=0.009200, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:59:39 2021-01-18 10:34:00 [INFO] [TRAIN] epoch=6, iter=3550/40000, loss=0.6587, lr=0.009198, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:59:14 2021-01-18 10:34:08 [INFO] [TRAIN] epoch=6, iter=3560/40000, loss=0.3985, lr=0.009196, batch_cost=0.7874, reader_cost=0.0001 | ETA 07:58:11 2021-01-18 10:34:16 [INFO] [TRAIN] epoch=6, iter=3570/40000, loss=0.4875, lr=0.009193, batch_cost=0.7925, reader_cost=0.0001 | ETA 08:01:09 2021-01-18 10:34:24 [INFO] [TRAIN] epoch=6, iter=3580/40000, loss=0.5499, lr=0.009191, batch_cost=0.7947, reader_cost=0.0001 | ETA 08:02:24 2021-01-18 10:34:32 [INFO] [TRAIN] epoch=6, iter=3590/40000, loss=0.5721, lr=0.009189, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:59:48 2021-01-18 10:34:40 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.3898, lr=0.009186, batch_cost=0.7838, reader_cost=0.0001 | ETA 07:55:30 2021-01-18 10:34:48 [INFO] [TRAIN] epoch=6, iter=3610/40000, loss=0.6024, lr=0.009184, batch_cost=0.7936, reader_cost=0.0001 | ETA 08:01:19 2021-01-18 10:34:56 [INFO] [TRAIN] epoch=6, iter=3620/40000, loss=0.4446, lr=0.009182, batch_cost=0.7858, reader_cost=0.0001 | ETA 07:56:27 2021-01-18 10:35:04 [INFO] [TRAIN] epoch=6, iter=3630/40000, loss=0.4308, lr=0.009180, batch_cost=0.7963, reader_cost=0.0001 | ETA 08:02:40 2021-01-18 10:35:12 [INFO] [TRAIN] epoch=6, iter=3640/40000, loss=0.7834, lr=0.009177, batch_cost=0.7949, reader_cost=0.0001 | ETA 08:01:42 2021-01-18 10:35:20 [INFO] [TRAIN] epoch=6, iter=3650/40000, loss=0.4765, lr=0.009175, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:58:35 2021-01-18 10:35:27 [INFO] [TRAIN] epoch=6, iter=3660/40000, loss=0.4355, lr=0.009173, batch_cost=0.7908, reader_cost=0.0001 | ETA 07:58:56 2021-01-18 10:35:35 [INFO] [TRAIN] epoch=6, iter=3670/40000, loss=0.5786, lr=0.009171, batch_cost=0.7933, reader_cost=0.0001 | ETA 08:00:19 2021-01-18 10:35:43 [INFO] [TRAIN] epoch=6, iter=3680/40000, loss=0.4503, lr=0.009168, batch_cost=0.7869, reader_cost=0.0001 | ETA 07:56:18 2021-01-18 10:35:51 [INFO] [TRAIN] epoch=6, iter=3690/40000, loss=0.3784, lr=0.009166, batch_cost=0.7880, reader_cost=0.0001 | ETA 07:56:51 2021-01-18 10:35:59 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.5422, lr=0.009164, batch_cost=0.7888, reader_cost=0.0001 | ETA 07:57:15 2021-01-18 10:36:07 [INFO] [TRAIN] epoch=6, iter=3710/40000, loss=0.5071, lr=0.009161, batch_cost=0.7879, reader_cost=0.0001 | ETA 07:56:32 2021-01-18 10:36:15 [INFO] [TRAIN] epoch=6, iter=3720/40000, loss=0.7798, lr=0.009159, batch_cost=0.7885, reader_cost=0.0001 | ETA 07:56:48 2021-01-18 10:36:23 [INFO] [TRAIN] epoch=6, iter=3730/40000, loss=0.4375, lr=0.009157, batch_cost=0.7927, reader_cost=0.0001 | ETA 07:59:10 2021-01-18 10:36:31 [INFO] [TRAIN] epoch=6, iter=3740/40000, loss=0.7082, lr=0.009155, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:59:45 2021-01-18 10:36:39 [INFO] [TRAIN] epoch=6, iter=3750/40000, loss=0.5686, lr=0.009152, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:59:32 2021-01-18 10:36:46 [INFO] [TRAIN] epoch=6, iter=3760/40000, loss=0.5373, lr=0.009150, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:58:56 2021-01-18 10:36:54 [INFO] [TRAIN] epoch=6, iter=3770/40000, loss=0.4839, lr=0.009148, batch_cost=0.7925, reader_cost=0.0001 | ETA 07:58:31 2021-01-18 10:37:02 [INFO] [TRAIN] epoch=6, iter=3780/40000, loss=0.7982, lr=0.009146, batch_cost=0.7895, reader_cost=0.0001 | ETA 07:56:35 2021-01-18 10:37:10 [INFO] [TRAIN] epoch=6, iter=3790/40000, loss=0.4481, lr=0.009143, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:56:39 2021-01-18 10:37:18 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.8514, lr=0.009141, batch_cost=0.7892, reader_cost=0.0001 | ETA 07:56:10 2021-01-18 10:37:26 [INFO] [TRAIN] epoch=6, iter=3810/40000, loss=0.5794, lr=0.009139, batch_cost=0.7942, reader_cost=0.0001 | ETA 07:59:01 2021-01-18 10:37:34 [INFO] [TRAIN] epoch=6, iter=3820/40000, loss=0.9120, lr=0.009136, batch_cost=0.7946, reader_cost=0.0001 | ETA 07:59:09 2021-01-18 10:37:42 [INFO] [TRAIN] epoch=6, iter=3830/40000, loss=0.8479, lr=0.009134, batch_cost=0.7929, reader_cost=0.0001 | ETA 07:58:00 2021-01-18 10:37:50 [INFO] [TRAIN] epoch=6, iter=3840/40000, loss=0.5738, lr=0.009132, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:57:53 2021-01-18 10:37:58 [INFO] [TRAIN] epoch=6, iter=3850/40000, loss=0.6945, lr=0.009130, batch_cost=0.7890, reader_cost=0.0001 | ETA 07:55:21 2021-01-18 10:38:06 [INFO] [TRAIN] epoch=6, iter=3860/40000, loss=0.6126, lr=0.009127, batch_cost=0.7931, reader_cost=0.0001 | ETA 07:57:43 2021-01-18 10:38:14 [INFO] [TRAIN] epoch=6, iter=3870/40000, loss=0.6451, lr=0.009125, batch_cost=0.7929, reader_cost=0.0001 | ETA 07:57:26 2021-01-18 10:38:22 [INFO] [TRAIN] epoch=6, iter=3880/40000, loss=0.4274, lr=0.009123, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:56:34 2021-01-18 10:38:29 [INFO] [TRAIN] epoch=6, iter=3890/40000, loss=0.6260, lr=0.009121, batch_cost=0.7913, reader_cost=0.0001 | ETA 07:56:13 2021-01-18 10:38:37 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.6786, lr=0.009118, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:54:54 2021-01-18 10:38:45 [INFO] [TRAIN] epoch=6, iter=3910/40000, loss=0.7126, lr=0.009116, batch_cost=0.7923, reader_cost=0.0001 | ETA 07:56:35 2021-01-18 10:38:53 [INFO] [TRAIN] epoch=6, iter=3920/40000, loss=0.3937, lr=0.009114, batch_cost=0.7908, reader_cost=0.0001 | ETA 07:55:33 2021-01-18 10:39:01 [INFO] [TRAIN] epoch=6, iter=3930/40000, loss=0.6643, lr=0.009111, batch_cost=0.7880, reader_cost=0.0001 | ETA 07:53:41 2021-01-18 10:39:09 [INFO] [TRAIN] epoch=6, iter=3940/40000, loss=0.9114, lr=0.009109, batch_cost=0.7931, reader_cost=0.0001 | ETA 07:56:39 2021-01-18 10:39:17 [INFO] [TRAIN] epoch=6, iter=3950/40000, loss=0.6589, lr=0.009107, batch_cost=0.7873, reader_cost=0.0001 | ETA 07:53:03 2021-01-18 10:39:25 [INFO] [TRAIN] epoch=6, iter=3960/40000, loss=0.5008, lr=0.009105, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:55:41 2021-01-18 10:39:33 [INFO] [TRAIN] epoch=7, iter=3970/40000, loss=0.6534, lr=0.009102, batch_cost=0.7761, reader_cost=0.0155 | ETA 07:46:01 2021-01-18 10:39:40 [INFO] [TRAIN] epoch=7, iter=3980/40000, loss=0.6032, lr=0.009100, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:56:57 2021-01-18 10:39:48 [INFO] [TRAIN] epoch=7, iter=3990/40000, loss=0.7503, lr=0.009098, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:55:57 2021-01-18 10:39:56 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.6327, lr=0.009096, batch_cost=0.7938, reader_cost=0.0001 | ETA 07:56:15 2021-01-18 10:39:56 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 10:40:26 [INFO] [EVAL] #Images=1449 mIoU=0.6352 Acc=0.9134 Kappa=0.8106 2021-01-18 10:40:26 [INFO] [EVAL] Class IoU: [0.9117 0.672 0.2972 0.7614 0.5607 0.7227 0.8074 0.7544 0.8183 0.2767 0.6723 0.5508 0.6993 0.7816 0.571 0.8049 0.3777 0.763 0.2725 0.7051 0.5591] 2021-01-18 10:40:26 [INFO] [EVAL] Class Acc: [0.9545 0.683 0.3115 0.8905 0.7393 0.8778 0.9025 0.8812 0.9248 0.562 0.9245 0.7909 0.7251 0.8576 0.9401 0.8656 0.4979 0.8618 0.8444 0.7701 0.6132] 2021-01-18 10:40:36 [INFO] [EVAL] The model with the best validation mIoU (0.6352) was saved at iter 4000. 2021-01-18 10:40:44 [INFO] [TRAIN] epoch=7, iter=4010/40000, loss=0.6976, lr=0.009093, batch_cost=0.7818, reader_cost=0.0001 | ETA 07:48:56 2021-01-18 10:40:51 [INFO] [TRAIN] epoch=7, iter=4020/40000, loss=0.5639, lr=0.009091, batch_cost=0.7868, reader_cost=0.0001 | ETA 07:51:49 2021-01-18 10:40:59 [INFO] [TRAIN] epoch=7, iter=4030/40000, loss=0.5333, lr=0.009089, batch_cost=0.7916, reader_cost=0.0001 | ETA 07:54:33 2021-01-18 10:41:07 [INFO] [TRAIN] epoch=7, iter=4040/40000, loss=0.5912, lr=0.009086, batch_cost=0.7925, reader_cost=0.0001 | ETA 07:54:59 2021-01-18 10:41:15 [INFO] [TRAIN] epoch=7, iter=4050/40000, loss=0.4140, lr=0.009084, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:55:39 2021-01-18 10:41:23 [INFO] [TRAIN] epoch=7, iter=4060/40000, loss=0.5404, lr=0.009082, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:52:33 2021-01-18 10:41:31 [INFO] [TRAIN] epoch=7, iter=4070/40000, loss=0.6422, lr=0.009080, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:51:55 2021-01-18 10:41:39 [INFO] [TRAIN] epoch=7, iter=4080/40000, loss=0.4411, lr=0.009077, batch_cost=0.7949, reader_cost=0.0001 | ETA 07:55:51 2021-01-18 10:41:47 [INFO] [TRAIN] epoch=7, iter=4090/40000, loss=0.5872, lr=0.009075, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:54:21 2021-01-18 10:41:55 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.5261, lr=0.009073, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:53:07 2021-01-18 10:42:03 [INFO] [TRAIN] epoch=7, iter=4110/40000, loss=0.7367, lr=0.009071, batch_cost=0.7965, reader_cost=0.0001 | ETA 07:56:26 2021-01-18 10:42:11 [INFO] [TRAIN] epoch=7, iter=4120/40000, loss=0.4520, lr=0.009068, batch_cost=0.7965, reader_cost=0.0001 | ETA 07:56:16 2021-01-18 10:42:19 [INFO] [TRAIN] epoch=7, iter=4130/40000, loss=0.6050, lr=0.009066, batch_cost=0.7921, reader_cost=0.0001 | ETA 07:53:33 2021-01-18 10:42:26 [INFO] [TRAIN] epoch=7, iter=4140/40000, loss=0.6008, lr=0.009064, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:51:58 2021-01-18 10:42:34 [INFO] [TRAIN] epoch=7, iter=4150/40000, loss=0.5983, lr=0.009061, batch_cost=0.7915, reader_cost=0.0001 | ETA 07:52:56 2021-01-18 10:42:42 [INFO] [TRAIN] epoch=7, iter=4160/40000, loss=0.4361, lr=0.009059, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:51:08 2021-01-18 10:42:50 [INFO] [TRAIN] epoch=7, iter=4170/40000, loss=0.3616, lr=0.009057, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:50:55 2021-01-18 10:42:58 [INFO] [TRAIN] epoch=7, iter=4180/40000, loss=0.6338, lr=0.009055, batch_cost=0.7857, reader_cost=0.0001 | ETA 07:49:04 2021-01-18 10:43:06 [INFO] [TRAIN] epoch=7, iter=4190/40000, loss=0.7312, lr=0.009052, batch_cost=0.7890, reader_cost=0.0001 | ETA 07:50:53 2021-01-18 10:43:14 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.4714, lr=0.009050, batch_cost=0.7934, reader_cost=0.0001 | ETA 07:53:23 2021-01-18 10:43:22 [INFO] [TRAIN] epoch=7, iter=4210/40000, loss=0.6045, lr=0.009048, batch_cost=0.7859, reader_cost=0.0001 | ETA 07:48:47 2021-01-18 10:43:30 [INFO] [TRAIN] epoch=7, iter=4220/40000, loss=0.6194, lr=0.009046, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:52:06 2021-01-18 10:43:38 [INFO] [TRAIN] epoch=7, iter=4230/40000, loss=0.5133, lr=0.009043, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:53:55 2021-01-18 10:43:45 [INFO] [TRAIN] epoch=7, iter=4240/40000, loss=0.4188, lr=0.009041, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:51:21 2021-01-18 10:43:53 [INFO] [TRAIN] epoch=7, iter=4250/40000, loss=0.4137, lr=0.009039, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:51:32 2021-01-18 10:44:01 [INFO] [TRAIN] epoch=7, iter=4260/40000, loss=0.4440, lr=0.009036, batch_cost=0.7855, reader_cost=0.0001 | ETA 07:47:52 2021-01-18 10:44:09 [INFO] [TRAIN] epoch=7, iter=4270/40000, loss=0.4459, lr=0.009034, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:51:33 2021-01-18 10:44:17 [INFO] [TRAIN] epoch=7, iter=4280/40000, loss=0.3554, lr=0.009032, batch_cost=0.7942, reader_cost=0.0001 | ETA 07:52:48 2021-01-18 10:44:25 [INFO] [TRAIN] epoch=7, iter=4290/40000, loss=0.5548, lr=0.009030, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:53:10 2021-01-18 10:44:33 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.5965, lr=0.009027, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:51:09 2021-01-18 10:44:41 [INFO] [TRAIN] epoch=7, iter=4310/40000, loss=0.6484, lr=0.009025, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:49:34 2021-01-18 10:44:49 [INFO] [TRAIN] epoch=7, iter=4320/40000, loss=0.5246, lr=0.009023, batch_cost=0.7857, reader_cost=0.0001 | ETA 07:47:12 2021-01-18 10:44:57 [INFO] [TRAIN] epoch=7, iter=4330/40000, loss=0.5897, lr=0.009020, batch_cost=0.7854, reader_cost=0.0001 | ETA 07:46:55 2021-01-18 10:45:05 [INFO] [TRAIN] epoch=7, iter=4340/40000, loss=0.6035, lr=0.009018, batch_cost=0.7946, reader_cost=0.0001 | ETA 07:52:15 2021-01-18 10:45:12 [INFO] [TRAIN] epoch=7, iter=4350/40000, loss=0.9299, lr=0.009016, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:51:17 2021-01-18 10:45:20 [INFO] [TRAIN] epoch=7, iter=4360/40000, loss=0.5520, lr=0.009014, batch_cost=0.7952, reader_cost=0.0001 | ETA 07:52:19 2021-01-18 10:45:28 [INFO] [TRAIN] epoch=7, iter=4370/40000, loss=0.6764, lr=0.009011, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:49:56 2021-01-18 10:45:36 [INFO] [TRAIN] epoch=7, iter=4380/40000, loss=0.5937, lr=0.009009, batch_cost=0.7855, reader_cost=0.0001 | ETA 07:46:18 2021-01-18 10:45:44 [INFO] [TRAIN] epoch=7, iter=4390/40000, loss=0.4867, lr=0.009007, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:50:39 2021-01-18 10:45:52 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.5520, lr=0.009005, batch_cost=0.7869, reader_cost=0.0001 | ETA 07:46:51 2021-01-18 10:46:00 [INFO] [TRAIN] epoch=7, iter=4410/40000, loss=0.5290, lr=0.009002, batch_cost=0.7928, reader_cost=0.0001 | ETA 07:50:16 2021-01-18 10:46:08 [INFO] [TRAIN] epoch=7, iter=4420/40000, loss=0.4139, lr=0.009000, batch_cost=0.7860, reader_cost=0.0001 | ETA 07:46:05 2021-01-18 10:46:16 [INFO] [TRAIN] epoch=7, iter=4430/40000, loss=0.5332, lr=0.008998, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:50:38 2021-01-18 10:46:24 [INFO] [TRAIN] epoch=7, iter=4440/40000, loss=0.6058, lr=0.008995, batch_cost=0.7944, reader_cost=0.0001 | ETA 07:50:48 2021-01-18 10:46:32 [INFO] [TRAIN] epoch=7, iter=4450/40000, loss=0.5780, lr=0.008993, batch_cost=0.7946, reader_cost=0.0001 | ETA 07:50:49 2021-01-18 10:46:39 [INFO] [TRAIN] epoch=7, iter=4460/40000, loss=0.5094, lr=0.008991, batch_cost=0.7883, reader_cost=0.0001 | ETA 07:46:57 2021-01-18 10:46:47 [INFO] [TRAIN] epoch=7, iter=4470/40000, loss=0.3847, lr=0.008989, batch_cost=0.7921, reader_cost=0.0001 | ETA 07:49:03 2021-01-18 10:46:55 [INFO] [TRAIN] epoch=7, iter=4480/40000, loss=0.4455, lr=0.008986, batch_cost=0.7912, reader_cost=0.0001 | ETA 07:48:21 2021-01-18 10:47:03 [INFO] [TRAIN] epoch=7, iter=4490/40000, loss=0.8102, lr=0.008984, batch_cost=0.7921, reader_cost=0.0001 | ETA 07:48:48 2021-01-18 10:47:11 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.4852, lr=0.008982, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:47:11 2021-01-18 10:47:19 [INFO] [TRAIN] epoch=7, iter=4510/40000, loss=0.5374, lr=0.008980, batch_cost=0.7948, reader_cost=0.0001 | ETA 07:50:08 2021-01-18 10:47:27 [INFO] [TRAIN] epoch=7, iter=4520/40000, loss=0.4278, lr=0.008977, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:47:09 2021-01-18 10:47:35 [INFO] [TRAIN] epoch=7, iter=4530/40000, loss=0.7809, lr=0.008975, batch_cost=0.7940, reader_cost=0.0001 | ETA 07:49:21 2021-01-18 10:47:43 [INFO] [TRAIN] epoch=7, iter=4540/40000, loss=0.4472, lr=0.008973, batch_cost=0.7876, reader_cost=0.0001 | ETA 07:45:29 2021-01-18 10:47:51 [INFO] [TRAIN] epoch=7, iter=4550/40000, loss=0.4800, lr=0.008970, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:47:09 2021-01-18 10:47:59 [INFO] [TRAIN] epoch=7, iter=4560/40000, loss=0.5146, lr=0.008968, batch_cost=0.7973, reader_cost=0.0001 | ETA 07:50:54 2021-01-18 10:48:07 [INFO] [TRAIN] epoch=7, iter=4570/40000, loss=0.5460, lr=0.008966, batch_cost=0.7927, reader_cost=0.0001 | ETA 07:48:06 2021-01-18 10:48:15 [INFO] [TRAIN] epoch=7, iter=4580/40000, loss=0.4086, lr=0.008964, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:45:32 2021-01-18 10:48:22 [INFO] [TRAIN] epoch=7, iter=4590/40000, loss=0.6528, lr=0.008961, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:48:53 2021-01-18 10:48:30 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.5526, lr=0.008959, batch_cost=0.7951, reader_cost=0.0001 | ETA 07:49:06 2021-01-18 10:48:38 [INFO] [TRAIN] epoch=7, iter=4610/40000, loss=0.7008, lr=0.008957, batch_cost=0.7974, reader_cost=0.0001 | ETA 07:50:21 2021-01-18 10:48:46 [INFO] [TRAIN] epoch=7, iter=4620/40000, loss=0.4091, lr=0.008954, batch_cost=0.7936, reader_cost=0.0001 | ETA 07:47:57 2021-01-18 10:48:54 [INFO] [TRAIN] epoch=8, iter=4630/40000, loss=0.7261, lr=0.008952, batch_cost=0.7808, reader_cost=0.0178 | ETA 07:40:18 2021-01-18 10:49:02 [INFO] [TRAIN] epoch=8, iter=4640/40000, loss=0.7911, lr=0.008950, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:46:34 2021-01-18 10:49:10 [INFO] [TRAIN] epoch=8, iter=4650/40000, loss=0.5870, lr=0.008948, batch_cost=0.7931, reader_cost=0.0001 | ETA 07:47:15 2021-01-18 10:49:18 [INFO] [TRAIN] epoch=8, iter=4660/40000, loss=0.6622, lr=0.008945, batch_cost=0.7903, reader_cost=0.0001 | ETA 07:45:28 2021-01-18 10:49:26 [INFO] [TRAIN] epoch=8, iter=4670/40000, loss=0.7308, lr=0.008943, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:46:10 2021-01-18 10:49:34 [INFO] [TRAIN] epoch=8, iter=4680/40000, loss=0.4661, lr=0.008941, batch_cost=0.7901, reader_cost=0.0001 | ETA 07:45:06 2021-01-18 10:49:42 [INFO] [TRAIN] epoch=8, iter=4690/40000, loss=0.6282, lr=0.008939, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:44:31 2021-01-18 10:49:50 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.4756, lr=0.008936, batch_cost=0.7941, reader_cost=0.0001 | ETA 07:47:13 2021-01-18 10:49:57 [INFO] [TRAIN] epoch=8, iter=4710/40000, loss=0.5884, lr=0.008934, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:43:48 2021-01-18 10:50:05 [INFO] [TRAIN] epoch=8, iter=4720/40000, loss=0.7661, lr=0.008932, batch_cost=0.7876, reader_cost=0.0001 | ETA 07:43:05 2021-01-18 10:50:13 [INFO] [TRAIN] epoch=8, iter=4730/40000, loss=0.4644, lr=0.008929, batch_cost=0.7851, reader_cost=0.0001 | ETA 07:41:30 2021-01-18 10:50:21 [INFO] [TRAIN] epoch=8, iter=4740/40000, loss=0.5112, lr=0.008927, batch_cost=0.7925, reader_cost=0.0001 | ETA 07:45:45 2021-01-18 10:50:29 [INFO] [TRAIN] epoch=8, iter=4750/40000, loss=0.6240, lr=0.008925, batch_cost=0.7904, reader_cost=0.0001 | ETA 07:44:22 2021-01-18 10:50:37 [INFO] [TRAIN] epoch=8, iter=4760/40000, loss=0.6347, lr=0.008923, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:43:38 2021-01-18 10:50:45 [INFO] [TRAIN] epoch=8, iter=4770/40000, loss=0.4892, lr=0.008920, batch_cost=0.7902, reader_cost=0.0001 | ETA 07:43:58 2021-01-18 10:50:53 [INFO] [TRAIN] epoch=8, iter=4780/40000, loss=0.4375, lr=0.008918, batch_cost=0.7892, reader_cost=0.0001 | ETA 07:43:14 2021-01-18 10:51:01 [INFO] [TRAIN] epoch=8, iter=4790/40000, loss=0.9411, lr=0.008916, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:43:35 2021-01-18 10:51:08 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.7881, lr=0.008913, batch_cost=0.7928, reader_cost=0.0001 | ETA 07:45:07 2021-01-18 10:51:16 [INFO] [TRAIN] epoch=8, iter=4810/40000, loss=0.6221, lr=0.008911, batch_cost=0.7853, reader_cost=0.0001 | ETA 07:40:33 2021-01-18 10:51:24 [INFO] [TRAIN] epoch=8, iter=4820/40000, loss=0.6478, lr=0.008909, batch_cost=0.7935, reader_cost=0.0001 | ETA 07:45:15 2021-01-18 10:51:32 [INFO] [TRAIN] epoch=8, iter=4830/40000, loss=0.4339, lr=0.008907, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:42:39 2021-01-18 10:51:40 [INFO] [TRAIN] epoch=8, iter=4840/40000, loss=0.6050, lr=0.008904, batch_cost=0.7935, reader_cost=0.0001 | ETA 07:44:57 2021-01-18 10:51:48 [INFO] [TRAIN] epoch=8, iter=4850/40000, loss=0.4218, lr=0.008902, batch_cost=0.7885, reader_cost=0.0001 | ETA 07:41:56 2021-01-18 10:51:56 [INFO] [TRAIN] epoch=8, iter=4860/40000, loss=0.6047, lr=0.008900, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:45:18 2021-01-18 10:52:04 [INFO] [TRAIN] epoch=8, iter=4870/40000, loss=0.7114, lr=0.008897, batch_cost=0.7921, reader_cost=0.0001 | ETA 07:43:45 2021-01-18 10:52:12 [INFO] [TRAIN] epoch=8, iter=4880/40000, loss=0.5316, lr=0.008895, batch_cost=0.7947, reader_cost=0.0001 | ETA 07:45:11 2021-01-18 10:52:20 [INFO] [TRAIN] epoch=8, iter=4890/40000, loss=0.5874, lr=0.008893, batch_cost=0.7954, reader_cost=0.0001 | ETA 07:45:26 2021-01-18 10:52:28 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.7351, lr=0.008891, batch_cost=0.7876, reader_cost=0.0001 | ETA 07:40:43 2021-01-18 10:52:36 [INFO] [TRAIN] epoch=8, iter=4910/40000, loss=0.4893, lr=0.008888, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:42:48 2021-01-18 10:52:43 [INFO] [TRAIN] epoch=8, iter=4920/40000, loss=0.5380, lr=0.008886, batch_cost=0.7928, reader_cost=0.0001 | ETA 07:43:32 2021-01-18 10:52:51 [INFO] [TRAIN] epoch=8, iter=4930/40000, loss=0.4179, lr=0.008884, batch_cost=0.7931, reader_cost=0.0001 | ETA 07:43:34 2021-01-18 10:52:59 [INFO] [TRAIN] epoch=8, iter=4940/40000, loss=0.6034, lr=0.008882, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:42:49 2021-01-18 10:53:07 [INFO] [TRAIN] epoch=8, iter=4950/40000, loss=0.5001, lr=0.008879, batch_cost=0.7947, reader_cost=0.0001 | ETA 07:44:15 2021-01-18 10:53:15 [INFO] [TRAIN] epoch=8, iter=4960/40000, loss=0.4950, lr=0.008877, batch_cost=0.7908, reader_cost=0.0001 | ETA 07:41:48 2021-01-18 10:53:23 [INFO] [TRAIN] epoch=8, iter=4970/40000, loss=0.6277, lr=0.008875, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:42:29 2021-01-18 10:53:31 [INFO] [TRAIN] epoch=8, iter=4980/40000, loss=0.4120, lr=0.008872, batch_cost=0.7845, reader_cost=0.0001 | ETA 07:37:52 2021-01-18 10:53:39 [INFO] [TRAIN] epoch=8, iter=4990/40000, loss=0.4810, lr=0.008870, batch_cost=0.7902, reader_cost=0.0001 | ETA 07:41:03 2021-01-18 10:53:47 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.3861, lr=0.008868, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:40:41 2021-01-18 10:53:47 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 10:54:14 [INFO] [EVAL] #Images=1449 mIoU=0.6741 Acc=0.9263 Kappa=0.8369 2021-01-18 10:54:14 [INFO] [EVAL] Class IoU: [0.9244 0.8154 0.3662 0.8631 0.5858 0.7105 0.8913 0.7488 0.8674 0.2964 0.7551 0.3703 0.7483 0.7884 0.5332 0.8219 0.5054 0.7565 0.4852 0.7984 0.5231] 2021-01-18 10:54:14 [INFO] [EVAL] Class Acc: [0.9563 0.845 0.3891 0.9084 0.8142 0.8542 0.9458 0.8068 0.8869 0.6477 0.8632 0.8631 0.8625 0.8555 0.9498 0.9284 0.6755 0.9091 0.6646 0.868 0.5885] 2021-01-18 10:54:24 [INFO] [EVAL] The model with the best validation mIoU (0.6741) was saved at iter 5000. 2021-01-18 10:54:32 [INFO] [TRAIN] epoch=8, iter=5010/40000, loss=0.4863, lr=0.008866, batch_cost=0.7842, reader_cost=0.0001 | ETA 07:37:20 2021-01-18 10:54:40 [INFO] [TRAIN] epoch=8, iter=5020/40000, loss=0.6082, lr=0.008863, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:41:45 2021-01-18 10:54:48 [INFO] [TRAIN] epoch=8, iter=5030/40000, loss=0.5522, lr=0.008861, batch_cost=0.7904, reader_cost=0.0001 | ETA 07:40:38 2021-01-18 10:54:56 [INFO] [TRAIN] epoch=8, iter=5040/40000, loss=0.4174, lr=0.008859, batch_cost=0.7928, reader_cost=0.0001 | ETA 07:41:57 2021-01-18 10:55:04 [INFO] [TRAIN] epoch=8, iter=5050/40000, loss=0.4338, lr=0.008856, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:41:20 2021-01-18 10:55:12 [INFO] [TRAIN] epoch=8, iter=5060/40000, loss=0.5568, lr=0.008854, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:41:19 2021-01-18 10:55:20 [INFO] [TRAIN] epoch=8, iter=5070/40000, loss=0.7227, lr=0.008852, batch_cost=0.7928, reader_cost=0.0001 | ETA 07:41:34 2021-01-18 10:55:28 [INFO] [TRAIN] epoch=8, iter=5080/40000, loss=0.6294, lr=0.008850, batch_cost=0.7903, reader_cost=0.0001 | ETA 07:39:57 2021-01-18 10:55:35 [INFO] [TRAIN] epoch=8, iter=5090/40000, loss=0.3909, lr=0.008847, batch_cost=0.7878, reader_cost=0.0001 | ETA 07:38:20 2021-01-18 10:55:43 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.3856, lr=0.008845, batch_cost=0.7923, reader_cost=0.0001 | ETA 07:40:49 2021-01-18 10:55:51 [INFO] [TRAIN] epoch=8, iter=5110/40000, loss=0.5205, lr=0.008843, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:41:38 2021-01-18 10:55:59 [INFO] [TRAIN] epoch=8, iter=5120/40000, loss=0.3458, lr=0.008840, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:40:03 2021-01-18 10:56:07 [INFO] [TRAIN] epoch=8, iter=5130/40000, loss=0.5828, lr=0.008838, batch_cost=0.7934, reader_cost=0.0001 | ETA 07:41:05 2021-01-18 10:56:15 [INFO] [TRAIN] epoch=8, iter=5140/40000, loss=0.3825, lr=0.008836, batch_cost=0.7944, reader_cost=0.0001 | ETA 07:41:33 2021-01-18 10:56:23 [INFO] [TRAIN] epoch=8, iter=5150/40000, loss=0.5973, lr=0.008834, batch_cost=0.7933, reader_cost=0.0001 | ETA 07:40:45 2021-01-18 10:56:31 [INFO] [TRAIN] epoch=8, iter=5160/40000, loss=0.5761, lr=0.008831, batch_cost=0.7915, reader_cost=0.0001 | ETA 07:39:34 2021-01-18 10:56:39 [INFO] [TRAIN] epoch=8, iter=5170/40000, loss=0.3927, lr=0.008829, batch_cost=0.7872, reader_cost=0.0001 | ETA 07:36:56 2021-01-18 10:56:47 [INFO] [TRAIN] epoch=8, iter=5180/40000, loss=0.4476, lr=0.008827, batch_cost=0.7905, reader_cost=0.0001 | ETA 07:38:45 2021-01-18 10:56:55 [INFO] [TRAIN] epoch=8, iter=5190/40000, loss=0.5354, lr=0.008825, batch_cost=0.7927, reader_cost=0.0001 | ETA 07:39:53 2021-01-18 10:57:03 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.5849, lr=0.008822, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:38:59 2021-01-18 10:57:11 [INFO] [TRAIN] epoch=8, iter=5210/40000, loss=0.7637, lr=0.008820, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:38:34 2021-01-18 10:57:18 [INFO] [TRAIN] epoch=8, iter=5220/40000, loss=0.5133, lr=0.008818, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:38:26 2021-01-18 10:57:26 [INFO] [TRAIN] epoch=8, iter=5230/40000, loss=0.6652, lr=0.008815, batch_cost=0.7888, reader_cost=0.0001 | ETA 07:37:07 2021-01-18 10:57:34 [INFO] [TRAIN] epoch=8, iter=5240/40000, loss=0.5047, lr=0.008813, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:39:23 2021-01-18 10:57:42 [INFO] [TRAIN] epoch=8, iter=5250/40000, loss=0.7571, lr=0.008811, batch_cost=0.7854, reader_cost=0.0001 | ETA 07:34:52 2021-01-18 10:57:50 [INFO] [TRAIN] epoch=8, iter=5260/40000, loss=0.4486, lr=0.008809, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:37:18 2021-01-18 10:57:58 [INFO] [TRAIN] epoch=8, iter=5270/40000, loss=0.5931, lr=0.008806, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:39:00 2021-01-18 10:58:06 [INFO] [TRAIN] epoch=8, iter=5280/40000, loss=0.5703, lr=0.008804, batch_cost=0.7958, reader_cost=0.0001 | ETA 07:40:29 2021-01-18 10:58:14 [INFO] [TRAIN] epoch=9, iter=5290/40000, loss=0.4594, lr=0.008802, batch_cost=0.7780, reader_cost=0.0176 | ETA 07:30:04 2021-01-18 10:58:22 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.5231, lr=0.008799, batch_cost=0.7854, reader_cost=0.0001 | ETA 07:34:13 2021-01-18 10:58:29 [INFO] [TRAIN] epoch=9, iter=5310/40000, loss=0.5583, lr=0.008797, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:36:39 2021-01-18 10:58:37 [INFO] [TRAIN] epoch=9, iter=5320/40000, loss=0.5883, lr=0.008795, batch_cost=0.7824, reader_cost=0.0001 | ETA 07:32:15 2021-01-18 10:58:45 [INFO] [TRAIN] epoch=9, iter=5330/40000, loss=0.5706, lr=0.008793, batch_cost=0.7924, reader_cost=0.0001 | ETA 07:37:51 2021-01-18 10:58:53 [INFO] [TRAIN] epoch=9, iter=5340/40000, loss=0.5230, lr=0.008790, batch_cost=0.7879, reader_cost=0.0001 | ETA 07:35:09 2021-01-18 10:59:01 [INFO] [TRAIN] epoch=9, iter=5350/40000, loss=0.6262, lr=0.008788, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:37:19 2021-01-18 10:59:09 [INFO] [TRAIN] epoch=9, iter=5360/40000, loss=0.3664, lr=0.008786, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:38:12 2021-01-18 10:59:17 [INFO] [TRAIN] epoch=9, iter=5370/40000, loss=0.5096, lr=0.008783, batch_cost=0.7923, reader_cost=0.0001 | ETA 07:37:16 2021-01-18 10:59:25 [INFO] [TRAIN] epoch=9, iter=5380/40000, loss=0.6279, lr=0.008781, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:36:19 2021-01-18 10:59:33 [INFO] [TRAIN] epoch=9, iter=5390/40000, loss=0.5624, lr=0.008779, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:35:34 2021-01-18 10:59:41 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.4448, lr=0.008777, batch_cost=0.7954, reader_cost=0.0001 | ETA 07:38:41 2021-01-18 10:59:48 [INFO] [TRAIN] epoch=9, iter=5410/40000, loss=0.5445, lr=0.008774, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:35:15 2021-01-18 10:59:56 [INFO] [TRAIN] epoch=9, iter=5420/40000, loss=0.2747, lr=0.008772, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:35:05 2021-01-18 11:00:04 [INFO] [TRAIN] epoch=9, iter=5430/40000, loss=0.4808, lr=0.008770, batch_cost=0.7912, reader_cost=0.0001 | ETA 07:35:51 2021-01-18 11:00:12 [INFO] [TRAIN] epoch=9, iter=5440/40000, loss=0.4743, lr=0.008767, batch_cost=0.7925, reader_cost=0.0001 | ETA 07:36:28 2021-01-18 11:00:20 [INFO] [TRAIN] epoch=9, iter=5450/40000, loss=0.4406, lr=0.008765, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:34:44 2021-01-18 11:00:28 [INFO] [TRAIN] epoch=9, iter=5460/40000, loss=0.5298, lr=0.008763, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:35:51 2021-01-18 11:00:36 [INFO] [TRAIN] epoch=9, iter=5470/40000, loss=0.4004, lr=0.008761, batch_cost=0.7867, reader_cost=0.0001 | ETA 07:32:44 2021-01-18 11:00:44 [INFO] [TRAIN] epoch=9, iter=5480/40000, loss=0.4397, lr=0.008758, batch_cost=0.7944, reader_cost=0.0001 | ETA 07:37:03 2021-01-18 11:00:52 [INFO] [TRAIN] epoch=9, iter=5490/40000, loss=0.4255, lr=0.008756, batch_cost=0.7964, reader_cost=0.0001 | ETA 07:38:03 2021-01-18 11:01:00 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.6086, lr=0.008754, batch_cost=0.7954, reader_cost=0.0001 | ETA 07:37:21 2021-01-18 11:01:08 [INFO] [TRAIN] epoch=9, iter=5510/40000, loss=0.5424, lr=0.008751, batch_cost=0.7943, reader_cost=0.0001 | ETA 07:36:34 2021-01-18 11:01:16 [INFO] [TRAIN] epoch=9, iter=5520/40000, loss=0.7782, lr=0.008749, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:33:38 2021-01-18 11:01:24 [INFO] [TRAIN] epoch=9, iter=5530/40000, loss=0.5248, lr=0.008747, batch_cost=0.7899, reader_cost=0.0001 | ETA 07:33:46 2021-01-18 11:01:31 [INFO] [TRAIN] epoch=9, iter=5540/40000, loss=0.5291, lr=0.008745, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:33:43 2021-01-18 11:01:39 [INFO] [TRAIN] epoch=9, iter=5550/40000, loss=0.2862, lr=0.008742, batch_cost=0.7880, reader_cost=0.0001 | ETA 07:32:27 2021-01-18 11:01:47 [INFO] [TRAIN] epoch=9, iter=5560/40000, loss=0.4471, lr=0.008740, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:36:02 2021-01-18 11:01:55 [INFO] [TRAIN] epoch=9, iter=5570/40000, loss=0.5229, lr=0.008738, batch_cost=0.7912, reader_cost=0.0001 | ETA 07:34:00 2021-01-18 11:02:03 [INFO] [TRAIN] epoch=9, iter=5580/40000, loss=0.3775, lr=0.008735, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:34:17 2021-01-18 11:02:11 [INFO] [TRAIN] epoch=9, iter=5590/40000, loss=0.4102, lr=0.008733, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:32:58 2021-01-18 11:02:19 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.4879, lr=0.008731, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:35:04 2021-01-18 11:02:27 [INFO] [TRAIN] epoch=9, iter=5610/40000, loss=0.4363, lr=0.008729, batch_cost=0.7915, reader_cost=0.0001 | ETA 07:33:41 2021-01-18 11:02:35 [INFO] [TRAIN] epoch=9, iter=5620/40000, loss=0.3637, lr=0.008726, batch_cost=0.7913, reader_cost=0.0001 | ETA 07:33:23 2021-01-18 11:02:43 [INFO] [TRAIN] epoch=9, iter=5630/40000, loss=0.5666, lr=0.008724, batch_cost=0.7877, reader_cost=0.0001 | ETA 07:31:13 2021-01-18 11:02:50 [INFO] [TRAIN] epoch=9, iter=5640/40000, loss=0.3975, lr=0.008722, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:31:38 2021-01-18 11:02:58 [INFO] [TRAIN] epoch=9, iter=5650/40000, loss=0.5336, lr=0.008719, batch_cost=0.7943, reader_cost=0.0001 | ETA 07:34:45 2021-01-18 11:03:06 [INFO] [TRAIN] epoch=9, iter=5660/40000, loss=0.3895, lr=0.008717, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:31:04 2021-01-18 11:03:14 [INFO] [TRAIN] epoch=9, iter=5670/40000, loss=0.5374, lr=0.008715, batch_cost=0.7892, reader_cost=0.0001 | ETA 07:31:34 2021-01-18 11:03:22 [INFO] [TRAIN] epoch=9, iter=5680/40000, loss=0.4212, lr=0.008713, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:32:21 2021-01-18 11:03:30 [INFO] [TRAIN] epoch=9, iter=5690/40000, loss=0.5625, lr=0.008710, batch_cost=0.7940, reader_cost=0.0001 | ETA 07:34:03 2021-01-18 11:03:38 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.4851, lr=0.008708, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:31:22 2021-01-18 11:03:46 [INFO] [TRAIN] epoch=9, iter=5710/40000, loss=0.7326, lr=0.008706, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:31:19 2021-01-18 11:03:54 [INFO] [TRAIN] epoch=9, iter=5720/40000, loss=0.6374, lr=0.008704, batch_cost=0.7910, reader_cost=0.0001 | ETA 07:31:56 2021-01-18 11:04:02 [INFO] [TRAIN] epoch=9, iter=5730/40000, loss=0.5112, lr=0.008701, batch_cost=0.7903, reader_cost=0.0001 | ETA 07:31:22 2021-01-18 11:04:10 [INFO] [TRAIN] epoch=9, iter=5740/40000, loss=0.4229, lr=0.008699, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:32:07 2021-01-18 11:04:17 [INFO] [TRAIN] epoch=9, iter=5750/40000, loss=0.6398, lr=0.008697, batch_cost=0.7843, reader_cost=0.0001 | ETA 07:27:41 2021-01-18 11:04:25 [INFO] [TRAIN] epoch=9, iter=5760/40000, loss=0.5977, lr=0.008694, batch_cost=0.7904, reader_cost=0.0001 | ETA 07:31:03 2021-01-18 11:04:33 [INFO] [TRAIN] epoch=9, iter=5770/40000, loss=0.4929, lr=0.008692, batch_cost=0.7915, reader_cost=0.0001 | ETA 07:31:32 2021-01-18 11:04:41 [INFO] [TRAIN] epoch=9, iter=5780/40000, loss=0.5193, lr=0.008690, batch_cost=0.7965, reader_cost=0.0001 | ETA 07:34:16 2021-01-18 11:04:49 [INFO] [TRAIN] epoch=9, iter=5790/40000, loss=0.5622, lr=0.008688, batch_cost=0.7911, reader_cost=0.0001 | ETA 07:31:01 2021-01-18 11:04:57 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.4373, lr=0.008685, batch_cost=0.7875, reader_cost=0.0001 | ETA 07:28:53 2021-01-18 11:05:05 [INFO] [TRAIN] epoch=9, iter=5810/40000, loss=0.6376, lr=0.008683, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:29:46 2021-01-18 11:05:13 [INFO] [TRAIN] epoch=9, iter=5820/40000, loss=0.4696, lr=0.008681, batch_cost=0.7874, reader_cost=0.0001 | ETA 07:28:34 2021-01-18 11:05:21 [INFO] [TRAIN] epoch=9, iter=5830/40000, loss=0.5693, lr=0.008678, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:31:09 2021-01-18 11:05:29 [INFO] [TRAIN] epoch=9, iter=5840/40000, loss=0.3240, lr=0.008676, batch_cost=0.7874, reader_cost=0.0001 | ETA 07:28:18 2021-01-18 11:05:36 [INFO] [TRAIN] epoch=9, iter=5850/40000, loss=0.5600, lr=0.008674, batch_cost=0.7905, reader_cost=0.0001 | ETA 07:29:56 2021-01-18 11:05:44 [INFO] [TRAIN] epoch=9, iter=5860/40000, loss=0.6239, lr=0.008672, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:29:17 2021-01-18 11:05:52 [INFO] [TRAIN] epoch=9, iter=5870/40000, loss=0.4542, lr=0.008669, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:29:01 2021-01-18 11:06:00 [INFO] [TRAIN] epoch=9, iter=5880/40000, loss=0.6031, lr=0.008667, batch_cost=0.7934, reader_cost=0.0001 | ETA 07:31:09 2021-01-18 11:06:08 [INFO] [TRAIN] epoch=9, iter=5890/40000, loss=0.5354, lr=0.008665, batch_cost=0.7911, reader_cost=0.0001 | ETA 07:29:45 2021-01-18 11:06:16 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.4867, lr=0.008662, batch_cost=0.7891, reader_cost=0.0001 | ETA 07:28:29 2021-01-18 11:06:24 [INFO] [TRAIN] epoch=9, iter=5910/40000, loss=0.5131, lr=0.008660, batch_cost=0.7911, reader_cost=0.0001 | ETA 07:29:27 2021-01-18 11:06:32 [INFO] [TRAIN] epoch=9, iter=5920/40000, loss=0.4606, lr=0.008658, batch_cost=0.7927, reader_cost=0.0001 | ETA 07:30:16 2021-01-18 11:06:40 [INFO] [TRAIN] epoch=9, iter=5930/40000, loss=0.4564, lr=0.008656, batch_cost=0.7959, reader_cost=0.0001 | ETA 07:31:55 2021-01-18 11:06:48 [INFO] [TRAIN] epoch=9, iter=5940/40000, loss=0.5361, lr=0.008653, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:29:36 2021-01-18 11:06:55 [INFO] [TRAIN] epoch=10, iter=5950/40000, loss=0.3842, lr=0.008651, batch_cost=0.7758, reader_cost=0.0167 | ETA 07:20:15 2021-01-18 11:07:03 [INFO] [TRAIN] epoch=10, iter=5960/40000, loss=0.5286, lr=0.008649, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:28:58 2021-01-18 11:07:11 [INFO] [TRAIN] epoch=10, iter=5970/40000, loss=0.4061, lr=0.008646, batch_cost=0.7874, reader_cost=0.0001 | ETA 07:26:35 2021-01-18 11:07:19 [INFO] [TRAIN] epoch=10, iter=5980/40000, loss=0.3788, lr=0.008644, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:28:18 2021-01-18 11:07:27 [INFO] [TRAIN] epoch=10, iter=5990/40000, loss=0.4926, lr=0.008642, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:27:46 2021-01-18 11:07:35 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.5387, lr=0.008639, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:29:45 2021-01-18 11:07:35 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 11:08:03 [INFO] [EVAL] #Images=1449 mIoU=0.6839 Acc=0.9273 Kappa=0.8374 2021-01-18 11:08:03 [INFO] [EVAL] Class IoU: [0.9282 0.8796 0.3828 0.676 0.647 0.7619 0.8407 0.7793 0.7683 0.3267 0.6519 0.4869 0.6679 0.7352 0.7914 0.8034 0.5724 0.7249 0.4603 0.7916 0.685 ] 2021-01-18 11:08:03 [INFO] [EVAL] Class Acc: [0.9544 0.9642 0.4108 0.8213 0.7941 0.8896 0.9816 0.8727 0.8473 0.6991 0.904 0.8931 0.7021 0.9514 0.9156 0.8901 0.7571 0.8798 0.747 0.8884 0.9311] 2021-01-18 11:08:13 [INFO] [EVAL] The model with the best validation mIoU (0.6839) was saved at iter 6000. 2021-01-18 11:08:21 [INFO] [TRAIN] epoch=10, iter=6010/40000, loss=0.6744, lr=0.008637, batch_cost=0.7872, reader_cost=0.0001 | ETA 07:25:58 2021-01-18 11:08:29 [INFO] [TRAIN] epoch=10, iter=6020/40000, loss=0.5327, lr=0.008635, batch_cost=0.7924, reader_cost=0.0001 | ETA 07:28:47 2021-01-18 11:08:37 [INFO] [TRAIN] epoch=10, iter=6030/40000, loss=0.5212, lr=0.008633, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:26:12 2021-01-18 11:08:45 [INFO] [TRAIN] epoch=10, iter=6040/40000, loss=0.3677, lr=0.008630, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:26:47 2021-01-18 11:08:52 [INFO] [TRAIN] epoch=10, iter=6050/40000, loss=0.4413, lr=0.008628, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:26:50 2021-01-18 11:09:00 [INFO] [TRAIN] epoch=10, iter=6060/40000, loss=0.4205, lr=0.008626, batch_cost=0.7972, reader_cost=0.0001 | ETA 07:30:55 2021-01-18 11:09:08 [INFO] [TRAIN] epoch=10, iter=6070/40000, loss=0.3974, lr=0.008623, batch_cost=0.7845, reader_cost=0.0001 | ETA 07:23:36 2021-01-18 11:09:16 [INFO] [TRAIN] epoch=10, iter=6080/40000, loss=0.5117, lr=0.008621, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:26:29 2021-01-18 11:09:24 [INFO] [TRAIN] epoch=10, iter=6090/40000, loss=0.6601, lr=0.008619, batch_cost=0.7947, reader_cost=0.0001 | ETA 07:29:08 2021-01-18 11:09:32 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.4283, lr=0.008617, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:26:08 2021-01-18 11:09:40 [INFO] [TRAIN] epoch=10, iter=6110/40000, loss=0.5330, lr=0.008614, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:27:14 2021-01-18 11:09:48 [INFO] [TRAIN] epoch=10, iter=6120/40000, loss=0.5520, lr=0.008612, batch_cost=0.7851, reader_cost=0.0001 | ETA 07:23:18 2021-01-18 11:09:56 [INFO] [TRAIN] epoch=10, iter=6130/40000, loss=0.4056, lr=0.008610, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:27:24 2021-01-18 11:10:04 [INFO] [TRAIN] epoch=10, iter=6140/40000, loss=0.2706, lr=0.008607, batch_cost=0.7948, reader_cost=0.0001 | ETA 07:28:31 2021-01-18 11:10:12 [INFO] [TRAIN] epoch=10, iter=6150/40000, loss=0.4204, lr=0.008605, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:24:38 2021-01-18 11:10:19 [INFO] [TRAIN] epoch=10, iter=6160/40000, loss=0.4397, lr=0.008603, batch_cost=0.7905, reader_cost=0.0001 | ETA 07:25:50 2021-01-18 11:10:27 [INFO] [TRAIN] epoch=10, iter=6170/40000, loss=0.3683, lr=0.008601, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:24:48 2021-01-18 11:10:35 [INFO] [TRAIN] epoch=10, iter=6180/40000, loss=0.3921, lr=0.008598, batch_cost=0.7908, reader_cost=0.0001 | ETA 07:25:44 2021-01-18 11:10:43 [INFO] [TRAIN] epoch=10, iter=6190/40000, loss=0.3569, lr=0.008596, batch_cost=0.7925, reader_cost=0.0001 | ETA 07:26:32 2021-01-18 11:10:51 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.3098, lr=0.008594, batch_cost=0.7912, reader_cost=0.0001 | ETA 07:25:41 2021-01-18 11:10:59 [INFO] [TRAIN] epoch=10, iter=6210/40000, loss=0.3202, lr=0.008591, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:24:16 2021-01-18 11:11:07 [INFO] [TRAIN] epoch=10, iter=6220/40000, loss=0.3497, lr=0.008589, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:25:53 2021-01-18 11:11:15 [INFO] [TRAIN] epoch=10, iter=6230/40000, loss=0.4327, lr=0.008587, batch_cost=0.7862, reader_cost=0.0001 | ETA 07:22:28 2021-01-18 11:11:23 [INFO] [TRAIN] epoch=10, iter=6240/40000, loss=0.4619, lr=0.008585, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:23:54 2021-01-18 11:11:31 [INFO] [TRAIN] epoch=10, iter=6250/40000, loss=0.5389, lr=0.008582, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:23:36 2021-01-18 11:11:38 [INFO] [TRAIN] epoch=10, iter=6260/40000, loss=0.4990, lr=0.008580, batch_cost=0.7863, reader_cost=0.0001 | ETA 07:22:09 2021-01-18 11:11:46 [INFO] [TRAIN] epoch=10, iter=6270/40000, loss=0.3860, lr=0.008578, batch_cost=0.7944, reader_cost=0.0001 | ETA 07:26:36 2021-01-18 11:11:54 [INFO] [TRAIN] epoch=10, iter=6280/40000, loss=0.3259, lr=0.008575, batch_cost=0.7949, reader_cost=0.0001 | ETA 07:26:45 2021-01-18 11:12:02 [INFO] [TRAIN] epoch=10, iter=6290/40000, loss=0.4369, lr=0.008573, batch_cost=0.7911, reader_cost=0.0001 | ETA 07:24:29 2021-01-18 11:12:10 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.6039, lr=0.008571, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:23:21 2021-01-18 11:12:18 [INFO] [TRAIN] epoch=10, iter=6310/40000, loss=0.5461, lr=0.008569, batch_cost=0.7903, reader_cost=0.0001 | ETA 07:23:46 2021-01-18 11:12:26 [INFO] [TRAIN] epoch=10, iter=6320/40000, loss=0.4776, lr=0.008566, batch_cost=0.7892, reader_cost=0.0001 | ETA 07:23:00 2021-01-18 11:12:34 [INFO] [TRAIN] epoch=10, iter=6330/40000, loss=0.5602, lr=0.008564, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:23:11 2021-01-18 11:12:42 [INFO] [TRAIN] epoch=10, iter=6340/40000, loss=0.6141, lr=0.008562, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:25:59 2021-01-18 11:12:50 [INFO] [TRAIN] epoch=10, iter=6350/40000, loss=0.5117, lr=0.008559, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:22:09 2021-01-18 11:12:58 [INFO] [TRAIN] epoch=10, iter=6360/40000, loss=0.3570, lr=0.008557, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:24:08 2021-01-18 11:13:05 [INFO] [TRAIN] epoch=10, iter=6370/40000, loss=0.6509, lr=0.008555, batch_cost=0.7913, reader_cost=0.0001 | ETA 07:23:29 2021-01-18 11:13:13 [INFO] [TRAIN] epoch=10, iter=6380/40000, loss=0.4273, lr=0.008553, batch_cost=0.7836, reader_cost=0.0001 | ETA 07:19:05 2021-01-18 11:13:21 [INFO] [TRAIN] epoch=10, iter=6390/40000, loss=0.5663, lr=0.008550, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:21:44 2021-01-18 11:13:29 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.6430, lr=0.008548, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:21:31 2021-01-18 11:13:37 [INFO] [TRAIN] epoch=10, iter=6410/40000, loss=0.6291, lr=0.008546, batch_cost=0.7852, reader_cost=0.0001 | ETA 07:19:36 2021-01-18 11:13:45 [INFO] [TRAIN] epoch=10, iter=6420/40000, loss=0.5940, lr=0.008543, batch_cost=0.7953, reader_cost=0.0001 | ETA 07:25:05 2021-01-18 11:13:53 [INFO] [TRAIN] epoch=10, iter=6430/40000, loss=0.5525, lr=0.008541, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:23:27 2021-01-18 11:14:01 [INFO] [TRAIN] epoch=10, iter=6440/40000, loss=0.5986, lr=0.008539, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:23:04 2021-01-18 11:14:09 [INFO] [TRAIN] epoch=10, iter=6450/40000, loss=0.6934, lr=0.008537, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:21:37 2021-01-18 11:14:17 [INFO] [TRAIN] epoch=10, iter=6460/40000, loss=0.5079, lr=0.008534, batch_cost=0.7923, reader_cost=0.0001 | ETA 07:22:52 2021-01-18 11:14:24 [INFO] [TRAIN] epoch=10, iter=6470/40000, loss=0.9491, lr=0.008532, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:24:00 2021-01-18 11:14:32 [INFO] [TRAIN] epoch=10, iter=6480/40000, loss=0.7166, lr=0.008530, batch_cost=0.7919, reader_cost=0.0001 | ETA 07:22:24 2021-01-18 11:14:40 [INFO] [TRAIN] epoch=10, iter=6490/40000, loss=0.4587, lr=0.008527, batch_cost=0.7924, reader_cost=0.0001 | ETA 07:22:32 2021-01-18 11:14:48 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.6307, lr=0.008525, batch_cost=0.7956, reader_cost=0.0001 | ETA 07:24:13 2021-01-18 11:14:56 [INFO] [TRAIN] epoch=10, iter=6510/40000, loss=0.7162, lr=0.008523, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:21:21 2021-01-18 11:15:04 [INFO] [TRAIN] epoch=10, iter=6520/40000, loss=0.5579, lr=0.008520, batch_cost=0.7871, reader_cost=0.0001 | ETA 07:19:11 2021-01-18 11:15:12 [INFO] [TRAIN] epoch=10, iter=6530/40000, loss=0.2887, lr=0.008518, batch_cost=0.7866, reader_cost=0.0001 | ETA 07:18:48 2021-01-18 11:15:20 [INFO] [TRAIN] epoch=10, iter=6540/40000, loss=0.6122, lr=0.008516, batch_cost=0.7901, reader_cost=0.0001 | ETA 07:20:37 2021-01-18 11:15:28 [INFO] [TRAIN] epoch=10, iter=6550/40000, loss=0.5387, lr=0.008514, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:22:28 2021-01-18 11:15:36 [INFO] [TRAIN] epoch=10, iter=6560/40000, loss=0.7115, lr=0.008511, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:20:19 2021-01-18 11:15:44 [INFO] [TRAIN] epoch=10, iter=6570/40000, loss=0.5537, lr=0.008509, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:19:16 2021-01-18 11:15:51 [INFO] [TRAIN] epoch=10, iter=6580/40000, loss=0.3547, lr=0.008507, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:19:51 2021-01-18 11:15:59 [INFO] [TRAIN] epoch=10, iter=6590/40000, loss=0.7615, lr=0.008504, batch_cost=0.7913, reader_cost=0.0001 | ETA 07:20:35 2021-01-18 11:16:07 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.5770, lr=0.008502, batch_cost=0.7943, reader_cost=0.0001 | ETA 07:22:08 2021-01-18 11:16:15 [INFO] [TRAIN] epoch=10, iter=6610/40000, loss=0.4929, lr=0.008500, batch_cost=0.7601, reader_cost=0.0001 | ETA 07:03:00 2021-01-18 11:16:23 [INFO] [TRAIN] epoch=11, iter=6620/40000, loss=0.4756, lr=0.008498, batch_cost=0.8067, reader_cost=0.0156 | ETA 07:28:47 2021-01-18 11:16:31 [INFO] [TRAIN] epoch=11, iter=6630/40000, loss=0.4854, lr=0.008495, batch_cost=0.7906, reader_cost=0.0001 | ETA 07:19:43 2021-01-18 11:16:39 [INFO] [TRAIN] epoch=11, iter=6640/40000, loss=0.4226, lr=0.008493, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:19:36 2021-01-18 11:16:47 [INFO] [TRAIN] epoch=11, iter=6650/40000, loss=0.4633, lr=0.008491, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:20:19 2021-01-18 11:16:55 [INFO] [TRAIN] epoch=11, iter=6660/40000, loss=0.5984, lr=0.008488, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:20:45 2021-01-18 11:17:03 [INFO] [TRAIN] epoch=11, iter=6670/40000, loss=0.5985, lr=0.008486, batch_cost=0.7880, reader_cost=0.0001 | ETA 07:17:43 2021-01-18 11:17:10 [INFO] [TRAIN] epoch=11, iter=6680/40000, loss=0.4584, lr=0.008484, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:18:21 2021-01-18 11:17:18 [INFO] [TRAIN] epoch=11, iter=6690/40000, loss=0.5171, lr=0.008482, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:18:11 2021-01-18 11:17:26 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.4591, lr=0.008479, batch_cost=0.7890, reader_cost=0.0001 | ETA 07:17:52 2021-01-18 11:17:34 [INFO] [TRAIN] epoch=11, iter=6710/40000, loss=0.4710, lr=0.008477, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:20:04 2021-01-18 11:17:42 [INFO] [TRAIN] epoch=11, iter=6720/40000, loss=0.3986, lr=0.008475, batch_cost=0.7872, reader_cost=0.0001 | ETA 07:16:36 2021-01-18 11:17:50 [INFO] [TRAIN] epoch=11, iter=6730/40000, loss=0.3511, lr=0.008472, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:19:04 2021-01-18 11:17:58 [INFO] [TRAIN] epoch=11, iter=6740/40000, loss=0.4191, lr=0.008470, batch_cost=0.7856, reader_cost=0.0001 | ETA 07:15:29 2021-01-18 11:18:06 [INFO] [TRAIN] epoch=11, iter=6750/40000, loss=0.5006, lr=0.008468, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:17:01 2021-01-18 11:18:14 [INFO] [TRAIN] epoch=11, iter=6760/40000, loss=0.4664, lr=0.008466, batch_cost=0.7903, reader_cost=0.0001 | ETA 07:17:51 2021-01-18 11:18:22 [INFO] [TRAIN] epoch=11, iter=6770/40000, loss=0.7494, lr=0.008463, batch_cost=0.7943, reader_cost=0.0001 | ETA 07:19:53 2021-01-18 11:18:29 [INFO] [TRAIN] epoch=11, iter=6780/40000, loss=0.3485, lr=0.008461, batch_cost=0.7871, reader_cost=0.0001 | ETA 07:15:46 2021-01-18 11:18:37 [INFO] [TRAIN] epoch=11, iter=6790/40000, loss=0.4465, lr=0.008459, batch_cost=0.7873, reader_cost=0.0001 | ETA 07:15:47 2021-01-18 11:18:45 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.4888, lr=0.008456, batch_cost=0.7849, reader_cost=0.0001 | ETA 07:14:17 2021-01-18 11:18:53 [INFO] [TRAIN] epoch=11, iter=6810/40000, loss=0.4620, lr=0.008454, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:18:26 2021-01-18 11:19:01 [INFO] [TRAIN] epoch=11, iter=6820/40000, loss=0.8280, lr=0.008452, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:17:52 2021-01-18 11:19:09 [INFO] [TRAIN] epoch=11, iter=6830/40000, loss=0.6565, lr=0.008449, batch_cost=0.7920, reader_cost=0.0001 | ETA 07:17:50 2021-01-18 11:19:17 [INFO] [TRAIN] epoch=11, iter=6840/40000, loss=0.7458, lr=0.008447, batch_cost=0.7951, reader_cost=0.0001 | ETA 07:19:24 2021-01-18 11:19:25 [INFO] [TRAIN] epoch=11, iter=6850/40000, loss=0.4640, lr=0.008445, batch_cost=0.7863, reader_cost=0.0001 | ETA 07:14:24 2021-01-18 11:19:33 [INFO] [TRAIN] epoch=11, iter=6860/40000, loss=0.5211, lr=0.008443, batch_cost=0.7899, reader_cost=0.0001 | ETA 07:16:17 2021-01-18 11:19:40 [INFO] [TRAIN] epoch=11, iter=6870/40000, loss=0.4866, lr=0.008440, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:15:29 2021-01-18 11:19:48 [INFO] [TRAIN] epoch=11, iter=6880/40000, loss=0.4487, lr=0.008438, batch_cost=0.7866, reader_cost=0.0001 | ETA 07:14:13 2021-01-18 11:19:56 [INFO] [TRAIN] epoch=11, iter=6890/40000, loss=0.3591, lr=0.008436, batch_cost=0.7905, reader_cost=0.0001 | ETA 07:16:14 2021-01-18 11:20:04 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.5214, lr=0.008433, batch_cost=0.7888, reader_cost=0.0001 | ETA 07:15:09 2021-01-18 11:20:12 [INFO] [TRAIN] epoch=11, iter=6910/40000, loss=0.3613, lr=0.008431, batch_cost=0.7881, reader_cost=0.0001 | ETA 07:14:37 2021-01-18 11:20:20 [INFO] [TRAIN] epoch=11, iter=6920/40000, loss=0.4694, lr=0.008429, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:17:20 2021-01-18 11:20:28 [INFO] [TRAIN] epoch=11, iter=6930/40000, loss=0.4643, lr=0.008427, batch_cost=0.7873, reader_cost=0.0001 | ETA 07:13:55 2021-01-18 11:20:36 [INFO] [TRAIN] epoch=11, iter=6940/40000, loss=0.5425, lr=0.008424, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:17:18 2021-01-18 11:20:44 [INFO] [TRAIN] epoch=11, iter=6950/40000, loss=0.4034, lr=0.008422, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:15:03 2021-01-18 11:20:52 [INFO] [TRAIN] epoch=11, iter=6960/40000, loss=0.5179, lr=0.008420, batch_cost=0.7902, reader_cost=0.0001 | ETA 07:15:08 2021-01-18 11:20:59 [INFO] [TRAIN] epoch=11, iter=6970/40000, loss=0.3573, lr=0.008417, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:14:44 2021-01-18 11:21:07 [INFO] [TRAIN] epoch=11, iter=6980/40000, loss=0.3900, lr=0.008415, batch_cost=0.7889, reader_cost=0.0001 | ETA 07:14:09 2021-01-18 11:21:15 [INFO] [TRAIN] epoch=11, iter=6990/40000, loss=0.4428, lr=0.008413, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:16:48 2021-01-18 11:21:23 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.2856, lr=0.008410, batch_cost=0.7880, reader_cost=0.0001 | ETA 07:13:23 2021-01-18 11:21:23 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 11:21:52 [INFO] [EVAL] #Images=1449 mIoU=0.6733 Acc=0.9241 Kappa=0.8339 2021-01-18 11:21:52 [INFO] [EVAL] Class IoU: [0.9281 0.8394 0.3678 0.7912 0.5928 0.6942 0.9343 0.8091 0.8125 0.3618 0.6007 0.5922 0.5959 0.5513 0.7307 0.8115 0.5244 0.7438 0.3692 0.8183 0.6703] 2021-01-18 11:21:52 [INFO] [EVAL] Class Acc: [0.9628 0.8726 0.378 0.9353 0.7196 0.755 0.9698 0.8994 0.8963 0.4749 0.6244 0.8003 0.6925 0.9427 0.9142 0.8985 0.8085 0.8119 0.8456 0.9571 0.851 ] 2021-01-18 11:21:58 [INFO] [EVAL] The model with the best validation mIoU (0.6839) was saved at iter 6000. 2021-01-18 11:22:06 [INFO] [TRAIN] epoch=11, iter=7010/40000, loss=0.5573, lr=0.008408, batch_cost=0.7851, reader_cost=0.0001 | ETA 07:11:39 2021-01-18 11:22:13 [INFO] [TRAIN] epoch=11, iter=7020/40000, loss=0.5205, lr=0.008406, batch_cost=0.7863, reader_cost=0.0001 | ETA 07:12:10 2021-01-18 11:22:21 [INFO] [TRAIN] epoch=11, iter=7030/40000, loss=0.4596, lr=0.008404, batch_cost=0.7899, reader_cost=0.0001 | ETA 07:14:02 2021-01-18 11:22:29 [INFO] [TRAIN] epoch=11, iter=7040/40000, loss=0.3811, lr=0.008401, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:13:53 2021-01-18 11:22:37 [INFO] [TRAIN] epoch=11, iter=7050/40000, loss=0.4699, lr=0.008399, batch_cost=0.7935, reader_cost=0.0001 | ETA 07:15:45 2021-01-18 11:22:45 [INFO] [TRAIN] epoch=11, iter=7060/40000, loss=0.4822, lr=0.008397, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:15:21 2021-01-18 11:22:53 [INFO] [TRAIN] epoch=11, iter=7070/40000, loss=0.4059, lr=0.008394, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:14:34 2021-01-18 11:23:01 [INFO] [TRAIN] epoch=11, iter=7080/40000, loss=0.5467, lr=0.008392, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:14:21 2021-01-18 11:23:09 [INFO] [TRAIN] epoch=11, iter=7090/40000, loss=0.5652, lr=0.008390, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:13:11 2021-01-18 11:23:17 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.4293, lr=0.008388, batch_cost=0.7855, reader_cost=0.0001 | ETA 07:10:44 2021-01-18 11:23:25 [INFO] [TRAIN] epoch=11, iter=7110/40000, loss=0.3483, lr=0.008385, batch_cost=0.7908, reader_cost=0.0001 | ETA 07:13:28 2021-01-18 11:23:32 [INFO] [TRAIN] epoch=11, iter=7120/40000, loss=0.3761, lr=0.008383, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:12:13 2021-01-18 11:23:40 [INFO] [TRAIN] epoch=11, iter=7130/40000, loss=0.5409, lr=0.008381, batch_cost=0.7937, reader_cost=0.0001 | ETA 07:14:48 2021-01-18 11:23:48 [INFO] [TRAIN] epoch=11, iter=7140/40000, loss=0.4338, lr=0.008378, batch_cost=0.7858, reader_cost=0.0001 | ETA 07:10:21 2021-01-18 11:23:56 [INFO] [TRAIN] epoch=11, iter=7150/40000, loss=0.4840, lr=0.008376, batch_cost=0.7912, reader_cost=0.0001 | ETA 07:13:11 2021-01-18 11:24:04 [INFO] [TRAIN] epoch=11, iter=7160/40000, loss=0.5366, lr=0.008374, batch_cost=0.7970, reader_cost=0.0001 | ETA 07:16:12 2021-01-18 11:24:12 [INFO] [TRAIN] epoch=11, iter=7170/40000, loss=0.6679, lr=0.008371, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:15:00 2021-01-18 11:24:20 [INFO] [TRAIN] epoch=11, iter=7180/40000, loss=0.5110, lr=0.008369, batch_cost=0.7894, reader_cost=0.0001 | ETA 07:11:48 2021-01-18 11:24:28 [INFO] [TRAIN] epoch=11, iter=7190/40000, loss=0.4507, lr=0.008367, batch_cost=0.7936, reader_cost=0.0001 | ETA 07:13:58 2021-01-18 11:24:36 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.4245, lr=0.008365, batch_cost=0.7856, reader_cost=0.0001 | ETA 07:09:27 2021-01-18 11:24:44 [INFO] [TRAIN] epoch=11, iter=7210/40000, loss=0.3715, lr=0.008362, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:13:08 2021-01-18 11:24:52 [INFO] [TRAIN] epoch=11, iter=7220/40000, loss=0.4815, lr=0.008360, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:12:30 2021-01-18 11:25:00 [INFO] [TRAIN] epoch=11, iter=7230/40000, loss=0.6322, lr=0.008358, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:14:12 2021-01-18 11:25:08 [INFO] [TRAIN] epoch=11, iter=7240/40000, loss=0.6969, lr=0.008355, batch_cost=0.7965, reader_cost=0.0001 | ETA 07:14:53 2021-01-18 11:25:15 [INFO] [TRAIN] epoch=11, iter=7250/40000, loss=0.4526, lr=0.008353, batch_cost=0.7906, reader_cost=0.0001 | ETA 07:11:32 2021-01-18 11:25:23 [INFO] [TRAIN] epoch=11, iter=7260/40000, loss=0.4199, lr=0.008351, batch_cost=0.7905, reader_cost=0.0001 | ETA 07:11:20 2021-01-18 11:25:31 [INFO] [TRAIN] epoch=11, iter=7270/40000, loss=0.6356, lr=0.008349, batch_cost=0.7636, reader_cost=0.0001 | ETA 06:56:32 2021-01-18 11:25:39 [INFO] [TRAIN] epoch=12, iter=7280/40000, loss=0.4509, lr=0.008346, batch_cost=0.8009, reader_cost=0.0168 | ETA 07:16:46 2021-01-18 11:25:47 [INFO] [TRAIN] epoch=12, iter=7290/40000, loss=0.3456, lr=0.008344, batch_cost=0.7936, reader_cost=0.0001 | ETA 07:12:39 2021-01-18 11:25:55 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.3464, lr=0.008342, batch_cost=0.7973, reader_cost=0.0001 | ETA 07:14:31 2021-01-18 11:26:03 [INFO] [TRAIN] epoch=12, iter=7310/40000, loss=0.4828, lr=0.008339, batch_cost=0.7895, reader_cost=0.0001 | ETA 07:10:09 2021-01-18 11:26:11 [INFO] [TRAIN] epoch=12, iter=7320/40000, loss=0.3755, lr=0.008337, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:09:34 2021-01-18 11:26:19 [INFO] [TRAIN] epoch=12, iter=7330/40000, loss=0.3053, lr=0.008335, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:11:21 2021-01-18 11:26:27 [INFO] [TRAIN] epoch=12, iter=7340/40000, loss=0.3648, lr=0.008332, batch_cost=0.7910, reader_cost=0.0001 | ETA 07:10:33 2021-01-18 11:26:34 [INFO] [TRAIN] epoch=12, iter=7350/40000, loss=0.4484, lr=0.008330, batch_cost=0.7885, reader_cost=0.0001 | ETA 07:09:04 2021-01-18 11:26:42 [INFO] [TRAIN] epoch=12, iter=7360/40000, loss=0.4740, lr=0.008328, batch_cost=0.7906, reader_cost=0.0001 | ETA 07:10:03 2021-01-18 11:26:50 [INFO] [TRAIN] epoch=12, iter=7370/40000, loss=0.3792, lr=0.008326, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:08:55 2021-01-18 11:26:58 [INFO] [TRAIN] epoch=12, iter=7380/40000, loss=0.3780, lr=0.008323, batch_cost=0.7878, reader_cost=0.0001 | ETA 07:08:17 2021-01-18 11:27:06 [INFO] [TRAIN] epoch=12, iter=7390/40000, loss=0.3267, lr=0.008321, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:09:09 2021-01-18 11:27:14 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.3817, lr=0.008319, batch_cost=0.7899, reader_cost=0.0001 | ETA 07:09:11 2021-01-18 11:27:22 [INFO] [TRAIN] epoch=12, iter=7410/40000, loss=0.5094, lr=0.008316, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:08:24 2021-01-18 11:27:30 [INFO] [TRAIN] epoch=12, iter=7420/40000, loss=0.4013, lr=0.008314, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:08:34 2021-01-18 11:27:38 [INFO] [TRAIN] epoch=12, iter=7430/40000, loss=0.5434, lr=0.008312, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:07:57 2021-01-18 11:27:46 [INFO] [TRAIN] epoch=12, iter=7440/40000, loss=0.4809, lr=0.008309, batch_cost=0.7990, reader_cost=0.0001 | ETA 07:13:34 2021-01-18 11:27:53 [INFO] [TRAIN] epoch=12, iter=7450/40000, loss=0.4928, lr=0.008307, batch_cost=0.7898, reader_cost=0.0001 | ETA 07:08:28 2021-01-18 11:28:01 [INFO] [TRAIN] epoch=12, iter=7460/40000, loss=0.5593, lr=0.008305, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:07:36 2021-01-18 11:28:09 [INFO] [TRAIN] epoch=12, iter=7470/40000, loss=0.4458, lr=0.008303, batch_cost=0.7867, reader_cost=0.0001 | ETA 07:06:30 2021-01-18 11:28:17 [INFO] [TRAIN] epoch=12, iter=7480/40000, loss=0.5483, lr=0.008300, batch_cost=0.7929, reader_cost=0.0001 | ETA 07:09:45 2021-01-18 11:28:25 [INFO] [TRAIN] epoch=12, iter=7490/40000, loss=0.3417, lr=0.008298, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:09:40 2021-01-18 11:28:33 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.5894, lr=0.008296, batch_cost=0.7906, reader_cost=0.0001 | ETA 07:08:15 2021-01-18 11:28:41 [INFO] [TRAIN] epoch=12, iter=7510/40000, loss=0.3890, lr=0.008293, batch_cost=0.7888, reader_cost=0.0001 | ETA 07:07:07 2021-01-18 11:28:49 [INFO] [TRAIN] epoch=12, iter=7520/40000, loss=0.3509, lr=0.008291, batch_cost=0.7938, reader_cost=0.0001 | ETA 07:09:41 2021-01-18 11:28:57 [INFO] [TRAIN] epoch=12, iter=7530/40000, loss=0.6258, lr=0.008289, batch_cost=0.7941, reader_cost=0.0001 | ETA 07:09:44 2021-01-18 11:29:05 [INFO] [TRAIN] epoch=12, iter=7540/40000, loss=0.4743, lr=0.008287, batch_cost=0.7856, reader_cost=0.0001 | ETA 07:05:00 2021-01-18 11:29:12 [INFO] [TRAIN] epoch=12, iter=7550/40000, loss=0.4881, lr=0.008284, batch_cost=0.7885, reader_cost=0.0001 | ETA 07:06:26 2021-01-18 11:29:20 [INFO] [TRAIN] epoch=12, iter=7560/40000, loss=0.3682, lr=0.008282, batch_cost=0.7886, reader_cost=0.0001 | ETA 07:06:21 2021-01-18 11:29:28 [INFO] [TRAIN] epoch=12, iter=7570/40000, loss=0.3932, lr=0.008280, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:09:42 2021-01-18 11:29:36 [INFO] [TRAIN] epoch=12, iter=7580/40000, loss=0.5956, lr=0.008277, batch_cost=0.7941, reader_cost=0.0001 | ETA 07:09:05 2021-01-18 11:29:44 [INFO] [TRAIN] epoch=12, iter=7590/40000, loss=0.4955, lr=0.008275, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:08:27 2021-01-18 11:29:52 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.5438, lr=0.008273, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:05:43 2021-01-18 11:30:00 [INFO] [TRAIN] epoch=12, iter=7610/40000, loss=0.4146, lr=0.008270, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:07:38 2021-01-18 11:30:08 [INFO] [TRAIN] epoch=12, iter=7620/40000, loss=0.4039, lr=0.008268, batch_cost=0.7877, reader_cost=0.0001 | ETA 07:05:04 2021-01-18 11:30:16 [INFO] [TRAIN] epoch=12, iter=7630/40000, loss=0.4659, lr=0.008266, batch_cost=0.7896, reader_cost=0.0001 | ETA 07:06:00 2021-01-18 11:30:24 [INFO] [TRAIN] epoch=12, iter=7640/40000, loss=0.5218, lr=0.008264, batch_cost=0.7902, reader_cost=0.0001 | ETA 07:06:12 2021-01-18 11:30:32 [INFO] [TRAIN] epoch=12, iter=7650/40000, loss=0.4192, lr=0.008261, batch_cost=0.7897, reader_cost=0.0001 | ETA 07:05:47 2021-01-18 11:30:40 [INFO] [TRAIN] epoch=12, iter=7660/40000, loss=0.3712, lr=0.008259, batch_cost=0.7975, reader_cost=0.0001 | ETA 07:09:51 2021-01-18 11:30:47 [INFO] [TRAIN] epoch=12, iter=7670/40000, loss=0.4128, lr=0.008257, batch_cost=0.7873, reader_cost=0.0001 | ETA 07:04:13 2021-01-18 11:30:55 [INFO] [TRAIN] epoch=12, iter=7680/40000, loss=0.5290, lr=0.008254, batch_cost=0.7891, reader_cost=0.0001 | ETA 07:05:05 2021-01-18 11:31:03 [INFO] [TRAIN] epoch=12, iter=7690/40000, loss=0.5283, lr=0.008252, batch_cost=0.7960, reader_cost=0.0001 | ETA 07:08:38 2021-01-18 11:31:11 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.5262, lr=0.008250, batch_cost=0.7918, reader_cost=0.0001 | ETA 07:06:14 2021-01-18 11:31:19 [INFO] [TRAIN] epoch=12, iter=7710/40000, loss=0.4761, lr=0.008247, batch_cost=0.7900, reader_cost=0.0001 | ETA 07:05:08 2021-01-18 11:31:27 [INFO] [TRAIN] epoch=12, iter=7720/40000, loss=0.5852, lr=0.008245, batch_cost=0.7864, reader_cost=0.0001 | ETA 07:03:04 2021-01-18 11:31:35 [INFO] [TRAIN] epoch=12, iter=7730/40000, loss=0.7166, lr=0.008243, batch_cost=0.7930, reader_cost=0.0001 | ETA 07:06:30 2021-01-18 11:31:43 [INFO] [TRAIN] epoch=12, iter=7740/40000, loss=0.3430, lr=0.008241, batch_cost=0.7979, reader_cost=0.0001 | ETA 07:09:00 2021-01-18 11:31:51 [INFO] [TRAIN] epoch=12, iter=7750/40000, loss=0.3568, lr=0.008238, batch_cost=0.7914, reader_cost=0.0001 | ETA 07:05:21 2021-01-18 11:31:59 [INFO] [TRAIN] epoch=12, iter=7760/40000, loss=0.3427, lr=0.008236, batch_cost=0.7910, reader_cost=0.0001 | ETA 07:05:00 2021-01-18 11:32:07 [INFO] [TRAIN] epoch=12, iter=7770/40000, loss=0.4049, lr=0.008234, batch_cost=0.7936, reader_cost=0.0001 | ETA 07:06:17 2021-01-18 11:32:15 [INFO] [TRAIN] epoch=12, iter=7780/40000, loss=0.4528, lr=0.008231, batch_cost=0.7888, reader_cost=0.0001 | ETA 07:03:35 2021-01-18 11:32:22 [INFO] [TRAIN] epoch=12, iter=7790/40000, loss=0.3941, lr=0.008229, batch_cost=0.7883, reader_cost=0.0001 | ETA 07:03:11 2021-01-18 11:32:30 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.3688, lr=0.008227, batch_cost=0.7934, reader_cost=0.0001 | ETA 07:05:46 2021-01-18 11:32:38 [INFO] [TRAIN] epoch=12, iter=7810/40000, loss=0.3942, lr=0.008224, batch_cost=0.7865, reader_cost=0.0001 | ETA 07:01:58 2021-01-18 11:32:46 [INFO] [TRAIN] epoch=12, iter=7820/40000, loss=0.4311, lr=0.008222, batch_cost=0.7939, reader_cost=0.0001 | ETA 07:05:49 2021-01-18 11:32:54 [INFO] [TRAIN] epoch=12, iter=7830/40000, loss=0.3826, lr=0.008220, batch_cost=0.7917, reader_cost=0.0001 | ETA 07:04:28 2021-01-18 11:33:02 [INFO] [TRAIN] epoch=12, iter=7840/40000, loss=0.4081, lr=0.008218, batch_cost=0.7943, reader_cost=0.0001 | ETA 07:05:45 2021-01-18 11:33:10 [INFO] [TRAIN] epoch=12, iter=7850/40000, loss=0.5719, lr=0.008215, batch_cost=0.7945, reader_cost=0.0001 | ETA 07:05:43 2021-01-18 11:33:18 [INFO] [TRAIN] epoch=12, iter=7860/40000, loss=0.3697, lr=0.008213, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:04:21 2021-01-18 11:33:26 [INFO] [TRAIN] epoch=12, iter=7870/40000, loss=0.3620, lr=0.008211, batch_cost=0.7884, reader_cost=0.0001 | ETA 07:02:10 2021-01-18 11:33:34 [INFO] [TRAIN] epoch=12, iter=7880/40000, loss=0.6043, lr=0.008208, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:02:32 2021-01-18 11:33:42 [INFO] [TRAIN] epoch=12, iter=7890/40000, loss=0.4023, lr=0.008206, batch_cost=0.7887, reader_cost=0.0001 | ETA 07:02:05 2021-01-18 11:33:49 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.3156, lr=0.008204, batch_cost=0.7866, reader_cost=0.0001 | ETA 07:00:48 2021-01-18 11:33:57 [INFO] [TRAIN] epoch=12, iter=7910/40000, loss=0.4049, lr=0.008201, batch_cost=0.7960, reader_cost=0.0001 | ETA 07:05:42 2021-01-18 11:34:05 [INFO] [TRAIN] epoch=12, iter=7920/40000, loss=0.5511, lr=0.008199, batch_cost=0.7950, reader_cost=0.0001 | ETA 07:05:04 2021-01-18 11:34:13 [INFO] [TRAIN] epoch=12, iter=7930/40000, loss=0.3684, lr=0.008197, batch_cost=0.7697, reader_cost=0.0001 | ETA 06:51:25 2021-01-18 11:34:21 [INFO] [TRAIN] epoch=13, iter=7940/40000, loss=0.4899, lr=0.008195, batch_cost=0.7962, reader_cost=0.0172 | ETA 07:05:26 2021-01-18 11:34:29 [INFO] [TRAIN] epoch=13, iter=7950/40000, loss=0.4846, lr=0.008192, batch_cost=0.7923, reader_cost=0.0001 | ETA 07:03:12 2021-01-18 11:34:37 [INFO] [TRAIN] epoch=13, iter=7960/40000, loss=0.4806, lr=0.008190, batch_cost=0.7904, reader_cost=0.0001 | ETA 07:02:03 2021-01-18 11:34:45 [INFO] [TRAIN] epoch=13, iter=7970/40000, loss=0.5217, lr=0.008188, batch_cost=0.7934, reader_cost=0.0001 | ETA 07:03:32 2021-01-18 11:34:53 [INFO] [TRAIN] epoch=13, iter=7980/40000, loss=0.2480, lr=0.008185, batch_cost=0.7929, reader_cost=0.0001 | ETA 07:03:10 2021-01-18 11:35:01 [INFO] [TRAIN] epoch=13, iter=7990/40000, loss=0.3994, lr=0.008183, batch_cost=0.7875, reader_cost=0.0001 | ETA 07:00:07 2021-01-18 11:35:08 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.3537, lr=0.008181, batch_cost=0.7893, reader_cost=0.0001 | ETA 07:00:58 2021-01-18 11:35:08 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 11:35:36 [INFO] [EVAL] #Images=1449 mIoU=0.7119 Acc=0.9347 Kappa=0.8548 2021-01-18 11:35:36 [INFO] [EVAL] Class IoU: [0.9306 0.8563 0.3798 0.7598 0.6103 0.796 0.9038 0.8234 0.8638 0.3233 0.7536 0.607 0.8176 0.8051 0.8147 0.8244 0.5273 0.6408 0.4128 0.8495 0.6508] 2021-01-18 11:35:36 [INFO] [EVAL] Class Acc: [0.9575 0.9286 0.3934 0.8201 0.7321 0.8934 0.9382 0.8867 0.9044 0.5853 0.8703 0.9008 0.9284 0.8707 0.8833 0.9543 0.6656 0.9505 0.72 0.9349 0.7473] 2021-01-18 11:35:46 [INFO] [EVAL] The model with the best validation mIoU (0.7119) was saved at iter 8000. 2021-01-18 11:35:54 [INFO] [TRAIN] epoch=13, iter=8010/40000, loss=0.5438, lr=0.008178, batch_cost=0.7895, reader_cost=0.0001 | ETA 07:00:57 2021-01-18 11:36:02 [INFO] [TRAIN] epoch=13, iter=8020/40000, loss=0.3432, lr=0.008176, batch_cost=0.7874, reader_cost=0.0001 | ETA 06:59:42 2021-01-18 11:36:10 [INFO] [TRAIN] epoch=13, iter=8030/40000, loss=0.4352, lr=0.008174, batch_cost=0.7891, reader_cost=0.0001 | ETA 07:00:28 2021-01-18 11:36:18 [INFO] [TRAIN] epoch=13, iter=8040/40000, loss=0.4393, lr=0.008172, batch_cost=0.7962, reader_cost=0.0001 | ETA 07:04:05 2021-01-18 11:36:26 [INFO] [TRAIN] epoch=13, iter=8050/40000, loss=0.4234, lr=0.008169, batch_cost=0.7922, reader_cost=0.0001 | ETA 07:01:49 2021-01-18 11:36:34 [INFO] [TRAIN] epoch=13, iter=8060/40000, loss=0.3265, lr=0.008167, batch_cost=0.7926, reader_cost=0.0001 | ETA 07:01:57 2021-01-18 11:36:42 [INFO] [TRAIN] epoch=13, iter=8070/40000, loss=0.4683, lr=0.008165, batch_cost=0.7932, reader_cost=0.0001 | ETA 07:02:05 2021-01-18 11:36:50 [INFO] [TRAIN] epoch=13, iter=8080/40000, loss=0.4088, lr=0.008162, batch_cost=0.7910, reader_cost=0.0001 | ETA 07:00:48 2021-01-18 11:36:58 [INFO] [TRAIN] epoch=13, iter=8090/40000, loss=0.5185, lr=0.008160, batch_cost=0.7909, reader_cost=0.0001 | ETA 07:00:37 2021-01-18 11:37:05 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.3946, lr=0.008158, batch_cost=0.7964, reader_cost=0.0001 | ETA 07:03:26 2021-01-18 11:37:13 [INFO] [TRAIN] epoch=13, iter=8110/40000, loss=0.4798, lr=0.008155, batch_cost=0.7871, reader_cost=0.0001 | ETA 06:58:21 2021-01-18 11:37:21 [INFO] [TRAIN] epoch=13, iter=8120/40000, loss=0.5443, lr=0.008153, batch_cost=0.7907, reader_cost=0.0001 | ETA 07:00:08 2021-01-18 11:37:29 [INFO] [TRAIN] epoch=13, iter=8130/40000, loss=0.5030, lr=0.008151, batch_cost=0.7904, reader_cost=0.0001 | ETA 06:59:49 2021-01-18 11:37:37 [INFO] [TRAIN] epoch=13, iter=8140/40000, loss=0.4192, lr=0.008149, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:59:47 2021-01-18 11:37:45 [INFO] [TRAIN] epoch=13, iter=8150/40000, loss=0.4683, lr=0.008146, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:59:25 2021-01-18 11:37:53 [INFO] [TRAIN] epoch=13, iter=8160/40000, loss=0.3642, lr=0.008144, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:58:39 2021-01-18 11:38:01 [INFO] [TRAIN] epoch=13, iter=8170/40000, loss=0.4346, lr=0.008142, batch_cost=0.7870, reader_cost=0.0001 | ETA 06:57:31 2021-01-18 11:38:09 [INFO] [TRAIN] epoch=13, iter=8180/40000, loss=0.2433, lr=0.008139, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:59:52 2021-01-18 11:38:17 [INFO] [TRAIN] epoch=13, iter=8190/40000, loss=0.3361, lr=0.008137, batch_cost=0.7883, reader_cost=0.0001 | ETA 06:57:55 2021-01-18 11:38:24 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.4364, lr=0.008135, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:58:03 2021-01-18 11:38:32 [INFO] [TRAIN] epoch=13, iter=8210/40000, loss=0.4596, lr=0.008132, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:57:17 2021-01-18 11:38:40 [INFO] [TRAIN] epoch=13, iter=8220/40000, loss=0.2489, lr=0.008130, batch_cost=0.7884, reader_cost=0.0001 | ETA 06:57:34 2021-01-18 11:38:48 [INFO] [TRAIN] epoch=13, iter=8230/40000, loss=0.3645, lr=0.008128, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:59:21 2021-01-18 11:38:56 [INFO] [TRAIN] epoch=13, iter=8240/40000, loss=0.4052, lr=0.008126, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:59:06 2021-01-18 11:39:04 [INFO] [TRAIN] epoch=13, iter=8250/40000, loss=0.4973, lr=0.008123, batch_cost=0.7923, reader_cost=0.0001 | ETA 06:59:16 2021-01-18 11:39:12 [INFO] [TRAIN] epoch=13, iter=8260/40000, loss=0.3982, lr=0.008121, batch_cost=0.7873, reader_cost=0.0001 | ETA 06:56:27 2021-01-18 11:39:20 [INFO] [TRAIN] epoch=13, iter=8270/40000, loss=0.4872, lr=0.008119, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:58:43 2021-01-18 11:39:28 [INFO] [TRAIN] epoch=13, iter=8280/40000, loss=0.4777, lr=0.008116, batch_cost=0.7948, reader_cost=0.0001 | ETA 07:00:10 2021-01-18 11:39:36 [INFO] [TRAIN] epoch=13, iter=8290/40000, loss=0.3678, lr=0.008114, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:59:04 2021-01-18 11:39:44 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.2228, lr=0.008112, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:57:25 2021-01-18 11:39:51 [INFO] [TRAIN] epoch=13, iter=8310/40000, loss=0.3941, lr=0.008109, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:57:33 2021-01-18 11:39:59 [INFO] [TRAIN] epoch=13, iter=8320/40000, loss=0.4404, lr=0.008107, batch_cost=0.7945, reader_cost=0.0001 | ETA 06:59:30 2021-01-18 11:40:07 [INFO] [TRAIN] epoch=13, iter=8330/40000, loss=0.4819, lr=0.008105, batch_cost=0.7900, reader_cost=0.0001 | ETA 06:57:00 2021-01-18 11:40:15 [INFO] [TRAIN] epoch=13, iter=8340/40000, loss=0.4319, lr=0.008102, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:55:39 2021-01-18 11:40:23 [INFO] [TRAIN] epoch=13, iter=8350/40000, loss=0.5576, lr=0.008100, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:56:07 2021-01-18 11:40:31 [INFO] [TRAIN] epoch=13, iter=8360/40000, loss=0.4708, lr=0.008098, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:56:29 2021-01-18 11:40:39 [INFO] [TRAIN] epoch=13, iter=8370/40000, loss=0.2719, lr=0.008096, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:57:21 2021-01-18 11:40:47 [INFO] [TRAIN] epoch=13, iter=8380/40000, loss=0.3868, lr=0.008093, batch_cost=0.7948, reader_cost=0.0001 | ETA 06:58:52 2021-01-18 11:40:55 [INFO] [TRAIN] epoch=13, iter=8390/40000, loss=0.6198, lr=0.008091, batch_cost=0.7908, reader_cost=0.0001 | ETA 06:56:38 2021-01-18 11:41:03 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.4672, lr=0.008089, batch_cost=0.7951, reader_cost=0.0001 | ETA 06:58:45 2021-01-18 11:41:11 [INFO] [TRAIN] epoch=13, iter=8410/40000, loss=0.3363, lr=0.008086, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:55:35 2021-01-18 11:41:18 [INFO] [TRAIN] epoch=13, iter=8420/40000, loss=0.4377, lr=0.008084, batch_cost=0.7855, reader_cost=0.0001 | ETA 06:53:27 2021-01-18 11:41:26 [INFO] [TRAIN] epoch=13, iter=8430/40000, loss=0.3873, lr=0.008082, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:55:58 2021-01-18 11:41:34 [INFO] [TRAIN] epoch=13, iter=8440/40000, loss=0.4197, lr=0.008079, batch_cost=0.7880, reader_cost=0.0001 | ETA 06:54:30 2021-01-18 11:41:42 [INFO] [TRAIN] epoch=13, iter=8450/40000, loss=0.3317, lr=0.008077, batch_cost=0.7951, reader_cost=0.0001 | ETA 06:58:05 2021-01-18 11:41:50 [INFO] [TRAIN] epoch=13, iter=8460/40000, loss=0.3038, lr=0.008075, batch_cost=0.7843, reader_cost=0.0001 | ETA 06:52:16 2021-01-18 11:41:58 [INFO] [TRAIN] epoch=13, iter=8470/40000, loss=0.2731, lr=0.008073, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:56:06 2021-01-18 11:42:06 [INFO] [TRAIN] epoch=13, iter=8480/40000, loss=0.5319, lr=0.008070, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:54:24 2021-01-18 11:42:14 [INFO] [TRAIN] epoch=13, iter=8490/40000, loss=0.3193, lr=0.008068, batch_cost=0.7895, reader_cost=0.0001 | ETA 06:54:36 2021-01-18 11:42:22 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.4312, lr=0.008066, batch_cost=0.7895, reader_cost=0.0001 | ETA 06:54:28 2021-01-18 11:42:30 [INFO] [TRAIN] epoch=13, iter=8510/40000, loss=0.4612, lr=0.008063, batch_cost=0.7900, reader_cost=0.0001 | ETA 06:54:36 2021-01-18 11:42:37 [INFO] [TRAIN] epoch=13, iter=8520/40000, loss=0.4490, lr=0.008061, batch_cost=0.7910, reader_cost=0.0001 | ETA 06:55:02 2021-01-18 11:42:45 [INFO] [TRAIN] epoch=13, iter=8530/40000, loss=0.2516, lr=0.008059, batch_cost=0.7862, reader_cost=0.0001 | ETA 06:52:21 2021-01-18 11:42:53 [INFO] [TRAIN] epoch=13, iter=8540/40000, loss=0.4601, lr=0.008056, batch_cost=0.7937, reader_cost=0.0001 | ETA 06:56:09 2021-01-18 11:43:01 [INFO] [TRAIN] epoch=13, iter=8550/40000, loss=0.4497, lr=0.008054, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:53:52 2021-01-18 11:43:09 [INFO] [TRAIN] epoch=13, iter=8560/40000, loss=0.3083, lr=0.008052, batch_cost=0.7897, reader_cost=0.0001 | ETA 06:53:48 2021-01-18 11:43:17 [INFO] [TRAIN] epoch=13, iter=8570/40000, loss=0.4322, lr=0.008049, batch_cost=0.7939, reader_cost=0.0001 | ETA 06:55:51 2021-01-18 11:43:25 [INFO] [TRAIN] epoch=13, iter=8580/40000, loss=0.5073, lr=0.008047, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:52:44 2021-01-18 11:43:33 [INFO] [TRAIN] epoch=13, iter=8590/40000, loss=0.4066, lr=0.008045, batch_cost=0.7772, reader_cost=0.0001 | ETA 06:46:51 2021-01-18 11:43:40 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.7177, lr=0.008043, batch_cost=0.7882, reader_cost=0.0143 | ETA 06:52:28 2021-01-18 11:43:48 [INFO] [TRAIN] epoch=14, iter=8610/40000, loss=0.6604, lr=0.008040, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:53:00 2021-01-18 11:43:56 [INFO] [TRAIN] epoch=14, iter=8620/40000, loss=0.4671, lr=0.008038, batch_cost=0.7904, reader_cost=0.0001 | ETA 06:53:23 2021-01-18 11:44:04 [INFO] [TRAIN] epoch=14, iter=8630/40000, loss=0.4450, lr=0.008036, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:53:35 2021-01-18 11:44:12 [INFO] [TRAIN] epoch=14, iter=8640/40000, loss=0.3420, lr=0.008033, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:52:27 2021-01-18 11:44:20 [INFO] [TRAIN] epoch=14, iter=8650/40000, loss=0.5740, lr=0.008031, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:51:48 2021-01-18 11:44:28 [INFO] [TRAIN] epoch=14, iter=8660/40000, loss=0.3293, lr=0.008029, batch_cost=0.8010, reader_cost=0.0001 | ETA 06:58:23 2021-01-18 11:44:36 [INFO] [TRAIN] epoch=14, iter=8670/40000, loss=0.4526, lr=0.008026, batch_cost=0.7905, reader_cost=0.0001 | ETA 06:52:45 2021-01-18 11:44:44 [INFO] [TRAIN] epoch=14, iter=8680/40000, loss=0.4078, lr=0.008024, batch_cost=0.7891, reader_cost=0.0001 | ETA 06:51:54 2021-01-18 11:44:52 [INFO] [TRAIN] epoch=14, iter=8690/40000, loss=0.3831, lr=0.008022, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:51:01 2021-01-18 11:45:00 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.5328, lr=0.008020, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:51:31 2021-01-18 11:45:07 [INFO] [TRAIN] epoch=14, iter=8710/40000, loss=0.3756, lr=0.008017, batch_cost=0.7947, reader_cost=0.0001 | ETA 06:54:26 2021-01-18 11:45:15 [INFO] [TRAIN] epoch=14, iter=8720/40000, loss=0.4209, lr=0.008015, batch_cost=0.7960, reader_cost=0.0001 | ETA 06:54:59 2021-01-18 11:45:23 [INFO] [TRAIN] epoch=14, iter=8730/40000, loss=0.3433, lr=0.008013, batch_cost=0.7886, reader_cost=0.0001 | ETA 06:51:00 2021-01-18 11:45:31 [INFO] [TRAIN] epoch=14, iter=8740/40000, loss=0.3200, lr=0.008010, batch_cost=0.7875, reader_cost=0.0001 | ETA 06:50:16 2021-01-18 11:45:39 [INFO] [TRAIN] epoch=14, iter=8750/40000, loss=0.5376, lr=0.008008, batch_cost=0.7952, reader_cost=0.0001 | ETA 06:54:09 2021-01-18 11:45:47 [INFO] [TRAIN] epoch=14, iter=8760/40000, loss=0.3612, lr=0.008006, batch_cost=0.7857, reader_cost=0.0001 | ETA 06:49:04 2021-01-18 11:45:55 [INFO] [TRAIN] epoch=14, iter=8770/40000, loss=0.4012, lr=0.008003, batch_cost=0.7915, reader_cost=0.0001 | ETA 06:51:58 2021-01-18 11:46:03 [INFO] [TRAIN] epoch=14, iter=8780/40000, loss=0.3150, lr=0.008001, batch_cost=0.7908, reader_cost=0.0001 | ETA 06:51:27 2021-01-18 11:46:11 [INFO] [TRAIN] epoch=14, iter=8790/40000, loss=0.2796, lr=0.007999, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:49:44 2021-01-18 11:46:19 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.2998, lr=0.007996, batch_cost=0.7910, reader_cost=0.0001 | ETA 06:51:19 2021-01-18 11:46:27 [INFO] [TRAIN] epoch=14, iter=8810/40000, loss=0.3991, lr=0.007994, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:50:22 2021-01-18 11:46:34 [INFO] [TRAIN] epoch=14, iter=8820/40000, loss=0.2817, lr=0.007992, batch_cost=0.7947, reader_cost=0.0001 | ETA 06:52:59 2021-01-18 11:46:42 [INFO] [TRAIN] epoch=14, iter=8830/40000, loss=0.5382, lr=0.007990, batch_cost=0.7912, reader_cost=0.0001 | ETA 06:51:02 2021-01-18 11:46:50 [INFO] [TRAIN] epoch=14, iter=8840/40000, loss=0.3076, lr=0.007987, batch_cost=0.7947, reader_cost=0.0001 | ETA 06:52:43 2021-01-18 11:46:58 [INFO] [TRAIN] epoch=14, iter=8850/40000, loss=0.3076, lr=0.007985, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:51:11 2021-01-18 11:47:06 [INFO] [TRAIN] epoch=14, iter=8860/40000, loss=0.3822, lr=0.007983, batch_cost=0.7926, reader_cost=0.0001 | ETA 06:51:22 2021-01-18 11:47:14 [INFO] [TRAIN] epoch=14, iter=8870/40000, loss=0.2368, lr=0.007980, batch_cost=0.7952, reader_cost=0.0001 | ETA 06:52:34 2021-01-18 11:47:22 [INFO] [TRAIN] epoch=14, iter=8880/40000, loss=0.3501, lr=0.007978, batch_cost=0.7866, reader_cost=0.0001 | ETA 06:47:58 2021-01-18 11:47:30 [INFO] [TRAIN] epoch=14, iter=8890/40000, loss=0.5774, lr=0.007976, batch_cost=0.7948, reader_cost=0.0001 | ETA 06:52:05 2021-01-18 11:47:38 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.6778, lr=0.007973, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:48:20 2021-01-18 11:47:46 [INFO] [TRAIN] epoch=14, iter=8910/40000, loss=0.3633, lr=0.007971, batch_cost=0.7924, reader_cost=0.0001 | ETA 06:50:35 2021-01-18 11:47:54 [INFO] [TRAIN] epoch=14, iter=8920/40000, loss=0.6368, lr=0.007969, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:50:14 2021-01-18 11:48:02 [INFO] [TRAIN] epoch=14, iter=8930/40000, loss=0.3698, lr=0.007966, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:47:53 2021-01-18 11:48:09 [INFO] [TRAIN] epoch=14, iter=8940/40000, loss=0.4905, lr=0.007964, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:48:39 2021-01-18 11:48:17 [INFO] [TRAIN] epoch=14, iter=8950/40000, loss=0.6375, lr=0.007962, batch_cost=0.7923, reader_cost=0.0001 | ETA 06:50:01 2021-01-18 11:48:25 [INFO] [TRAIN] epoch=14, iter=8960/40000, loss=0.4119, lr=0.007960, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:48:21 2021-01-18 11:48:33 [INFO] [TRAIN] epoch=14, iter=8970/40000, loss=0.4621, lr=0.007957, batch_cost=0.7921, reader_cost=0.0001 | ETA 06:49:40 2021-01-18 11:48:41 [INFO] [TRAIN] epoch=14, iter=8980/40000, loss=0.3951, lr=0.007955, batch_cost=0.7916, reader_cost=0.0001 | ETA 06:49:15 2021-01-18 11:48:49 [INFO] [TRAIN] epoch=14, iter=8990/40000, loss=0.3306, lr=0.007953, batch_cost=0.7936, reader_cost=0.0001 | ETA 06:50:09 2021-01-18 11:48:57 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.3887, lr=0.007950, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:48:07 2021-01-18 11:48:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 11:49:24 [INFO] [EVAL] #Images=1449 mIoU=0.7388 Acc=0.9393 Kappa=0.8673 2021-01-18 11:49:24 [INFO] [EVAL] Class IoU: [0.9355 0.8811 0.4022 0.8701 0.6956 0.7526 0.9243 0.8169 0.8382 0.3629 0.872 0.5665 0.7881 0.8221 0.7656 0.8499 0.5806 0.8202 0.3925 0.8894 0.6892] 2021-01-18 11:49:24 [INFO] [EVAL] Class Acc: [0.9672 0.9228 0.4218 0.9234 0.7493 0.8982 0.9478 0.9408 0.8653 0.5036 0.9546 0.8388 0.9315 0.8484 0.8007 0.9046 0.7379 0.893 0.7869 0.9631 0.7896] 2021-01-18 11:49:35 [INFO] [EVAL] The model with the best validation mIoU (0.7388) was saved at iter 9000. 2021-01-18 11:49:43 [INFO] [TRAIN] epoch=14, iter=9010/40000, loss=0.3953, lr=0.007948, batch_cost=0.7804, reader_cost=0.0001 | ETA 06:43:05 2021-01-18 11:49:51 [INFO] [TRAIN] epoch=14, iter=9020/40000, loss=0.3190, lr=0.007946, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:47:14 2021-01-18 11:49:58 [INFO] [TRAIN] epoch=14, iter=9030/40000, loss=0.4425, lr=0.007943, batch_cost=0.7946, reader_cost=0.0001 | ETA 06:50:09 2021-01-18 11:50:06 [INFO] [TRAIN] epoch=14, iter=9040/40000, loss=0.3961, lr=0.007941, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:46:30 2021-01-18 11:50:14 [INFO] [TRAIN] epoch=14, iter=9050/40000, loss=0.4156, lr=0.007939, batch_cost=0.7915, reader_cost=0.0001 | ETA 06:48:17 2021-01-18 11:50:22 [INFO] [TRAIN] epoch=14, iter=9060/40000, loss=0.5169, lr=0.007936, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:47:28 2021-01-18 11:50:30 [INFO] [TRAIN] epoch=14, iter=9070/40000, loss=0.6075, lr=0.007934, batch_cost=0.7921, reader_cost=0.0001 | ETA 06:48:20 2021-01-18 11:50:38 [INFO] [TRAIN] epoch=14, iter=9080/40000, loss=0.4708, lr=0.007932, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:47:58 2021-01-18 11:50:46 [INFO] [TRAIN] epoch=14, iter=9090/40000, loss=0.4962, lr=0.007930, batch_cost=0.7859, reader_cost=0.0001 | ETA 06:44:53 2021-01-18 11:50:54 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.5195, lr=0.007927, batch_cost=0.7886, reader_cost=0.0001 | ETA 06:46:07 2021-01-18 11:51:02 [INFO] [TRAIN] epoch=14, iter=9110/40000, loss=0.3355, lr=0.007925, batch_cost=0.7932, reader_cost=0.0001 | ETA 06:48:23 2021-01-18 11:51:10 [INFO] [TRAIN] epoch=14, iter=9120/40000, loss=0.3827, lr=0.007923, batch_cost=0.7868, reader_cost=0.0001 | ETA 06:44:56 2021-01-18 11:51:17 [INFO] [TRAIN] epoch=14, iter=9130/40000, loss=0.4270, lr=0.007920, batch_cost=0.7886, reader_cost=0.0001 | ETA 06:45:43 2021-01-18 11:51:25 [INFO] [TRAIN] epoch=14, iter=9140/40000, loss=0.4590, lr=0.007918, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:46:21 2021-01-18 11:51:33 [INFO] [TRAIN] epoch=14, iter=9150/40000, loss=0.3209, lr=0.007916, batch_cost=0.7871, reader_cost=0.0001 | ETA 06:44:41 2021-01-18 11:51:41 [INFO] [TRAIN] epoch=14, iter=9160/40000, loss=0.3512, lr=0.007913, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:46:07 2021-01-18 11:51:49 [INFO] [TRAIN] epoch=14, iter=9170/40000, loss=0.4168, lr=0.007911, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:45:30 2021-01-18 11:51:57 [INFO] [TRAIN] epoch=14, iter=9180/40000, loss=0.4198, lr=0.007909, batch_cost=0.7858, reader_cost=0.0001 | ETA 06:43:36 2021-01-18 11:52:05 [INFO] [TRAIN] epoch=14, iter=9190/40000, loss=0.4917, lr=0.007906, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:46:15 2021-01-18 11:52:13 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.3954, lr=0.007904, batch_cost=0.7861, reader_cost=0.0001 | ETA 06:43:31 2021-01-18 11:52:21 [INFO] [TRAIN] epoch=14, iter=9210/40000, loss=0.4291, lr=0.007902, batch_cost=0.7857, reader_cost=0.0001 | ETA 06:43:10 2021-01-18 11:52:28 [INFO] [TRAIN] epoch=14, iter=9220/40000, loss=0.4897, lr=0.007900, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:46:43 2021-01-18 11:52:36 [INFO] [TRAIN] epoch=14, iter=9230/40000, loss=0.3881, lr=0.007897, batch_cost=0.7890, reader_cost=0.0001 | ETA 06:44:37 2021-01-18 11:52:44 [INFO] [TRAIN] epoch=14, iter=9240/40000, loss=0.5629, lr=0.007895, batch_cost=0.7976, reader_cost=0.0001 | ETA 06:48:54 2021-01-18 11:52:52 [INFO] [TRAIN] epoch=14, iter=9250/40000, loss=0.3871, lr=0.007893, batch_cost=0.7844, reader_cost=0.0001 | ETA 06:42:01 2021-01-18 11:53:00 [INFO] [TRAIN] epoch=15, iter=9260/40000, loss=0.5184, lr=0.007890, batch_cost=0.7776, reader_cost=0.0125 | ETA 06:38:24 2021-01-18 11:53:08 [INFO] [TRAIN] epoch=15, iter=9270/40000, loss=0.4859, lr=0.007888, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:44:42 2021-01-18 11:53:16 [INFO] [TRAIN] epoch=15, iter=9280/40000, loss=0.4686, lr=0.007886, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:45:58 2021-01-18 11:53:24 [INFO] [TRAIN] epoch=15, iter=9290/40000, loss=0.5012, lr=0.007883, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:43:25 2021-01-18 11:53:32 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.5657, lr=0.007881, batch_cost=0.7921, reader_cost=0.0001 | ETA 06:45:17 2021-01-18 11:53:40 [INFO] [TRAIN] epoch=15, iter=9310/40000, loss=0.3937, lr=0.007879, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:45:30 2021-01-18 11:53:47 [INFO] [TRAIN] epoch=15, iter=9320/40000, loss=0.4414, lr=0.007876, batch_cost=0.7873, reader_cost=0.0001 | ETA 06:42:35 2021-01-18 11:53:55 [INFO] [TRAIN] epoch=15, iter=9330/40000, loss=0.3335, lr=0.007874, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:43:10 2021-01-18 11:54:03 [INFO] [TRAIN] epoch=15, iter=9340/40000, loss=0.3714, lr=0.007872, batch_cost=0.7954, reader_cost=0.0001 | ETA 06:46:25 2021-01-18 11:54:11 [INFO] [TRAIN] epoch=15, iter=9350/40000, loss=0.3701, lr=0.007869, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:43:25 2021-01-18 11:54:19 [INFO] [TRAIN] epoch=15, iter=9360/40000, loss=0.3787, lr=0.007867, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:44:52 2021-01-18 11:54:27 [INFO] [TRAIN] epoch=15, iter=9370/40000, loss=0.3819, lr=0.007865, batch_cost=0.7891, reader_cost=0.0001 | ETA 06:42:51 2021-01-18 11:54:35 [INFO] [TRAIN] epoch=15, iter=9380/40000, loss=0.4179, lr=0.007863, batch_cost=0.7927, reader_cost=0.0001 | ETA 06:44:32 2021-01-18 11:54:43 [INFO] [TRAIN] epoch=15, iter=9390/40000, loss=0.3959, lr=0.007860, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:44:00 2021-01-18 11:54:51 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.5066, lr=0.007858, batch_cost=0.7921, reader_cost=0.0001 | ETA 06:43:59 2021-01-18 11:54:59 [INFO] [TRAIN] epoch=15, iter=9410/40000, loss=0.5157, lr=0.007856, batch_cost=0.7893, reader_cost=0.0001 | ETA 06:42:26 2021-01-18 11:55:06 [INFO] [TRAIN] epoch=15, iter=9420/40000, loss=0.4233, lr=0.007853, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:41:29 2021-01-18 11:55:14 [INFO] [TRAIN] epoch=15, iter=9430/40000, loss=0.3490, lr=0.007851, batch_cost=0.7948, reader_cost=0.0001 | ETA 06:44:56 2021-01-18 11:55:22 [INFO] [TRAIN] epoch=15, iter=9440/40000, loss=0.3468, lr=0.007849, batch_cost=0.7879, reader_cost=0.0001 | ETA 06:41:19 2021-01-18 11:55:30 [INFO] [TRAIN] epoch=15, iter=9450/40000, loss=0.3628, lr=0.007846, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:41:08 2021-01-18 11:55:38 [INFO] [TRAIN] epoch=15, iter=9460/40000, loss=0.3630, lr=0.007844, batch_cost=0.7832, reader_cost=0.0001 | ETA 06:38:38 2021-01-18 11:55:46 [INFO] [TRAIN] epoch=15, iter=9470/40000, loss=0.3334, lr=0.007842, batch_cost=0.7905, reader_cost=0.0001 | ETA 06:42:13 2021-01-18 11:55:54 [INFO] [TRAIN] epoch=15, iter=9480/40000, loss=0.3127, lr=0.007839, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:43:38 2021-01-18 11:56:02 [INFO] [TRAIN] epoch=15, iter=9490/40000, loss=0.5061, lr=0.007837, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:40:46 2021-01-18 11:56:10 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.3566, lr=0.007835, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:42:31 2021-01-18 11:56:18 [INFO] [TRAIN] epoch=15, iter=9510/40000, loss=0.2560, lr=0.007833, batch_cost=0.7936, reader_cost=0.0001 | ETA 06:43:17 2021-01-18 11:56:25 [INFO] [TRAIN] epoch=15, iter=9520/40000, loss=0.3426, lr=0.007830, batch_cost=0.7818, reader_cost=0.0001 | ETA 06:37:10 2021-01-18 11:56:33 [INFO] [TRAIN] epoch=15, iter=9530/40000, loss=0.5575, lr=0.007828, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:42:04 2021-01-18 11:56:41 [INFO] [TRAIN] epoch=15, iter=9540/40000, loss=0.4259, lr=0.007826, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:42:04 2021-01-18 11:56:49 [INFO] [TRAIN] epoch=15, iter=9550/40000, loss=0.5058, lr=0.007823, batch_cost=0.7939, reader_cost=0.0001 | ETA 06:42:55 2021-01-18 11:56:57 [INFO] [TRAIN] epoch=15, iter=9560/40000, loss=0.4339, lr=0.007821, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:40:09 2021-01-18 11:57:05 [INFO] [TRAIN] epoch=15, iter=9570/40000, loss=0.3296, lr=0.007819, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:42:08 2021-01-18 11:57:13 [INFO] [TRAIN] epoch=15, iter=9580/40000, loss=0.5193, lr=0.007816, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:42:00 2021-01-18 11:57:21 [INFO] [TRAIN] epoch=15, iter=9590/40000, loss=0.3182, lr=0.007814, batch_cost=0.7846, reader_cost=0.0001 | ETA 06:37:39 2021-01-18 11:57:29 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.4907, lr=0.007812, batch_cost=0.7926, reader_cost=0.0001 | ETA 06:41:34 2021-01-18 11:57:37 [INFO] [TRAIN] epoch=15, iter=9610/40000, loss=0.3667, lr=0.007809, batch_cost=0.7858, reader_cost=0.0001 | ETA 06:38:00 2021-01-18 11:57:44 [INFO] [TRAIN] epoch=15, iter=9620/40000, loss=0.3284, lr=0.007807, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:40:06 2021-01-18 11:57:52 [INFO] [TRAIN] epoch=15, iter=9630/40000, loss=0.2884, lr=0.007805, batch_cost=0.7927, reader_cost=0.0001 | ETA 06:41:15 2021-01-18 11:58:00 [INFO] [TRAIN] epoch=15, iter=9640/40000, loss=0.2519, lr=0.007802, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:40:02 2021-01-18 11:58:08 [INFO] [TRAIN] epoch=15, iter=9650/40000, loss=0.4931, lr=0.007800, batch_cost=0.7880, reader_cost=0.0001 | ETA 06:38:35 2021-01-18 11:58:16 [INFO] [TRAIN] epoch=15, iter=9660/40000, loss=0.3511, lr=0.007798, batch_cost=0.7844, reader_cost=0.0001 | ETA 06:36:37 2021-01-18 11:58:24 [INFO] [TRAIN] epoch=15, iter=9670/40000, loss=0.3965, lr=0.007796, batch_cost=0.7912, reader_cost=0.0001 | ETA 06:39:58 2021-01-18 11:58:32 [INFO] [TRAIN] epoch=15, iter=9680/40000, loss=0.3751, lr=0.007793, batch_cost=0.7905, reader_cost=0.0001 | ETA 06:39:28 2021-01-18 11:58:40 [INFO] [TRAIN] epoch=15, iter=9690/40000, loss=0.2803, lr=0.007791, batch_cost=0.7856, reader_cost=0.0001 | ETA 06:36:50 2021-01-18 11:58:48 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.4529, lr=0.007789, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:39:53 2021-01-18 11:58:56 [INFO] [TRAIN] epoch=15, iter=9710/40000, loss=0.5742, lr=0.007786, batch_cost=0.7922, reader_cost=0.0001 | ETA 06:39:55 2021-01-18 11:59:03 [INFO] [TRAIN] epoch=15, iter=9720/40000, loss=0.3991, lr=0.007784, batch_cost=0.7847, reader_cost=0.0001 | ETA 06:36:01 2021-01-18 11:59:11 [INFO] [TRAIN] epoch=15, iter=9730/40000, loss=0.4161, lr=0.007782, batch_cost=0.7886, reader_cost=0.0001 | ETA 06:37:50 2021-01-18 11:59:19 [INFO] [TRAIN] epoch=15, iter=9740/40000, loss=0.3788, lr=0.007779, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:38:59 2021-01-18 11:59:27 [INFO] [TRAIN] epoch=15, iter=9750/40000, loss=0.4406, lr=0.007777, batch_cost=0.7958, reader_cost=0.0001 | ETA 06:41:11 2021-01-18 11:59:35 [INFO] [TRAIN] epoch=15, iter=9760/40000, loss=0.3249, lr=0.007775, batch_cost=0.7921, reader_cost=0.0001 | ETA 06:39:13 2021-01-18 11:59:43 [INFO] [TRAIN] epoch=15, iter=9770/40000, loss=0.3248, lr=0.007772, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:38:59 2021-01-18 11:59:51 [INFO] [TRAIN] epoch=15, iter=9780/40000, loss=0.4356, lr=0.007770, batch_cost=0.7957, reader_cost=0.0001 | ETA 06:40:46 2021-01-18 11:59:59 [INFO] [TRAIN] epoch=15, iter=9790/40000, loss=0.3782, lr=0.007768, batch_cost=0.7846, reader_cost=0.0001 | ETA 06:35:03 2021-01-18 12:00:07 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.3899, lr=0.007765, batch_cost=0.7926, reader_cost=0.0001 | ETA 06:38:56 2021-01-18 12:00:15 [INFO] [TRAIN] epoch=15, iter=9810/40000, loss=0.4998, lr=0.007763, batch_cost=0.7904, reader_cost=0.0001 | ETA 06:37:40 2021-01-18 12:00:23 [INFO] [TRAIN] epoch=15, iter=9820/40000, loss=0.3027, lr=0.007761, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:37:56 2021-01-18 12:00:30 [INFO] [TRAIN] epoch=15, iter=9830/40000, loss=0.4270, lr=0.007758, batch_cost=0.7945, reader_cost=0.0001 | ETA 06:39:29 2021-01-18 12:00:38 [INFO] [TRAIN] epoch=15, iter=9840/40000, loss=0.3799, lr=0.007756, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:38:07 2021-01-18 12:00:46 [INFO] [TRAIN] epoch=15, iter=9850/40000, loss=0.5632, lr=0.007754, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:37:54 2021-01-18 12:00:54 [INFO] [TRAIN] epoch=15, iter=9860/40000, loss=0.3698, lr=0.007752, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:35:56 2021-01-18 12:01:02 [INFO] [TRAIN] epoch=15, iter=9870/40000, loss=0.4085, lr=0.007749, batch_cost=0.7914, reader_cost=0.0001 | ETA 06:37:23 2021-01-18 12:01:10 [INFO] [TRAIN] epoch=15, iter=9880/40000, loss=0.5487, lr=0.007747, batch_cost=0.7924, reader_cost=0.0001 | ETA 06:37:46 2021-01-18 12:01:18 [INFO] [TRAIN] epoch=15, iter=9890/40000, loss=0.4171, lr=0.007745, batch_cost=0.7936, reader_cost=0.0001 | ETA 06:38:16 2021-01-18 12:01:26 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.3023, lr=0.007742, batch_cost=0.7927, reader_cost=0.0001 | ETA 06:37:40 2021-01-18 12:01:34 [INFO] [TRAIN] epoch=15, iter=9910/40000, loss=0.4908, lr=0.007740, batch_cost=0.7867, reader_cost=0.0001 | ETA 06:34:30 2021-01-18 12:01:42 [INFO] [TRAIN] epoch=16, iter=9920/40000, loss=0.4632, lr=0.007738, batch_cost=0.7726, reader_cost=0.0154 | ETA 06:27:18 2021-01-18 12:01:49 [INFO] [TRAIN] epoch=16, iter=9930/40000, loss=0.3021, lr=0.007735, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:34:44 2021-01-18 12:01:57 [INFO] [TRAIN] epoch=16, iter=9940/40000, loss=0.5364, lr=0.007733, batch_cost=0.7946, reader_cost=0.0001 | ETA 06:38:06 2021-01-18 12:02:05 [INFO] [TRAIN] epoch=16, iter=9950/40000, loss=0.4227, lr=0.007731, batch_cost=0.7855, reader_cost=0.0001 | ETA 06:33:23 2021-01-18 12:02:13 [INFO] [TRAIN] epoch=16, iter=9960/40000, loss=0.3523, lr=0.007728, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:35:21 2021-01-18 12:02:21 [INFO] [TRAIN] epoch=16, iter=9970/40000, loss=0.3362, lr=0.007726, batch_cost=0.7932, reader_cost=0.0001 | ETA 06:36:59 2021-01-18 12:02:29 [INFO] [TRAIN] epoch=16, iter=9980/40000, loss=0.2793, lr=0.007724, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:35:42 2021-01-18 12:02:37 [INFO] [TRAIN] epoch=16, iter=9990/40000, loss=0.2711, lr=0.007721, batch_cost=0.7834, reader_cost=0.0001 | ETA 06:31:49 2021-01-18 12:02:45 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.4099, lr=0.007719, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:34:22 2021-01-18 12:02:45 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 12:03:12 [INFO] [EVAL] #Images=1449 mIoU=0.7339 Acc=0.9378 Kappa=0.8644 2021-01-18 12:03:12 [INFO] [EVAL] Class IoU: [0.9351 0.8494 0.3898 0.8389 0.6528 0.7651 0.9237 0.8309 0.8492 0.289 0.866 0.5804 0.7867 0.8938 0.8101 0.8463 0.5383 0.7659 0.4863 0.8761 0.6382] 2021-01-18 12:03:12 [INFO] [EVAL] Class Acc: [0.9678 0.899 0.4226 0.8957 0.7971 0.8264 0.9626 0.9301 0.9087 0.4128 0.9192 0.7092 0.8756 0.949 0.9087 0.9514 0.8337 0.9619 0.6275 0.9323 0.7625] 2021-01-18 12:03:19 [INFO] [EVAL] The model with the best validation mIoU (0.7388) was saved at iter 9000. 2021-01-18 12:03:27 [INFO] [TRAIN] epoch=16, iter=10010/40000, loss=0.2893, lr=0.007717, batch_cost=0.7827, reader_cost=0.0001 | ETA 06:31:14 2021-01-18 12:03:34 [INFO] [TRAIN] epoch=16, iter=10020/40000, loss=0.5277, lr=0.007714, batch_cost=0.7871, reader_cost=0.0001 | ETA 06:33:17 2021-01-18 12:03:42 [INFO] [TRAIN] epoch=16, iter=10030/40000, loss=0.3526, lr=0.007712, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:34:40 2021-01-18 12:03:50 [INFO] [TRAIN] epoch=16, iter=10040/40000, loss=0.2964, lr=0.007710, batch_cost=0.7873, reader_cost=0.0001 | ETA 06:33:06 2021-01-18 12:03:58 [INFO] [TRAIN] epoch=16, iter=10050/40000, loss=0.3879, lr=0.007708, batch_cost=0.7868, reader_cost=0.0001 | ETA 06:32:44 2021-01-18 12:04:06 [INFO] [TRAIN] epoch=16, iter=10060/40000, loss=0.4554, lr=0.007705, batch_cost=0.7860, reader_cost=0.0001 | ETA 06:32:12 2021-01-18 12:04:14 [INFO] [TRAIN] epoch=16, iter=10070/40000, loss=0.7100, lr=0.007703, batch_cost=0.7904, reader_cost=0.0001 | ETA 06:34:15 2021-01-18 12:04:22 [INFO] [TRAIN] epoch=16, iter=10080/40000, loss=0.3360, lr=0.007701, batch_cost=0.7960, reader_cost=0.0001 | ETA 06:36:57 2021-01-18 12:04:30 [INFO] [TRAIN] epoch=16, iter=10090/40000, loss=0.4008, lr=0.007698, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:33:44 2021-01-18 12:04:38 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.4585, lr=0.007696, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:33:04 2021-01-18 12:04:45 [INFO] [TRAIN] epoch=16, iter=10110/40000, loss=0.4829, lr=0.007694, batch_cost=0.7897, reader_cost=0.0001 | ETA 06:33:25 2021-01-18 12:04:53 [INFO] [TRAIN] epoch=16, iter=10120/40000, loss=0.4365, lr=0.007691, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:34:20 2021-01-18 12:05:01 [INFO] [TRAIN] epoch=16, iter=10130/40000, loss=0.4037, lr=0.007689, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:32:53 2021-01-18 12:05:09 [INFO] [TRAIN] epoch=16, iter=10140/40000, loss=0.5148, lr=0.007687, batch_cost=0.7940, reader_cost=0.0001 | ETA 06:35:08 2021-01-18 12:05:17 [INFO] [TRAIN] epoch=16, iter=10150/40000, loss=0.3710, lr=0.007684, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:32:55 2021-01-18 12:05:25 [INFO] [TRAIN] epoch=16, iter=10160/40000, loss=0.3722, lr=0.007682, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:32:58 2021-01-18 12:05:33 [INFO] [TRAIN] epoch=16, iter=10170/40000, loss=0.3571, lr=0.007680, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:32:52 2021-01-18 12:05:41 [INFO] [TRAIN] epoch=16, iter=10180/40000, loss=0.5222, lr=0.007677, batch_cost=0.7881, reader_cost=0.0001 | ETA 06:31:40 2021-01-18 12:05:49 [INFO] [TRAIN] epoch=16, iter=10190/40000, loss=0.3634, lr=0.007675, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:33:57 2021-01-18 12:05:57 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.3376, lr=0.007673, batch_cost=0.7897, reader_cost=0.0001 | ETA 06:32:12 2021-01-18 12:06:04 [INFO] [TRAIN] epoch=16, iter=10210/40000, loss=0.3602, lr=0.007670, batch_cost=0.7841, reader_cost=0.0001 | ETA 06:29:16 2021-01-18 12:06:12 [INFO] [TRAIN] epoch=16, iter=10220/40000, loss=0.4580, lr=0.007668, batch_cost=0.7875, reader_cost=0.0001 | ETA 06:30:50 2021-01-18 12:06:20 [INFO] [TRAIN] epoch=16, iter=10230/40000, loss=0.2589, lr=0.007666, batch_cost=0.7948, reader_cost=0.0001 | ETA 06:34:20 2021-01-18 12:06:28 [INFO] [TRAIN] epoch=16, iter=10240/40000, loss=0.3762, lr=0.007664, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:33:16 2021-01-18 12:06:36 [INFO] [TRAIN] epoch=16, iter=10250/40000, loss=0.4591, lr=0.007661, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:31:19 2021-01-18 12:06:44 [INFO] [TRAIN] epoch=16, iter=10260/40000, loss=0.3770, lr=0.007659, batch_cost=0.7955, reader_cost=0.0001 | ETA 06:34:18 2021-01-18 12:06:52 [INFO] [TRAIN] epoch=16, iter=10270/40000, loss=0.4442, lr=0.007657, batch_cost=0.7863, reader_cost=0.0001 | ETA 06:29:36 2021-01-18 12:07:00 [INFO] [TRAIN] epoch=16, iter=10280/40000, loss=0.2888, lr=0.007654, batch_cost=0.7884, reader_cost=0.0001 | ETA 06:30:30 2021-01-18 12:07:08 [INFO] [TRAIN] epoch=16, iter=10290/40000, loss=0.5002, lr=0.007652, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:30:33 2021-01-18 12:07:16 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.3538, lr=0.007650, batch_cost=0.7930, reader_cost=0.0001 | ETA 06:32:31 2021-01-18 12:07:24 [INFO] [TRAIN] epoch=16, iter=10310/40000, loss=0.3210, lr=0.007647, batch_cost=0.7850, reader_cost=0.0001 | ETA 06:28:27 2021-01-18 12:07:31 [INFO] [TRAIN] epoch=16, iter=10320/40000, loss=0.4024, lr=0.007645, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:30:08 2021-01-18 12:07:39 [INFO] [TRAIN] epoch=16, iter=10330/40000, loss=0.4284, lr=0.007643, batch_cost=0.7880, reader_cost=0.0001 | ETA 06:29:38 2021-01-18 12:07:47 [INFO] [TRAIN] epoch=16, iter=10340/40000, loss=0.3147, lr=0.007640, batch_cost=0.7910, reader_cost=0.0001 | ETA 06:31:01 2021-01-18 12:07:55 [INFO] [TRAIN] epoch=16, iter=10350/40000, loss=0.5078, lr=0.007638, batch_cost=0.7926, reader_cost=0.0001 | ETA 06:31:40 2021-01-18 12:08:03 [INFO] [TRAIN] epoch=16, iter=10360/40000, loss=0.3848, lr=0.007636, batch_cost=0.7908, reader_cost=0.0001 | ETA 06:30:39 2021-01-18 12:08:11 [INFO] [TRAIN] epoch=16, iter=10370/40000, loss=0.4472, lr=0.007633, batch_cost=0.7912, reader_cost=0.0001 | ETA 06:30:44 2021-01-18 12:08:19 [INFO] [TRAIN] epoch=16, iter=10380/40000, loss=0.3731, lr=0.007631, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:28:49 2021-01-18 12:08:27 [INFO] [TRAIN] epoch=16, iter=10390/40000, loss=0.3908, lr=0.007629, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:29:48 2021-01-18 12:08:35 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.4510, lr=0.007626, batch_cost=0.7963, reader_cost=0.0001 | ETA 06:32:51 2021-01-18 12:08:43 [INFO] [TRAIN] epoch=16, iter=10410/40000, loss=0.5197, lr=0.007624, batch_cost=0.7964, reader_cost=0.0001 | ETA 06:32:44 2021-01-18 12:08:51 [INFO] [TRAIN] epoch=16, iter=10420/40000, loss=0.4192, lr=0.007622, batch_cost=0.7882, reader_cost=0.0001 | ETA 06:28:35 2021-01-18 12:08:58 [INFO] [TRAIN] epoch=16, iter=10430/40000, loss=0.3717, lr=0.007619, batch_cost=0.7956, reader_cost=0.0001 | ETA 06:32:06 2021-01-18 12:09:06 [INFO] [TRAIN] epoch=16, iter=10440/40000, loss=0.4537, lr=0.007617, batch_cost=0.7905, reader_cost=0.0001 | ETA 06:29:27 2021-01-18 12:09:14 [INFO] [TRAIN] epoch=16, iter=10450/40000, loss=0.3862, lr=0.007615, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:29:31 2021-01-18 12:09:22 [INFO] [TRAIN] epoch=16, iter=10460/40000, loss=0.2844, lr=0.007613, batch_cost=0.7941, reader_cost=0.0001 | ETA 06:30:57 2021-01-18 12:09:30 [INFO] [TRAIN] epoch=16, iter=10470/40000, loss=0.4433, lr=0.007610, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:28:55 2021-01-18 12:09:38 [INFO] [TRAIN] epoch=16, iter=10480/40000, loss=0.2658, lr=0.007608, batch_cost=0.7952, reader_cost=0.0001 | ETA 06:31:13 2021-01-18 12:09:46 [INFO] [TRAIN] epoch=16, iter=10490/40000, loss=0.5513, lr=0.007606, batch_cost=0.7968, reader_cost=0.0001 | ETA 06:31:52 2021-01-18 12:09:54 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.3854, lr=0.007603, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:27:45 2021-01-18 12:10:02 [INFO] [TRAIN] epoch=16, iter=10510/40000, loss=0.2487, lr=0.007601, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:28:43 2021-01-18 12:10:10 [INFO] [TRAIN] epoch=16, iter=10520/40000, loss=0.3685, lr=0.007599, batch_cost=0.7879, reader_cost=0.0001 | ETA 06:27:06 2021-01-18 12:10:18 [INFO] [TRAIN] epoch=16, iter=10530/40000, loss=0.5709, lr=0.007596, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:27:54 2021-01-18 12:10:25 [INFO] [TRAIN] epoch=16, iter=10540/40000, loss=0.3995, lr=0.007594, batch_cost=0.7856, reader_cost=0.0001 | ETA 06:25:43 2021-01-18 12:10:33 [INFO] [TRAIN] epoch=16, iter=10550/40000, loss=0.3029, lr=0.007592, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:28:34 2021-01-18 12:10:41 [INFO] [TRAIN] epoch=16, iter=10560/40000, loss=0.3552, lr=0.007589, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:27:41 2021-01-18 12:10:49 [INFO] [TRAIN] epoch=16, iter=10570/40000, loss=0.2709, lr=0.007587, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:27:26 2021-01-18 12:10:57 [INFO] [TRAIN] epoch=17, iter=10580/40000, loss=0.3969, lr=0.007585, batch_cost=0.7775, reader_cost=0.0120 | ETA 06:21:15 2021-01-18 12:11:05 [INFO] [TRAIN] epoch=17, iter=10590/40000, loss=0.3526, lr=0.007582, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:27:18 2021-01-18 12:11:13 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.4177, lr=0.007580, batch_cost=0.7890, reader_cost=0.0001 | ETA 06:26:36 2021-01-18 12:11:21 [INFO] [TRAIN] epoch=17, iter=10610/40000, loss=0.2580, lr=0.007578, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:27:51 2021-01-18 12:11:29 [INFO] [TRAIN] epoch=17, iter=10620/40000, loss=0.3469, lr=0.007575, batch_cost=0.7861, reader_cost=0.0001 | ETA 06:24:54 2021-01-18 12:11:36 [INFO] [TRAIN] epoch=17, iter=10630/40000, loss=0.3717, lr=0.007573, batch_cost=0.7938, reader_cost=0.0001 | ETA 06:28:33 2021-01-18 12:11:44 [INFO] [TRAIN] epoch=17, iter=10640/40000, loss=0.4155, lr=0.007571, batch_cost=0.7883, reader_cost=0.0001 | ETA 06:25:43 2021-01-18 12:11:52 [INFO] [TRAIN] epoch=17, iter=10650/40000, loss=0.3276, lr=0.007568, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:28:08 2021-01-18 12:12:00 [INFO] [TRAIN] epoch=17, iter=10660/40000, loss=0.4597, lr=0.007566, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:28:26 2021-01-18 12:12:08 [INFO] [TRAIN] epoch=17, iter=10670/40000, loss=0.2241, lr=0.007564, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:25:05 2021-01-18 12:12:16 [INFO] [TRAIN] epoch=17, iter=10680/40000, loss=0.3883, lr=0.007561, batch_cost=0.7865, reader_cost=0.0001 | ETA 06:24:20 2021-01-18 12:12:24 [INFO] [TRAIN] epoch=17, iter=10690/40000, loss=0.6007, lr=0.007559, batch_cost=0.7883, reader_cost=0.0001 | ETA 06:25:03 2021-01-18 12:12:32 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.3451, lr=0.007557, batch_cost=0.7924, reader_cost=0.0001 | ETA 06:26:56 2021-01-18 12:12:40 [INFO] [TRAIN] epoch=17, iter=10710/40000, loss=0.3090, lr=0.007555, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:25:56 2021-01-18 12:12:48 [INFO] [TRAIN] epoch=17, iter=10720/40000, loss=0.5661, lr=0.007552, batch_cost=0.7908, reader_cost=0.0001 | ETA 06:25:56 2021-01-18 12:12:55 [INFO] [TRAIN] epoch=17, iter=10730/40000, loss=0.3095, lr=0.007550, batch_cost=0.7824, reader_cost=0.0001 | ETA 06:21:42 2021-01-18 12:13:03 [INFO] [TRAIN] epoch=17, iter=10740/40000, loss=0.2759, lr=0.007548, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:25:02 2021-01-18 12:13:11 [INFO] [TRAIN] epoch=17, iter=10750/40000, loss=0.2690, lr=0.007545, batch_cost=0.7945, reader_cost=0.0001 | ETA 06:27:19 2021-01-18 12:13:19 [INFO] [TRAIN] epoch=17, iter=10760/40000, loss=0.3765, lr=0.007543, batch_cost=0.7847, reader_cost=0.0001 | ETA 06:22:23 2021-01-18 12:13:27 [INFO] [TRAIN] epoch=17, iter=10770/40000, loss=0.3702, lr=0.007541, batch_cost=0.7852, reader_cost=0.0001 | ETA 06:22:30 2021-01-18 12:13:35 [INFO] [TRAIN] epoch=17, iter=10780/40000, loss=0.3911, lr=0.007538, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:24:09 2021-01-18 12:13:43 [INFO] [TRAIN] epoch=17, iter=10790/40000, loss=0.4401, lr=0.007536, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:24:02 2021-01-18 12:13:51 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.4109, lr=0.007534, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:24:45 2021-01-18 12:13:59 [INFO] [TRAIN] epoch=17, iter=10810/40000, loss=0.3635, lr=0.007531, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:24:51 2021-01-18 12:14:07 [INFO] [TRAIN] epoch=17, iter=10820/40000, loss=0.3786, lr=0.007529, batch_cost=0.7933, reader_cost=0.0001 | ETA 06:25:48 2021-01-18 12:14:14 [INFO] [TRAIN] epoch=17, iter=10830/40000, loss=0.3374, lr=0.007527, batch_cost=0.7931, reader_cost=0.0001 | ETA 06:25:33 2021-01-18 12:14:22 [INFO] [TRAIN] epoch=17, iter=10840/40000, loss=0.3666, lr=0.007524, batch_cost=0.7930, reader_cost=0.0001 | ETA 06:25:24 2021-01-18 12:14:30 [INFO] [TRAIN] epoch=17, iter=10850/40000, loss=0.3758, lr=0.007522, batch_cost=0.7864, reader_cost=0.0001 | ETA 06:22:02 2021-01-18 12:14:38 [INFO] [TRAIN] epoch=17, iter=10860/40000, loss=0.3861, lr=0.007520, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:25:49 2021-01-18 12:14:46 [INFO] [TRAIN] epoch=17, iter=10870/40000, loss=0.4957, lr=0.007517, batch_cost=0.7887, reader_cost=0.0001 | ETA 06:22:55 2021-01-18 12:14:54 [INFO] [TRAIN] epoch=17, iter=10880/40000, loss=0.2498, lr=0.007515, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:24:48 2021-01-18 12:15:02 [INFO] [TRAIN] epoch=17, iter=10890/40000, loss=0.3225, lr=0.007513, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:23:22 2021-01-18 12:15:10 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.3401, lr=0.007510, batch_cost=0.7904, reader_cost=0.0001 | ETA 06:23:20 2021-01-18 12:15:18 [INFO] [TRAIN] epoch=17, iter=10910/40000, loss=0.3238, lr=0.007508, batch_cost=0.7866, reader_cost=0.0001 | ETA 06:21:21 2021-01-18 12:15:26 [INFO] [TRAIN] epoch=17, iter=10920/40000, loss=0.3777, lr=0.007506, batch_cost=0.7833, reader_cost=0.0001 | ETA 06:19:37 2021-01-18 12:15:33 [INFO] [TRAIN] epoch=17, iter=10930/40000, loss=0.3507, lr=0.007503, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:23:15 2021-01-18 12:15:41 [INFO] [TRAIN] epoch=17, iter=10940/40000, loss=0.3009, lr=0.007501, batch_cost=0.7911, reader_cost=0.0001 | ETA 06:23:09 2021-01-18 12:15:49 [INFO] [TRAIN] epoch=17, iter=10950/40000, loss=0.2604, lr=0.007499, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:22:32 2021-01-18 12:15:57 [INFO] [TRAIN] epoch=17, iter=10960/40000, loss=0.3465, lr=0.007496, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:22:23 2021-01-18 12:16:05 [INFO] [TRAIN] epoch=17, iter=10970/40000, loss=0.3717, lr=0.007494, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:22:03 2021-01-18 12:16:13 [INFO] [TRAIN] epoch=17, iter=10980/40000, loss=0.4867, lr=0.007492, batch_cost=0.7938, reader_cost=0.0001 | ETA 06:23:54 2021-01-18 12:16:21 [INFO] [TRAIN] epoch=17, iter=10990/40000, loss=0.2920, lr=0.007489, batch_cost=0.7959, reader_cost=0.0001 | ETA 06:24:47 2021-01-18 12:16:29 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.2405, lr=0.007487, batch_cost=0.7927, reader_cost=0.0001 | ETA 06:23:07 2021-01-18 12:16:29 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 12:16:56 [INFO] [EVAL] #Images=1449 mIoU=0.7495 Acc=0.9435 Kappa=0.8764 2021-01-18 12:16:56 [INFO] [EVAL] Class IoU: [0.9401 0.8053 0.4135 0.8728 0.6227 0.7929 0.9446 0.8324 0.8663 0.3441 0.837 0.5771 0.7994 0.8773 0.8295 0.8498 0.5986 0.8013 0.5492 0.8738 0.7115] 2021-01-18 12:16:56 [INFO] [EVAL] Class Acc: [0.9695 0.8318 0.4342 0.9361 0.7347 0.8532 0.9738 0.8928 0.957 0.567 0.8942 0.8433 0.8417 0.924 0.9144 0.922 0.7854 0.9462 0.7087 0.9402 0.9022] 2021-01-18 12:17:07 [INFO] [EVAL] The model with the best validation mIoU (0.7495) was saved at iter 11000. 2021-01-18 12:17:15 [INFO] [TRAIN] epoch=17, iter=11010/40000, loss=0.5847, lr=0.007485, batch_cost=0.7805, reader_cost=0.0001 | ETA 06:17:07 2021-01-18 12:17:23 [INFO] [TRAIN] epoch=17, iter=11020/40000, loss=0.3339, lr=0.007483, batch_cost=0.7922, reader_cost=0.0001 | ETA 06:22:36 2021-01-18 12:17:31 [INFO] [TRAIN] epoch=17, iter=11030/40000, loss=0.2798, lr=0.007480, batch_cost=0.7855, reader_cost=0.0001 | ETA 06:19:14 2021-01-18 12:17:38 [INFO] [TRAIN] epoch=17, iter=11040/40000, loss=0.2800, lr=0.007478, batch_cost=0.7932, reader_cost=0.0001 | ETA 06:22:50 2021-01-18 12:17:46 [INFO] [TRAIN] epoch=17, iter=11050/40000, loss=0.4261, lr=0.007476, batch_cost=0.7886, reader_cost=0.0001 | ETA 06:20:30 2021-01-18 12:17:54 [INFO] [TRAIN] epoch=17, iter=11060/40000, loss=0.5700, lr=0.007473, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:20:45 2021-01-18 12:18:02 [INFO] [TRAIN] epoch=17, iter=11070/40000, loss=0.3521, lr=0.007471, batch_cost=0.7939, reader_cost=0.0001 | ETA 06:22:47 2021-01-18 12:18:10 [INFO] [TRAIN] epoch=17, iter=11080/40000, loss=0.3823, lr=0.007469, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:20:50 2021-01-18 12:18:18 [INFO] [TRAIN] epoch=17, iter=11090/40000, loss=0.4201, lr=0.007466, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:21:59 2021-01-18 12:18:26 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.3337, lr=0.007464, batch_cost=0.7923, reader_cost=0.0001 | ETA 06:21:36 2021-01-18 12:18:34 [INFO] [TRAIN] epoch=17, iter=11110/40000, loss=0.7134, lr=0.007462, batch_cost=0.7964, reader_cost=0.0001 | ETA 06:23:28 2021-01-18 12:18:42 [INFO] [TRAIN] epoch=17, iter=11120/40000, loss=0.5252, lr=0.007459, batch_cost=0.7950, reader_cost=0.0001 | ETA 06:22:40 2021-01-18 12:18:50 [INFO] [TRAIN] epoch=17, iter=11130/40000, loss=0.4843, lr=0.007457, batch_cost=0.7934, reader_cost=0.0001 | ETA 06:21:46 2021-01-18 12:18:58 [INFO] [TRAIN] epoch=17, iter=11140/40000, loss=0.3231, lr=0.007455, batch_cost=0.7880, reader_cost=0.0001 | ETA 06:19:00 2021-01-18 12:19:06 [INFO] [TRAIN] epoch=17, iter=11150/40000, loss=0.4363, lr=0.007452, batch_cost=0.7949, reader_cost=0.0001 | ETA 06:22:11 2021-01-18 12:19:14 [INFO] [TRAIN] epoch=17, iter=11160/40000, loss=0.3955, lr=0.007450, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:21:50 2021-01-18 12:19:21 [INFO] [TRAIN] epoch=17, iter=11170/40000, loss=0.3309, lr=0.007448, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:20:29 2021-01-18 12:19:29 [INFO] [TRAIN] epoch=17, iter=11180/40000, loss=0.3127, lr=0.007445, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:18:21 2021-01-18 12:19:37 [INFO] [TRAIN] epoch=17, iter=11190/40000, loss=0.4226, lr=0.007443, batch_cost=0.7937, reader_cost=0.0001 | ETA 06:21:05 2021-01-18 12:19:45 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.4709, lr=0.007441, batch_cost=0.7878, reader_cost=0.0001 | ETA 06:18:08 2021-01-18 12:19:53 [INFO] [TRAIN] epoch=17, iter=11210/40000, loss=0.2611, lr=0.007438, batch_cost=0.7863, reader_cost=0.0001 | ETA 06:17:18 2021-01-18 12:20:01 [INFO] [TRAIN] epoch=17, iter=11220/40000, loss=0.4077, lr=0.007436, batch_cost=0.7895, reader_cost=0.0001 | ETA 06:18:41 2021-01-18 12:20:09 [INFO] [TRAIN] epoch=17, iter=11230/40000, loss=0.3249, lr=0.007434, batch_cost=0.7931, reader_cost=0.0001 | ETA 06:20:17 2021-01-18 12:20:17 [INFO] [TRAIN] epoch=18, iter=11240/40000, loss=0.2629, lr=0.007431, batch_cost=0.7754, reader_cost=0.0152 | ETA 06:11:40 2021-01-18 12:20:24 [INFO] [TRAIN] epoch=18, iter=11250/40000, loss=0.3134, lr=0.007429, batch_cost=0.7857, reader_cost=0.0001 | ETA 06:16:28 2021-01-18 12:20:32 [INFO] [TRAIN] epoch=18, iter=11260/40000, loss=0.3755, lr=0.007427, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:18:17 2021-01-18 12:20:40 [INFO] [TRAIN] epoch=18, iter=11270/40000, loss=0.2952, lr=0.007424, batch_cost=0.7973, reader_cost=0.0001 | ETA 06:21:45 2021-01-18 12:20:48 [INFO] [TRAIN] epoch=18, iter=11280/40000, loss=0.3574, lr=0.007422, batch_cost=0.7891, reader_cost=0.0001 | ETA 06:17:43 2021-01-18 12:20:56 [INFO] [TRAIN] epoch=18, iter=11290/40000, loss=0.3544, lr=0.007420, batch_cost=0.7917, reader_cost=0.0001 | ETA 06:18:50 2021-01-18 12:21:04 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.2395, lr=0.007417, batch_cost=0.7949, reader_cost=0.0001 | ETA 06:20:12 2021-01-18 12:21:12 [INFO] [TRAIN] epoch=18, iter=11310/40000, loss=0.3723, lr=0.007415, batch_cost=0.7954, reader_cost=0.0001 | ETA 06:20:18 2021-01-18 12:21:20 [INFO] [TRAIN] epoch=18, iter=11320/40000, loss=0.4152, lr=0.007413, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:17:01 2021-01-18 12:21:28 [INFO] [TRAIN] epoch=18, iter=11330/40000, loss=0.4120, lr=0.007410, batch_cost=0.7939, reader_cost=0.0001 | ETA 06:19:21 2021-01-18 12:21:36 [INFO] [TRAIN] epoch=18, iter=11340/40000, loss=0.3414, lr=0.007408, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:19:27 2021-01-18 12:21:44 [INFO] [TRAIN] epoch=18, iter=11350/40000, loss=0.3076, lr=0.007406, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:16:40 2021-01-18 12:21:52 [INFO] [TRAIN] epoch=18, iter=11360/40000, loss=0.3634, lr=0.007403, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:17:01 2021-01-18 12:22:00 [INFO] [TRAIN] epoch=18, iter=11370/40000, loss=0.3622, lr=0.007401, batch_cost=0.7910, reader_cost=0.0001 | ETA 06:17:25 2021-01-18 12:22:07 [INFO] [TRAIN] epoch=18, iter=11380/40000, loss=0.2922, lr=0.007399, batch_cost=0.7929, reader_cost=0.0001 | ETA 06:18:11 2021-01-18 12:22:15 [INFO] [TRAIN] epoch=18, iter=11390/40000, loss=0.2864, lr=0.007396, batch_cost=0.7949, reader_cost=0.0001 | ETA 06:19:01 2021-01-18 12:22:23 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.3332, lr=0.007394, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:17:31 2021-01-18 12:22:31 [INFO] [TRAIN] epoch=18, iter=11410/40000, loss=0.4467, lr=0.007392, batch_cost=0.7922, reader_cost=0.0001 | ETA 06:17:29 2021-01-18 12:22:39 [INFO] [TRAIN] epoch=18, iter=11420/40000, loss=0.4445, lr=0.007390, batch_cost=0.7937, reader_cost=0.0001 | ETA 06:18:03 2021-01-18 12:22:47 [INFO] [TRAIN] epoch=18, iter=11430/40000, loss=0.2547, lr=0.007387, batch_cost=0.7953, reader_cost=0.0001 | ETA 06:18:41 2021-01-18 12:22:55 [INFO] [TRAIN] epoch=18, iter=11440/40000, loss=0.3749, lr=0.007385, batch_cost=0.7953, reader_cost=0.0001 | ETA 06:18:35 2021-01-18 12:23:03 [INFO] [TRAIN] epoch=18, iter=11450/40000, loss=0.3028, lr=0.007383, batch_cost=0.7894, reader_cost=0.0001 | ETA 06:15:37 2021-01-18 12:23:11 [INFO] [TRAIN] epoch=18, iter=11460/40000, loss=0.3357, lr=0.007380, batch_cost=0.7865, reader_cost=0.0001 | ETA 06:14:07 2021-01-18 12:23:19 [INFO] [TRAIN] epoch=18, iter=11470/40000, loss=0.3166, lr=0.007378, batch_cost=0.7914, reader_cost=0.0001 | ETA 06:16:18 2021-01-18 12:23:27 [INFO] [TRAIN] epoch=18, iter=11480/40000, loss=0.2441, lr=0.007376, batch_cost=0.7947, reader_cost=0.0001 | ETA 06:17:44 2021-01-18 12:23:35 [INFO] [TRAIN] epoch=18, iter=11490/40000, loss=0.2582, lr=0.007373, batch_cost=0.7908, reader_cost=0.0001 | ETA 06:15:45 2021-01-18 12:23:43 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.2948, lr=0.007371, batch_cost=0.7906, reader_cost=0.0001 | ETA 06:15:31 2021-01-18 12:23:50 [INFO] [TRAIN] epoch=18, iter=11510/40000, loss=0.2814, lr=0.007369, batch_cost=0.7933, reader_cost=0.0001 | ETA 06:16:39 2021-01-18 12:23:58 [INFO] [TRAIN] epoch=18, iter=11520/40000, loss=0.3690, lr=0.007366, batch_cost=0.7931, reader_cost=0.0001 | ETA 06:16:28 2021-01-18 12:24:06 [INFO] [TRAIN] epoch=18, iter=11530/40000, loss=0.4283, lr=0.007364, batch_cost=0.7877, reader_cost=0.0001 | ETA 06:13:46 2021-01-18 12:24:14 [INFO] [TRAIN] epoch=18, iter=11540/40000, loss=0.2935, lr=0.007362, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:14:32 2021-01-18 12:24:22 [INFO] [TRAIN] epoch=18, iter=11550/40000, loss=0.2838, lr=0.007359, batch_cost=0.7884, reader_cost=0.0001 | ETA 06:13:48 2021-01-18 12:24:30 [INFO] [TRAIN] epoch=18, iter=11560/40000, loss=0.2263, lr=0.007357, batch_cost=0.8014, reader_cost=0.0001 | ETA 06:19:51 2021-01-18 12:24:38 [INFO] [TRAIN] epoch=18, iter=11570/40000, loss=0.4753, lr=0.007355, batch_cost=0.7946, reader_cost=0.0001 | ETA 06:16:29 2021-01-18 12:24:46 [INFO] [TRAIN] epoch=18, iter=11580/40000, loss=0.4203, lr=0.007352, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:15:51 2021-01-18 12:24:54 [INFO] [TRAIN] epoch=18, iter=11590/40000, loss=0.2460, lr=0.007350, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:14:29 2021-01-18 12:25:02 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.4659, lr=0.007348, batch_cost=0.7893, reader_cost=0.0001 | ETA 06:13:36 2021-01-18 12:25:10 [INFO] [TRAIN] epoch=18, iter=11610/40000, loss=0.4793, lr=0.007345, batch_cost=0.7879, reader_cost=0.0001 | ETA 06:12:49 2021-01-18 12:25:17 [INFO] [TRAIN] epoch=18, iter=11620/40000, loss=0.3447, lr=0.007343, batch_cost=0.7871, reader_cost=0.0001 | ETA 06:12:18 2021-01-18 12:25:25 [INFO] [TRAIN] epoch=18, iter=11630/40000, loss=0.3365, lr=0.007341, batch_cost=0.7913, reader_cost=0.0001 | ETA 06:14:07 2021-01-18 12:25:33 [INFO] [TRAIN] epoch=18, iter=11640/40000, loss=0.4723, lr=0.007338, batch_cost=0.7870, reader_cost=0.0001 | ETA 06:11:58 2021-01-18 12:25:41 [INFO] [TRAIN] epoch=18, iter=11650/40000, loss=0.2417, lr=0.007336, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:13:19 2021-01-18 12:25:49 [INFO] [TRAIN] epoch=18, iter=11660/40000, loss=0.4531, lr=0.007334, batch_cost=0.7865, reader_cost=0.0001 | ETA 06:11:30 2021-01-18 12:25:57 [INFO] [TRAIN] epoch=18, iter=11670/40000, loss=0.2923, lr=0.007331, batch_cost=0.7880, reader_cost=0.0001 | ETA 06:12:03 2021-01-18 12:26:05 [INFO] [TRAIN] epoch=18, iter=11680/40000, loss=0.4958, lr=0.007329, batch_cost=0.7875, reader_cost=0.0001 | ETA 06:11:42 2021-01-18 12:26:13 [INFO] [TRAIN] epoch=18, iter=11690/40000, loss=0.3459, lr=0.007327, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:13:35 2021-01-18 12:26:21 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.3210, lr=0.007324, batch_cost=0.7949, reader_cost=0.0001 | ETA 06:14:55 2021-01-18 12:26:29 [INFO] [TRAIN] epoch=18, iter=11710/40000, loss=0.3291, lr=0.007322, batch_cost=0.7914, reader_cost=0.0001 | ETA 06:13:08 2021-01-18 12:26:36 [INFO] [TRAIN] epoch=18, iter=11720/40000, loss=0.3535, lr=0.007320, batch_cost=0.7914, reader_cost=0.0001 | ETA 06:12:59 2021-01-18 12:26:44 [INFO] [TRAIN] epoch=18, iter=11730/40000, loss=0.3274, lr=0.007317, batch_cost=0.7936, reader_cost=0.0001 | ETA 06:13:55 2021-01-18 12:26:52 [INFO] [TRAIN] epoch=18, iter=11740/40000, loss=0.3984, lr=0.007315, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:14:10 2021-01-18 12:27:00 [INFO] [TRAIN] epoch=18, iter=11750/40000, loss=0.4268, lr=0.007313, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:10:49 2021-01-18 12:27:08 [INFO] [TRAIN] epoch=18, iter=11760/40000, loss=0.4387, lr=0.007310, batch_cost=0.7812, reader_cost=0.0001 | ETA 06:07:40 2021-01-18 12:27:16 [INFO] [TRAIN] epoch=18, iter=11770/40000, loss=0.4874, lr=0.007308, batch_cost=0.7907, reader_cost=0.0001 | ETA 06:12:00 2021-01-18 12:27:24 [INFO] [TRAIN] epoch=18, iter=11780/40000, loss=0.4797, lr=0.007306, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:12:51 2021-01-18 12:27:32 [INFO] [TRAIN] epoch=18, iter=11790/40000, loss=0.3017, lr=0.007303, batch_cost=0.7864, reader_cost=0.0001 | ETA 06:09:43 2021-01-18 12:27:40 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.3758, lr=0.007301, batch_cost=0.7879, reader_cost=0.0001 | ETA 06:10:19 2021-01-18 12:27:48 [INFO] [TRAIN] epoch=18, iter=11810/40000, loss=0.4274, lr=0.007299, batch_cost=0.7895, reader_cost=0.0001 | ETA 06:10:56 2021-01-18 12:27:55 [INFO] [TRAIN] epoch=18, iter=11820/40000, loss=0.4149, lr=0.007296, batch_cost=0.7854, reader_cost=0.0001 | ETA 06:08:53 2021-01-18 12:28:03 [INFO] [TRAIN] epoch=18, iter=11830/40000, loss=0.5533, lr=0.007294, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:10:51 2021-01-18 12:28:11 [INFO] [TRAIN] epoch=18, iter=11840/40000, loss=0.3170, lr=0.007292, batch_cost=0.7873, reader_cost=0.0001 | ETA 06:09:30 2021-01-18 12:28:19 [INFO] [TRAIN] epoch=18, iter=11850/40000, loss=0.4139, lr=0.007289, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:12:17 2021-01-18 12:28:27 [INFO] [TRAIN] epoch=18, iter=11860/40000, loss=0.3239, lr=0.007287, batch_cost=0.7885, reader_cost=0.0001 | ETA 06:09:49 2021-01-18 12:28:35 [INFO] [TRAIN] epoch=18, iter=11870/40000, loss=0.2599, lr=0.007285, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:09:49 2021-01-18 12:28:43 [INFO] [TRAIN] epoch=18, iter=11880/40000, loss=0.4148, lr=0.007282, batch_cost=0.7897, reader_cost=0.0001 | ETA 06:10:07 2021-01-18 12:28:51 [INFO] [TRAIN] epoch=18, iter=11890/40000, loss=0.3887, lr=0.007280, batch_cost=0.7920, reader_cost=0.0001 | ETA 06:11:03 2021-01-18 12:28:58 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.4256, lr=0.007278, batch_cost=0.7777, reader_cost=0.0166 | ETA 06:04:12 2021-01-18 12:29:06 [INFO] [TRAIN] epoch=19, iter=11910/40000, loss=0.2921, lr=0.007275, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:09:29 2021-01-18 12:29:14 [INFO] [TRAIN] epoch=19, iter=11920/40000, loss=0.5249, lr=0.007273, batch_cost=0.7864, reader_cost=0.0001 | ETA 06:08:01 2021-01-18 12:29:22 [INFO] [TRAIN] epoch=19, iter=11930/40000, loss=0.2517, lr=0.007271, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:09:59 2021-01-18 12:29:30 [INFO] [TRAIN] epoch=19, iter=11940/40000, loss=0.2892, lr=0.007268, batch_cost=0.7866, reader_cost=0.0001 | ETA 06:07:53 2021-01-18 12:29:38 [INFO] [TRAIN] epoch=19, iter=11950/40000, loss=0.2953, lr=0.007266, batch_cost=0.7901, reader_cost=0.0001 | ETA 06:09:21 2021-01-18 12:29:46 [INFO] [TRAIN] epoch=19, iter=11960/40000, loss=0.3652, lr=0.007264, batch_cost=0.7873, reader_cost=0.0001 | ETA 06:07:54 2021-01-18 12:29:54 [INFO] [TRAIN] epoch=19, iter=11970/40000, loss=0.4394, lr=0.007261, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:09:09 2021-01-18 12:30:02 [INFO] [TRAIN] epoch=19, iter=11980/40000, loss=0.3817, lr=0.007259, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:08:22 2021-01-18 12:30:09 [INFO] [TRAIN] epoch=19, iter=11990/40000, loss=0.2387, lr=0.007257, batch_cost=0.7910, reader_cost=0.0001 | ETA 06:09:15 2021-01-18 12:30:17 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.4806, lr=0.007254, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:08:06 2021-01-18 12:30:17 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 12:30:47 [INFO] [EVAL] #Images=1449 mIoU=0.7025 Acc=0.9347 Kappa=0.8555 2021-01-18 12:30:47 [INFO] [EVAL] Class IoU: [0.9327 0.8342 0.3024 0.8529 0.5664 0.698 0.8736 0.6876 0.879 0.3786 0.8428 0.4735 0.8313 0.85 0.622 0.841 0.5584 0.7745 0.4323 0.8448 0.6755] 2021-01-18 12:30:47 [INFO] [EVAL] Class Acc: [0.9608 0.8813 0.4002 0.9427 0.6278 0.8562 0.8869 0.7465 0.921 0.5976 0.947 0.9435 0.9132 0.9166 0.759 0.938 0.8692 0.8324 0.7399 0.8781 0.8756] 2021-01-18 12:30:53 [INFO] [EVAL] The model with the best validation mIoU (0.7495) was saved at iter 11000. 2021-01-18 12:31:01 [INFO] [TRAIN] epoch=19, iter=12010/40000, loss=0.3870, lr=0.007252, batch_cost=0.7830, reader_cost=0.0001 | ETA 06:05:15 2021-01-18 12:31:09 [INFO] [TRAIN] epoch=19, iter=12020/40000, loss=0.3213, lr=0.007250, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:09:41 2021-01-18 12:31:17 [INFO] [TRAIN] epoch=19, iter=12030/40000, loss=0.3058, lr=0.007247, batch_cost=0.7928, reader_cost=0.0001 | ETA 06:09:34 2021-01-18 12:31:25 [INFO] [TRAIN] epoch=19, iter=12040/40000, loss=0.5076, lr=0.007245, batch_cost=0.7949, reader_cost=0.0001 | ETA 06:10:24 2021-01-18 12:31:33 [INFO] [TRAIN] epoch=19, iter=12050/40000, loss=0.3591, lr=0.007243, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:07:29 2021-01-18 12:31:41 [INFO] [TRAIN] epoch=19, iter=12060/40000, loss=0.3413, lr=0.007240, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:07:58 2021-01-18 12:31:49 [INFO] [TRAIN] epoch=19, iter=12070/40000, loss=0.3220, lr=0.007238, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:09:23 2021-01-18 12:31:56 [INFO] [TRAIN] epoch=19, iter=12080/40000, loss=0.3443, lr=0.007236, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:07:43 2021-01-18 12:32:04 [INFO] [TRAIN] epoch=19, iter=12090/40000, loss=0.2945, lr=0.007233, batch_cost=0.7935, reader_cost=0.0001 | ETA 06:09:06 2021-01-18 12:32:12 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.3236, lr=0.007231, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:08:13 2021-01-18 12:32:20 [INFO] [TRAIN] epoch=19, iter=12110/40000, loss=0.2858, lr=0.007229, batch_cost=0.7919, reader_cost=0.0001 | ETA 06:08:05 2021-01-18 12:32:28 [INFO] [TRAIN] epoch=19, iter=12120/40000, loss=0.4900, lr=0.007226, batch_cost=0.7936, reader_cost=0.0001 | ETA 06:08:44 2021-01-18 12:32:36 [INFO] [TRAIN] epoch=19, iter=12130/40000, loss=0.4724, lr=0.007224, batch_cost=0.7912, reader_cost=0.0001 | ETA 06:07:31 2021-01-18 12:32:44 [INFO] [TRAIN] epoch=19, iter=12140/40000, loss=0.3782, lr=0.007222, batch_cost=0.7890, reader_cost=0.0001 | ETA 06:06:21 2021-01-18 12:32:52 [INFO] [TRAIN] epoch=19, iter=12150/40000, loss=0.3528, lr=0.007219, batch_cost=0.7927, reader_cost=0.0001 | ETA 06:07:56 2021-01-18 12:33:00 [INFO] [TRAIN] epoch=19, iter=12160/40000, loss=0.4756, lr=0.007217, batch_cost=0.7914, reader_cost=0.0001 | ETA 06:07:12 2021-01-18 12:33:08 [INFO] [TRAIN] epoch=19, iter=12170/40000, loss=0.3856, lr=0.007215, batch_cost=0.7890, reader_cost=0.0001 | ETA 06:05:58 2021-01-18 12:33:16 [INFO] [TRAIN] epoch=19, iter=12180/40000, loss=0.5807, lr=0.007212, batch_cost=0.7861, reader_cost=0.0001 | ETA 06:04:28 2021-01-18 12:33:23 [INFO] [TRAIN] epoch=19, iter=12190/40000, loss=0.3659, lr=0.007210, batch_cost=0.7903, reader_cost=0.0001 | ETA 06:06:18 2021-01-18 12:33:31 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.4370, lr=0.007208, batch_cost=0.7893, reader_cost=0.0001 | ETA 06:05:42 2021-01-18 12:33:39 [INFO] [TRAIN] epoch=19, iter=12210/40000, loss=0.4862, lr=0.007205, batch_cost=0.7912, reader_cost=0.0001 | ETA 06:06:27 2021-01-18 12:33:47 [INFO] [TRAIN] epoch=19, iter=12220/40000, loss=0.3208, lr=0.007203, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:07:47 2021-01-18 12:33:55 [INFO] [TRAIN] epoch=19, iter=12230/40000, loss=0.2813, lr=0.007201, batch_cost=0.7952, reader_cost=0.0001 | ETA 06:08:02 2021-01-18 12:34:03 [INFO] [TRAIN] epoch=19, iter=12240/40000, loss=0.3700, lr=0.007198, batch_cost=0.7867, reader_cost=0.0001 | ETA 06:03:58 2021-01-18 12:34:11 [INFO] [TRAIN] epoch=19, iter=12250/40000, loss=0.3928, lr=0.007196, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:04:59 2021-01-18 12:34:19 [INFO] [TRAIN] epoch=19, iter=12260/40000, loss=0.3616, lr=0.007194, batch_cost=0.7897, reader_cost=0.0001 | ETA 06:05:05 2021-01-18 12:34:27 [INFO] [TRAIN] epoch=19, iter=12270/40000, loss=0.4183, lr=0.007191, batch_cost=0.7945, reader_cost=0.0001 | ETA 06:07:11 2021-01-18 12:34:35 [INFO] [TRAIN] epoch=19, iter=12280/40000, loss=0.2646, lr=0.007189, batch_cost=0.7899, reader_cost=0.0001 | ETA 06:04:55 2021-01-18 12:34:43 [INFO] [TRAIN] epoch=19, iter=12290/40000, loss=0.2451, lr=0.007187, batch_cost=0.7943, reader_cost=0.0001 | ETA 06:06:48 2021-01-18 12:34:50 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.2424, lr=0.007184, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:04:37 2021-01-18 12:34:58 [INFO] [TRAIN] epoch=19, iter=12310/40000, loss=0.2476, lr=0.007182, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:04:24 2021-01-18 12:35:06 [INFO] [TRAIN] epoch=19, iter=12320/40000, loss=0.2943, lr=0.007180, batch_cost=0.7898, reader_cost=0.0001 | ETA 06:04:22 2021-01-18 12:35:14 [INFO] [TRAIN] epoch=19, iter=12330/40000, loss=0.3783, lr=0.007177, batch_cost=0.7941, reader_cost=0.0001 | ETA 06:06:11 2021-01-18 12:35:22 [INFO] [TRAIN] epoch=19, iter=12340/40000, loss=0.3803, lr=0.007175, batch_cost=0.7829, reader_cost=0.0001 | ETA 06:00:53 2021-01-18 12:35:30 [INFO] [TRAIN] epoch=19, iter=12350/40000, loss=0.2496, lr=0.007173, batch_cost=0.7944, reader_cost=0.0001 | ETA 06:06:04 2021-01-18 12:35:38 [INFO] [TRAIN] epoch=19, iter=12360/40000, loss=0.3637, lr=0.007170, batch_cost=0.7875, reader_cost=0.0001 | ETA 06:02:46 2021-01-18 12:35:46 [INFO] [TRAIN] epoch=19, iter=12370/40000, loss=0.3450, lr=0.007168, batch_cost=0.7842, reader_cost=0.0001 | ETA 06:01:07 2021-01-18 12:35:54 [INFO] [TRAIN] epoch=19, iter=12380/40000, loss=0.3465, lr=0.007166, batch_cost=0.7891, reader_cost=0.0001 | ETA 06:03:15 2021-01-18 12:36:01 [INFO] [TRAIN] epoch=19, iter=12390/40000, loss=0.2636, lr=0.007163, batch_cost=0.7859, reader_cost=0.0001 | ETA 06:01:38 2021-01-18 12:36:09 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.3307, lr=0.007161, batch_cost=0.7874, reader_cost=0.0001 | ETA 06:02:11 2021-01-18 12:36:17 [INFO] [TRAIN] epoch=19, iter=12410/40000, loss=0.3302, lr=0.007159, batch_cost=0.7896, reader_cost=0.0001 | ETA 06:03:05 2021-01-18 12:36:25 [INFO] [TRAIN] epoch=19, iter=12420/40000, loss=0.2928, lr=0.007156, batch_cost=0.7889, reader_cost=0.0001 | ETA 06:02:37 2021-01-18 12:36:33 [INFO] [TRAIN] epoch=19, iter=12430/40000, loss=0.3783, lr=0.007154, batch_cost=0.7845, reader_cost=0.0001 | ETA 06:00:28 2021-01-18 12:36:41 [INFO] [TRAIN] epoch=19, iter=12440/40000, loss=0.3103, lr=0.007152, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:02:19 2021-01-18 12:36:49 [INFO] [TRAIN] epoch=19, iter=12450/40000, loss=0.5460, lr=0.007149, batch_cost=0.7969, reader_cost=0.0001 | ETA 06:05:53 2021-01-18 12:36:57 [INFO] [TRAIN] epoch=19, iter=12460/40000, loss=0.2355, lr=0.007147, batch_cost=0.7864, reader_cost=0.0001 | ETA 06:00:56 2021-01-18 12:37:05 [INFO] [TRAIN] epoch=19, iter=12470/40000, loss=0.7339, lr=0.007145, batch_cost=0.7893, reader_cost=0.0001 | ETA 06:02:09 2021-01-18 12:37:12 [INFO] [TRAIN] epoch=19, iter=12480/40000, loss=0.3612, lr=0.007142, batch_cost=0.7884, reader_cost=0.0001 | ETA 06:01:35 2021-01-18 12:37:20 [INFO] [TRAIN] epoch=19, iter=12490/40000, loss=0.4358, lr=0.007140, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:01:05 2021-01-18 12:37:28 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.3609, lr=0.007138, batch_cost=0.7876, reader_cost=0.0001 | ETA 06:00:58 2021-01-18 12:37:36 [INFO] [TRAIN] epoch=19, iter=12510/40000, loss=0.3842, lr=0.007135, batch_cost=0.7870, reader_cost=0.0001 | ETA 06:00:33 2021-01-18 12:37:44 [INFO] [TRAIN] epoch=19, iter=12520/40000, loss=0.4313, lr=0.007133, batch_cost=0.7888, reader_cost=0.0001 | ETA 06:01:16 2021-01-18 12:37:52 [INFO] [TRAIN] epoch=19, iter=12530/40000, loss=0.2603, lr=0.007131, batch_cost=0.7902, reader_cost=0.0001 | ETA 06:01:45 2021-01-18 12:38:00 [INFO] [TRAIN] epoch=19, iter=12540/40000, loss=0.3123, lr=0.007128, batch_cost=0.7903, reader_cost=0.0001 | ETA 06:01:41 2021-01-18 12:38:08 [INFO] [TRAIN] epoch=19, iter=12550/40000, loss=0.3632, lr=0.007126, batch_cost=0.7892, reader_cost=0.0001 | ETA 06:01:02 2021-01-18 12:38:15 [INFO] [TRAIN] epoch=20, iter=12560/40000, loss=0.3014, lr=0.007124, batch_cost=0.7769, reader_cost=0.0154 | ETA 05:55:17 2021-01-18 12:38:23 [INFO] [TRAIN] epoch=20, iter=12570/40000, loss=0.4719, lr=0.007121, batch_cost=0.7916, reader_cost=0.0001 | ETA 06:01:54 2021-01-18 12:38:31 [INFO] [TRAIN] epoch=20, iter=12580/40000, loss=0.4386, lr=0.007119, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:59:54 2021-01-18 12:38:39 [INFO] [TRAIN] epoch=20, iter=12590/40000, loss=0.3840, lr=0.007117, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:59:47 2021-01-18 12:38:47 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.3724, lr=0.007114, batch_cost=0.7872, reader_cost=0.0001 | ETA 05:59:28 2021-01-18 12:38:55 [INFO] [TRAIN] epoch=20, iter=12610/40000, loss=0.3221, lr=0.007112, batch_cost=0.7909, reader_cost=0.0001 | ETA 06:01:02 2021-01-18 12:39:03 [INFO] [TRAIN] epoch=20, iter=12620/40000, loss=0.2606, lr=0.007110, batch_cost=0.7830, reader_cost=0.0001 | ETA 05:57:18 2021-01-18 12:39:11 [INFO] [TRAIN] epoch=20, iter=12630/40000, loss=0.2132, lr=0.007107, batch_cost=0.7870, reader_cost=0.0001 | ETA 05:58:59 2021-01-18 12:39:18 [INFO] [TRAIN] epoch=20, iter=12640/40000, loss=0.2279, lr=0.007105, batch_cost=0.7869, reader_cost=0.0001 | ETA 05:58:49 2021-01-18 12:39:26 [INFO] [TRAIN] epoch=20, iter=12650/40000, loss=0.2539, lr=0.007103, batch_cost=0.7879, reader_cost=0.0001 | ETA 05:59:09 2021-01-18 12:39:34 [INFO] [TRAIN] epoch=20, iter=12660/40000, loss=0.2495, lr=0.007100, batch_cost=0.7903, reader_cost=0.0001 | ETA 06:00:06 2021-01-18 12:39:42 [INFO] [TRAIN] epoch=20, iter=12670/40000, loss=0.3307, lr=0.007098, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:59:29 2021-01-18 12:39:50 [INFO] [TRAIN] epoch=20, iter=12680/40000, loss=0.2539, lr=0.007096, batch_cost=0.7933, reader_cost=0.0001 | ETA 06:01:11 2021-01-18 12:39:58 [INFO] [TRAIN] epoch=20, iter=12690/40000, loss=0.2789, lr=0.007093, batch_cost=0.7918, reader_cost=0.0001 | ETA 06:00:24 2021-01-18 12:40:06 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.3015, lr=0.007091, batch_cost=0.7851, reader_cost=0.0001 | ETA 05:57:13 2021-01-18 12:40:14 [INFO] [TRAIN] epoch=20, iter=12710/40000, loss=0.2205, lr=0.007089, batch_cost=0.7945, reader_cost=0.0001 | ETA 06:01:22 2021-01-18 12:40:22 [INFO] [TRAIN] epoch=20, iter=12720/40000, loss=0.2118, lr=0.007086, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:59:15 2021-01-18 12:40:30 [INFO] [TRAIN] epoch=20, iter=12730/40000, loss=0.2474, lr=0.007084, batch_cost=0.7898, reader_cost=0.0001 | ETA 05:58:58 2021-01-18 12:40:37 [INFO] [TRAIN] epoch=20, iter=12740/40000, loss=0.2790, lr=0.007082, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:57:49 2021-01-18 12:40:45 [INFO] [TRAIN] epoch=20, iter=12750/40000, loss=0.2355, lr=0.007079, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:58:26 2021-01-18 12:40:53 [INFO] [TRAIN] epoch=20, iter=12760/40000, loss=0.1868, lr=0.007077, batch_cost=0.7894, reader_cost=0.0001 | ETA 05:58:23 2021-01-18 12:41:01 [INFO] [TRAIN] epoch=20, iter=12770/40000, loss=0.3714, lr=0.007075, batch_cost=0.7858, reader_cost=0.0001 | ETA 05:56:36 2021-01-18 12:41:09 [INFO] [TRAIN] epoch=20, iter=12780/40000, loss=0.3584, lr=0.007072, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:59:27 2021-01-18 12:41:17 [INFO] [TRAIN] epoch=20, iter=12790/40000, loss=0.3536, lr=0.007070, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:58:51 2021-01-18 12:41:25 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.2581, lr=0.007068, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:57:41 2021-01-18 12:41:33 [INFO] [TRAIN] epoch=20, iter=12810/40000, loss=0.3743, lr=0.007065, batch_cost=0.7835, reader_cost=0.0001 | ETA 05:55:03 2021-01-18 12:41:41 [INFO] [TRAIN] epoch=20, iter=12820/40000, loss=0.3526, lr=0.007063, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:57:16 2021-01-18 12:41:48 [INFO] [TRAIN] epoch=20, iter=12830/40000, loss=0.1938, lr=0.007061, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:57:36 2021-01-18 12:41:56 [INFO] [TRAIN] epoch=20, iter=12840/40000, loss=0.3567, lr=0.007058, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:58:13 2021-01-18 12:42:04 [INFO] [TRAIN] epoch=20, iter=12850/40000, loss=0.3741, lr=0.007056, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:57:34 2021-01-18 12:42:12 [INFO] [TRAIN] epoch=20, iter=12860/40000, loss=0.4130, lr=0.007054, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:58:56 2021-01-18 12:42:20 [INFO] [TRAIN] epoch=20, iter=12870/40000, loss=0.3260, lr=0.007051, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:56:37 2021-01-18 12:42:28 [INFO] [TRAIN] epoch=20, iter=12880/40000, loss=0.2848, lr=0.007049, batch_cost=0.7899, reader_cost=0.0001 | ETA 05:57:01 2021-01-18 12:42:36 [INFO] [TRAIN] epoch=20, iter=12890/40000, loss=0.4611, lr=0.007047, batch_cost=0.7920, reader_cost=0.0001 | ETA 05:57:52 2021-01-18 12:42:44 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.4450, lr=0.007044, batch_cost=0.7873, reader_cost=0.0001 | ETA 05:55:36 2021-01-18 12:42:52 [INFO] [TRAIN] epoch=20, iter=12910/40000, loss=0.4024, lr=0.007042, batch_cost=0.7882, reader_cost=0.0001 | ETA 05:55:53 2021-01-18 12:43:00 [INFO] [TRAIN] epoch=20, iter=12920/40000, loss=0.2801, lr=0.007040, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:58:05 2021-01-18 12:43:07 [INFO] [TRAIN] epoch=20, iter=12930/40000, loss=0.3336, lr=0.007037, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:55:55 2021-01-18 12:43:15 [INFO] [TRAIN] epoch=20, iter=12940/40000, loss=0.3176, lr=0.007035, batch_cost=0.7883, reader_cost=0.0001 | ETA 05:55:31 2021-01-18 12:43:23 [INFO] [TRAIN] epoch=20, iter=12950/40000, loss=0.3170, lr=0.007033, batch_cost=0.7858, reader_cost=0.0001 | ETA 05:54:16 2021-01-18 12:43:31 [INFO] [TRAIN] epoch=20, iter=12960/40000, loss=0.3397, lr=0.007030, batch_cost=0.7878, reader_cost=0.0001 | ETA 05:55:01 2021-01-18 12:43:39 [INFO] [TRAIN] epoch=20, iter=12970/40000, loss=0.4431, lr=0.007028, batch_cost=0.7841, reader_cost=0.0001 | ETA 05:53:14 2021-01-18 12:43:47 [INFO] [TRAIN] epoch=20, iter=12980/40000, loss=0.3494, lr=0.007026, batch_cost=0.7938, reader_cost=0.0001 | ETA 05:57:28 2021-01-18 12:43:55 [INFO] [TRAIN] epoch=20, iter=12990/40000, loss=0.3542, lr=0.007023, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:56:53 2021-01-18 12:44:03 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.3080, lr=0.007021, batch_cost=0.7976, reader_cost=0.0001 | ETA 05:58:55 2021-01-18 12:44:03 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 12:44:31 [INFO] [EVAL] #Images=1449 mIoU=0.7530 Acc=0.9432 Kappa=0.8746 2021-01-18 12:44:31 [INFO] [EVAL] Class IoU: [0.9385 0.862 0.4101 0.8821 0.6866 0.7995 0.9351 0.8385 0.876 0.3964 0.8645 0.3823 0.8372 0.8888 0.8425 0.8617 0.5985 0.824 0.44 0.908 0.7405] 2021-01-18 12:44:31 [INFO] [EVAL] Class Acc: [0.9647 0.8869 0.4222 0.9387 0.8812 0.8958 0.9707 0.8982 0.9034 0.489 0.9461 0.8047 0.9092 0.9401 0.9195 0.9364 0.7804 0.9186 0.8138 0.9522 0.9101] 2021-01-18 12:44:41 [INFO] [EVAL] The model with the best validation mIoU (0.7530) was saved at iter 13000. 2021-01-18 12:44:49 [INFO] [TRAIN] epoch=20, iter=13010/40000, loss=0.2391, lr=0.007018, batch_cost=0.7825, reader_cost=0.0001 | ETA 05:51:59 2021-01-18 12:44:57 [INFO] [TRAIN] epoch=20, iter=13020/40000, loss=0.2852, lr=0.007016, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:55:53 2021-01-18 12:45:05 [INFO] [TRAIN] epoch=20, iter=13030/40000, loss=0.2773, lr=0.007014, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:55:30 2021-01-18 12:45:13 [INFO] [TRAIN] epoch=20, iter=13040/40000, loss=0.2960, lr=0.007011, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:55:34 2021-01-18 12:45:21 [INFO] [TRAIN] epoch=20, iter=13050/40000, loss=0.2308, lr=0.007009, batch_cost=0.7910, reader_cost=0.0001 | ETA 05:55:18 2021-01-18 12:45:29 [INFO] [TRAIN] epoch=20, iter=13060/40000, loss=0.2594, lr=0.007007, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:55:44 2021-01-18 12:45:37 [INFO] [TRAIN] epoch=20, iter=13070/40000, loss=0.3657, lr=0.007004, batch_cost=0.7888, reader_cost=0.0001 | ETA 05:54:01 2021-01-18 12:45:45 [INFO] [TRAIN] epoch=20, iter=13080/40000, loss=0.3372, lr=0.007002, batch_cost=0.7844, reader_cost=0.0001 | ETA 05:51:56 2021-01-18 12:45:52 [INFO] [TRAIN] epoch=20, iter=13090/40000, loss=0.3631, lr=0.007000, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:54:07 2021-01-18 12:46:00 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.4099, lr=0.006997, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:55:36 2021-01-18 12:46:08 [INFO] [TRAIN] epoch=20, iter=13110/40000, loss=0.3161, lr=0.006995, batch_cost=0.7955, reader_cost=0.0001 | ETA 05:56:31 2021-01-18 12:46:16 [INFO] [TRAIN] epoch=20, iter=13120/40000, loss=0.1836, lr=0.006993, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:52:52 2021-01-18 12:46:24 [INFO] [TRAIN] epoch=20, iter=13130/40000, loss=0.2032, lr=0.006990, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:55:06 2021-01-18 12:46:32 [INFO] [TRAIN] epoch=20, iter=13140/40000, loss=0.3696, lr=0.006988, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:52:38 2021-01-18 12:46:40 [INFO] [TRAIN] epoch=20, iter=13150/40000, loss=0.2398, lr=0.006986, batch_cost=0.7888, reader_cost=0.0001 | ETA 05:53:00 2021-01-18 12:46:48 [INFO] [TRAIN] epoch=20, iter=13160/40000, loss=0.2768, lr=0.006983, batch_cost=0.7946, reader_cost=0.0001 | ETA 05:55:28 2021-01-18 12:46:56 [INFO] [TRAIN] epoch=20, iter=13170/40000, loss=0.2374, lr=0.006981, batch_cost=0.7848, reader_cost=0.0001 | ETA 05:50:56 2021-01-18 12:47:04 [INFO] [TRAIN] epoch=20, iter=13180/40000, loss=0.2044, lr=0.006979, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:53:44 2021-01-18 12:47:11 [INFO] [TRAIN] epoch=20, iter=13190/40000, loss=0.2618, lr=0.006976, batch_cost=0.7907, reader_cost=0.0001 | ETA 05:53:19 2021-01-18 12:47:19 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.2207, lr=0.006974, batch_cost=0.7875, reader_cost=0.0001 | ETA 05:51:46 2021-01-18 12:47:27 [INFO] [TRAIN] epoch=20, iter=13210/40000, loss=0.2599, lr=0.006972, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:52:48 2021-01-18 12:47:35 [INFO] [TRAIN] epoch=20, iter=13220/40000, loss=0.3494, lr=0.006969, batch_cost=0.7596, reader_cost=0.0001 | ETA 05:39:02 2021-01-18 12:47:43 [INFO] [TRAIN] epoch=21, iter=13230/40000, loss=0.2604, lr=0.006967, batch_cost=0.8079, reader_cost=0.0155 | ETA 06:00:26 2021-01-18 12:47:51 [INFO] [TRAIN] epoch=21, iter=13240/40000, loss=0.3571, lr=0.006965, batch_cost=0.7937, reader_cost=0.0001 | ETA 05:53:59 2021-01-18 12:47:59 [INFO] [TRAIN] epoch=21, iter=13250/40000, loss=0.4245, lr=0.006962, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:51:58 2021-01-18 12:48:07 [INFO] [TRAIN] epoch=21, iter=13260/40000, loss=0.2502, lr=0.006960, batch_cost=0.7899, reader_cost=0.0001 | ETA 05:52:02 2021-01-18 12:48:15 [INFO] [TRAIN] epoch=21, iter=13270/40000, loss=0.2368, lr=0.006958, batch_cost=0.7941, reader_cost=0.0001 | ETA 05:53:47 2021-01-18 12:48:23 [INFO] [TRAIN] epoch=21, iter=13280/40000, loss=0.2045, lr=0.006955, batch_cost=0.7903, reader_cost=0.0001 | ETA 05:51:57 2021-01-18 12:48:30 [INFO] [TRAIN] epoch=21, iter=13290/40000, loss=0.2212, lr=0.006953, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:51:56 2021-01-18 12:48:38 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.2812, lr=0.006951, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:52:59 2021-01-18 12:48:46 [INFO] [TRAIN] epoch=21, iter=13310/40000, loss=0.2131, lr=0.006948, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:52:55 2021-01-18 12:48:54 [INFO] [TRAIN] epoch=21, iter=13320/40000, loss=0.3090, lr=0.006946, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:51:09 2021-01-18 12:49:02 [INFO] [TRAIN] epoch=21, iter=13330/40000, loss=0.3557, lr=0.006944, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:50:08 2021-01-18 12:49:10 [INFO] [TRAIN] epoch=21, iter=13340/40000, loss=0.2064, lr=0.006941, batch_cost=0.7931, reader_cost=0.0001 | ETA 05:52:23 2021-01-18 12:49:18 [INFO] [TRAIN] epoch=21, iter=13350/40000, loss=0.2808, lr=0.006939, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:52:14 2021-01-18 12:49:26 [INFO] [TRAIN] epoch=21, iter=13360/40000, loss=0.3522, lr=0.006937, batch_cost=0.7850, reader_cost=0.0001 | ETA 05:48:33 2021-01-18 12:49:34 [INFO] [TRAIN] epoch=21, iter=13370/40000, loss=0.4398, lr=0.006934, batch_cost=0.7918, reader_cost=0.0001 | ETA 05:51:25 2021-01-18 12:49:42 [INFO] [TRAIN] epoch=21, iter=13380/40000, loss=0.3938, lr=0.006932, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:51:04 2021-01-18 12:49:49 [INFO] [TRAIN] epoch=21, iter=13390/40000, loss=0.3120, lr=0.006929, batch_cost=0.7858, reader_cost=0.0001 | ETA 05:48:31 2021-01-18 12:49:57 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.3617, lr=0.006927, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:50:06 2021-01-18 12:50:05 [INFO] [TRAIN] epoch=21, iter=13410/40000, loss=0.3848, lr=0.006925, batch_cost=0.7894, reader_cost=0.0001 | ETA 05:49:50 2021-01-18 12:50:13 [INFO] [TRAIN] epoch=21, iter=13420/40000, loss=0.2861, lr=0.006922, batch_cost=0.7917, reader_cost=0.0001 | ETA 05:50:42 2021-01-18 12:50:21 [INFO] [TRAIN] epoch=21, iter=13430/40000, loss=0.3264, lr=0.006920, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:49:14 2021-01-18 12:50:29 [INFO] [TRAIN] epoch=21, iter=13440/40000, loss=0.2763, lr=0.006918, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:50:10 2021-01-18 12:50:37 [INFO] [TRAIN] epoch=21, iter=13450/40000, loss=0.3108, lr=0.006915, batch_cost=0.7862, reader_cost=0.0001 | ETA 05:47:52 2021-01-18 12:50:45 [INFO] [TRAIN] epoch=21, iter=13460/40000, loss=0.2400, lr=0.006913, batch_cost=0.7936, reader_cost=0.0001 | ETA 05:51:01 2021-01-18 12:50:53 [INFO] [TRAIN] epoch=21, iter=13470/40000, loss=0.2652, lr=0.006911, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:50:43 2021-01-18 12:51:01 [INFO] [TRAIN] epoch=21, iter=13480/40000, loss=0.4227, lr=0.006908, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:48:41 2021-01-18 12:51:09 [INFO] [TRAIN] epoch=21, iter=13490/40000, loss=0.2928, lr=0.006906, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:49:14 2021-01-18 12:51:16 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.4432, lr=0.006904, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:48:27 2021-01-18 12:51:24 [INFO] [TRAIN] epoch=21, iter=13510/40000, loss=0.2942, lr=0.006901, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:48:58 2021-01-18 12:51:32 [INFO] [TRAIN] epoch=21, iter=13520/40000, loss=0.3336, lr=0.006899, batch_cost=0.7869, reader_cost=0.0001 | ETA 05:47:15 2021-01-18 12:51:40 [INFO] [TRAIN] epoch=21, iter=13530/40000, loss=0.2171, lr=0.006897, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:48:24 2021-01-18 12:51:48 [INFO] [TRAIN] epoch=21, iter=13540/40000, loss=0.3596, lr=0.006894, batch_cost=0.7946, reader_cost=0.0001 | ETA 05:50:25 2021-01-18 12:51:56 [INFO] [TRAIN] epoch=21, iter=13550/40000, loss=0.3748, lr=0.006892, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:48:20 2021-01-18 12:52:04 [INFO] [TRAIN] epoch=21, iter=13560/40000, loss=0.2377, lr=0.006890, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:47:54 2021-01-18 12:52:12 [INFO] [TRAIN] epoch=21, iter=13570/40000, loss=0.2629, lr=0.006887, batch_cost=0.7872, reader_cost=0.0001 | ETA 05:46:46 2021-01-18 12:52:20 [INFO] [TRAIN] epoch=21, iter=13580/40000, loss=0.3579, lr=0.006885, batch_cost=0.7866, reader_cost=0.0001 | ETA 05:46:22 2021-01-18 12:52:27 [INFO] [TRAIN] epoch=21, iter=13590/40000, loss=0.2612, lr=0.006883, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:46:40 2021-01-18 12:52:35 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.2951, lr=0.006880, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:48:12 2021-01-18 12:52:43 [INFO] [TRAIN] epoch=21, iter=13610/40000, loss=0.2881, lr=0.006878, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:48:01 2021-01-18 12:52:51 [INFO] [TRAIN] epoch=21, iter=13620/40000, loss=0.2812, lr=0.006876, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:48:53 2021-01-18 12:52:59 [INFO] [TRAIN] epoch=21, iter=13630/40000, loss=0.2250, lr=0.006873, batch_cost=0.7903, reader_cost=0.0001 | ETA 05:47:20 2021-01-18 12:53:07 [INFO] [TRAIN] epoch=21, iter=13640/40000, loss=0.2729, lr=0.006871, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:47:14 2021-01-18 12:53:15 [INFO] [TRAIN] epoch=21, iter=13650/40000, loss=0.2698, lr=0.006869, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:47:34 2021-01-18 12:53:23 [INFO] [TRAIN] epoch=21, iter=13660/40000, loss=0.4154, lr=0.006866, batch_cost=0.7865, reader_cost=0.0001 | ETA 05:45:15 2021-01-18 12:53:31 [INFO] [TRAIN] epoch=21, iter=13670/40000, loss=0.2434, lr=0.006864, batch_cost=0.7946, reader_cost=0.0001 | ETA 05:48:41 2021-01-18 12:53:39 [INFO] [TRAIN] epoch=21, iter=13680/40000, loss=0.3346, lr=0.006861, batch_cost=0.7859, reader_cost=0.0001 | ETA 05:44:45 2021-01-18 12:53:47 [INFO] [TRAIN] epoch=21, iter=13690/40000, loss=0.2462, lr=0.006859, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:47:30 2021-01-18 12:53:54 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.3481, lr=0.006857, batch_cost=0.7919, reader_cost=0.0001 | ETA 05:47:07 2021-01-18 12:54:02 [INFO] [TRAIN] epoch=21, iter=13710/40000, loss=0.3802, lr=0.006854, batch_cost=0.7933, reader_cost=0.0001 | ETA 05:47:35 2021-01-18 12:54:10 [INFO] [TRAIN] epoch=21, iter=13720/40000, loss=0.3502, lr=0.006852, batch_cost=0.7949, reader_cost=0.0001 | ETA 05:48:10 2021-01-18 12:54:18 [INFO] [TRAIN] epoch=21, iter=13730/40000, loss=0.2855, lr=0.006850, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:47:23 2021-01-18 12:54:26 [INFO] [TRAIN] epoch=21, iter=13740/40000, loss=0.3086, lr=0.006847, batch_cost=0.7871, reader_cost=0.0001 | ETA 05:44:28 2021-01-18 12:54:34 [INFO] [TRAIN] epoch=21, iter=13750/40000, loss=0.2247, lr=0.006845, batch_cost=0.7986, reader_cost=0.0001 | ETA 05:49:22 2021-01-18 12:54:42 [INFO] [TRAIN] epoch=21, iter=13760/40000, loss=0.2545, lr=0.006843, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:45:08 2021-01-18 12:54:50 [INFO] [TRAIN] epoch=21, iter=13770/40000, loss=0.2016, lr=0.006840, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:46:35 2021-01-18 12:54:58 [INFO] [TRAIN] epoch=21, iter=13780/40000, loss=0.2679, lr=0.006838, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:45:05 2021-01-18 12:55:06 [INFO] [TRAIN] epoch=21, iter=13790/40000, loss=0.3950, lr=0.006836, batch_cost=0.7921, reader_cost=0.0001 | ETA 05:46:01 2021-01-18 12:55:14 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.2748, lr=0.006833, batch_cost=0.7940, reader_cost=0.0001 | ETA 05:46:43 2021-01-18 12:55:22 [INFO] [TRAIN] epoch=21, iter=13810/40000, loss=0.4010, lr=0.006831, batch_cost=0.7912, reader_cost=0.0001 | ETA 05:45:21 2021-01-18 12:55:29 [INFO] [TRAIN] epoch=21, iter=13820/40000, loss=0.3472, lr=0.006829, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:44:12 2021-01-18 12:55:37 [INFO] [TRAIN] epoch=21, iter=13830/40000, loss=0.2970, lr=0.006826, batch_cost=0.7873, reader_cost=0.0001 | ETA 05:43:23 2021-01-18 12:55:45 [INFO] [TRAIN] epoch=21, iter=13840/40000, loss=0.2546, lr=0.006824, batch_cost=0.7880, reader_cost=0.0001 | ETA 05:43:35 2021-01-18 12:55:53 [INFO] [TRAIN] epoch=21, iter=13850/40000, loss=0.3428, lr=0.006822, batch_cost=0.7915, reader_cost=0.0001 | ETA 05:44:56 2021-01-18 12:56:01 [INFO] [TRAIN] epoch=21, iter=13860/40000, loss=0.2244, lr=0.006819, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:45:40 2021-01-18 12:56:09 [INFO] [TRAIN] epoch=21, iter=13870/40000, loss=0.3274, lr=0.006817, batch_cost=0.7933, reader_cost=0.0001 | ETA 05:45:28 2021-01-18 12:56:17 [INFO] [TRAIN] epoch=21, iter=13880/40000, loss=0.2712, lr=0.006815, batch_cost=0.7604, reader_cost=0.0001 | ETA 05:31:01 2021-01-18 12:56:25 [INFO] [TRAIN] epoch=22, iter=13890/40000, loss=0.2959, lr=0.006812, batch_cost=0.8027, reader_cost=0.0151 | ETA 05:49:19 2021-01-18 12:56:33 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.2942, lr=0.006810, batch_cost=0.7970, reader_cost=0.0001 | ETA 05:46:40 2021-01-18 12:56:41 [INFO] [TRAIN] epoch=22, iter=13910/40000, loss=0.3439, lr=0.006808, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:44:05 2021-01-18 12:56:48 [INFO] [TRAIN] epoch=22, iter=13920/40000, loss=0.2675, lr=0.006805, batch_cost=0.7926, reader_cost=0.0001 | ETA 05:44:29 2021-01-18 12:56:56 [INFO] [TRAIN] epoch=22, iter=13930/40000, loss=0.3123, lr=0.006803, batch_cost=0.7903, reader_cost=0.0001 | ETA 05:43:23 2021-01-18 12:57:04 [INFO] [TRAIN] epoch=22, iter=13940/40000, loss=0.3423, lr=0.006800, batch_cost=0.7884, reader_cost=0.0001 | ETA 05:42:26 2021-01-18 12:57:12 [INFO] [TRAIN] epoch=22, iter=13950/40000, loss=0.5380, lr=0.006798, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:42:38 2021-01-18 12:57:20 [INFO] [TRAIN] epoch=22, iter=13960/40000, loss=0.3502, lr=0.006796, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:44:21 2021-01-18 12:57:28 [INFO] [TRAIN] epoch=22, iter=13970/40000, loss=0.2850, lr=0.006793, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:42:58 2021-01-18 12:57:36 [INFO] [TRAIN] epoch=22, iter=13980/40000, loss=0.1883, lr=0.006791, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:41:35 2021-01-18 12:57:44 [INFO] [TRAIN] epoch=22, iter=13990/40000, loss=0.3772, lr=0.006789, batch_cost=0.7939, reader_cost=0.0001 | ETA 05:44:10 2021-01-18 12:57:52 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.2098, lr=0.006786, batch_cost=0.7921, reader_cost=0.0001 | ETA 05:43:15 2021-01-18 12:57:52 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 12:58:20 [INFO] [EVAL] #Images=1449 mIoU=0.7564 Acc=0.9458 Kappa=0.8800 2021-01-18 12:58:20 [INFO] [EVAL] Class IoU: [0.9411 0.8998 0.4129 0.8579 0.658 0.763 0.917 0.8586 0.9051 0.3528 0.8819 0.5339 0.846 0.8882 0.8591 0.8599 0.5062 0.8629 0.5234 0.8237 0.7331] 2021-01-18 12:58:20 [INFO] [EVAL] Class Acc: [0.9647 0.9395 0.4297 0.9086 0.7616 0.855 0.9362 0.9176 0.9284 0.5599 0.9428 0.9093 0.9319 0.9449 0.9217 0.9466 0.7858 0.9341 0.7465 0.9447 0.911 ] 2021-01-18 12:58:31 [INFO] [EVAL] The model with the best validation mIoU (0.7564) was saved at iter 14000. 2021-01-18 12:58:38 [INFO] [TRAIN] epoch=22, iter=14010/40000, loss=0.3627, lr=0.006784, batch_cost=0.7838, reader_cost=0.0001 | ETA 05:39:32 2021-01-18 12:58:46 [INFO] [TRAIN] epoch=22, iter=14020/40000, loss=0.2311, lr=0.006782, batch_cost=0.7839, reader_cost=0.0001 | ETA 05:39:26 2021-01-18 12:58:54 [INFO] [TRAIN] epoch=22, iter=14030/40000, loss=0.2681, lr=0.006779, batch_cost=0.7867, reader_cost=0.0001 | ETA 05:40:30 2021-01-18 12:59:02 [INFO] [TRAIN] epoch=22, iter=14040/40000, loss=0.3148, lr=0.006777, batch_cost=0.7856, reader_cost=0.0001 | ETA 05:39:54 2021-01-18 12:59:10 [INFO] [TRAIN] epoch=22, iter=14050/40000, loss=0.2649, lr=0.006775, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:42:15 2021-01-18 12:59:18 [INFO] [TRAIN] epoch=22, iter=14060/40000, loss=0.2287, lr=0.006772, batch_cost=0.7927, reader_cost=0.0001 | ETA 05:42:42 2021-01-18 12:59:26 [INFO] [TRAIN] epoch=22, iter=14070/40000, loss=0.2813, lr=0.006770, batch_cost=0.7960, reader_cost=0.0001 | ETA 05:44:00 2021-01-18 12:59:34 [INFO] [TRAIN] epoch=22, iter=14080/40000, loss=0.3169, lr=0.006768, batch_cost=0.7867, reader_cost=0.0001 | ETA 05:39:50 2021-01-18 12:59:41 [INFO] [TRAIN] epoch=22, iter=14090/40000, loss=0.2392, lr=0.006765, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:40:10 2021-01-18 12:59:49 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.3596, lr=0.006763, batch_cost=0.7891, reader_cost=0.0001 | ETA 05:40:36 2021-01-18 12:59:57 [INFO] [TRAIN] epoch=22, iter=14110/40000, loss=0.3219, lr=0.006761, batch_cost=0.7950, reader_cost=0.0001 | ETA 05:43:02 2021-01-18 13:00:05 [INFO] [TRAIN] epoch=22, iter=14120/40000, loss=0.2376, lr=0.006758, batch_cost=0.7875, reader_cost=0.0001 | ETA 05:39:39 2021-01-18 13:00:13 [INFO] [TRAIN] epoch=22, iter=14130/40000, loss=0.3170, lr=0.006756, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:41:05 2021-01-18 13:00:21 [INFO] [TRAIN] epoch=22, iter=14140/40000, loss=0.2379, lr=0.006753, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:41:34 2021-01-18 13:00:29 [INFO] [TRAIN] epoch=22, iter=14150/40000, loss=0.4419, lr=0.006751, batch_cost=0.7862, reader_cost=0.0001 | ETA 05:38:43 2021-01-18 13:00:37 [INFO] [TRAIN] epoch=22, iter=14160/40000, loss=0.3300, lr=0.006749, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:39:39 2021-01-18 13:00:45 [INFO] [TRAIN] epoch=22, iter=14170/40000, loss=0.2893, lr=0.006746, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:40:22 2021-01-18 13:00:53 [INFO] [TRAIN] epoch=22, iter=14180/40000, loss=0.2568, lr=0.006744, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:39:28 2021-01-18 13:01:00 [INFO] [TRAIN] epoch=22, iter=14190/40000, loss=0.3377, lr=0.006742, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:39:39 2021-01-18 13:01:08 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.2681, lr=0.006739, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:40:40 2021-01-18 13:01:16 [INFO] [TRAIN] epoch=22, iter=14210/40000, loss=0.2616, lr=0.006737, batch_cost=0.7862, reader_cost=0.0001 | ETA 05:37:56 2021-01-18 13:01:24 [INFO] [TRAIN] epoch=22, iter=14220/40000, loss=0.2916, lr=0.006735, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:39:01 2021-01-18 13:01:32 [INFO] [TRAIN] epoch=22, iter=14230/40000, loss=0.3430, lr=0.006732, batch_cost=0.7927, reader_cost=0.0001 | ETA 05:40:26 2021-01-18 13:01:40 [INFO] [TRAIN] epoch=22, iter=14240/40000, loss=0.3339, lr=0.006730, batch_cost=0.7908, reader_cost=0.0001 | ETA 05:39:29 2021-01-18 13:01:48 [INFO] [TRAIN] epoch=22, iter=14250/40000, loss=0.2610, lr=0.006728, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:40:08 2021-01-18 13:01:56 [INFO] [TRAIN] epoch=22, iter=14260/40000, loss=0.6494, lr=0.006725, batch_cost=0.7938, reader_cost=0.0001 | ETA 05:40:33 2021-01-18 13:02:04 [INFO] [TRAIN] epoch=22, iter=14270/40000, loss=0.2117, lr=0.006723, batch_cost=0.7900, reader_cost=0.0001 | ETA 05:38:45 2021-01-18 13:02:12 [INFO] [TRAIN] epoch=22, iter=14280/40000, loss=0.3833, lr=0.006721, batch_cost=0.7900, reader_cost=0.0001 | ETA 05:38:39 2021-01-18 13:02:20 [INFO] [TRAIN] epoch=22, iter=14290/40000, loss=0.2789, lr=0.006718, batch_cost=0.7874, reader_cost=0.0001 | ETA 05:37:23 2021-01-18 13:02:27 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.3537, lr=0.006716, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:39:44 2021-01-18 13:02:35 [INFO] [TRAIN] epoch=22, iter=14310/40000, loss=0.2864, lr=0.006713, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:37:12 2021-01-18 13:02:43 [INFO] [TRAIN] epoch=22, iter=14320/40000, loss=0.2920, lr=0.006711, batch_cost=0.7857, reader_cost=0.0001 | ETA 05:36:16 2021-01-18 13:02:51 [INFO] [TRAIN] epoch=22, iter=14330/40000, loss=0.3588, lr=0.006709, batch_cost=0.7905, reader_cost=0.0001 | ETA 05:38:12 2021-01-18 13:02:59 [INFO] [TRAIN] epoch=22, iter=14340/40000, loss=0.2716, lr=0.006706, batch_cost=0.7910, reader_cost=0.0001 | ETA 05:38:16 2021-01-18 13:03:07 [INFO] [TRAIN] epoch=22, iter=14350/40000, loss=0.3003, lr=0.006704, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:38:56 2021-01-18 13:03:15 [INFO] [TRAIN] epoch=22, iter=14360/40000, loss=0.2444, lr=0.006702, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:37:08 2021-01-18 13:03:23 [INFO] [TRAIN] epoch=22, iter=14370/40000, loss=0.2672, lr=0.006699, batch_cost=0.7957, reader_cost=0.0001 | ETA 05:39:54 2021-01-18 13:03:31 [INFO] [TRAIN] epoch=22, iter=14380/40000, loss=0.2015, lr=0.006697, batch_cost=0.7843, reader_cost=0.0001 | ETA 05:34:53 2021-01-18 13:03:39 [INFO] [TRAIN] epoch=22, iter=14390/40000, loss=0.2935, lr=0.006695, batch_cost=0.7901, reader_cost=0.0001 | ETA 05:37:15 2021-01-18 13:03:46 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.4424, lr=0.006692, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:36:35 2021-01-18 13:03:54 [INFO] [TRAIN] epoch=22, iter=14410/40000, loss=0.2393, lr=0.006690, batch_cost=0.7997, reader_cost=0.0001 | ETA 05:41:03 2021-01-18 13:04:02 [INFO] [TRAIN] epoch=22, iter=14420/40000, loss=0.3686, lr=0.006688, batch_cost=0.7845, reader_cost=0.0001 | ETA 05:34:28 2021-01-18 13:04:10 [INFO] [TRAIN] epoch=22, iter=14430/40000, loss=0.3067, lr=0.006685, batch_cost=0.7894, reader_cost=0.0001 | ETA 05:36:24 2021-01-18 13:04:18 [INFO] [TRAIN] epoch=22, iter=14440/40000, loss=0.2959, lr=0.006683, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:35:57 2021-01-18 13:04:26 [INFO] [TRAIN] epoch=22, iter=14450/40000, loss=0.3651, lr=0.006681, batch_cost=0.7908, reader_cost=0.0001 | ETA 05:36:44 2021-01-18 13:04:34 [INFO] [TRAIN] epoch=22, iter=14460/40000, loss=0.2217, lr=0.006678, batch_cost=0.7931, reader_cost=0.0001 | ETA 05:37:36 2021-01-18 13:04:42 [INFO] [TRAIN] epoch=22, iter=14470/40000, loss=0.2335, lr=0.006676, batch_cost=0.7945, reader_cost=0.0001 | ETA 05:38:03 2021-01-18 13:04:50 [INFO] [TRAIN] epoch=22, iter=14480/40000, loss=0.2165, lr=0.006674, batch_cost=0.7878, reader_cost=0.0001 | ETA 05:35:04 2021-01-18 13:04:58 [INFO] [TRAIN] epoch=22, iter=14490/40000, loss=0.3205, lr=0.006671, batch_cost=0.7884, reader_cost=0.0001 | ETA 05:35:12 2021-01-18 13:05:05 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.3278, lr=0.006669, batch_cost=0.7893, reader_cost=0.0001 | ETA 05:35:26 2021-01-18 13:05:13 [INFO] [TRAIN] epoch=22, iter=14510/40000, loss=0.3627, lr=0.006666, batch_cost=0.7857, reader_cost=0.0001 | ETA 05:33:46 2021-01-18 13:05:21 [INFO] [TRAIN] epoch=22, iter=14520/40000, loss=0.2347, lr=0.006664, batch_cost=0.7918, reader_cost=0.0001 | ETA 05:36:15 2021-01-18 13:05:29 [INFO] [TRAIN] epoch=22, iter=14530/40000, loss=0.2428, lr=0.006662, batch_cost=0.7943, reader_cost=0.0001 | ETA 05:37:10 2021-01-18 13:05:37 [INFO] [TRAIN] epoch=22, iter=14540/40000, loss=0.3020, lr=0.006659, batch_cost=0.7689, reader_cost=0.0001 | ETA 05:26:14 2021-01-18 13:05:45 [INFO] [TRAIN] epoch=23, iter=14550/40000, loss=0.3697, lr=0.006657, batch_cost=0.7985, reader_cost=0.0118 | ETA 05:38:43 2021-01-18 13:05:53 [INFO] [TRAIN] epoch=23, iter=14560/40000, loss=0.3103, lr=0.006655, batch_cost=0.7916, reader_cost=0.0001 | ETA 05:35:38 2021-01-18 13:06:01 [INFO] [TRAIN] epoch=23, iter=14570/40000, loss=0.2738, lr=0.006652, batch_cost=0.7883, reader_cost=0.0001 | ETA 05:34:07 2021-01-18 13:06:09 [INFO] [TRAIN] epoch=23, iter=14580/40000, loss=0.2297, lr=0.006650, batch_cost=0.7873, reader_cost=0.0001 | ETA 05:33:33 2021-01-18 13:06:16 [INFO] [TRAIN] epoch=23, iter=14590/40000, loss=0.2373, lr=0.006648, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:34:27 2021-01-18 13:06:24 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.2419, lr=0.006645, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:35:23 2021-01-18 13:06:32 [INFO] [TRAIN] epoch=23, iter=14610/40000, loss=0.3291, lr=0.006643, batch_cost=0.7966, reader_cost=0.0001 | ETA 05:37:04 2021-01-18 13:06:40 [INFO] [TRAIN] epoch=23, iter=14620/40000, loss=0.3267, lr=0.006641, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:34:00 2021-01-18 13:06:48 [INFO] [TRAIN] epoch=23, iter=14630/40000, loss=0.2776, lr=0.006638, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:33:41 2021-01-18 13:06:56 [INFO] [TRAIN] epoch=23, iter=14640/40000, loss=0.1872, lr=0.006636, batch_cost=0.7856, reader_cost=0.0001 | ETA 05:32:01 2021-01-18 13:07:04 [INFO] [TRAIN] epoch=23, iter=14650/40000, loss=0.3435, lr=0.006633, batch_cost=0.7937, reader_cost=0.0001 | ETA 05:35:19 2021-01-18 13:07:12 [INFO] [TRAIN] epoch=23, iter=14660/40000, loss=0.3907, lr=0.006631, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:32:39 2021-01-18 13:07:20 [INFO] [TRAIN] epoch=23, iter=14670/40000, loss=0.3125, lr=0.006629, batch_cost=0.7893, reader_cost=0.0001 | ETA 05:33:13 2021-01-18 13:07:28 [INFO] [TRAIN] epoch=23, iter=14680/40000, loss=0.2933, lr=0.006626, batch_cost=0.7962, reader_cost=0.0001 | ETA 05:35:59 2021-01-18 13:07:36 [INFO] [TRAIN] epoch=23, iter=14690/40000, loss=0.2127, lr=0.006624, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:32:39 2021-01-18 13:07:43 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.2723, lr=0.006622, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:33:21 2021-01-18 13:07:51 [INFO] [TRAIN] epoch=23, iter=14710/40000, loss=0.2959, lr=0.006619, batch_cost=0.7959, reader_cost=0.0001 | ETA 05:35:27 2021-01-18 13:07:59 [INFO] [TRAIN] epoch=23, iter=14720/40000, loss=0.3543, lr=0.006617, batch_cost=0.7851, reader_cost=0.0001 | ETA 05:30:47 2021-01-18 13:08:07 [INFO] [TRAIN] epoch=23, iter=14730/40000, loss=0.2789, lr=0.006615, batch_cost=0.7942, reader_cost=0.0001 | ETA 05:34:28 2021-01-18 13:08:15 [INFO] [TRAIN] epoch=23, iter=14740/40000, loss=0.3891, lr=0.006612, batch_cost=0.7853, reader_cost=0.0001 | ETA 05:30:37 2021-01-18 13:08:23 [INFO] [TRAIN] epoch=23, iter=14750/40000, loss=0.4222, lr=0.006610, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:32:21 2021-01-18 13:08:31 [INFO] [TRAIN] epoch=23, iter=14760/40000, loss=0.4363, lr=0.006608, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:32:47 2021-01-18 13:08:39 [INFO] [TRAIN] epoch=23, iter=14770/40000, loss=0.2574, lr=0.006605, batch_cost=0.7850, reader_cost=0.0001 | ETA 05:30:05 2021-01-18 13:08:47 [INFO] [TRAIN] epoch=23, iter=14780/40000, loss=0.2779, lr=0.006603, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:33:19 2021-01-18 13:08:55 [INFO] [TRAIN] epoch=23, iter=14790/40000, loss=0.4504, lr=0.006601, batch_cost=0.7863, reader_cost=0.0001 | ETA 05:30:23 2021-01-18 13:09:02 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.4423, lr=0.006598, batch_cost=0.7916, reader_cost=0.0001 | ETA 05:32:29 2021-01-18 13:09:10 [INFO] [TRAIN] epoch=23, iter=14810/40000, loss=0.2708, lr=0.006596, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:31:27 2021-01-18 13:09:18 [INFO] [TRAIN] epoch=23, iter=14820/40000, loss=0.1720, lr=0.006593, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:31:53 2021-01-18 13:09:26 [INFO] [TRAIN] epoch=23, iter=14830/40000, loss=0.3055, lr=0.006591, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:31:33 2021-01-18 13:09:34 [INFO] [TRAIN] epoch=23, iter=14840/40000, loss=0.3184, lr=0.006589, batch_cost=0.7918, reader_cost=0.0001 | ETA 05:32:00 2021-01-18 13:09:42 [INFO] [TRAIN] epoch=23, iter=14850/40000, loss=0.2028, lr=0.006586, batch_cost=0.7878, reader_cost=0.0001 | ETA 05:30:13 2021-01-18 13:09:50 [INFO] [TRAIN] epoch=23, iter=14860/40000, loss=0.2441, lr=0.006584, batch_cost=0.7916, reader_cost=0.0001 | ETA 05:31:40 2021-01-18 13:09:58 [INFO] [TRAIN] epoch=23, iter=14870/40000, loss=0.2980, lr=0.006582, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:31:56 2021-01-18 13:10:06 [INFO] [TRAIN] epoch=23, iter=14880/40000, loss=0.2037, lr=0.006579, batch_cost=0.7917, reader_cost=0.0001 | ETA 05:31:27 2021-01-18 13:10:14 [INFO] [TRAIN] epoch=23, iter=14890/40000, loss=0.2577, lr=0.006577, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:31:35 2021-01-18 13:10:22 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.2814, lr=0.006575, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:30:57 2021-01-18 13:10:29 [INFO] [TRAIN] epoch=23, iter=14910/40000, loss=0.2205, lr=0.006572, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:30:37 2021-01-18 13:10:37 [INFO] [TRAIN] epoch=23, iter=14920/40000, loss=0.2306, lr=0.006570, batch_cost=0.7919, reader_cost=0.0001 | ETA 05:31:00 2021-01-18 13:10:45 [INFO] [TRAIN] epoch=23, iter=14930/40000, loss=0.2559, lr=0.006568, batch_cost=0.7884, reader_cost=0.0001 | ETA 05:29:24 2021-01-18 13:10:53 [INFO] [TRAIN] epoch=23, iter=14940/40000, loss=0.2912, lr=0.006565, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:31:12 2021-01-18 13:11:01 [INFO] [TRAIN] epoch=23, iter=14950/40000, loss=0.2938, lr=0.006563, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:31:05 2021-01-18 13:11:09 [INFO] [TRAIN] epoch=23, iter=14960/40000, loss=0.3061, lr=0.006560, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:29:56 2021-01-18 13:11:17 [INFO] [TRAIN] epoch=23, iter=14970/40000, loss=0.3695, lr=0.006558, batch_cost=0.7836, reader_cost=0.0001 | ETA 05:26:54 2021-01-18 13:11:25 [INFO] [TRAIN] epoch=23, iter=14980/40000, loss=0.2307, lr=0.006556, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:30:23 2021-01-18 13:11:33 [INFO] [TRAIN] epoch=23, iter=14990/40000, loss=0.2353, lr=0.006553, batch_cost=0.7962, reader_cost=0.0001 | ETA 05:31:53 2021-01-18 13:11:41 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.3089, lr=0.006551, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:28:35 2021-01-18 13:11:41 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 13:12:09 [INFO] [EVAL] #Images=1449 mIoU=0.7550 Acc=0.9455 Kappa=0.8792 2021-01-18 13:12:09 [INFO] [EVAL] Class IoU: [0.9408 0.8916 0.4186 0.8603 0.5939 0.7752 0.921 0.8815 0.8899 0.3656 0.8876 0.5628 0.8279 0.8689 0.8724 0.8617 0.6069 0.7874 0.4563 0.8452 0.7403] 2021-01-18 13:12:09 [INFO] [EVAL] Class Acc: [0.9644 0.9353 0.432 0.8821 0.884 0.8459 0.9786 0.9621 0.9187 0.6116 0.961 0.8779 0.9399 0.9305 0.9443 0.9211 0.7673 0.8075 0.7495 0.928 0.9414] 2021-01-18 13:12:15 [INFO] [EVAL] The model with the best validation mIoU (0.7564) was saved at iter 14000. 2021-01-18 13:12:23 [INFO] [TRAIN] epoch=23, iter=15010/40000, loss=0.1968, lr=0.006549, batch_cost=0.7827, reader_cost=0.0001 | ETA 05:25:58 2021-01-18 13:12:31 [INFO] [TRAIN] epoch=23, iter=15020/40000, loss=0.4460, lr=0.006546, batch_cost=0.7874, reader_cost=0.0001 | ETA 05:27:49 2021-01-18 13:12:39 [INFO] [TRAIN] epoch=23, iter=15030/40000, loss=0.2672, lr=0.006544, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:28:11 2021-01-18 13:12:46 [INFO] [TRAIN] epoch=23, iter=15040/40000, loss=0.2450, lr=0.006542, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:30:02 2021-01-18 13:12:54 [INFO] [TRAIN] epoch=23, iter=15050/40000, loss=0.3157, lr=0.006539, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:27:58 2021-01-18 13:13:02 [INFO] [TRAIN] epoch=23, iter=15060/40000, loss=0.3466, lr=0.006537, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:27:54 2021-01-18 13:13:10 [INFO] [TRAIN] epoch=23, iter=15070/40000, loss=0.2385, lr=0.006534, batch_cost=0.7916, reader_cost=0.0001 | ETA 05:28:54 2021-01-18 13:13:18 [INFO] [TRAIN] epoch=23, iter=15080/40000, loss=0.3332, lr=0.006532, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:29:25 2021-01-18 13:13:26 [INFO] [TRAIN] epoch=23, iter=15090/40000, loss=0.2617, lr=0.006530, batch_cost=0.7969, reader_cost=0.0001 | ETA 05:30:49 2021-01-18 13:13:34 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.3958, lr=0.006527, batch_cost=0.7919, reader_cost=0.0001 | ETA 05:28:39 2021-01-18 13:13:42 [INFO] [TRAIN] epoch=23, iter=15110/40000, loss=0.2886, lr=0.006525, batch_cost=0.7968, reader_cost=0.0001 | ETA 05:30:32 2021-01-18 13:13:50 [INFO] [TRAIN] epoch=23, iter=15120/40000, loss=0.3489, lr=0.006523, batch_cost=0.7894, reader_cost=0.0001 | ETA 05:27:20 2021-01-18 13:13:58 [INFO] [TRAIN] epoch=23, iter=15130/40000, loss=0.1774, lr=0.006520, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:27:36 2021-01-18 13:14:06 [INFO] [TRAIN] epoch=23, iter=15140/40000, loss=0.2430, lr=0.006518, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:26:59 2021-01-18 13:14:14 [INFO] [TRAIN] epoch=23, iter=15150/40000, loss=0.2357, lr=0.006516, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:26:45 2021-01-18 13:14:21 [INFO] [TRAIN] epoch=23, iter=15160/40000, loss=0.4510, lr=0.006513, batch_cost=0.7924, reader_cost=0.0001 | ETA 05:28:02 2021-01-18 13:14:29 [INFO] [TRAIN] epoch=23, iter=15170/40000, loss=0.3304, lr=0.006511, batch_cost=0.7881, reader_cost=0.0001 | ETA 05:26:08 2021-01-18 13:14:37 [INFO] [TRAIN] epoch=23, iter=15180/40000, loss=0.4739, lr=0.006509, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:27:09 2021-01-18 13:14:45 [INFO] [TRAIN] epoch=23, iter=15190/40000, loss=0.3502, lr=0.006506, batch_cost=0.7948, reader_cost=0.0001 | ETA 05:28:38 2021-01-18 13:14:53 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.3667, lr=0.006504, batch_cost=0.7782, reader_cost=0.0001 | ETA 05:21:38 2021-01-18 13:15:01 [INFO] [TRAIN] epoch=24, iter=15210/40000, loss=0.2744, lr=0.006501, batch_cost=0.7898, reader_cost=0.0176 | ETA 05:26:18 2021-01-18 13:15:09 [INFO] [TRAIN] epoch=24, iter=15220/40000, loss=0.3005, lr=0.006499, batch_cost=0.7908, reader_cost=0.0001 | ETA 05:26:36 2021-01-18 13:15:17 [INFO] [TRAIN] epoch=24, iter=15230/40000, loss=0.3315, lr=0.006497, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:25:55 2021-01-18 13:15:25 [INFO] [TRAIN] epoch=24, iter=15240/40000, loss=0.2565, lr=0.006494, batch_cost=0.7926, reader_cost=0.0001 | ETA 05:27:03 2021-01-18 13:15:32 [INFO] [TRAIN] epoch=24, iter=15250/40000, loss=0.3250, lr=0.006492, batch_cost=0.7843, reader_cost=0.0001 | ETA 05:23:30 2021-01-18 13:15:40 [INFO] [TRAIN] epoch=24, iter=15260/40000, loss=0.3174, lr=0.006490, batch_cost=0.7917, reader_cost=0.0001 | ETA 05:26:27 2021-01-18 13:15:48 [INFO] [TRAIN] epoch=24, iter=15270/40000, loss=0.2069, lr=0.006487, batch_cost=0.7872, reader_cost=0.0001 | ETA 05:24:26 2021-01-18 13:15:56 [INFO] [TRAIN] epoch=24, iter=15280/40000, loss=0.2217, lr=0.006485, batch_cost=0.7869, reader_cost=0.0001 | ETA 05:24:12 2021-01-18 13:16:04 [INFO] [TRAIN] epoch=24, iter=15290/40000, loss=0.2517, lr=0.006483, batch_cost=0.7872, reader_cost=0.0001 | ETA 05:24:11 2021-01-18 13:16:12 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.2552, lr=0.006480, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:24:13 2021-01-18 13:16:20 [INFO] [TRAIN] epoch=24, iter=15310/40000, loss=0.3115, lr=0.006478, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:24:04 2021-01-18 13:16:28 [INFO] [TRAIN] epoch=24, iter=15320/40000, loss=0.2633, lr=0.006475, batch_cost=0.7916, reader_cost=0.0001 | ETA 05:25:37 2021-01-18 13:16:36 [INFO] [TRAIN] epoch=24, iter=15330/40000, loss=0.2202, lr=0.006473, batch_cost=0.7922, reader_cost=0.0001 | ETA 05:25:43 2021-01-18 13:16:43 [INFO] [TRAIN] epoch=24, iter=15340/40000, loss=0.2872, lr=0.006471, batch_cost=0.7918, reader_cost=0.0001 | ETA 05:25:24 2021-01-18 13:16:51 [INFO] [TRAIN] epoch=24, iter=15350/40000, loss=0.2355, lr=0.006468, batch_cost=0.7874, reader_cost=0.0001 | ETA 05:23:28 2021-01-18 13:16:59 [INFO] [TRAIN] epoch=24, iter=15360/40000, loss=0.2511, lr=0.006466, batch_cost=0.7901, reader_cost=0.0001 | ETA 05:24:28 2021-01-18 13:17:07 [INFO] [TRAIN] epoch=24, iter=15370/40000, loss=0.2826, lr=0.006464, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:25:42 2021-01-18 13:17:15 [INFO] [TRAIN] epoch=24, iter=15380/40000, loss=0.1823, lr=0.006461, batch_cost=0.7919, reader_cost=0.0001 | ETA 05:24:56 2021-01-18 13:17:23 [INFO] [TRAIN] epoch=24, iter=15390/40000, loss=0.2695, lr=0.006459, batch_cost=0.7905, reader_cost=0.0001 | ETA 05:24:15 2021-01-18 13:17:31 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.3232, lr=0.006457, batch_cost=0.7901, reader_cost=0.0001 | ETA 05:23:56 2021-01-18 13:17:39 [INFO] [TRAIN] epoch=24, iter=15410/40000, loss=0.3047, lr=0.006454, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:25:12 2021-01-18 13:17:47 [INFO] [TRAIN] epoch=24, iter=15420/40000, loss=0.1748, lr=0.006452, batch_cost=0.7941, reader_cost=0.0001 | ETA 05:25:19 2021-01-18 13:17:55 [INFO] [TRAIN] epoch=24, iter=15430/40000, loss=0.3627, lr=0.006450, batch_cost=0.7833, reader_cost=0.0001 | ETA 05:20:46 2021-01-18 13:18:02 [INFO] [TRAIN] epoch=24, iter=15440/40000, loss=0.3537, lr=0.006447, batch_cost=0.7809, reader_cost=0.0001 | ETA 05:19:39 2021-01-18 13:18:10 [INFO] [TRAIN] epoch=24, iter=15450/40000, loss=0.2259, lr=0.006445, batch_cost=0.7945, reader_cost=0.0001 | ETA 05:25:05 2021-01-18 13:18:18 [INFO] [TRAIN] epoch=24, iter=15460/40000, loss=0.3236, lr=0.006442, batch_cost=0.7878, reader_cost=0.0001 | ETA 05:22:12 2021-01-18 13:18:26 [INFO] [TRAIN] epoch=24, iter=15470/40000, loss=0.2266, lr=0.006440, batch_cost=0.7917, reader_cost=0.0001 | ETA 05:23:40 2021-01-18 13:18:34 [INFO] [TRAIN] epoch=24, iter=15480/40000, loss=0.3579, lr=0.006438, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:22:56 2021-01-18 13:18:42 [INFO] [TRAIN] epoch=24, iter=15490/40000, loss=0.3066, lr=0.006435, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:21:44 2021-01-18 13:18:50 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.2082, lr=0.006433, batch_cost=0.7941, reader_cost=0.0001 | ETA 05:24:16 2021-01-18 13:18:58 [INFO] [TRAIN] epoch=24, iter=15510/40000, loss=0.2365, lr=0.006431, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:23:45 2021-01-18 13:19:06 [INFO] [TRAIN] epoch=24, iter=15520/40000, loss=0.2235, lr=0.006428, batch_cost=0.7921, reader_cost=0.0001 | ETA 05:23:11 2021-01-18 13:19:14 [INFO] [TRAIN] epoch=24, iter=15530/40000, loss=0.2398, lr=0.006426, batch_cost=0.7856, reader_cost=0.0001 | ETA 05:20:22 2021-01-18 13:19:21 [INFO] [TRAIN] epoch=24, iter=15540/40000, loss=0.2259, lr=0.006424, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:21:32 2021-01-18 13:19:29 [INFO] [TRAIN] epoch=24, iter=15550/40000, loss=0.3616, lr=0.006421, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:21:49 2021-01-18 13:19:37 [INFO] [TRAIN] epoch=24, iter=15560/40000, loss=0.2417, lr=0.006419, batch_cost=0.7866, reader_cost=0.0001 | ETA 05:20:24 2021-01-18 13:19:45 [INFO] [TRAIN] epoch=24, iter=15570/40000, loss=0.2569, lr=0.006416, batch_cost=0.7854, reader_cost=0.0001 | ETA 05:19:47 2021-01-18 13:19:53 [INFO] [TRAIN] epoch=24, iter=15580/40000, loss=0.2130, lr=0.006414, batch_cost=0.7964, reader_cost=0.0001 | ETA 05:24:07 2021-01-18 13:20:01 [INFO] [TRAIN] epoch=24, iter=15590/40000, loss=0.2125, lr=0.006412, batch_cost=0.7848, reader_cost=0.0001 | ETA 05:19:15 2021-01-18 13:20:09 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.1679, lr=0.006409, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:20:49 2021-01-18 13:20:17 [INFO] [TRAIN] epoch=24, iter=15610/40000, loss=0.2077, lr=0.006407, batch_cost=0.7898, reader_cost=0.0001 | ETA 05:21:03 2021-01-18 13:20:25 [INFO] [TRAIN] epoch=24, iter=15620/40000, loss=0.2180, lr=0.006405, batch_cost=0.7941, reader_cost=0.0001 | ETA 05:22:40 2021-01-18 13:20:33 [INFO] [TRAIN] epoch=24, iter=15630/40000, loss=0.2347, lr=0.006402, batch_cost=0.7872, reader_cost=0.0001 | ETA 05:19:43 2021-01-18 13:20:40 [INFO] [TRAIN] epoch=24, iter=15640/40000, loss=0.2067, lr=0.006400, batch_cost=0.7877, reader_cost=0.0001 | ETA 05:19:49 2021-01-18 13:20:48 [INFO] [TRAIN] epoch=24, iter=15650/40000, loss=0.2584, lr=0.006398, batch_cost=0.7956, reader_cost=0.0001 | ETA 05:22:52 2021-01-18 13:20:56 [INFO] [TRAIN] epoch=24, iter=15660/40000, loss=0.3533, lr=0.006395, batch_cost=0.7869, reader_cost=0.0001 | ETA 05:19:12 2021-01-18 13:21:04 [INFO] [TRAIN] epoch=24, iter=15670/40000, loss=0.2638, lr=0.006393, batch_cost=0.7962, reader_cost=0.0001 | ETA 05:22:52 2021-01-18 13:21:12 [INFO] [TRAIN] epoch=24, iter=15680/40000, loss=0.2804, lr=0.006390, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:19:39 2021-01-18 13:21:20 [INFO] [TRAIN] epoch=24, iter=15690/40000, loss=0.4027, lr=0.006388, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:21:06 2021-01-18 13:21:28 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.3947, lr=0.006386, batch_cost=0.7899, reader_cost=0.0001 | ETA 05:19:54 2021-01-18 13:21:36 [INFO] [TRAIN] epoch=24, iter=15710/40000, loss=0.2674, lr=0.006383, batch_cost=0.7946, reader_cost=0.0001 | ETA 05:21:41 2021-01-18 13:21:44 [INFO] [TRAIN] epoch=24, iter=15720/40000, loss=0.3956, lr=0.006381, batch_cost=0.7940, reader_cost=0.0001 | ETA 05:21:17 2021-01-18 13:21:52 [INFO] [TRAIN] epoch=24, iter=15730/40000, loss=0.2689, lr=0.006379, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:19:47 2021-01-18 13:22:00 [INFO] [TRAIN] epoch=24, iter=15740/40000, loss=0.1677, lr=0.006376, batch_cost=0.7952, reader_cost=0.0001 | ETA 05:21:32 2021-01-18 13:22:08 [INFO] [TRAIN] epoch=24, iter=15750/40000, loss=0.4064, lr=0.006374, batch_cost=0.7917, reader_cost=0.0001 | ETA 05:19:57 2021-01-18 13:22:15 [INFO] [TRAIN] epoch=24, iter=15760/40000, loss=0.2102, lr=0.006371, batch_cost=0.7859, reader_cost=0.0001 | ETA 05:17:29 2021-01-18 13:22:23 [INFO] [TRAIN] epoch=24, iter=15770/40000, loss=0.1994, lr=0.006369, batch_cost=0.7947, reader_cost=0.0001 | ETA 05:20:55 2021-01-18 13:22:31 [INFO] [TRAIN] epoch=24, iter=15780/40000, loss=0.3325, lr=0.006367, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:19:15 2021-01-18 13:22:39 [INFO] [TRAIN] epoch=24, iter=15790/40000, loss=0.2080, lr=0.006364, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:19:42 2021-01-18 13:22:47 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.2797, lr=0.006362, batch_cost=0.7881, reader_cost=0.0001 | ETA 05:17:53 2021-01-18 13:22:55 [INFO] [TRAIN] epoch=24, iter=15810/40000, loss=0.2671, lr=0.006360, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:19:25 2021-01-18 13:23:03 [INFO] [TRAIN] epoch=24, iter=15820/40000, loss=0.3671, lr=0.006357, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:18:15 2021-01-18 13:23:11 [INFO] [TRAIN] epoch=24, iter=15830/40000, loss=0.2387, lr=0.006355, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:19:10 2021-01-18 13:23:19 [INFO] [TRAIN] epoch=24, iter=15840/40000, loss=0.2966, lr=0.006353, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:17:59 2021-01-18 13:23:27 [INFO] [TRAIN] epoch=24, iter=15850/40000, loss=0.3364, lr=0.006350, batch_cost=0.7929, reader_cost=0.0001 | ETA 05:19:09 2021-01-18 13:23:35 [INFO] [TRAIN] epoch=24, iter=15860/40000, loss=0.4199, lr=0.006348, batch_cost=0.7838, reader_cost=0.0001 | ETA 05:15:21 2021-01-18 13:23:42 [INFO] [TRAIN] epoch=25, iter=15870/40000, loss=0.2986, lr=0.006345, batch_cost=0.7826, reader_cost=0.0136 | ETA 05:14:44 2021-01-18 13:23:50 [INFO] [TRAIN] epoch=25, iter=15880/40000, loss=0.3049, lr=0.006343, batch_cost=0.7841, reader_cost=0.0001 | ETA 05:15:12 2021-01-18 13:23:58 [INFO] [TRAIN] epoch=25, iter=15890/40000, loss=0.2292, lr=0.006341, batch_cost=0.7938, reader_cost=0.0001 | ETA 05:18:58 2021-01-18 13:24:06 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.3321, lr=0.006338, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:17:51 2021-01-18 13:24:14 [INFO] [TRAIN] epoch=25, iter=15910/40000, loss=0.1831, lr=0.006336, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:16:37 2021-01-18 13:24:22 [INFO] [TRAIN] epoch=25, iter=15920/40000, loss=0.3519, lr=0.006334, batch_cost=0.7970, reader_cost=0.0001 | ETA 05:19:52 2021-01-18 13:24:30 [INFO] [TRAIN] epoch=25, iter=15930/40000, loss=0.2634, lr=0.006331, batch_cost=0.7875, reader_cost=0.0001 | ETA 05:15:54 2021-01-18 13:24:38 [INFO] [TRAIN] epoch=25, iter=15940/40000, loss=0.2563, lr=0.006329, batch_cost=0.7947, reader_cost=0.0001 | ETA 05:18:39 2021-01-18 13:24:46 [INFO] [TRAIN] epoch=25, iter=15950/40000, loss=0.2927, lr=0.006327, batch_cost=0.7850, reader_cost=0.0001 | ETA 05:14:39 2021-01-18 13:24:53 [INFO] [TRAIN] epoch=25, iter=15960/40000, loss=0.2025, lr=0.006324, batch_cost=0.7891, reader_cost=0.0001 | ETA 05:16:10 2021-01-18 13:25:01 [INFO] [TRAIN] epoch=25, iter=15970/40000, loss=0.3233, lr=0.006322, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:17:40 2021-01-18 13:25:09 [INFO] [TRAIN] epoch=25, iter=15980/40000, loss=0.3484, lr=0.006319, batch_cost=0.7893, reader_cost=0.0001 | ETA 05:15:59 2021-01-18 13:25:17 [INFO] [TRAIN] epoch=25, iter=15990/40000, loss=0.2144, lr=0.006317, batch_cost=0.7880, reader_cost=0.0001 | ETA 05:15:19 2021-01-18 13:25:25 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.2350, lr=0.006315, batch_cost=0.7922, reader_cost=0.0001 | ETA 05:16:52 2021-01-18 13:25:25 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 13:25:54 [INFO] [EVAL] #Images=1449 mIoU=0.7617 Acc=0.9465 Kappa=0.8821 2021-01-18 13:25:54 [INFO] [EVAL] Class IoU: [0.9421 0.8544 0.4275 0.877 0.7263 0.8071 0.8883 0.8539 0.911 0.3519 0.8333 0.6238 0.8472 0.8676 0.8484 0.8592 0.6211 0.7494 0.508 0.8517 0.7464] 2021-01-18 13:25:54 [INFO] [EVAL] Class Acc: [0.9676 0.8816 0.437 0.9141 0.8336 0.8656 0.9768 0.9093 0.9656 0.7302 0.8798 0.7488 0.9291 0.9202 0.9 0.9531 0.7521 0.8935 0.7303 0.9033 0.9202] 2021-01-18 13:26:05 [INFO] [EVAL] The model with the best validation mIoU (0.7617) was saved at iter 16000. 2021-01-18 13:26:13 [INFO] [TRAIN] epoch=25, iter=16010/40000, loss=0.2596, lr=0.006312, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:14:55 2021-01-18 13:26:21 [INFO] [TRAIN] epoch=25, iter=16020/40000, loss=0.2428, lr=0.006310, batch_cost=0.7865, reader_cost=0.0001 | ETA 05:14:20 2021-01-18 13:26:29 [INFO] [TRAIN] epoch=25, iter=16030/40000, loss=0.2210, lr=0.006308, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:16:47 2021-01-18 13:26:37 [INFO] [TRAIN] epoch=25, iter=16040/40000, loss=0.2188, lr=0.006305, batch_cost=0.7937, reader_cost=0.0001 | ETA 05:16:57 2021-01-18 13:26:45 [INFO] [TRAIN] epoch=25, iter=16050/40000, loss=0.3229, lr=0.006303, batch_cost=0.7939, reader_cost=0.0001 | ETA 05:16:54 2021-01-18 13:26:52 [INFO] [TRAIN] epoch=25, iter=16060/40000, loss=0.2912, lr=0.006300, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:15:00 2021-01-18 13:27:00 [INFO] [TRAIN] epoch=25, iter=16070/40000, loss=0.2126, lr=0.006298, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:14:54 2021-01-18 13:27:08 [INFO] [TRAIN] epoch=25, iter=16080/40000, loss=0.2872, lr=0.006296, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:14:36 2021-01-18 13:27:16 [INFO] [TRAIN] epoch=25, iter=16090/40000, loss=0.2657, lr=0.006293, batch_cost=0.7866, reader_cost=0.0001 | ETA 05:13:28 2021-01-18 13:27:24 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.2152, lr=0.006291, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:15:53 2021-01-18 13:27:32 [INFO] [TRAIN] epoch=25, iter=16110/40000, loss=0.2532, lr=0.006289, batch_cost=0.7899, reader_cost=0.0001 | ETA 05:14:31 2021-01-18 13:27:40 [INFO] [TRAIN] epoch=25, iter=16120/40000, loss=0.1717, lr=0.006286, batch_cost=0.7910, reader_cost=0.0001 | ETA 05:14:49 2021-01-18 13:27:48 [INFO] [TRAIN] epoch=25, iter=16130/40000, loss=0.4892, lr=0.006284, batch_cost=0.7882, reader_cost=0.0001 | ETA 05:13:34 2021-01-18 13:27:56 [INFO] [TRAIN] epoch=25, iter=16140/40000, loss=0.4022, lr=0.006282, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:15:25 2021-01-18 13:28:04 [INFO] [TRAIN] epoch=25, iter=16150/40000, loss=0.5298, lr=0.006279, batch_cost=0.7926, reader_cost=0.0001 | ETA 05:15:03 2021-01-18 13:28:11 [INFO] [TRAIN] epoch=25, iter=16160/40000, loss=0.2491, lr=0.006277, batch_cost=0.7929, reader_cost=0.0001 | ETA 05:15:02 2021-01-18 13:28:19 [INFO] [TRAIN] epoch=25, iter=16170/40000, loss=0.4446, lr=0.006274, batch_cost=0.7856, reader_cost=0.0001 | ETA 05:12:01 2021-01-18 13:28:27 [INFO] [TRAIN] epoch=25, iter=16180/40000, loss=0.2968, lr=0.006272, batch_cost=0.7941, reader_cost=0.0001 | ETA 05:15:15 2021-01-18 13:28:35 [INFO] [TRAIN] epoch=25, iter=16190/40000, loss=0.3525, lr=0.006270, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:14:42 2021-01-18 13:28:43 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.2916, lr=0.006267, batch_cost=0.7893, reader_cost=0.0001 | ETA 05:13:05 2021-01-18 13:28:51 [INFO] [TRAIN] epoch=25, iter=16210/40000, loss=0.2664, lr=0.006265, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:12:17 2021-01-18 13:28:59 [INFO] [TRAIN] epoch=25, iter=16220/40000, loss=0.2762, lr=0.006263, batch_cost=0.7852, reader_cost=0.0001 | ETA 05:11:12 2021-01-18 13:29:07 [INFO] [TRAIN] epoch=25, iter=16230/40000, loss=0.3862, lr=0.006260, batch_cost=0.7871, reader_cost=0.0001 | ETA 05:11:49 2021-01-18 13:29:15 [INFO] [TRAIN] epoch=25, iter=16240/40000, loss=0.2475, lr=0.006258, batch_cost=0.7937, reader_cost=0.0001 | ETA 05:14:19 2021-01-18 13:29:23 [INFO] [TRAIN] epoch=25, iter=16250/40000, loss=0.2648, lr=0.006255, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:13:09 2021-01-18 13:29:30 [INFO] [TRAIN] epoch=25, iter=16260/40000, loss=0.2951, lr=0.006253, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:12:25 2021-01-18 13:29:38 [INFO] [TRAIN] epoch=25, iter=16270/40000, loss=0.3419, lr=0.006251, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:11:55 2021-01-18 13:29:46 [INFO] [TRAIN] epoch=25, iter=16280/40000, loss=0.2334, lr=0.006248, batch_cost=0.7900, reader_cost=0.0001 | ETA 05:12:17 2021-01-18 13:29:54 [INFO] [TRAIN] epoch=25, iter=16290/40000, loss=0.3370, lr=0.006246, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:13:16 2021-01-18 13:30:02 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.2682, lr=0.006244, batch_cost=0.7884, reader_cost=0.0001 | ETA 05:11:25 2021-01-18 13:30:10 [INFO] [TRAIN] epoch=25, iter=16310/40000, loss=0.2185, lr=0.006241, batch_cost=0.7859, reader_cost=0.0001 | ETA 05:10:18 2021-01-18 13:30:18 [INFO] [TRAIN] epoch=25, iter=16320/40000, loss=0.2732, lr=0.006239, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:13:09 2021-01-18 13:30:26 [INFO] [TRAIN] epoch=25, iter=16330/40000, loss=0.2384, lr=0.006237, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:11:09 2021-01-18 13:30:34 [INFO] [TRAIN] epoch=25, iter=16340/40000, loss=0.2898, lr=0.006234, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:11:35 2021-01-18 13:30:42 [INFO] [TRAIN] epoch=25, iter=16350/40000, loss=0.2823, lr=0.006232, batch_cost=0.7867, reader_cost=0.0001 | ETA 05:10:06 2021-01-18 13:30:49 [INFO] [TRAIN] epoch=25, iter=16360/40000, loss=0.2712, lr=0.006229, batch_cost=0.7851, reader_cost=0.0001 | ETA 05:09:19 2021-01-18 13:30:57 [INFO] [TRAIN] epoch=25, iter=16370/40000, loss=0.2865, lr=0.006227, batch_cost=0.7915, reader_cost=0.0001 | ETA 05:11:44 2021-01-18 13:31:05 [INFO] [TRAIN] epoch=25, iter=16380/40000, loss=0.2129, lr=0.006225, batch_cost=0.7919, reader_cost=0.0001 | ETA 05:11:44 2021-01-18 13:31:13 [INFO] [TRAIN] epoch=25, iter=16390/40000, loss=0.2321, lr=0.006222, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:11:24 2021-01-18 13:31:21 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.2600, lr=0.006220, batch_cost=0.7934, reader_cost=0.0001 | ETA 05:12:04 2021-01-18 13:31:29 [INFO] [TRAIN] epoch=25, iter=16410/40000, loss=0.2965, lr=0.006218, batch_cost=0.7906, reader_cost=0.0001 | ETA 05:10:49 2021-01-18 13:31:37 [INFO] [TRAIN] epoch=25, iter=16420/40000, loss=0.2490, lr=0.006215, batch_cost=0.7882, reader_cost=0.0001 | ETA 05:09:44 2021-01-18 13:31:45 [INFO] [TRAIN] epoch=25, iter=16430/40000, loss=0.2890, lr=0.006213, batch_cost=0.7897, reader_cost=0.0001 | ETA 05:10:13 2021-01-18 13:31:53 [INFO] [TRAIN] epoch=25, iter=16440/40000, loss=0.2038, lr=0.006210, batch_cost=0.7944, reader_cost=0.0001 | ETA 05:11:56 2021-01-18 13:32:01 [INFO] [TRAIN] epoch=25, iter=16450/40000, loss=0.3129, lr=0.006208, batch_cost=0.7943, reader_cost=0.0001 | ETA 05:11:44 2021-01-18 13:32:09 [INFO] [TRAIN] epoch=25, iter=16460/40000, loss=0.2335, lr=0.006206, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:09:37 2021-01-18 13:32:16 [INFO] [TRAIN] epoch=25, iter=16470/40000, loss=0.2487, lr=0.006203, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:09:23 2021-01-18 13:32:24 [INFO] [TRAIN] epoch=25, iter=16480/40000, loss=0.3555, lr=0.006201, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:09:17 2021-01-18 13:32:32 [INFO] [TRAIN] epoch=25, iter=16490/40000, loss=0.3011, lr=0.006199, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:09:59 2021-01-18 13:32:40 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.2684, lr=0.006196, batch_cost=0.7926, reader_cost=0.0001 | ETA 05:10:26 2021-01-18 13:32:48 [INFO] [TRAIN] epoch=25, iter=16510/40000, loss=0.2362, lr=0.006194, batch_cost=0.7885, reader_cost=0.0001 | ETA 05:08:41 2021-01-18 13:32:56 [INFO] [TRAIN] epoch=25, iter=16520/40000, loss=0.2238, lr=0.006191, batch_cost=0.7932, reader_cost=0.0001 | ETA 05:10:23 2021-01-18 13:33:04 [INFO] [TRAIN] epoch=26, iter=16530/40000, loss=0.2749, lr=0.006189, batch_cost=0.7779, reader_cost=0.0151 | ETA 05:04:18 2021-01-18 13:33:12 [INFO] [TRAIN] epoch=26, iter=16540/40000, loss=0.2896, lr=0.006187, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:08:23 2021-01-18 13:33:20 [INFO] [TRAIN] epoch=26, iter=16550/40000, loss=0.2065, lr=0.006184, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:09:50 2021-01-18 13:33:27 [INFO] [TRAIN] epoch=26, iter=16560/40000, loss=0.2818, lr=0.006182, batch_cost=0.7860, reader_cost=0.0001 | ETA 05:07:03 2021-01-18 13:33:35 [INFO] [TRAIN] epoch=26, iter=16570/40000, loss=0.3194, lr=0.006180, batch_cost=0.7972, reader_cost=0.0001 | ETA 05:11:17 2021-01-18 13:33:43 [INFO] [TRAIN] epoch=26, iter=16580/40000, loss=0.2492, lr=0.006177, batch_cost=0.7888, reader_cost=0.0001 | ETA 05:07:53 2021-01-18 13:33:51 [INFO] [TRAIN] epoch=26, iter=16590/40000, loss=0.1818, lr=0.006175, batch_cost=0.7945, reader_cost=0.0001 | ETA 05:09:58 2021-01-18 13:33:59 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.2416, lr=0.006172, batch_cost=0.7876, reader_cost=0.0001 | ETA 05:07:09 2021-01-18 13:34:07 [INFO] [TRAIN] epoch=26, iter=16610/40000, loss=0.3260, lr=0.006170, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:07:27 2021-01-18 13:34:15 [INFO] [TRAIN] epoch=26, iter=16620/40000, loss=0.2470, lr=0.006168, batch_cost=0.7967, reader_cost=0.0001 | ETA 05:10:25 2021-01-18 13:34:23 [INFO] [TRAIN] epoch=26, iter=16630/40000, loss=0.3407, lr=0.006165, batch_cost=0.7894, reader_cost=0.0001 | ETA 05:07:29 2021-01-18 13:34:31 [INFO] [TRAIN] epoch=26, iter=16640/40000, loss=0.2473, lr=0.006163, batch_cost=0.7952, reader_cost=0.0001 | ETA 05:09:35 2021-01-18 13:34:39 [INFO] [TRAIN] epoch=26, iter=16650/40000, loss=0.2351, lr=0.006161, batch_cost=0.7893, reader_cost=0.0001 | ETA 05:07:10 2021-01-18 13:34:47 [INFO] [TRAIN] epoch=26, iter=16660/40000, loss=0.3197, lr=0.006158, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:08:29 2021-01-18 13:34:55 [INFO] [TRAIN] epoch=26, iter=16670/40000, loss=0.1312, lr=0.006156, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:07:30 2021-01-18 13:35:02 [INFO] [TRAIN] epoch=26, iter=16680/40000, loss=0.2001, lr=0.006153, batch_cost=0.7943, reader_cost=0.0001 | ETA 05:08:43 2021-01-18 13:35:10 [INFO] [TRAIN] epoch=26, iter=16690/40000, loss=0.2397, lr=0.006151, batch_cost=0.7875, reader_cost=0.0001 | ETA 05:05:57 2021-01-18 13:35:18 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.2054, lr=0.006149, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:06:14 2021-01-18 13:35:26 [INFO] [TRAIN] epoch=26, iter=16710/40000, loss=0.2595, lr=0.006146, batch_cost=0.7908, reader_cost=0.0001 | ETA 05:06:58 2021-01-18 13:35:34 [INFO] [TRAIN] epoch=26, iter=16720/40000, loss=0.2046, lr=0.006144, batch_cost=0.7869, reader_cost=0.0001 | ETA 05:05:19 2021-01-18 13:35:42 [INFO] [TRAIN] epoch=26, iter=16730/40000, loss=0.3030, lr=0.006142, batch_cost=0.7962, reader_cost=0.0001 | ETA 05:08:48 2021-01-18 13:35:50 [INFO] [TRAIN] epoch=26, iter=16740/40000, loss=0.2045, lr=0.006139, batch_cost=0.7891, reader_cost=0.0001 | ETA 05:05:54 2021-01-18 13:35:58 [INFO] [TRAIN] epoch=26, iter=16750/40000, loss=0.3026, lr=0.006137, batch_cost=0.7962, reader_cost=0.0001 | ETA 05:08:31 2021-01-18 13:36:06 [INFO] [TRAIN] epoch=26, iter=16760/40000, loss=0.2656, lr=0.006134, batch_cost=0.7895, reader_cost=0.0001 | ETA 05:05:48 2021-01-18 13:36:14 [INFO] [TRAIN] epoch=26, iter=16770/40000, loss=0.2652, lr=0.006132, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:07:01 2021-01-18 13:36:22 [INFO] [TRAIN] epoch=26, iter=16780/40000, loss=0.3042, lr=0.006130, batch_cost=0.7922, reader_cost=0.0001 | ETA 05:06:35 2021-01-18 13:36:29 [INFO] [TRAIN] epoch=26, iter=16790/40000, loss=0.2501, lr=0.006127, batch_cost=0.7867, reader_cost=0.0001 | ETA 05:04:20 2021-01-18 13:36:37 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.2672, lr=0.006125, batch_cost=0.7846, reader_cost=0.0001 | ETA 05:03:22 2021-01-18 13:36:45 [INFO] [TRAIN] epoch=26, iter=16810/40000, loss=0.2400, lr=0.006123, batch_cost=0.7863, reader_cost=0.0001 | ETA 05:03:54 2021-01-18 13:36:53 [INFO] [TRAIN] epoch=26, iter=16820/40000, loss=0.2241, lr=0.006120, batch_cost=0.7867, reader_cost=0.0001 | ETA 05:03:55 2021-01-18 13:37:01 [INFO] [TRAIN] epoch=26, iter=16830/40000, loss=0.2351, lr=0.006118, batch_cost=0.7882, reader_cost=0.0001 | ETA 05:04:21 2021-01-18 13:37:09 [INFO] [TRAIN] epoch=26, iter=16840/40000, loss=0.2350, lr=0.006115, batch_cost=0.7885, reader_cost=0.0001 | ETA 05:04:20 2021-01-18 13:37:17 [INFO] [TRAIN] epoch=26, iter=16850/40000, loss=0.3221, lr=0.006113, batch_cost=0.7923, reader_cost=0.0001 | ETA 05:05:41 2021-01-18 13:37:25 [INFO] [TRAIN] epoch=26, iter=16860/40000, loss=0.4224, lr=0.006111, batch_cost=0.7889, reader_cost=0.0001 | ETA 05:04:14 2021-01-18 13:37:32 [INFO] [TRAIN] epoch=26, iter=16870/40000, loss=0.2479, lr=0.006108, batch_cost=0.7847, reader_cost=0.0001 | ETA 05:02:30 2021-01-18 13:37:40 [INFO] [TRAIN] epoch=26, iter=16880/40000, loss=0.3236, lr=0.006106, batch_cost=0.7864, reader_cost=0.0001 | ETA 05:03:02 2021-01-18 13:37:48 [INFO] [TRAIN] epoch=26, iter=16890/40000, loss=0.2996, lr=0.006104, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:04:27 2021-01-18 13:37:56 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.2363, lr=0.006101, batch_cost=0.7914, reader_cost=0.0001 | ETA 05:04:41 2021-01-18 13:38:04 [INFO] [TRAIN] epoch=26, iter=16910/40000, loss=0.3208, lr=0.006099, batch_cost=0.7951, reader_cost=0.0001 | ETA 05:05:59 2021-01-18 13:38:12 [INFO] [TRAIN] epoch=26, iter=16920/40000, loss=0.2168, lr=0.006096, batch_cost=0.7925, reader_cost=0.0001 | ETA 05:04:50 2021-01-18 13:38:20 [INFO] [TRAIN] epoch=26, iter=16930/40000, loss=0.3002, lr=0.006094, batch_cost=0.7904, reader_cost=0.0001 | ETA 05:03:54 2021-01-18 13:38:28 [INFO] [TRAIN] epoch=26, iter=16940/40000, loss=0.2699, lr=0.006092, batch_cost=0.7881, reader_cost=0.0001 | ETA 05:02:53 2021-01-18 13:38:36 [INFO] [TRAIN] epoch=26, iter=16950/40000, loss=0.2587, lr=0.006089, batch_cost=0.7892, reader_cost=0.0001 | ETA 05:03:11 2021-01-18 13:38:44 [INFO] [TRAIN] epoch=26, iter=16960/40000, loss=0.2380, lr=0.006087, batch_cost=0.7842, reader_cost=0.0001 | ETA 05:01:07 2021-01-18 13:38:51 [INFO] [TRAIN] epoch=26, iter=16970/40000, loss=0.1995, lr=0.006085, batch_cost=0.7888, reader_cost=0.0001 | ETA 05:02:45 2021-01-18 13:38:59 [INFO] [TRAIN] epoch=26, iter=16980/40000, loss=0.2696, lr=0.006082, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:03:30 2021-01-18 13:39:07 [INFO] [TRAIN] epoch=26, iter=16990/40000, loss=0.2386, lr=0.006080, batch_cost=0.7888, reader_cost=0.0001 | ETA 05:02:30 2021-01-18 13:39:15 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.2235, lr=0.006077, batch_cost=0.7903, reader_cost=0.0001 | ETA 05:02:57 2021-01-18 13:39:15 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 13:39:43 [INFO] [EVAL] #Images=1449 mIoU=0.7669 Acc=0.9485 Kappa=0.8866 2021-01-18 13:39:43 [INFO] [EVAL] Class IoU: [0.9448 0.8814 0.4192 0.9059 0.6735 0.7714 0.9177 0.8359 0.9178 0.364 0.8777 0.6048 0.8535 0.8957 0.8543 0.8629 0.6206 0.8024 0.4979 0.8641 0.7383] 2021-01-18 13:39:43 [INFO] [EVAL] Class Acc: [0.9695 0.9088 0.4426 0.9469 0.8206 0.8027 0.9758 0.9051 0.9366 0.5526 0.9193 0.8545 0.9055 0.9384 0.9329 0.936 0.8171 0.8845 0.796 0.9523 0.8822] 2021-01-18 13:39:53 [INFO] [EVAL] The model with the best validation mIoU (0.7669) was saved at iter 17000. 2021-01-18 13:40:01 [INFO] [TRAIN] epoch=26, iter=17010/40000, loss=0.2621, lr=0.006075, batch_cost=0.7905, reader_cost=0.0001 | ETA 05:02:54 2021-01-18 13:40:09 [INFO] [TRAIN] epoch=26, iter=17020/40000, loss=0.1929, lr=0.006073, batch_cost=0.7886, reader_cost=0.0001 | ETA 05:02:03 2021-01-18 13:40:17 [INFO] [TRAIN] epoch=26, iter=17030/40000, loss=0.1917, lr=0.006070, batch_cost=0.7887, reader_cost=0.0001 | ETA 05:01:56 2021-01-18 13:40:25 [INFO] [TRAIN] epoch=26, iter=17040/40000, loss=0.2287, lr=0.006068, batch_cost=0.7902, reader_cost=0.0001 | ETA 05:02:23 2021-01-18 13:40:33 [INFO] [TRAIN] epoch=26, iter=17050/40000, loss=0.2186, lr=0.006066, batch_cost=0.7859, reader_cost=0.0001 | ETA 05:00:35 2021-01-18 13:40:41 [INFO] [TRAIN] epoch=26, iter=17060/40000, loss=0.3177, lr=0.006063, batch_cost=0.7871, reader_cost=0.0001 | ETA 05:00:57 2021-01-18 13:40:49 [INFO] [TRAIN] epoch=26, iter=17070/40000, loss=0.2642, lr=0.006061, batch_cost=0.7918, reader_cost=0.0001 | ETA 05:02:34 2021-01-18 13:40:57 [INFO] [TRAIN] epoch=26, iter=17080/40000, loss=0.2985, lr=0.006058, batch_cost=0.7980, reader_cost=0.0001 | ETA 05:04:50 2021-01-18 13:41:05 [INFO] [TRAIN] epoch=26, iter=17090/40000, loss=0.2378, lr=0.006056, batch_cost=0.7865, reader_cost=0.0001 | ETA 05:00:17 2021-01-18 13:41:12 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.3343, lr=0.006054, batch_cost=0.7950, reader_cost=0.0001 | ETA 05:03:26 2021-01-18 13:41:20 [INFO] [TRAIN] epoch=26, iter=17110/40000, loss=0.2235, lr=0.006051, batch_cost=0.7890, reader_cost=0.0001 | ETA 05:01:00 2021-01-18 13:41:28 [INFO] [TRAIN] epoch=26, iter=17120/40000, loss=0.3534, lr=0.006049, batch_cost=0.7885, reader_cost=0.0001 | ETA 05:00:41 2021-01-18 13:41:36 [INFO] [TRAIN] epoch=26, iter=17130/40000, loss=0.2416, lr=0.006046, batch_cost=0.7913, reader_cost=0.0001 | ETA 05:01:36 2021-01-18 13:41:44 [INFO] [TRAIN] epoch=26, iter=17140/40000, loss=0.3076, lr=0.006044, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:02:20 2021-01-18 13:41:52 [INFO] [TRAIN] epoch=26, iter=17150/40000, loss=0.2326, lr=0.006042, batch_cost=0.7930, reader_cost=0.0001 | ETA 05:02:00 2021-01-18 13:42:00 [INFO] [TRAIN] epoch=26, iter=17160/40000, loss=0.1901, lr=0.006039, batch_cost=0.7909, reader_cost=0.0001 | ETA 05:01:04 2021-01-18 13:42:08 [INFO] [TRAIN] epoch=26, iter=17170/40000, loss=0.4270, lr=0.006037, batch_cost=0.7896, reader_cost=0.0001 | ETA 05:00:27 2021-01-18 13:42:16 [INFO] [TRAIN] epoch=26, iter=17180/40000, loss=0.2040, lr=0.006035, batch_cost=0.7935, reader_cost=0.0001 | ETA 05:01:47 2021-01-18 13:42:24 [INFO] [TRAIN] epoch=27, iter=17190/40000, loss=0.3060, lr=0.006032, batch_cost=0.7818, reader_cost=0.0152 | ETA 04:57:12 2021-01-18 13:42:32 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.2335, lr=0.006030, batch_cost=0.7949, reader_cost=0.0001 | ETA 05:02:03 2021-01-18 13:42:39 [INFO] [TRAIN] epoch=27, iter=17210/40000, loss=0.2965, lr=0.006027, batch_cost=0.7911, reader_cost=0.0001 | ETA 05:00:28 2021-01-18 13:42:47 [INFO] [TRAIN] epoch=27, iter=17220/40000, loss=0.2247, lr=0.006025, batch_cost=0.7897, reader_cost=0.0001 | ETA 04:59:49 2021-01-18 13:42:55 [INFO] [TRAIN] epoch=27, iter=17230/40000, loss=0.3296, lr=0.006023, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:59:15 2021-01-18 13:43:03 [INFO] [TRAIN] epoch=27, iter=17240/40000, loss=0.2612, lr=0.006020, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:58:18 2021-01-18 13:43:11 [INFO] [TRAIN] epoch=27, iter=17250/40000, loss=0.2770, lr=0.006018, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:59:52 2021-01-18 13:43:19 [INFO] [TRAIN] epoch=27, iter=17260/40000, loss=0.2953, lr=0.006016, batch_cost=0.7928, reader_cost=0.0001 | ETA 05:00:28 2021-01-18 13:43:27 [INFO] [TRAIN] epoch=27, iter=17270/40000, loss=0.2036, lr=0.006013, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:59:38 2021-01-18 13:43:35 [INFO] [TRAIN] epoch=27, iter=17280/40000, loss=0.1851, lr=0.006011, batch_cost=0.7835, reader_cost=0.0001 | ETA 04:56:42 2021-01-18 13:43:43 [INFO] [TRAIN] epoch=27, iter=17290/40000, loss=0.2727, lr=0.006008, batch_cost=0.7943, reader_cost=0.0001 | ETA 05:00:37 2021-01-18 13:43:51 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.2749, lr=0.006006, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:58:13 2021-01-18 13:43:58 [INFO] [TRAIN] epoch=27, iter=17310/40000, loss=0.2934, lr=0.006004, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:59:07 2021-01-18 13:44:06 [INFO] [TRAIN] epoch=27, iter=17320/40000, loss=0.2442, lr=0.006001, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:58:40 2021-01-18 13:44:14 [INFO] [TRAIN] epoch=27, iter=17330/40000, loss=0.2506, lr=0.005999, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:58:45 2021-01-18 13:44:22 [INFO] [TRAIN] epoch=27, iter=17340/40000, loss=0.2564, lr=0.005996, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:58:09 2021-01-18 13:44:30 [INFO] [TRAIN] epoch=27, iter=17350/40000, loss=0.2672, lr=0.005994, batch_cost=0.7868, reader_cost=0.0001 | ETA 04:57:01 2021-01-18 13:44:38 [INFO] [TRAIN] epoch=27, iter=17360/40000, loss=0.2438, lr=0.005992, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:58:50 2021-01-18 13:44:46 [INFO] [TRAIN] epoch=27, iter=17370/40000, loss=0.2929, lr=0.005989, batch_cost=0.7915, reader_cost=0.0001 | ETA 04:58:31 2021-01-18 13:44:54 [INFO] [TRAIN] epoch=27, iter=17380/40000, loss=0.2626, lr=0.005987, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:59:03 2021-01-18 13:45:02 [INFO] [TRAIN] epoch=27, iter=17390/40000, loss=0.2194, lr=0.005985, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:57:00 2021-01-18 13:45:10 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.2832, lr=0.005982, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:58:29 2021-01-18 13:45:18 [INFO] [TRAIN] epoch=27, iter=17410/40000, loss=0.2892, lr=0.005980, batch_cost=0.7928, reader_cost=0.0001 | ETA 04:58:28 2021-01-18 13:45:25 [INFO] [TRAIN] epoch=27, iter=17420/40000, loss=0.2459, lr=0.005977, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:56:49 2021-01-18 13:45:33 [INFO] [TRAIN] epoch=27, iter=17430/40000, loss=0.2743, lr=0.005975, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:55:47 2021-01-18 13:45:41 [INFO] [TRAIN] epoch=27, iter=17440/40000, loss=0.2278, lr=0.005973, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:57:12 2021-01-18 13:45:49 [INFO] [TRAIN] epoch=27, iter=17450/40000, loss=0.2532, lr=0.005970, batch_cost=0.7936, reader_cost=0.0001 | ETA 04:58:15 2021-01-18 13:45:57 [INFO] [TRAIN] epoch=27, iter=17460/40000, loss=0.2404, lr=0.005968, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:57:51 2021-01-18 13:46:05 [INFO] [TRAIN] epoch=27, iter=17470/40000, loss=0.2481, lr=0.005966, batch_cost=0.7932, reader_cost=0.0001 | ETA 04:57:49 2021-01-18 13:46:13 [INFO] [TRAIN] epoch=27, iter=17480/40000, loss=0.2386, lr=0.005963, batch_cost=0.7856, reader_cost=0.0001 | ETA 04:54:52 2021-01-18 13:46:21 [INFO] [TRAIN] epoch=27, iter=17490/40000, loss=0.3371, lr=0.005961, batch_cost=0.7954, reader_cost=0.0001 | ETA 04:58:24 2021-01-18 13:46:29 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.4102, lr=0.005958, batch_cost=0.7930, reader_cost=0.0001 | ETA 04:57:22 2021-01-18 13:46:37 [INFO] [TRAIN] epoch=27, iter=17510/40000, loss=0.2316, lr=0.005956, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:57:13 2021-01-18 13:46:45 [INFO] [TRAIN] epoch=27, iter=17520/40000, loss=0.2002, lr=0.005954, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:56:08 2021-01-18 13:46:52 [INFO] [TRAIN] epoch=27, iter=17530/40000, loss=0.2338, lr=0.005951, batch_cost=0.7916, reader_cost=0.0001 | ETA 04:56:27 2021-01-18 13:47:00 [INFO] [TRAIN] epoch=27, iter=17540/40000, loss=0.2226, lr=0.005949, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:55:39 2021-01-18 13:47:08 [INFO] [TRAIN] epoch=27, iter=17550/40000, loss=0.2101, lr=0.005946, batch_cost=0.7880, reader_cost=0.0001 | ETA 04:54:49 2021-01-18 13:47:16 [INFO] [TRAIN] epoch=27, iter=17560/40000, loss=0.3399, lr=0.005944, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:55:15 2021-01-18 13:47:24 [INFO] [TRAIN] epoch=27, iter=17570/40000, loss=0.2409, lr=0.005942, batch_cost=0.7911, reader_cost=0.0001 | ETA 04:55:45 2021-01-18 13:47:32 [INFO] [TRAIN] epoch=27, iter=17580/40000, loss=0.2468, lr=0.005939, batch_cost=0.7826, reader_cost=0.0001 | ETA 04:52:26 2021-01-18 13:47:40 [INFO] [TRAIN] epoch=27, iter=17590/40000, loss=0.3382, lr=0.005937, batch_cost=0.7900, reader_cost=0.0001 | ETA 04:55:04 2021-01-18 13:47:48 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.2631, lr=0.005935, batch_cost=0.7951, reader_cost=0.0001 | ETA 04:56:50 2021-01-18 13:47:56 [INFO] [TRAIN] epoch=27, iter=17610/40000, loss=0.2378, lr=0.005932, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:55:04 2021-01-18 13:48:04 [INFO] [TRAIN] epoch=27, iter=17620/40000, loss=0.1718, lr=0.005930, batch_cost=0.7943, reader_cost=0.0001 | ETA 04:56:17 2021-01-18 13:48:11 [INFO] [TRAIN] epoch=27, iter=17630/40000, loss=0.3090, lr=0.005927, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:54:21 2021-01-18 13:48:19 [INFO] [TRAIN] epoch=27, iter=17640/40000, loss=0.1989, lr=0.005925, batch_cost=0.7879, reader_cost=0.0001 | ETA 04:53:37 2021-01-18 13:48:27 [INFO] [TRAIN] epoch=27, iter=17650/40000, loss=0.3758, lr=0.005923, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:54:21 2021-01-18 13:48:35 [INFO] [TRAIN] epoch=27, iter=17660/40000, loss=0.2805, lr=0.005920, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:53:37 2021-01-18 13:48:43 [INFO] [TRAIN] epoch=27, iter=17670/40000, loss=0.2298, lr=0.005918, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:52:40 2021-01-18 13:48:51 [INFO] [TRAIN] epoch=27, iter=17680/40000, loss=0.2212, lr=0.005915, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:54:01 2021-01-18 13:48:59 [INFO] [TRAIN] epoch=27, iter=17690/40000, loss=0.2797, lr=0.005913, batch_cost=0.7832, reader_cost=0.0001 | ETA 04:51:12 2021-01-18 13:49:07 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.2536, lr=0.005911, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:53:42 2021-01-18 13:49:15 [INFO] [TRAIN] epoch=27, iter=17710/40000, loss=0.2440, lr=0.005908, batch_cost=0.7885, reader_cost=0.0001 | ETA 04:52:55 2021-01-18 13:49:22 [INFO] [TRAIN] epoch=27, iter=17720/40000, loss=0.2619, lr=0.005906, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:53:33 2021-01-18 13:49:30 [INFO] [TRAIN] epoch=27, iter=17730/40000, loss=0.2507, lr=0.005904, batch_cost=0.7868, reader_cost=0.0001 | ETA 04:52:01 2021-01-18 13:49:38 [INFO] [TRAIN] epoch=27, iter=17740/40000, loss=0.3635, lr=0.005901, batch_cost=0.7940, reader_cost=0.0001 | ETA 04:54:35 2021-01-18 13:49:46 [INFO] [TRAIN] epoch=27, iter=17750/40000, loss=0.4041, lr=0.005899, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:53:24 2021-01-18 13:49:54 [INFO] [TRAIN] epoch=27, iter=17760/40000, loss=0.2718, lr=0.005896, batch_cost=0.7934, reader_cost=0.0001 | ETA 04:54:04 2021-01-18 13:50:02 [INFO] [TRAIN] epoch=27, iter=17770/40000, loss=0.2421, lr=0.005894, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:52:29 2021-01-18 13:50:10 [INFO] [TRAIN] epoch=27, iter=17780/40000, loss=0.1923, lr=0.005892, batch_cost=0.7970, reader_cost=0.0001 | ETA 04:55:09 2021-01-18 13:50:18 [INFO] [TRAIN] epoch=27, iter=17790/40000, loss=0.3773, lr=0.005889, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:52:52 2021-01-18 13:50:26 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.2270, lr=0.005887, batch_cost=0.7863, reader_cost=0.0001 | ETA 04:50:56 2021-01-18 13:50:34 [INFO] [TRAIN] epoch=27, iter=17810/40000, loss=0.2905, lr=0.005884, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:50:50 2021-01-18 13:50:41 [INFO] [TRAIN] epoch=27, iter=17820/40000, loss=0.2279, lr=0.005882, batch_cost=0.7888, reader_cost=0.0001 | ETA 04:51:35 2021-01-18 13:50:49 [INFO] [TRAIN] epoch=27, iter=17830/40000, loss=0.2283, lr=0.005880, batch_cost=0.8006, reader_cost=0.0001 | ETA 04:55:49 2021-01-18 13:50:57 [INFO] [TRAIN] epoch=27, iter=17840/40000, loss=0.2179, lr=0.005877, batch_cost=0.7964, reader_cost=0.0001 | ETA 04:54:08 2021-01-18 13:51:05 [INFO] [TRAIN] epoch=28, iter=17850/40000, loss=0.2240, lr=0.005875, batch_cost=0.7799, reader_cost=0.0183 | ETA 04:47:53 2021-01-18 13:51:13 [INFO] [TRAIN] epoch=28, iter=17860/40000, loss=0.2262, lr=0.005873, batch_cost=0.7956, reader_cost=0.0001 | ETA 04:53:34 2021-01-18 13:51:21 [INFO] [TRAIN] epoch=28, iter=17870/40000, loss=0.2331, lr=0.005870, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:50:41 2021-01-18 13:51:29 [INFO] [TRAIN] epoch=28, iter=17880/40000, loss=0.2425, lr=0.005868, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:50:33 2021-01-18 13:51:37 [INFO] [TRAIN] epoch=28, iter=17890/40000, loss=0.4398, lr=0.005865, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:51:16 2021-01-18 13:51:45 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.2011, lr=0.005863, batch_cost=0.7961, reader_cost=0.0001 | ETA 04:53:14 2021-01-18 13:51:53 [INFO] [TRAIN] epoch=28, iter=17910/40000, loss=0.2166, lr=0.005861, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:50:38 2021-01-18 13:52:01 [INFO] [TRAIN] epoch=28, iter=17920/40000, loss=0.2357, lr=0.005858, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:51:00 2021-01-18 13:52:09 [INFO] [TRAIN] epoch=28, iter=17930/40000, loss=0.2751, lr=0.005856, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:51:19 2021-01-18 13:52:16 [INFO] [TRAIN] epoch=28, iter=17940/40000, loss=0.3633, lr=0.005853, batch_cost=0.7934, reader_cost=0.0001 | ETA 04:51:42 2021-01-18 13:52:24 [INFO] [TRAIN] epoch=28, iter=17950/40000, loss=0.1926, lr=0.005851, batch_cost=0.7911, reader_cost=0.0001 | ETA 04:50:43 2021-01-18 13:52:32 [INFO] [TRAIN] epoch=28, iter=17960/40000, loss=0.2849, lr=0.005849, batch_cost=0.7897, reader_cost=0.0001 | ETA 04:50:03 2021-01-18 13:52:40 [INFO] [TRAIN] epoch=28, iter=17970/40000, loss=0.1850, lr=0.005846, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:50:16 2021-01-18 13:52:48 [INFO] [TRAIN] epoch=28, iter=17980/40000, loss=0.2941, lr=0.005844, batch_cost=0.7931, reader_cost=0.0001 | ETA 04:51:03 2021-01-18 13:52:56 [INFO] [TRAIN] epoch=28, iter=17990/40000, loss=0.1990, lr=0.005841, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:49:53 2021-01-18 13:53:04 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.2535, lr=0.005839, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:50:02 2021-01-18 13:53:04 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 13:53:32 [INFO] [EVAL] #Images=1449 mIoU=0.7491 Acc=0.9435 Kappa=0.8757 2021-01-18 13:53:32 [INFO] [EVAL] Class IoU: [0.941 0.8424 0.4077 0.8783 0.666 0.7377 0.9285 0.7726 0.863 0.3844 0.8557 0.6366 0.7855 0.9007 0.6472 0.8609 0.5977 0.807 0.5729 0.8898 0.755 ] 2021-01-18 13:53:32 [INFO] [EVAL] Class Acc: [0.9677 0.9702 0.4303 0.919 0.7437 0.9017 0.9816 0.8195 0.8893 0.5698 0.9069 0.8763 0.9325 0.951 0.9313 0.9566 0.8887 0.9367 0.6327 0.9644 0.902 ] 2021-01-18 13:53:38 [INFO] [EVAL] The model with the best validation mIoU (0.7669) was saved at iter 17000. 2021-01-18 13:53:46 [INFO] [TRAIN] epoch=28, iter=18010/40000, loss=0.2285, lr=0.005837, batch_cost=0.7846, reader_cost=0.0001 | ETA 04:47:34 2021-01-18 13:53:54 [INFO] [TRAIN] epoch=28, iter=18020/40000, loss=0.2960, lr=0.005834, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:50:27 2021-01-18 13:54:02 [INFO] [TRAIN] epoch=28, iter=18030/40000, loss=0.4409, lr=0.005832, batch_cost=0.7852, reader_cost=0.0001 | ETA 04:47:31 2021-01-18 13:54:10 [INFO] [TRAIN] epoch=28, iter=18040/40000, loss=0.2641, lr=0.005830, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:48:54 2021-01-18 13:54:18 [INFO] [TRAIN] epoch=28, iter=18050/40000, loss=0.2443, lr=0.005827, batch_cost=0.7885, reader_cost=0.0001 | ETA 04:48:26 2021-01-18 13:54:26 [INFO] [TRAIN] epoch=28, iter=18060/40000, loss=0.2635, lr=0.005825, batch_cost=0.7874, reader_cost=0.0001 | ETA 04:47:56 2021-01-18 13:54:33 [INFO] [TRAIN] epoch=28, iter=18070/40000, loss=0.3147, lr=0.005822, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:48:21 2021-01-18 13:54:41 [INFO] [TRAIN] epoch=28, iter=18080/40000, loss=0.2046, lr=0.005820, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:48:12 2021-01-18 13:54:49 [INFO] [TRAIN] epoch=28, iter=18090/40000, loss=0.4345, lr=0.005818, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:48:38 2021-01-18 13:54:57 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.2699, lr=0.005815, batch_cost=0.7863, reader_cost=0.0001 | ETA 04:46:59 2021-01-18 13:55:05 [INFO] [TRAIN] epoch=28, iter=18110/40000, loss=0.1943, lr=0.005813, batch_cost=0.7964, reader_cost=0.0001 | ETA 04:50:33 2021-01-18 13:55:13 [INFO] [TRAIN] epoch=28, iter=18120/40000, loss=0.2926, lr=0.005810, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:47:35 2021-01-18 13:55:21 [INFO] [TRAIN] epoch=28, iter=18130/40000, loss=0.3138, lr=0.005808, batch_cost=0.7918, reader_cost=0.0001 | ETA 04:48:36 2021-01-18 13:55:29 [INFO] [TRAIN] epoch=28, iter=18140/40000, loss=0.2649, lr=0.005806, batch_cost=0.7873, reader_cost=0.0001 | ETA 04:46:49 2021-01-18 13:55:37 [INFO] [TRAIN] epoch=28, iter=18150/40000, loss=0.2672, lr=0.005803, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:48:28 2021-01-18 13:55:45 [INFO] [TRAIN] epoch=28, iter=18160/40000, loss=0.3261, lr=0.005801, batch_cost=0.7860, reader_cost=0.0001 | ETA 04:46:05 2021-01-18 13:55:52 [INFO] [TRAIN] epoch=28, iter=18170/40000, loss=0.3083, lr=0.005798, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:46:45 2021-01-18 13:56:00 [INFO] [TRAIN] epoch=28, iter=18180/40000, loss=0.2719, lr=0.005796, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:46:46 2021-01-18 13:56:08 [INFO] [TRAIN] epoch=28, iter=18190/40000, loss=0.2120, lr=0.005794, batch_cost=0.7872, reader_cost=0.0001 | ETA 04:46:09 2021-01-18 13:56:16 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.2185, lr=0.005791, batch_cost=0.7867, reader_cost=0.0001 | ETA 04:45:51 2021-01-18 13:56:24 [INFO] [TRAIN] epoch=28, iter=18210/40000, loss=0.2268, lr=0.005789, batch_cost=0.7938, reader_cost=0.0001 | ETA 04:48:15 2021-01-18 13:56:32 [INFO] [TRAIN] epoch=28, iter=18220/40000, loss=0.2707, lr=0.005787, batch_cost=0.7930, reader_cost=0.0001 | ETA 04:47:51 2021-01-18 13:56:40 [INFO] [TRAIN] epoch=28, iter=18230/40000, loss=0.2129, lr=0.005784, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:46:59 2021-01-18 13:56:48 [INFO] [TRAIN] epoch=28, iter=18240/40000, loss=0.1865, lr=0.005782, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:46:21 2021-01-18 13:56:56 [INFO] [TRAIN] epoch=28, iter=18250/40000, loss=0.2025, lr=0.005779, batch_cost=0.7859, reader_cost=0.0001 | ETA 04:44:52 2021-01-18 13:57:03 [INFO] [TRAIN] epoch=28, iter=18260/40000, loss=0.3143, lr=0.005777, batch_cost=0.7921, reader_cost=0.0001 | ETA 04:47:00 2021-01-18 13:57:11 [INFO] [TRAIN] epoch=28, iter=18270/40000, loss=0.2425, lr=0.005775, batch_cost=0.7945, reader_cost=0.0001 | ETA 04:47:44 2021-01-18 13:57:19 [INFO] [TRAIN] epoch=28, iter=18280/40000, loss=0.3200, lr=0.005772, batch_cost=0.7890, reader_cost=0.0001 | ETA 04:45:37 2021-01-18 13:57:27 [INFO] [TRAIN] epoch=28, iter=18290/40000, loss=0.2146, lr=0.005770, batch_cost=0.7899, reader_cost=0.0001 | ETA 04:45:47 2021-01-18 13:57:35 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.1747, lr=0.005767, batch_cost=0.7930, reader_cost=0.0001 | ETA 04:46:49 2021-01-18 13:57:43 [INFO] [TRAIN] epoch=28, iter=18310/40000, loss=0.1538, lr=0.005765, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:45:38 2021-01-18 13:57:51 [INFO] [TRAIN] epoch=28, iter=18320/40000, loss=0.2571, lr=0.005763, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:45:16 2021-01-18 13:57:59 [INFO] [TRAIN] epoch=28, iter=18330/40000, loss=0.4063, lr=0.005760, batch_cost=0.7917, reader_cost=0.0001 | ETA 04:45:56 2021-01-18 13:58:07 [INFO] [TRAIN] epoch=28, iter=18340/40000, loss=0.3044, lr=0.005758, batch_cost=0.7897, reader_cost=0.0001 | ETA 04:45:04 2021-01-18 13:58:15 [INFO] [TRAIN] epoch=28, iter=18350/40000, loss=0.3406, lr=0.005755, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:45:16 2021-01-18 13:58:23 [INFO] [TRAIN] epoch=28, iter=18360/40000, loss=0.1914, lr=0.005753, batch_cost=0.7958, reader_cost=0.0001 | ETA 04:47:01 2021-01-18 13:58:30 [INFO] [TRAIN] epoch=28, iter=18370/40000, loss=0.2868, lr=0.005751, batch_cost=0.7815, reader_cost=0.0001 | ETA 04:41:44 2021-01-18 13:58:38 [INFO] [TRAIN] epoch=28, iter=18380/40000, loss=0.2650, lr=0.005748, batch_cost=0.7944, reader_cost=0.0001 | ETA 04:46:15 2021-01-18 13:58:46 [INFO] [TRAIN] epoch=28, iter=18390/40000, loss=0.3153, lr=0.005746, batch_cost=0.7917, reader_cost=0.0001 | ETA 04:45:09 2021-01-18 13:58:54 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.2317, lr=0.005743, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:44:38 2021-01-18 13:59:02 [INFO] [TRAIN] epoch=28, iter=18410/40000, loss=0.1970, lr=0.005741, batch_cost=0.7931, reader_cost=0.0001 | ETA 04:45:22 2021-01-18 13:59:10 [INFO] [TRAIN] epoch=28, iter=18420/40000, loss=0.3011, lr=0.005739, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:44:11 2021-01-18 13:59:18 [INFO] [TRAIN] epoch=28, iter=18430/40000, loss=0.2080, lr=0.005736, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:44:21 2021-01-18 13:59:26 [INFO] [TRAIN] epoch=28, iter=18440/40000, loss=0.2985, lr=0.005734, batch_cost=0.7880, reader_cost=0.0001 | ETA 04:43:09 2021-01-18 13:59:34 [INFO] [TRAIN] epoch=28, iter=18450/40000, loss=0.2093, lr=0.005731, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:43:14 2021-01-18 13:59:42 [INFO] [TRAIN] epoch=28, iter=18460/40000, loss=0.2518, lr=0.005729, batch_cost=0.7871, reader_cost=0.0001 | ETA 04:42:34 2021-01-18 13:59:49 [INFO] [TRAIN] epoch=28, iter=18470/40000, loss=0.2249, lr=0.005727, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:43:48 2021-01-18 13:59:57 [INFO] [TRAIN] epoch=28, iter=18480/40000, loss=0.2279, lr=0.005724, batch_cost=0.7860, reader_cost=0.0001 | ETA 04:41:54 2021-01-18 14:00:05 [INFO] [TRAIN] epoch=28, iter=18490/40000, loss=0.2833, lr=0.005722, batch_cost=0.7943, reader_cost=0.0001 | ETA 04:44:45 2021-01-18 14:00:13 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.2229, lr=0.005720, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:43:15 2021-01-18 14:00:21 [INFO] [TRAIN] epoch=29, iter=18510/40000, loss=0.3018, lr=0.005717, batch_cost=0.7821, reader_cost=0.0139 | ETA 04:40:08 2021-01-18 14:00:29 [INFO] [TRAIN] epoch=29, iter=18520/40000, loss=0.2162, lr=0.005715, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:43:14 2021-01-18 14:00:37 [INFO] [TRAIN] epoch=29, iter=18530/40000, loss=0.1899, lr=0.005712, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:42:02 2021-01-18 14:00:45 [INFO] [TRAIN] epoch=29, iter=18540/40000, loss=0.2199, lr=0.005710, batch_cost=0.7932, reader_cost=0.0001 | ETA 04:43:42 2021-01-18 14:00:53 [INFO] [TRAIN] epoch=29, iter=18550/40000, loss=0.3208, lr=0.005708, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:42:35 2021-01-18 14:01:01 [INFO] [TRAIN] epoch=29, iter=18560/40000, loss=0.2316, lr=0.005705, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:42:08 2021-01-18 14:01:08 [INFO] [TRAIN] epoch=29, iter=18570/40000, loss=0.2185, lr=0.005703, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:42:52 2021-01-18 14:01:16 [INFO] [TRAIN] epoch=29, iter=18580/40000, loss=0.2643, lr=0.005700, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:41:38 2021-01-18 14:01:24 [INFO] [TRAIN] epoch=29, iter=18590/40000, loss=0.2763, lr=0.005698, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:41:18 2021-01-18 14:01:32 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.2495, lr=0.005696, batch_cost=0.7916, reader_cost=0.0001 | ETA 04:42:20 2021-01-18 14:01:40 [INFO] [TRAIN] epoch=29, iter=18610/40000, loss=0.2770, lr=0.005693, batch_cost=0.7941, reader_cost=0.0001 | ETA 04:43:05 2021-01-18 14:01:48 [INFO] [TRAIN] epoch=29, iter=18620/40000, loss=0.1931, lr=0.005691, batch_cost=0.7877, reader_cost=0.0001 | ETA 04:40:40 2021-01-18 14:01:56 [INFO] [TRAIN] epoch=29, iter=18630/40000, loss=0.3866, lr=0.005688, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:41:44 2021-01-18 14:02:04 [INFO] [TRAIN] epoch=29, iter=18640/40000, loss=0.3099, lr=0.005686, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:41:15 2021-01-18 14:02:12 [INFO] [TRAIN] epoch=29, iter=18650/40000, loss=0.2196, lr=0.005684, batch_cost=0.7956, reader_cost=0.0001 | ETA 04:43:05 2021-01-18 14:02:20 [INFO] [TRAIN] epoch=29, iter=18660/40000, loss=0.2184, lr=0.005681, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:41:07 2021-01-18 14:02:28 [INFO] [TRAIN] epoch=29, iter=18670/40000, loss=0.2232, lr=0.005679, batch_cost=0.7918, reader_cost=0.0001 | ETA 04:41:29 2021-01-18 14:02:35 [INFO] [TRAIN] epoch=29, iter=18680/40000, loss=0.2046, lr=0.005676, batch_cost=0.7849, reader_cost=0.0001 | ETA 04:38:53 2021-01-18 14:02:43 [INFO] [TRAIN] epoch=29, iter=18690/40000, loss=0.1717, lr=0.005674, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:39:19 2021-01-18 14:02:51 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.2134, lr=0.005672, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:40:39 2021-01-18 14:02:59 [INFO] [TRAIN] epoch=29, iter=18710/40000, loss=0.5140, lr=0.005669, batch_cost=0.7899, reader_cost=0.0001 | ETA 04:40:16 2021-01-18 14:03:07 [INFO] [TRAIN] epoch=29, iter=18720/40000, loss=0.2342, lr=0.005667, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:40:59 2021-01-18 14:03:15 [INFO] [TRAIN] epoch=29, iter=18730/40000, loss=0.2739, lr=0.005664, batch_cost=0.7831, reader_cost=0.0001 | ETA 04:37:36 2021-01-18 14:03:23 [INFO] [TRAIN] epoch=29, iter=18740/40000, loss=0.3176, lr=0.005662, batch_cost=0.7847, reader_cost=0.0001 | ETA 04:38:03 2021-01-18 14:03:31 [INFO] [TRAIN] epoch=29, iter=18750/40000, loss=0.1924, lr=0.005660, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:39:36 2021-01-18 14:03:39 [INFO] [TRAIN] epoch=29, iter=18760/40000, loss=0.3255, lr=0.005657, batch_cost=0.7956, reader_cost=0.0001 | ETA 04:41:39 2021-01-18 14:03:46 [INFO] [TRAIN] epoch=29, iter=18770/40000, loss=0.2602, lr=0.005655, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:39:42 2021-01-18 14:03:54 [INFO] [TRAIN] epoch=29, iter=18780/40000, loss=0.2948, lr=0.005652, batch_cost=0.7963, reader_cost=0.0001 | ETA 04:41:37 2021-01-18 14:04:02 [INFO] [TRAIN] epoch=29, iter=18790/40000, loss=0.2445, lr=0.005650, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:39:30 2021-01-18 14:04:10 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.3351, lr=0.005648, batch_cost=0.7843, reader_cost=0.0001 | ETA 04:37:07 2021-01-18 14:04:18 [INFO] [TRAIN] epoch=29, iter=18810/40000, loss=0.3497, lr=0.005645, batch_cost=0.7844, reader_cost=0.0001 | ETA 04:37:00 2021-01-18 14:04:26 [INFO] [TRAIN] epoch=29, iter=18820/40000, loss=0.2265, lr=0.005643, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:38:16 2021-01-18 14:04:34 [INFO] [TRAIN] epoch=29, iter=18830/40000, loss=0.7536, lr=0.005640, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:39:13 2021-01-18 14:04:42 [INFO] [TRAIN] epoch=29, iter=18840/40000, loss=0.5852, lr=0.005638, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:38:13 2021-01-18 14:04:50 [INFO] [TRAIN] epoch=29, iter=18850/40000, loss=0.4237, lr=0.005636, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:38:06 2021-01-18 14:04:57 [INFO] [TRAIN] epoch=29, iter=18860/40000, loss=0.5541, lr=0.005633, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:38:07 2021-01-18 14:05:05 [INFO] [TRAIN] epoch=29, iter=18870/40000, loss=0.3054, lr=0.005631, batch_cost=0.7953, reader_cost=0.0001 | ETA 04:40:05 2021-01-18 14:05:13 [INFO] [TRAIN] epoch=29, iter=18880/40000, loss=0.3827, lr=0.005628, batch_cost=0.7966, reader_cost=0.0001 | ETA 04:40:23 2021-01-18 14:05:21 [INFO] [TRAIN] epoch=29, iter=18890/40000, loss=0.2776, lr=0.005626, batch_cost=0.7949, reader_cost=0.0001 | ETA 04:39:39 2021-01-18 14:05:29 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.2516, lr=0.005624, batch_cost=0.7937, reader_cost=0.0001 | ETA 04:39:07 2021-01-18 14:05:37 [INFO] [TRAIN] epoch=29, iter=18910/40000, loss=0.3017, lr=0.005621, batch_cost=0.7903, reader_cost=0.0001 | ETA 04:37:46 2021-01-18 14:05:45 [INFO] [TRAIN] epoch=29, iter=18920/40000, loss=0.2796, lr=0.005619, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:37:52 2021-01-18 14:05:53 [INFO] [TRAIN] epoch=29, iter=18930/40000, loss=0.2791, lr=0.005616, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:37:50 2021-01-18 14:06:01 [INFO] [TRAIN] epoch=29, iter=18940/40000, loss=0.2806, lr=0.005614, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:36:37 2021-01-18 14:06:09 [INFO] [TRAIN] epoch=29, iter=18950/40000, loss=0.2147, lr=0.005612, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:37:54 2021-01-18 14:06:17 [INFO] [TRAIN] epoch=29, iter=18960/40000, loss=0.3742, lr=0.005609, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:36:34 2021-01-18 14:06:25 [INFO] [TRAIN] epoch=29, iter=18970/40000, loss=0.2336, lr=0.005607, batch_cost=0.7899, reader_cost=0.0001 | ETA 04:36:51 2021-01-18 14:06:32 [INFO] [TRAIN] epoch=29, iter=18980/40000, loss=0.4017, lr=0.005604, batch_cost=0.7885, reader_cost=0.0001 | ETA 04:36:13 2021-01-18 14:06:40 [INFO] [TRAIN] epoch=29, iter=18990/40000, loss=0.2292, lr=0.005602, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:36:55 2021-01-18 14:06:48 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.3164, lr=0.005600, batch_cost=0.7870, reader_cost=0.0001 | ETA 04:35:26 2021-01-18 14:06:48 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 14:07:16 [INFO] [EVAL] #Images=1449 mIoU=0.7659 Acc=0.9480 Kappa=0.8850 2021-01-18 14:07:16 [INFO] [EVAL] Class IoU: [0.9421 0.8847 0.4271 0.8679 0.6746 0.8223 0.9188 0.8432 0.9102 0.3351 0.8916 0.5905 0.877 0.8695 0.8252 0.8598 0.5647 0.8994 0.5433 0.8279 0.7083] 2021-01-18 14:07:16 [INFO] [EVAL] Class Acc: [0.9663 0.9295 0.4621 0.9159 0.7788 0.8889 0.9821 0.9262 0.9582 0.7357 0.9505 0.8558 0.9436 0.9259 0.9206 0.9221 0.8271 0.9357 0.757 0.8494 0.8478] 2021-01-18 14:07:23 [INFO] [EVAL] The model with the best validation mIoU (0.7669) was saved at iter 17000. 2021-01-18 14:07:30 [INFO] [TRAIN] epoch=29, iter=19010/40000, loss=0.3587, lr=0.005597, batch_cost=0.7880, reader_cost=0.0001 | ETA 04:35:39 2021-01-18 14:07:38 [INFO] [TRAIN] epoch=29, iter=19020/40000, loss=0.3821, lr=0.005595, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:35:34 2021-01-18 14:07:46 [INFO] [TRAIN] epoch=29, iter=19030/40000, loss=0.2592, lr=0.005592, batch_cost=0.7918, reader_cost=0.0001 | ETA 04:36:44 2021-01-18 14:07:54 [INFO] [TRAIN] epoch=29, iter=19040/40000, loss=0.2904, lr=0.005590, batch_cost=0.7951, reader_cost=0.0001 | ETA 04:37:44 2021-01-18 14:08:02 [INFO] [TRAIN] epoch=29, iter=19050/40000, loss=0.2043, lr=0.005588, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:35:55 2021-01-18 14:08:10 [INFO] [TRAIN] epoch=29, iter=19060/40000, loss=0.2280, lr=0.005585, batch_cost=0.7944, reader_cost=0.0001 | ETA 04:37:15 2021-01-18 14:08:18 [INFO] [TRAIN] epoch=29, iter=19070/40000, loss=0.2497, lr=0.005583, batch_cost=0.7940, reader_cost=0.0001 | ETA 04:36:57 2021-01-18 14:08:26 [INFO] [TRAIN] epoch=29, iter=19080/40000, loss=0.4110, lr=0.005580, batch_cost=0.7903, reader_cost=0.0001 | ETA 04:35:33 2021-01-18 14:08:34 [INFO] [TRAIN] epoch=29, iter=19090/40000, loss=0.2312, lr=0.005578, batch_cost=0.7878, reader_cost=0.0001 | ETA 04:34:33 2021-01-18 14:08:42 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.2042, lr=0.005576, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:36:19 2021-01-18 14:08:50 [INFO] [TRAIN] epoch=29, iter=19110/40000, loss=0.2856, lr=0.005573, batch_cost=0.7873, reader_cost=0.0001 | ETA 04:34:06 2021-01-18 14:08:57 [INFO] [TRAIN] epoch=29, iter=19120/40000, loss=0.3416, lr=0.005571, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:36:04 2021-01-18 14:09:05 [INFO] [TRAIN] epoch=29, iter=19130/40000, loss=0.4825, lr=0.005568, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:35:40 2021-01-18 14:09:13 [INFO] [TRAIN] epoch=29, iter=19140/40000, loss=0.2824, lr=0.005566, batch_cost=0.7893, reader_cost=0.0001 | ETA 04:34:24 2021-01-18 14:09:21 [INFO] [TRAIN] epoch=29, iter=19150/40000, loss=0.2130, lr=0.005564, batch_cost=0.7946, reader_cost=0.0001 | ETA 04:36:08 2021-01-18 14:09:29 [INFO] [TRAIN] epoch=29, iter=19160/40000, loss=0.2903, lr=0.005561, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:33:55 2021-01-18 14:09:37 [INFO] [TRAIN] epoch=30, iter=19170/40000, loss=0.2857, lr=0.005559, batch_cost=0.7761, reader_cost=0.0155 | ETA 04:29:26 2021-01-18 14:09:45 [INFO] [TRAIN] epoch=30, iter=19180/40000, loss=0.2930, lr=0.005556, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:33:31 2021-01-18 14:09:53 [INFO] [TRAIN] epoch=30, iter=19190/40000, loss=0.1856, lr=0.005554, batch_cost=0.7917, reader_cost=0.0001 | ETA 04:34:35 2021-01-18 14:10:01 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.2505, lr=0.005552, batch_cost=0.7915, reader_cost=0.0001 | ETA 04:34:24 2021-01-18 14:10:08 [INFO] [TRAIN] epoch=30, iter=19210/40000, loss=0.2127, lr=0.005549, batch_cost=0.7877, reader_cost=0.0001 | ETA 04:32:56 2021-01-18 14:10:16 [INFO] [TRAIN] epoch=30, iter=19220/40000, loss=0.2719, lr=0.005547, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:34:21 2021-01-18 14:10:24 [INFO] [TRAIN] epoch=30, iter=19230/40000, loss=0.2169, lr=0.005544, batch_cost=0.7942, reader_cost=0.0001 | ETA 04:34:54 2021-01-18 14:10:32 [INFO] [TRAIN] epoch=30, iter=19240/40000, loss=0.2776, lr=0.005542, batch_cost=0.7940, reader_cost=0.0001 | ETA 04:34:43 2021-01-18 14:10:40 [INFO] [TRAIN] epoch=30, iter=19250/40000, loss=0.2025, lr=0.005540, batch_cost=0.7913, reader_cost=0.0001 | ETA 04:33:39 2021-01-18 14:10:48 [INFO] [TRAIN] epoch=30, iter=19260/40000, loss=0.2614, lr=0.005537, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:31:49 2021-01-18 14:10:56 [INFO] [TRAIN] epoch=30, iter=19270/40000, loss=0.2500, lr=0.005535, batch_cost=0.7900, reader_cost=0.0001 | ETA 04:32:56 2021-01-18 14:11:04 [INFO] [TRAIN] epoch=30, iter=19280/40000, loss=0.2492, lr=0.005532, batch_cost=0.7875, reader_cost=0.0001 | ETA 04:31:56 2021-01-18 14:11:12 [INFO] [TRAIN] epoch=30, iter=19290/40000, loss=0.2331, lr=0.005530, batch_cost=0.7930, reader_cost=0.0001 | ETA 04:33:42 2021-01-18 14:11:20 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.1960, lr=0.005528, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:32:41 2021-01-18 14:11:28 [INFO] [TRAIN] epoch=30, iter=19310/40000, loss=0.1902, lr=0.005525, batch_cost=0.7911, reader_cost=0.0001 | ETA 04:32:47 2021-01-18 14:11:36 [INFO] [TRAIN] epoch=30, iter=19320/40000, loss=0.4212, lr=0.005523, batch_cost=0.7946, reader_cost=0.0001 | ETA 04:33:52 2021-01-18 14:11:43 [INFO] [TRAIN] epoch=30, iter=19330/40000, loss=0.2561, lr=0.005520, batch_cost=0.7943, reader_cost=0.0001 | ETA 04:33:37 2021-01-18 14:11:51 [INFO] [TRAIN] epoch=30, iter=19340/40000, loss=0.2414, lr=0.005518, batch_cost=0.7923, reader_cost=0.0001 | ETA 04:32:48 2021-01-18 14:11:59 [INFO] [TRAIN] epoch=30, iter=19350/40000, loss=0.2090, lr=0.005516, batch_cost=0.7888, reader_cost=0.0001 | ETA 04:31:28 2021-01-18 14:12:07 [INFO] [TRAIN] epoch=30, iter=19360/40000, loss=0.3042, lr=0.005513, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:32:26 2021-01-18 14:12:15 [INFO] [TRAIN] epoch=30, iter=19370/40000, loss=0.2571, lr=0.005511, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:32:44 2021-01-18 14:12:23 [INFO] [TRAIN] epoch=30, iter=19380/40000, loss=0.2586, lr=0.005508, batch_cost=0.7951, reader_cost=0.0001 | ETA 04:33:14 2021-01-18 14:12:31 [INFO] [TRAIN] epoch=30, iter=19390/40000, loss=0.2151, lr=0.005506, batch_cost=0.7890, reader_cost=0.0001 | ETA 04:31:00 2021-01-18 14:12:39 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.2056, lr=0.005504, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:31:33 2021-01-18 14:12:47 [INFO] [TRAIN] epoch=30, iter=19410/40000, loss=0.2710, lr=0.005501, batch_cost=0.7923, reader_cost=0.0001 | ETA 04:31:52 2021-01-18 14:12:55 [INFO] [TRAIN] epoch=30, iter=19420/40000, loss=0.2280, lr=0.005499, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:30:54 2021-01-18 14:13:03 [INFO] [TRAIN] epoch=30, iter=19430/40000, loss=0.1970, lr=0.005496, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:31:14 2021-01-18 14:13:11 [INFO] [TRAIN] epoch=30, iter=19440/40000, loss=0.2691, lr=0.005494, batch_cost=0.7945, reader_cost=0.0001 | ETA 04:32:15 2021-01-18 14:13:19 [INFO] [TRAIN] epoch=30, iter=19450/40000, loss=0.2224, lr=0.005492, batch_cost=0.7962, reader_cost=0.0001 | ETA 04:32:42 2021-01-18 14:13:26 [INFO] [TRAIN] epoch=30, iter=19460/40000, loss=0.2241, lr=0.005489, batch_cost=0.7854, reader_cost=0.0001 | ETA 04:28:51 2021-01-18 14:13:34 [INFO] [TRAIN] epoch=30, iter=19470/40000, loss=0.2369, lr=0.005487, batch_cost=0.7891, reader_cost=0.0001 | ETA 04:30:00 2021-01-18 14:13:42 [INFO] [TRAIN] epoch=30, iter=19480/40000, loss=0.2887, lr=0.005484, batch_cost=0.7903, reader_cost=0.0001 | ETA 04:30:16 2021-01-18 14:13:50 [INFO] [TRAIN] epoch=30, iter=19490/40000, loss=0.2457, lr=0.005482, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:30:32 2021-01-18 14:13:58 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.2416, lr=0.005480, batch_cost=0.7844, reader_cost=0.0001 | ETA 04:27:59 2021-01-18 14:14:06 [INFO] [TRAIN] epoch=30, iter=19510/40000, loss=0.2002, lr=0.005477, batch_cost=0.7873, reader_cost=0.0001 | ETA 04:28:50 2021-01-18 14:14:14 [INFO] [TRAIN] epoch=30, iter=19520/40000, loss=0.2134, lr=0.005475, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:29:51 2021-01-18 14:14:22 [INFO] [TRAIN] epoch=30, iter=19530/40000, loss=0.3211, lr=0.005472, batch_cost=0.7940, reader_cost=0.0001 | ETA 04:30:52 2021-01-18 14:14:30 [INFO] [TRAIN] epoch=30, iter=19540/40000, loss=0.2742, lr=0.005470, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:29:11 2021-01-18 14:14:37 [INFO] [TRAIN] epoch=30, iter=19550/40000, loss=0.1906, lr=0.005468, batch_cost=0.7878, reader_cost=0.0001 | ETA 04:28:30 2021-01-18 14:14:45 [INFO] [TRAIN] epoch=30, iter=19560/40000, loss=0.2086, lr=0.005465, batch_cost=0.7841, reader_cost=0.0001 | ETA 04:27:06 2021-01-18 14:14:53 [INFO] [TRAIN] epoch=30, iter=19570/40000, loss=0.2673, lr=0.005463, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:29:50 2021-01-18 14:15:01 [INFO] [TRAIN] epoch=30, iter=19580/40000, loss=0.1808, lr=0.005460, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:29:11 2021-01-18 14:15:09 [INFO] [TRAIN] epoch=30, iter=19590/40000, loss=0.2059, lr=0.005458, batch_cost=0.7893, reader_cost=0.0001 | ETA 04:28:29 2021-01-18 14:15:17 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.3631, lr=0.005455, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:29:43 2021-01-18 14:15:25 [INFO] [TRAIN] epoch=30, iter=19610/40000, loss=0.1934, lr=0.005453, batch_cost=0.7870, reader_cost=0.0001 | ETA 04:27:26 2021-01-18 14:15:33 [INFO] [TRAIN] epoch=30, iter=19620/40000, loss=0.2370, lr=0.005451, batch_cost=0.7892, reader_cost=0.0001 | ETA 04:28:03 2021-01-18 14:15:41 [INFO] [TRAIN] epoch=30, iter=19630/40000, loss=0.2364, lr=0.005448, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:28:29 2021-01-18 14:15:49 [INFO] [TRAIN] epoch=30, iter=19640/40000, loss=0.2591, lr=0.005446, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:28:21 2021-01-18 14:15:56 [INFO] [TRAIN] epoch=30, iter=19650/40000, loss=0.3781, lr=0.005443, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:27:19 2021-01-18 14:16:04 [INFO] [TRAIN] epoch=30, iter=19660/40000, loss=0.2329, lr=0.005441, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:28:01 2021-01-18 14:16:12 [INFO] [TRAIN] epoch=30, iter=19670/40000, loss=0.1795, lr=0.005439, batch_cost=0.7874, reader_cost=0.0001 | ETA 04:26:47 2021-01-18 14:16:20 [INFO] [TRAIN] epoch=30, iter=19680/40000, loss=0.2805, lr=0.005436, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:27:43 2021-01-18 14:16:28 [INFO] [TRAIN] epoch=30, iter=19690/40000, loss=0.2826, lr=0.005434, batch_cost=0.7870, reader_cost=0.0001 | ETA 04:26:23 2021-01-18 14:16:36 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.2205, lr=0.005431, batch_cost=0.7876, reader_cost=0.0001 | ETA 04:26:27 2021-01-18 14:16:44 [INFO] [TRAIN] epoch=30, iter=19710/40000, loss=0.2262, lr=0.005429, batch_cost=0.7919, reader_cost=0.0001 | ETA 04:27:46 2021-01-18 14:16:52 [INFO] [TRAIN] epoch=30, iter=19720/40000, loss=0.3032, lr=0.005427, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:27:02 2021-01-18 14:17:00 [INFO] [TRAIN] epoch=30, iter=19730/40000, loss=0.1669, lr=0.005424, batch_cost=0.7919, reader_cost=0.0001 | ETA 04:27:31 2021-01-18 14:17:08 [INFO] [TRAIN] epoch=30, iter=19740/40000, loss=0.2233, lr=0.005422, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:27:03 2021-01-18 14:17:15 [INFO] [TRAIN] epoch=30, iter=19750/40000, loss=0.1767, lr=0.005419, batch_cost=0.7959, reader_cost=0.0001 | ETA 04:28:37 2021-01-18 14:17:23 [INFO] [TRAIN] epoch=30, iter=19760/40000, loss=0.2726, lr=0.005417, batch_cost=0.7877, reader_cost=0.0001 | ETA 04:25:42 2021-01-18 14:17:31 [INFO] [TRAIN] epoch=30, iter=19770/40000, loss=0.2214, lr=0.005415, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:25:47 2021-01-18 14:17:39 [INFO] [TRAIN] epoch=30, iter=19780/40000, loss=0.2558, lr=0.005412, batch_cost=0.7903, reader_cost=0.0001 | ETA 04:26:20 2021-01-18 14:17:47 [INFO] [TRAIN] epoch=30, iter=19790/40000, loss=0.2422, lr=0.005410, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:26:33 2021-01-18 14:17:55 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.1908, lr=0.005407, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:26:12 2021-01-18 14:18:03 [INFO] [TRAIN] epoch=30, iter=19810/40000, loss=0.2203, lr=0.005405, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:26:30 2021-01-18 14:18:11 [INFO] [TRAIN] epoch=30, iter=19820/40000, loss=0.1928, lr=0.005402, batch_cost=0.7931, reader_cost=0.0001 | ETA 04:26:45 2021-01-18 14:18:18 [INFO] [TRAIN] epoch=30, iter=19830/40000, loss=0.2204, lr=0.005400, batch_cost=0.7561, reader_cost=0.0001 | ETA 04:14:11 2021-01-18 14:18:26 [INFO] [TRAIN] epoch=31, iter=19840/40000, loss=0.1998, lr=0.005398, batch_cost=0.8092, reader_cost=0.0149 | ETA 04:31:52 2021-01-18 14:18:34 [INFO] [TRAIN] epoch=31, iter=19850/40000, loss=0.1574, lr=0.005395, batch_cost=0.7943, reader_cost=0.0001 | ETA 04:26:44 2021-01-18 14:18:42 [INFO] [TRAIN] epoch=31, iter=19860/40000, loss=0.1540, lr=0.005393, batch_cost=0.7961, reader_cost=0.0001 | ETA 04:27:12 2021-01-18 14:18:50 [INFO] [TRAIN] epoch=31, iter=19870/40000, loss=0.1998, lr=0.005390, batch_cost=0.7869, reader_cost=0.0001 | ETA 04:24:00 2021-01-18 14:18:58 [INFO] [TRAIN] epoch=31, iter=19880/40000, loss=0.3055, lr=0.005388, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:24:47 2021-01-18 14:19:06 [INFO] [TRAIN] epoch=31, iter=19890/40000, loss=0.2638, lr=0.005386, batch_cost=0.7892, reader_cost=0.0001 | ETA 04:24:31 2021-01-18 14:19:14 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.2033, lr=0.005383, batch_cost=0.7969, reader_cost=0.0001 | ETA 04:26:57 2021-01-18 14:19:22 [INFO] [TRAIN] epoch=31, iter=19910/40000, loss=0.2232, lr=0.005381, batch_cost=0.7877, reader_cost=0.0001 | ETA 04:23:45 2021-01-18 14:19:30 [INFO] [TRAIN] epoch=31, iter=19920/40000, loss=0.1895, lr=0.005378, batch_cost=0.7915, reader_cost=0.0001 | ETA 04:24:53 2021-01-18 14:19:38 [INFO] [TRAIN] epoch=31, iter=19930/40000, loss=0.2394, lr=0.005376, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:24:22 2021-01-18 14:19:46 [INFO] [TRAIN] epoch=31, iter=19940/40000, loss=0.1885, lr=0.005374, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:24:20 2021-01-18 14:19:53 [INFO] [TRAIN] epoch=31, iter=19950/40000, loss=0.2087, lr=0.005371, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:24:10 2021-01-18 14:20:01 [INFO] [TRAIN] epoch=31, iter=19960/40000, loss=0.2639, lr=0.005369, batch_cost=0.7897, reader_cost=0.0001 | ETA 04:23:45 2021-01-18 14:20:09 [INFO] [TRAIN] epoch=31, iter=19970/40000, loss=0.1746, lr=0.005366, batch_cost=0.7981, reader_cost=0.0001 | ETA 04:26:26 2021-01-18 14:20:17 [INFO] [TRAIN] epoch=31, iter=19980/40000, loss=0.2101, lr=0.005364, batch_cost=0.7906, reader_cost=0.0001 | ETA 04:23:48 2021-01-18 14:20:25 [INFO] [TRAIN] epoch=31, iter=19990/40000, loss=0.1735, lr=0.005362, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:23:41 2021-01-18 14:20:33 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.2460, lr=0.005359, batch_cost=0.7880, reader_cost=0.0001 | ETA 04:22:40 2021-01-18 14:20:33 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 14:21:01 [INFO] [EVAL] #Images=1449 mIoU=0.7761 Acc=0.9490 Kappa=0.8879 2021-01-18 14:21:01 [INFO] [EVAL] Class IoU: [0.9439 0.8453 0.4285 0.8988 0.7309 0.8482 0.8978 0.8199 0.8989 0.4053 0.8659 0.5987 0.8449 0.8899 0.8709 0.8743 0.6378 0.8402 0.5393 0.8598 0.7595] 2021-01-18 14:21:01 [INFO] [EVAL] Class Acc: [0.9697 0.8684 0.441 0.9516 0.8459 0.8985 0.9763 0.8505 0.9245 0.5969 0.9197 0.8479 0.9272 0.9396 0.926 0.9521 0.8155 0.9411 0.7926 0.901 0.921 ] 2021-01-18 14:21:11 [INFO] [EVAL] The model with the best validation mIoU (0.7761) was saved at iter 20000. 2021-01-18 14:21:19 [INFO] [TRAIN] epoch=31, iter=20010/40000, loss=0.2599, lr=0.005357, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:23:04 2021-01-18 14:21:27 [INFO] [TRAIN] epoch=31, iter=20020/40000, loss=0.2010, lr=0.005354, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:22:51 2021-01-18 14:21:35 [INFO] [TRAIN] epoch=31, iter=20030/40000, loss=0.1559, lr=0.005352, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:23:23 2021-01-18 14:21:43 [INFO] [TRAIN] epoch=31, iter=20040/40000, loss=0.2059, lr=0.005349, batch_cost=0.7954, reader_cost=0.0001 | ETA 04:24:35 2021-01-18 14:21:51 [INFO] [TRAIN] epoch=31, iter=20050/40000, loss=0.2245, lr=0.005347, batch_cost=0.7923, reader_cost=0.0001 | ETA 04:23:27 2021-01-18 14:21:59 [INFO] [TRAIN] epoch=31, iter=20060/40000, loss=0.1648, lr=0.005345, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:22:59 2021-01-18 14:22:07 [INFO] [TRAIN] epoch=31, iter=20070/40000, loss=0.1882, lr=0.005342, batch_cost=0.7846, reader_cost=0.0001 | ETA 04:20:36 2021-01-18 14:22:15 [INFO] [TRAIN] epoch=31, iter=20080/40000, loss=0.2327, lr=0.005340, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:21:39 2021-01-18 14:22:22 [INFO] [TRAIN] epoch=31, iter=20090/40000, loss=0.1745, lr=0.005337, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:21:32 2021-01-18 14:22:30 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.1994, lr=0.005335, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:21:56 2021-01-18 14:22:38 [INFO] [TRAIN] epoch=31, iter=20110/40000, loss=0.1505, lr=0.005333, batch_cost=0.7892, reader_cost=0.0001 | ETA 04:21:36 2021-01-18 14:22:46 [INFO] [TRAIN] epoch=31, iter=20120/40000, loss=0.2162, lr=0.005330, batch_cost=0.7894, reader_cost=0.0001 | ETA 04:21:33 2021-01-18 14:22:54 [INFO] [TRAIN] epoch=31, iter=20130/40000, loss=0.2430, lr=0.005328, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:22:05 2021-01-18 14:23:02 [INFO] [TRAIN] epoch=31, iter=20140/40000, loss=0.2197, lr=0.005325, batch_cost=0.7870, reader_cost=0.0001 | ETA 04:20:30 2021-01-18 14:23:10 [INFO] [TRAIN] epoch=31, iter=20150/40000, loss=0.1869, lr=0.005323, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:20:53 2021-01-18 14:23:18 [INFO] [TRAIN] epoch=31, iter=20160/40000, loss=0.1690, lr=0.005321, batch_cost=0.7935, reader_cost=0.0001 | ETA 04:22:23 2021-01-18 14:23:26 [INFO] [TRAIN] epoch=31, iter=20170/40000, loss=0.2325, lr=0.005318, batch_cost=0.7915, reader_cost=0.0001 | ETA 04:21:35 2021-01-18 14:23:34 [INFO] [TRAIN] epoch=31, iter=20180/40000, loss=0.2865, lr=0.005316, batch_cost=0.7893, reader_cost=0.0001 | ETA 04:20:44 2021-01-18 14:23:41 [INFO] [TRAIN] epoch=31, iter=20190/40000, loss=0.2228, lr=0.005313, batch_cost=0.7911, reader_cost=0.0001 | ETA 04:21:11 2021-01-18 14:23:49 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.1434, lr=0.005311, batch_cost=0.7928, reader_cost=0.0001 | ETA 04:21:36 2021-01-18 14:23:57 [INFO] [TRAIN] epoch=31, iter=20210/40000, loss=0.1819, lr=0.005308, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:20:36 2021-01-18 14:24:05 [INFO] [TRAIN] epoch=31, iter=20220/40000, loss=0.2409, lr=0.005306, batch_cost=0.7935, reader_cost=0.0001 | ETA 04:21:35 2021-01-18 14:24:13 [INFO] [TRAIN] epoch=31, iter=20230/40000, loss=0.2704, lr=0.005304, batch_cost=0.7879, reader_cost=0.0001 | ETA 04:19:36 2021-01-18 14:24:21 [INFO] [TRAIN] epoch=31, iter=20240/40000, loss=0.2284, lr=0.005301, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:20:19 2021-01-18 14:24:29 [INFO] [TRAIN] epoch=31, iter=20250/40000, loss=0.2328, lr=0.005299, batch_cost=0.7900, reader_cost=0.0001 | ETA 04:20:02 2021-01-18 14:24:37 [INFO] [TRAIN] epoch=31, iter=20260/40000, loss=0.2465, lr=0.005296, batch_cost=0.7891, reader_cost=0.0001 | ETA 04:19:36 2021-01-18 14:24:45 [INFO] [TRAIN] epoch=31, iter=20270/40000, loss=0.2487, lr=0.005294, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:19:37 2021-01-18 14:24:53 [INFO] [TRAIN] epoch=31, iter=20280/40000, loss=0.3876, lr=0.005292, batch_cost=0.7921, reader_cost=0.0001 | ETA 04:20:20 2021-01-18 14:25:01 [INFO] [TRAIN] epoch=31, iter=20290/40000, loss=0.2099, lr=0.005289, batch_cost=0.7888, reader_cost=0.0001 | ETA 04:19:07 2021-01-18 14:25:08 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.2775, lr=0.005287, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:19:01 2021-01-18 14:25:16 [INFO] [TRAIN] epoch=31, iter=20310/40000, loss=0.2917, lr=0.005284, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:20:12 2021-01-18 14:25:24 [INFO] [TRAIN] epoch=31, iter=20320/40000, loss=0.2637, lr=0.005282, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:18:59 2021-01-18 14:25:32 [INFO] [TRAIN] epoch=31, iter=20330/40000, loss=0.2711, lr=0.005279, batch_cost=0.7952, reader_cost=0.0001 | ETA 04:20:41 2021-01-18 14:25:40 [INFO] [TRAIN] epoch=31, iter=20340/40000, loss=0.2132, lr=0.005277, batch_cost=0.7878, reader_cost=0.0001 | ETA 04:18:08 2021-01-18 14:25:48 [INFO] [TRAIN] epoch=31, iter=20350/40000, loss=0.2683, lr=0.005275, batch_cost=0.7935, reader_cost=0.0001 | ETA 04:19:52 2021-01-18 14:25:56 [INFO] [TRAIN] epoch=31, iter=20360/40000, loss=0.1772, lr=0.005272, batch_cost=0.7924, reader_cost=0.0001 | ETA 04:19:23 2021-01-18 14:26:04 [INFO] [TRAIN] epoch=31, iter=20370/40000, loss=0.2875, lr=0.005270, batch_cost=0.7869, reader_cost=0.0001 | ETA 04:17:26 2021-01-18 14:26:12 [INFO] [TRAIN] epoch=31, iter=20380/40000, loss=0.2309, lr=0.005267, batch_cost=0.7920, reader_cost=0.0001 | ETA 04:18:59 2021-01-18 14:26:20 [INFO] [TRAIN] epoch=31, iter=20390/40000, loss=0.1715, lr=0.005265, batch_cost=0.7915, reader_cost=0.0001 | ETA 04:18:40 2021-01-18 14:26:28 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.2006, lr=0.005263, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:18:20 2021-01-18 14:26:35 [INFO] [TRAIN] epoch=31, iter=20410/40000, loss=0.2964, lr=0.005260, batch_cost=0.7884, reader_cost=0.0001 | ETA 04:17:25 2021-01-18 14:26:43 [INFO] [TRAIN] epoch=31, iter=20420/40000, loss=0.1859, lr=0.005258, batch_cost=0.7911, reader_cost=0.0001 | ETA 04:18:09 2021-01-18 14:26:51 [INFO] [TRAIN] epoch=31, iter=20430/40000, loss=0.1681, lr=0.005255, batch_cost=0.7880, reader_cost=0.0001 | ETA 04:17:00 2021-01-18 14:26:59 [INFO] [TRAIN] epoch=31, iter=20440/40000, loss=0.2822, lr=0.005253, batch_cost=0.7969, reader_cost=0.0001 | ETA 04:19:46 2021-01-18 14:27:07 [INFO] [TRAIN] epoch=31, iter=20450/40000, loss=0.2094, lr=0.005250, batch_cost=0.7934, reader_cost=0.0001 | ETA 04:18:30 2021-01-18 14:27:15 [INFO] [TRAIN] epoch=31, iter=20460/40000, loss=0.2476, lr=0.005248, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:17:30 2021-01-18 14:27:23 [INFO] [TRAIN] epoch=31, iter=20470/40000, loss=0.2346, lr=0.005246, batch_cost=0.7849, reader_cost=0.0001 | ETA 04:15:29 2021-01-18 14:27:31 [INFO] [TRAIN] epoch=31, iter=20480/40000, loss=0.2224, lr=0.005243, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:17:14 2021-01-18 14:27:38 [INFO] [TRAIN] epoch=31, iter=20490/40000, loss=0.1951, lr=0.005241, batch_cost=0.7607, reader_cost=0.0001 | ETA 04:07:20 2021-01-18 14:27:46 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.2269, lr=0.005238, batch_cost=0.7990, reader_cost=0.0136 | ETA 04:19:40 2021-01-18 14:27:54 [INFO] [TRAIN] epoch=32, iter=20510/40000, loss=0.2412, lr=0.005236, batch_cost=0.7879, reader_cost=0.0001 | ETA 04:15:55 2021-01-18 14:28:02 [INFO] [TRAIN] epoch=32, iter=20520/40000, loss=0.2375, lr=0.005234, batch_cost=0.7936, reader_cost=0.0001 | ETA 04:17:39 2021-01-18 14:28:10 [INFO] [TRAIN] epoch=32, iter=20530/40000, loss=0.1353, lr=0.005231, batch_cost=0.7859, reader_cost=0.0001 | ETA 04:15:01 2021-01-18 14:28:18 [INFO] [TRAIN] epoch=32, iter=20540/40000, loss=0.1457, lr=0.005229, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:15:48 2021-01-18 14:28:26 [INFO] [TRAIN] epoch=32, iter=20550/40000, loss=0.2591, lr=0.005226, batch_cost=0.7882, reader_cost=0.0001 | ETA 04:15:31 2021-01-18 14:28:34 [INFO] [TRAIN] epoch=32, iter=20560/40000, loss=0.1699, lr=0.005224, batch_cost=0.7936, reader_cost=0.0001 | ETA 04:17:06 2021-01-18 14:28:42 [INFO] [TRAIN] epoch=32, iter=20570/40000, loss=0.1990, lr=0.005221, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:15:52 2021-01-18 14:28:50 [INFO] [TRAIN] epoch=32, iter=20580/40000, loss=0.1830, lr=0.005219, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:15:43 2021-01-18 14:28:57 [INFO] [TRAIN] epoch=32, iter=20590/40000, loss=0.1670, lr=0.005217, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:15:07 2021-01-18 14:29:05 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.1631, lr=0.005214, batch_cost=0.7963, reader_cost=0.0001 | ETA 04:17:28 2021-01-18 14:29:13 [INFO] [TRAIN] epoch=32, iter=20610/40000, loss=0.2186, lr=0.005212, batch_cost=0.7939, reader_cost=0.0001 | ETA 04:16:32 2021-01-18 14:29:21 [INFO] [TRAIN] epoch=32, iter=20620/40000, loss=0.1628, lr=0.005209, batch_cost=0.7895, reader_cost=0.0001 | ETA 04:15:01 2021-01-18 14:29:29 [INFO] [TRAIN] epoch=32, iter=20630/40000, loss=0.2229, lr=0.005207, batch_cost=0.7903, reader_cost=0.0001 | ETA 04:15:08 2021-01-18 14:29:37 [INFO] [TRAIN] epoch=32, iter=20640/40000, loss=0.1591, lr=0.005205, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:15:03 2021-01-18 14:29:45 [INFO] [TRAIN] epoch=32, iter=20650/40000, loss=0.1952, lr=0.005202, batch_cost=0.7918, reader_cost=0.0001 | ETA 04:15:20 2021-01-18 14:29:53 [INFO] [TRAIN] epoch=32, iter=20660/40000, loss=0.1814, lr=0.005200, batch_cost=0.7848, reader_cost=0.0001 | ETA 04:12:58 2021-01-18 14:30:01 [INFO] [TRAIN] epoch=32, iter=20670/40000, loss=0.1954, lr=0.005197, batch_cost=0.7864, reader_cost=0.0001 | ETA 04:13:20 2021-01-18 14:30:09 [INFO] [TRAIN] epoch=32, iter=20680/40000, loss=0.2011, lr=0.005195, batch_cost=0.7890, reader_cost=0.0001 | ETA 04:14:03 2021-01-18 14:30:17 [INFO] [TRAIN] epoch=32, iter=20690/40000, loss=0.2037, lr=0.005192, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:14:17 2021-01-18 14:30:24 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.2573, lr=0.005190, batch_cost=0.7856, reader_cost=0.0001 | ETA 04:12:42 2021-01-18 14:30:32 [INFO] [TRAIN] epoch=32, iter=20710/40000, loss=0.1864, lr=0.005188, batch_cost=0.7908, reader_cost=0.0001 | ETA 04:14:14 2021-01-18 14:30:40 [INFO] [TRAIN] epoch=32, iter=20720/40000, loss=0.1598, lr=0.005185, batch_cost=0.7917, reader_cost=0.0001 | ETA 04:14:23 2021-01-18 14:30:48 [INFO] [TRAIN] epoch=32, iter=20730/40000, loss=0.1874, lr=0.005183, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:13:44 2021-01-18 14:30:56 [INFO] [TRAIN] epoch=32, iter=20740/40000, loss=0.2441, lr=0.005180, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:14:18 2021-01-18 14:31:04 [INFO] [TRAIN] epoch=32, iter=20750/40000, loss=0.1922, lr=0.005178, batch_cost=0.7956, reader_cost=0.0001 | ETA 04:15:16 2021-01-18 14:31:12 [INFO] [TRAIN] epoch=32, iter=20760/40000, loss=0.2564, lr=0.005175, batch_cost=0.7887, reader_cost=0.0001 | ETA 04:12:54 2021-01-18 14:31:20 [INFO] [TRAIN] epoch=32, iter=20770/40000, loss=0.2487, lr=0.005173, batch_cost=0.7867, reader_cost=0.0001 | ETA 04:12:08 2021-01-18 14:31:28 [INFO] [TRAIN] epoch=32, iter=20780/40000, loss=0.1960, lr=0.005171, batch_cost=0.7926, reader_cost=0.0001 | ETA 04:13:53 2021-01-18 14:31:36 [INFO] [TRAIN] epoch=32, iter=20790/40000, loss=0.2175, lr=0.005168, batch_cost=0.7950, reader_cost=0.0001 | ETA 04:14:32 2021-01-18 14:31:44 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.2291, lr=0.005166, batch_cost=0.7927, reader_cost=0.0001 | ETA 04:13:38 2021-01-18 14:31:51 [INFO] [TRAIN] epoch=32, iter=20810/40000, loss=0.1372, lr=0.005163, batch_cost=0.7879, reader_cost=0.0001 | ETA 04:11:59 2021-01-18 14:31:59 [INFO] [TRAIN] epoch=32, iter=20820/40000, loss=0.2009, lr=0.005161, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:11:59 2021-01-18 14:32:07 [INFO] [TRAIN] epoch=32, iter=20830/40000, loss=0.2306, lr=0.005159, batch_cost=0.7938, reader_cost=0.0001 | ETA 04:13:37 2021-01-18 14:32:15 [INFO] [TRAIN] epoch=32, iter=20840/40000, loss=0.1543, lr=0.005156, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:13:18 2021-01-18 14:32:23 [INFO] [TRAIN] epoch=32, iter=20850/40000, loss=0.2704, lr=0.005154, batch_cost=0.7881, reader_cost=0.0001 | ETA 04:11:32 2021-01-18 14:32:31 [INFO] [TRAIN] epoch=32, iter=20860/40000, loss=0.1924, lr=0.005151, batch_cost=0.7837, reader_cost=0.0001 | ETA 04:10:00 2021-01-18 14:32:39 [INFO] [TRAIN] epoch=32, iter=20870/40000, loss=0.1470, lr=0.005149, batch_cost=0.7937, reader_cost=0.0001 | ETA 04:13:02 2021-01-18 14:32:47 [INFO] [TRAIN] epoch=32, iter=20880/40000, loss=0.1650, lr=0.005146, batch_cost=0.7900, reader_cost=0.0001 | ETA 04:11:44 2021-01-18 14:32:55 [INFO] [TRAIN] epoch=32, iter=20890/40000, loss=0.2415, lr=0.005144, batch_cost=0.7952, reader_cost=0.0001 | ETA 04:13:16 2021-01-18 14:33:03 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.2003, lr=0.005142, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:11:55 2021-01-18 14:33:10 [INFO] [TRAIN] epoch=32, iter=20910/40000, loss=0.2361, lr=0.005139, batch_cost=0.7857, reader_cost=0.0001 | ETA 04:09:58 2021-01-18 14:33:18 [INFO] [TRAIN] epoch=32, iter=20920/40000, loss=0.2340, lr=0.005137, batch_cost=0.7877, reader_cost=0.0001 | ETA 04:10:29 2021-01-18 14:33:26 [INFO] [TRAIN] epoch=32, iter=20930/40000, loss=0.1914, lr=0.005134, batch_cost=0.7928, reader_cost=0.0001 | ETA 04:11:59 2021-01-18 14:33:34 [INFO] [TRAIN] epoch=32, iter=20940/40000, loss=0.2768, lr=0.005132, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:11:07 2021-01-18 14:33:42 [INFO] [TRAIN] epoch=32, iter=20950/40000, loss=0.1733, lr=0.005129, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:11:52 2021-01-18 14:33:50 [INFO] [TRAIN] epoch=32, iter=20960/40000, loss=0.2311, lr=0.005127, batch_cost=0.7866, reader_cost=0.0001 | ETA 04:09:37 2021-01-18 14:33:58 [INFO] [TRAIN] epoch=32, iter=20970/40000, loss=0.2423, lr=0.005125, batch_cost=0.7889, reader_cost=0.0001 | ETA 04:10:13 2021-01-18 14:34:06 [INFO] [TRAIN] epoch=32, iter=20980/40000, loss=0.1962, lr=0.005122, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:10:48 2021-01-18 14:34:14 [INFO] [TRAIN] epoch=32, iter=20990/40000, loss=0.3146, lr=0.005120, batch_cost=0.7921, reader_cost=0.0001 | ETA 04:10:58 2021-01-18 14:34:22 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.2390, lr=0.005117, batch_cost=0.7869, reader_cost=0.0001 | ETA 04:09:10 2021-01-18 14:34:22 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 14:34:50 [INFO] [EVAL] #Images=1449 mIoU=0.7745 Acc=0.9502 Kappa=0.8891 2021-01-18 14:34:50 [INFO] [EVAL] Class IoU: [0.9442 0.8902 0.4399 0.8847 0.6877 0.8144 0.9342 0.8568 0.9044 0.3949 0.8448 0.5888 0.8698 0.8623 0.8621 0.8795 0.6579 0.8333 0.4757 0.8864 0.753 ] 2021-01-18 14:34:50 [INFO] [EVAL] Class Acc: [0.9639 0.9283 0.4539 0.9383 0.8156 0.8893 0.9801 0.916 0.9587 0.7356 0.9364 0.8355 0.9357 0.9077 0.9553 0.9421 0.8614 0.9469 0.8317 0.9399 0.9111] 2021-01-18 14:34:56 [INFO] [EVAL] The model with the best validation mIoU (0.7761) was saved at iter 20000. 2021-01-18 14:35:04 [INFO] [TRAIN] epoch=32, iter=21010/40000, loss=0.1612, lr=0.005115, batch_cost=0.7873, reader_cost=0.0001 | ETA 04:09:11 2021-01-18 14:35:12 [INFO] [TRAIN] epoch=32, iter=21020/40000, loss=0.1782, lr=0.005112, batch_cost=0.7883, reader_cost=0.0001 | ETA 04:09:22 2021-01-18 14:35:20 [INFO] [TRAIN] epoch=32, iter=21030/40000, loss=0.3184, lr=0.005110, batch_cost=0.7912, reader_cost=0.0001 | ETA 04:10:08 2021-01-18 14:35:28 [INFO] [TRAIN] epoch=32, iter=21040/40000, loss=0.1691, lr=0.005108, batch_cost=0.7965, reader_cost=0.0001 | ETA 04:11:42 2021-01-18 14:35:36 [INFO] [TRAIN] epoch=32, iter=21050/40000, loss=0.2129, lr=0.005105, batch_cost=0.7959, reader_cost=0.0001 | ETA 04:11:22 2021-01-18 14:35:44 [INFO] [TRAIN] epoch=32, iter=21060/40000, loss=0.2336, lr=0.005103, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:10:10 2021-01-18 14:35:52 [INFO] [TRAIN] epoch=32, iter=21070/40000, loss=0.2973, lr=0.005100, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:09:17 2021-01-18 14:35:59 [INFO] [TRAIN] epoch=32, iter=21080/40000, loss=0.2049, lr=0.005098, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:09:53 2021-01-18 14:36:07 [INFO] [TRAIN] epoch=32, iter=21090/40000, loss=0.2839, lr=0.005096, batch_cost=0.7917, reader_cost=0.0001 | ETA 04:09:31 2021-01-18 14:36:15 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.3437, lr=0.005093, batch_cost=0.7945, reader_cost=0.0001 | ETA 04:10:15 2021-01-18 14:36:23 [INFO] [TRAIN] epoch=32, iter=21110/40000, loss=0.2357, lr=0.005091, batch_cost=0.7899, reader_cost=0.0001 | ETA 04:08:42 2021-01-18 14:36:31 [INFO] [TRAIN] epoch=32, iter=21120/40000, loss=0.3303, lr=0.005088, batch_cost=0.7866, reader_cost=0.0001 | ETA 04:07:30 2021-01-18 14:36:39 [INFO] [TRAIN] epoch=32, iter=21130/40000, loss=0.1831, lr=0.005086, batch_cost=0.7925, reader_cost=0.0001 | ETA 04:09:13 2021-01-18 14:36:47 [INFO] [TRAIN] epoch=32, iter=21140/40000, loss=0.2028, lr=0.005083, batch_cost=0.7863, reader_cost=0.0001 | ETA 04:07:09 2021-01-18 14:36:55 [INFO] [TRAIN] epoch=32, iter=21150/40000, loss=0.1533, lr=0.005081, batch_cost=0.7672, reader_cost=0.0001 | ETA 04:01:01 2021-01-18 14:37:02 [INFO] [TRAIN] epoch=33, iter=21160/40000, loss=0.2113, lr=0.005079, batch_cost=0.7942, reader_cost=0.0126 | ETA 04:09:23 2021-01-18 14:37:10 [INFO] [TRAIN] epoch=33, iter=21170/40000, loss=0.2737, lr=0.005076, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:08:21 2021-01-18 14:37:18 [INFO] [TRAIN] epoch=33, iter=21180/40000, loss=0.1863, lr=0.005074, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:07:52 2021-01-18 14:37:26 [INFO] [TRAIN] epoch=33, iter=21190/40000, loss=0.1210, lr=0.005071, batch_cost=0.7888, reader_cost=0.0001 | ETA 04:07:17 2021-01-18 14:37:34 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.1815, lr=0.005069, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:07:49 2021-01-18 14:37:42 [INFO] [TRAIN] epoch=33, iter=21210/40000, loss=0.2032, lr=0.005066, batch_cost=0.7885, reader_cost=0.0001 | ETA 04:06:55 2021-01-18 14:37:50 [INFO] [TRAIN] epoch=33, iter=21220/40000, loss=0.3075, lr=0.005064, batch_cost=0.7934, reader_cost=0.0001 | ETA 04:08:20 2021-01-18 14:37:58 [INFO] [TRAIN] epoch=33, iter=21230/40000, loss=0.4889, lr=0.005062, batch_cost=0.7932, reader_cost=0.0001 | ETA 04:08:08 2021-01-18 14:38:06 [INFO] [TRAIN] epoch=33, iter=21240/40000, loss=0.3783, lr=0.005059, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:07:18 2021-01-18 14:38:14 [INFO] [TRAIN] epoch=33, iter=21250/40000, loss=0.2371, lr=0.005057, batch_cost=0.7946, reader_cost=0.0001 | ETA 04:08:19 2021-01-18 14:38:22 [INFO] [TRAIN] epoch=33, iter=21260/40000, loss=0.2117, lr=0.005054, batch_cost=0.7952, reader_cost=0.0001 | ETA 04:08:22 2021-01-18 14:38:30 [INFO] [TRAIN] epoch=33, iter=21270/40000, loss=0.2249, lr=0.005052, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:07:17 2021-01-18 14:38:37 [INFO] [TRAIN] epoch=33, iter=21280/40000, loss=0.2703, lr=0.005049, batch_cost=0.7892, reader_cost=0.0001 | ETA 04:06:13 2021-01-18 14:38:45 [INFO] [TRAIN] epoch=33, iter=21290/40000, loss=0.2040, lr=0.005047, batch_cost=0.7874, reader_cost=0.0001 | ETA 04:05:32 2021-01-18 14:38:53 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.4658, lr=0.005045, batch_cost=0.7910, reader_cost=0.0001 | ETA 04:06:31 2021-01-18 14:39:01 [INFO] [TRAIN] epoch=33, iter=21310/40000, loss=0.2588, lr=0.005042, batch_cost=0.7970, reader_cost=0.0001 | ETA 04:08:15 2021-01-18 14:39:09 [INFO] [TRAIN] epoch=33, iter=21320/40000, loss=0.2131, lr=0.005040, batch_cost=0.7949, reader_cost=0.0001 | ETA 04:07:28 2021-01-18 14:39:17 [INFO] [TRAIN] epoch=33, iter=21330/40000, loss=0.4027, lr=0.005037, batch_cost=0.7879, reader_cost=0.0001 | ETA 04:05:10 2021-01-18 14:39:25 [INFO] [TRAIN] epoch=33, iter=21340/40000, loss=0.2079, lr=0.005035, batch_cost=0.7891, reader_cost=0.0001 | ETA 04:05:24 2021-01-18 14:39:33 [INFO] [TRAIN] epoch=33, iter=21350/40000, loss=0.2227, lr=0.005032, batch_cost=0.7923, reader_cost=0.0001 | ETA 04:06:15 2021-01-18 14:39:41 [INFO] [TRAIN] epoch=33, iter=21360/40000, loss=0.1910, lr=0.005030, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:05:22 2021-01-18 14:39:49 [INFO] [TRAIN] epoch=33, iter=21370/40000, loss=0.2248, lr=0.005028, batch_cost=0.7905, reader_cost=0.0001 | ETA 04:05:27 2021-01-18 14:39:57 [INFO] [TRAIN] epoch=33, iter=21380/40000, loss=0.3570, lr=0.005025, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:06:10 2021-01-18 14:40:05 [INFO] [TRAIN] epoch=33, iter=21390/40000, loss=0.1353, lr=0.005023, batch_cost=0.7930, reader_cost=0.0001 | ETA 04:05:57 2021-01-18 14:40:12 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.2478, lr=0.005020, batch_cost=0.7957, reader_cost=0.0001 | ETA 04:06:40 2021-01-18 14:40:20 [INFO] [TRAIN] epoch=33, iter=21410/40000, loss=0.2299, lr=0.005018, batch_cost=0.7919, reader_cost=0.0001 | ETA 04:05:21 2021-01-18 14:40:28 [INFO] [TRAIN] epoch=33, iter=21420/40000, loss=0.3234, lr=0.005015, batch_cost=0.7902, reader_cost=0.0001 | ETA 04:04:42 2021-01-18 14:40:36 [INFO] [TRAIN] epoch=33, iter=21430/40000, loss=0.2834, lr=0.005013, batch_cost=0.7914, reader_cost=0.0001 | ETA 04:04:55 2021-01-18 14:40:44 [INFO] [TRAIN] epoch=33, iter=21440/40000, loss=0.1848, lr=0.005011, batch_cost=0.7918, reader_cost=0.0001 | ETA 04:04:56 2021-01-18 14:40:52 [INFO] [TRAIN] epoch=33, iter=21450/40000, loss=0.2798, lr=0.005008, batch_cost=0.7913, reader_cost=0.0001 | ETA 04:04:37 2021-01-18 14:41:00 [INFO] [TRAIN] epoch=33, iter=21460/40000, loss=0.2098, lr=0.005006, batch_cost=0.7907, reader_cost=0.0001 | ETA 04:04:20 2021-01-18 14:41:08 [INFO] [TRAIN] epoch=33, iter=21470/40000, loss=0.2901, lr=0.005003, batch_cost=0.7846, reader_cost=0.0001 | ETA 04:02:17 2021-01-18 14:41:16 [INFO] [TRAIN] epoch=33, iter=21480/40000, loss=0.1873, lr=0.005001, batch_cost=0.7852, reader_cost=0.0001 | ETA 04:02:22 2021-01-18 14:41:24 [INFO] [TRAIN] epoch=33, iter=21490/40000, loss=0.1493, lr=0.004998, batch_cost=0.7943, reader_cost=0.0001 | ETA 04:05:02 2021-01-18 14:41:32 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.1971, lr=0.004996, batch_cost=0.7897, reader_cost=0.0001 | ETA 04:03:28 2021-01-18 14:41:39 [INFO] [TRAIN] epoch=33, iter=21510/40000, loss=0.1544, lr=0.004994, batch_cost=0.7890, reader_cost=0.0001 | ETA 04:03:09 2021-01-18 14:41:47 [INFO] [TRAIN] epoch=33, iter=21520/40000, loss=0.1742, lr=0.004991, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:03:15 2021-01-18 14:41:55 [INFO] [TRAIN] epoch=33, iter=21530/40000, loss=0.2050, lr=0.004989, batch_cost=0.7872, reader_cost=0.0001 | ETA 04:02:19 2021-01-18 14:42:03 [INFO] [TRAIN] epoch=33, iter=21540/40000, loss=0.2680, lr=0.004986, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:03:19 2021-01-18 14:42:11 [INFO] [TRAIN] epoch=33, iter=21550/40000, loss=0.2952, lr=0.004984, batch_cost=0.7871, reader_cost=0.0001 | ETA 04:02:01 2021-01-18 14:42:19 [INFO] [TRAIN] epoch=33, iter=21560/40000, loss=0.2451, lr=0.004981, batch_cost=0.7924, reader_cost=0.0001 | ETA 04:03:31 2021-01-18 14:42:27 [INFO] [TRAIN] epoch=33, iter=21570/40000, loss=0.2021, lr=0.004979, batch_cost=0.7928, reader_cost=0.0001 | ETA 04:03:30 2021-01-18 14:42:35 [INFO] [TRAIN] epoch=33, iter=21580/40000, loss=0.2853, lr=0.004977, batch_cost=0.7827, reader_cost=0.0001 | ETA 04:00:17 2021-01-18 14:42:43 [INFO] [TRAIN] epoch=33, iter=21590/40000, loss=0.2064, lr=0.004974, batch_cost=0.7892, reader_cost=0.0001 | ETA 04:02:09 2021-01-18 14:42:50 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.1613, lr=0.004972, batch_cost=0.7909, reader_cost=0.0001 | ETA 04:02:32 2021-01-18 14:42:58 [INFO] [TRAIN] epoch=33, iter=21610/40000, loss=0.1892, lr=0.004969, batch_cost=0.7896, reader_cost=0.0001 | ETA 04:02:01 2021-01-18 14:43:06 [INFO] [TRAIN] epoch=33, iter=21620/40000, loss=0.2316, lr=0.004967, batch_cost=0.7939, reader_cost=0.0001 | ETA 04:03:12 2021-01-18 14:43:14 [INFO] [TRAIN] epoch=33, iter=21630/40000, loss=0.1547, lr=0.004964, batch_cost=0.7940, reader_cost=0.0001 | ETA 04:03:05 2021-01-18 14:43:22 [INFO] [TRAIN] epoch=33, iter=21640/40000, loss=0.2296, lr=0.004962, batch_cost=0.7898, reader_cost=0.0001 | ETA 04:01:40 2021-01-18 14:43:30 [INFO] [TRAIN] epoch=33, iter=21650/40000, loss=0.1673, lr=0.004960, batch_cost=0.7886, reader_cost=0.0001 | ETA 04:01:11 2021-01-18 14:43:38 [INFO] [TRAIN] epoch=33, iter=21660/40000, loss=0.4608, lr=0.004957, batch_cost=0.7885, reader_cost=0.0001 | ETA 04:01:00 2021-01-18 14:43:46 [INFO] [TRAIN] epoch=33, iter=21670/40000, loss=0.1810, lr=0.004955, batch_cost=0.7913, reader_cost=0.0001 | ETA 04:01:44 2021-01-18 14:43:54 [INFO] [TRAIN] epoch=33, iter=21680/40000, loss=0.2423, lr=0.004952, batch_cost=0.7922, reader_cost=0.0001 | ETA 04:01:53 2021-01-18 14:44:02 [INFO] [TRAIN] epoch=33, iter=21690/40000, loss=0.2693, lr=0.004950, batch_cost=0.7933, reader_cost=0.0001 | ETA 04:02:04 2021-01-18 14:44:10 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.2496, lr=0.004947, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:01:50 2021-01-18 14:44:17 [INFO] [TRAIN] epoch=33, iter=21710/40000, loss=0.1311, lr=0.004945, batch_cost=0.7904, reader_cost=0.0001 | ETA 04:00:55 2021-01-18 14:44:25 [INFO] [TRAIN] epoch=33, iter=21720/40000, loss=0.2078, lr=0.004942, batch_cost=0.7929, reader_cost=0.0001 | ETA 04:01:34 2021-01-18 14:44:33 [INFO] [TRAIN] epoch=33, iter=21730/40000, loss=0.2017, lr=0.004940, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:59:44 2021-01-18 14:44:41 [INFO] [TRAIN] epoch=33, iter=21740/40000, loss=0.2869, lr=0.004938, batch_cost=0.7937, reader_cost=0.0001 | ETA 04:01:32 2021-01-18 14:44:49 [INFO] [TRAIN] epoch=33, iter=21750/40000, loss=0.2605, lr=0.004935, batch_cost=0.7901, reader_cost=0.0001 | ETA 04:00:19 2021-01-18 14:44:57 [INFO] [TRAIN] epoch=33, iter=21760/40000, loss=0.2880, lr=0.004933, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:59:50 2021-01-18 14:45:05 [INFO] [TRAIN] epoch=33, iter=21770/40000, loss=0.2196, lr=0.004930, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:59:52 2021-01-18 14:45:13 [INFO] [TRAIN] epoch=33, iter=21780/40000, loss=0.3707, lr=0.004928, batch_cost=0.7936, reader_cost=0.0001 | ETA 04:00:59 2021-01-18 14:45:21 [INFO] [TRAIN] epoch=33, iter=21790/40000, loss=0.2849, lr=0.004925, batch_cost=0.7887, reader_cost=0.0001 | ETA 03:59:22 2021-01-18 14:45:29 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.1967, lr=0.004923, batch_cost=0.7913, reader_cost=0.0001 | ETA 04:00:02 2021-01-18 14:45:36 [INFO] [TRAIN] epoch=33, iter=21810/40000, loss=0.2639, lr=0.004921, batch_cost=0.7807, reader_cost=0.0001 | ETA 03:56:41 2021-01-18 14:45:44 [INFO] [TRAIN] epoch=34, iter=21820/40000, loss=0.2317, lr=0.004918, batch_cost=0.7888, reader_cost=0.0144 | ETA 03:59:00 2021-01-18 14:45:52 [INFO] [TRAIN] epoch=34, iter=21830/40000, loss=0.2140, lr=0.004916, batch_cost=0.7946, reader_cost=0.0001 | ETA 04:00:37 2021-01-18 14:46:00 [INFO] [TRAIN] epoch=34, iter=21840/40000, loss=0.1790, lr=0.004913, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:58:57 2021-01-18 14:46:08 [INFO] [TRAIN] epoch=34, iter=21850/40000, loss=0.2025, lr=0.004911, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:59:11 2021-01-18 14:46:16 [INFO] [TRAIN] epoch=34, iter=21860/40000, loss=0.2515, lr=0.004908, batch_cost=0.7939, reader_cost=0.0001 | ETA 04:00:01 2021-01-18 14:46:24 [INFO] [TRAIN] epoch=34, iter=21870/40000, loss=0.2003, lr=0.004906, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:59:37 2021-01-18 14:46:32 [INFO] [TRAIN] epoch=34, iter=21880/40000, loss=0.3291, lr=0.004904, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:58:16 2021-01-18 14:46:40 [INFO] [TRAIN] epoch=34, iter=21890/40000, loss=0.1099, lr=0.004901, batch_cost=0.7885, reader_cost=0.0001 | ETA 03:57:58 2021-01-18 14:46:48 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.2323, lr=0.004899, batch_cost=0.7914, reader_cost=0.0001 | ETA 03:58:44 2021-01-18 14:46:56 [INFO] [TRAIN] epoch=34, iter=21910/40000, loss=0.1711, lr=0.004896, batch_cost=0.7872, reader_cost=0.0001 | ETA 03:57:20 2021-01-18 14:47:03 [INFO] [TRAIN] epoch=34, iter=21920/40000, loss=0.1928, lr=0.004894, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:57:45 2021-01-18 14:47:11 [INFO] [TRAIN] epoch=34, iter=21930/40000, loss=0.2242, lr=0.004891, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:58:14 2021-01-18 14:47:19 [INFO] [TRAIN] epoch=34, iter=21940/40000, loss=0.2029, lr=0.004889, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:58:20 2021-01-18 14:47:27 [INFO] [TRAIN] epoch=34, iter=21950/40000, loss=0.1830, lr=0.004886, batch_cost=0.7918, reader_cost=0.0001 | ETA 03:58:11 2021-01-18 14:47:35 [INFO] [TRAIN] epoch=34, iter=21960/40000, loss=0.2553, lr=0.004884, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:57:43 2021-01-18 14:47:43 [INFO] [TRAIN] epoch=34, iter=21970/40000, loss=0.2759, lr=0.004882, batch_cost=0.7867, reader_cost=0.0001 | ETA 03:56:24 2021-01-18 14:47:51 [INFO] [TRAIN] epoch=34, iter=21980/40000, loss=0.2042, lr=0.004879, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:56:28 2021-01-18 14:47:59 [INFO] [TRAIN] epoch=34, iter=21990/40000, loss=0.2438, lr=0.004877, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:56:30 2021-01-18 14:48:07 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.1658, lr=0.004874, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:57:34 2021-01-18 14:48:07 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 14:48:34 [INFO] [EVAL] #Images=1449 mIoU=0.7833 Acc=0.9512 Kappa=0.8922 2021-01-18 14:48:34 [INFO] [EVAL] Class IoU: [0.9448 0.8808 0.4274 0.8826 0.6769 0.8435 0.9429 0.8529 0.9206 0.4419 0.8874 0.5686 0.8889 0.8982 0.8602 0.8694 0.6678 0.8329 0.5018 0.8996 0.7597] 2021-01-18 14:48:34 [INFO] [EVAL] Class Acc: [0.9674 0.9042 0.4438 0.9144 0.7633 0.9027 0.9774 0.8994 0.9524 0.6498 0.9275 0.9132 0.9454 0.9687 0.9272 0.9517 0.803 0.9409 0.8197 0.9327 0.8985] 2021-01-18 14:48:45 [INFO] [EVAL] The model with the best validation mIoU (0.7833) was saved at iter 22000. 2021-01-18 14:48:53 [INFO] [TRAIN] epoch=34, iter=22010/40000, loss=0.1510, lr=0.004872, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:57:26 2021-01-18 14:49:01 [INFO] [TRAIN] epoch=34, iter=22020/40000, loss=0.1820, lr=0.004869, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:56:10 2021-01-18 14:49:09 [INFO] [TRAIN] epoch=34, iter=22030/40000, loss=0.1555, lr=0.004867, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:55:49 2021-01-18 14:49:17 [INFO] [TRAIN] epoch=34, iter=22040/40000, loss=0.1491, lr=0.004865, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:56:26 2021-01-18 14:49:24 [INFO] [TRAIN] epoch=34, iter=22050/40000, loss=0.2144, lr=0.004862, batch_cost=0.7844, reader_cost=0.0001 | ETA 03:54:39 2021-01-18 14:49:32 [INFO] [TRAIN] epoch=34, iter=22060/40000, loss=0.2445, lr=0.004860, batch_cost=0.7910, reader_cost=0.0001 | ETA 03:56:31 2021-01-18 14:49:40 [INFO] [TRAIN] epoch=34, iter=22070/40000, loss=0.2379, lr=0.004857, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:56:28 2021-01-18 14:49:48 [INFO] [TRAIN] epoch=34, iter=22080/40000, loss=0.1787, lr=0.004855, batch_cost=0.7924, reader_cost=0.0001 | ETA 03:56:38 2021-01-18 14:49:56 [INFO] [TRAIN] epoch=34, iter=22090/40000, loss=0.1751, lr=0.004852, batch_cost=0.7884, reader_cost=0.0001 | ETA 03:55:20 2021-01-18 14:50:04 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.1604, lr=0.004850, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:55:37 2021-01-18 14:50:12 [INFO] [TRAIN] epoch=34, iter=22110/40000, loss=0.3032, lr=0.004847, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:56:38 2021-01-18 14:50:20 [INFO] [TRAIN] epoch=34, iter=22120/40000, loss=0.2247, lr=0.004845, batch_cost=0.7946, reader_cost=0.0001 | ETA 03:56:47 2021-01-18 14:50:28 [INFO] [TRAIN] epoch=34, iter=22130/40000, loss=0.2160, lr=0.004843, batch_cost=0.7933, reader_cost=0.0001 | ETA 03:56:15 2021-01-18 14:50:36 [INFO] [TRAIN] epoch=34, iter=22140/40000, loss=0.2475, lr=0.004840, batch_cost=0.7927, reader_cost=0.0001 | ETA 03:55:58 2021-01-18 14:50:44 [INFO] [TRAIN] epoch=34, iter=22150/40000, loss=0.3100, lr=0.004838, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:54:30 2021-01-18 14:50:52 [INFO] [TRAIN] epoch=34, iter=22160/40000, loss=0.1623, lr=0.004835, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:55:32 2021-01-18 14:50:59 [INFO] [TRAIN] epoch=34, iter=22170/40000, loss=0.1662, lr=0.004833, batch_cost=0.7870, reader_cost=0.0001 | ETA 03:53:51 2021-01-18 14:51:07 [INFO] [TRAIN] epoch=34, iter=22180/40000, loss=0.1952, lr=0.004830, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:55:23 2021-01-18 14:51:15 [INFO] [TRAIN] epoch=34, iter=22190/40000, loss=0.1559, lr=0.004828, batch_cost=0.7860, reader_cost=0.0001 | ETA 03:53:18 2021-01-18 14:51:23 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.1530, lr=0.004826, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:54:26 2021-01-18 14:51:31 [INFO] [TRAIN] epoch=34, iter=22210/40000, loss=0.2196, lr=0.004823, batch_cost=0.7916, reader_cost=0.0001 | ETA 03:54:42 2021-01-18 14:51:39 [INFO] [TRAIN] epoch=34, iter=22220/40000, loss=0.1837, lr=0.004821, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:54:32 2021-01-18 14:51:47 [INFO] [TRAIN] epoch=34, iter=22230/40000, loss=0.1430, lr=0.004818, batch_cost=0.7945, reader_cost=0.0001 | ETA 03:55:18 2021-01-18 14:51:55 [INFO] [TRAIN] epoch=34, iter=22240/40000, loss=0.1985, lr=0.004816, batch_cost=0.7959, reader_cost=0.0001 | ETA 03:55:34 2021-01-18 14:52:03 [INFO] [TRAIN] epoch=34, iter=22250/40000, loss=0.1971, lr=0.004813, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:54:27 2021-01-18 14:52:11 [INFO] [TRAIN] epoch=34, iter=22260/40000, loss=0.1718, lr=0.004811, batch_cost=0.7888, reader_cost=0.0001 | ETA 03:53:13 2021-01-18 14:52:19 [INFO] [TRAIN] epoch=34, iter=22270/40000, loss=0.1921, lr=0.004808, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:54:19 2021-01-18 14:52:26 [INFO] [TRAIN] epoch=34, iter=22280/40000, loss=0.1980, lr=0.004806, batch_cost=0.7893, reader_cost=0.0001 | ETA 03:53:07 2021-01-18 14:52:34 [INFO] [TRAIN] epoch=34, iter=22290/40000, loss=0.2136, lr=0.004804, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:54:04 2021-01-18 14:52:42 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.2036, lr=0.004801, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:53:15 2021-01-18 14:52:50 [INFO] [TRAIN] epoch=34, iter=22310/40000, loss=0.1694, lr=0.004799, batch_cost=0.7937, reader_cost=0.0001 | ETA 03:54:00 2021-01-18 14:52:58 [INFO] [TRAIN] epoch=34, iter=22320/40000, loss=0.1527, lr=0.004796, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:52:07 2021-01-18 14:53:06 [INFO] [TRAIN] epoch=34, iter=22330/40000, loss=0.1934, lr=0.004794, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:52:31 2021-01-18 14:53:14 [INFO] [TRAIN] epoch=34, iter=22340/40000, loss=0.2549, lr=0.004791, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:52:21 2021-01-18 14:53:22 [INFO] [TRAIN] epoch=34, iter=22350/40000, loss=0.2037, lr=0.004789, batch_cost=0.7853, reader_cost=0.0001 | ETA 03:51:00 2021-01-18 14:53:30 [INFO] [TRAIN] epoch=34, iter=22360/40000, loss=0.2047, lr=0.004786, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:52:27 2021-01-18 14:53:38 [INFO] [TRAIN] epoch=34, iter=22370/40000, loss=0.1530, lr=0.004784, batch_cost=0.7954, reader_cost=0.0001 | ETA 03:53:43 2021-01-18 14:53:46 [INFO] [TRAIN] epoch=34, iter=22380/40000, loss=0.1389, lr=0.004782, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:52:06 2021-01-18 14:53:53 [INFO] [TRAIN] epoch=34, iter=22390/40000, loss=0.1354, lr=0.004779, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:51:12 2021-01-18 14:54:01 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.2880, lr=0.004777, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:51:12 2021-01-18 14:54:09 [INFO] [TRAIN] epoch=34, iter=22410/40000, loss=0.2946, lr=0.004774, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:51:32 2021-01-18 14:54:17 [INFO] [TRAIN] epoch=34, iter=22420/40000, loss=0.2002, lr=0.004772, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:51:44 2021-01-18 14:54:25 [INFO] [TRAIN] epoch=34, iter=22430/40000, loss=0.1503, lr=0.004769, batch_cost=0.7946, reader_cost=0.0001 | ETA 03:52:41 2021-01-18 14:54:33 [INFO] [TRAIN] epoch=34, iter=22440/40000, loss=0.2200, lr=0.004767, batch_cost=0.7954, reader_cost=0.0001 | ETA 03:52:48 2021-01-18 14:54:41 [INFO] [TRAIN] epoch=34, iter=22450/40000, loss=0.2042, lr=0.004765, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:51:12 2021-01-18 14:54:49 [INFO] [TRAIN] epoch=34, iter=22460/40000, loss=0.2045, lr=0.004762, batch_cost=0.7892, reader_cost=0.0001 | ETA 03:50:42 2021-01-18 14:54:57 [INFO] [TRAIN] epoch=34, iter=22470/40000, loss=0.1264, lr=0.004760, batch_cost=0.7851, reader_cost=0.0001 | ETA 03:49:22 2021-01-18 14:55:04 [INFO] [TRAIN] epoch=35, iter=22480/40000, loss=0.2181, lr=0.004757, batch_cost=0.7785, reader_cost=0.0188 | ETA 03:47:18 2021-01-18 14:55:12 [INFO] [TRAIN] epoch=35, iter=22490/40000, loss=0.1616, lr=0.004755, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:49:53 2021-01-18 14:55:20 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.1720, lr=0.004752, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:50:07 2021-01-18 14:55:28 [INFO] [TRAIN] epoch=35, iter=22510/40000, loss=0.1864, lr=0.004750, batch_cost=0.7933, reader_cost=0.0001 | ETA 03:51:15 2021-01-18 14:55:36 [INFO] [TRAIN] epoch=35, iter=22520/40000, loss=0.1846, lr=0.004747, batch_cost=0.7924, reader_cost=0.0001 | ETA 03:50:50 2021-01-18 14:55:44 [INFO] [TRAIN] epoch=35, iter=22530/40000, loss=0.1632, lr=0.004745, batch_cost=0.7928, reader_cost=0.0001 | ETA 03:50:49 2021-01-18 14:55:52 [INFO] [TRAIN] epoch=35, iter=22540/40000, loss=0.3382, lr=0.004743, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:50:00 2021-01-18 14:56:00 [INFO] [TRAIN] epoch=35, iter=22550/40000, loss=0.1728, lr=0.004740, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:49:12 2021-01-18 14:56:08 [INFO] [TRAIN] epoch=35, iter=22560/40000, loss=0.1466, lr=0.004738, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:50:36 2021-01-18 14:56:16 [INFO] [TRAIN] epoch=35, iter=22570/40000, loss=0.1805, lr=0.004735, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:50:36 2021-01-18 14:56:24 [INFO] [TRAIN] epoch=35, iter=22580/40000, loss=0.1590, lr=0.004733, batch_cost=0.7875, reader_cost=0.0001 | ETA 03:48:38 2021-01-18 14:56:31 [INFO] [TRAIN] epoch=35, iter=22590/40000, loss=0.2110, lr=0.004730, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:49:39 2021-01-18 14:56:39 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.1712, lr=0.004728, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:50:01 2021-01-18 14:56:47 [INFO] [TRAIN] epoch=35, iter=22610/40000, loss=0.1869, lr=0.004725, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:49:13 2021-01-18 14:56:55 [INFO] [TRAIN] epoch=35, iter=22620/40000, loss=0.1623, lr=0.004723, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:48:53 2021-01-18 14:57:03 [INFO] [TRAIN] epoch=35, iter=22630/40000, loss=0.2326, lr=0.004721, batch_cost=0.7866, reader_cost=0.0001 | ETA 03:47:42 2021-01-18 14:57:11 [INFO] [TRAIN] epoch=35, iter=22640/40000, loss=0.2137, lr=0.004718, batch_cost=0.7924, reader_cost=0.0001 | ETA 03:49:16 2021-01-18 14:57:19 [INFO] [TRAIN] epoch=35, iter=22650/40000, loss=0.2503, lr=0.004716, batch_cost=0.7916, reader_cost=0.0001 | ETA 03:48:54 2021-01-18 14:57:27 [INFO] [TRAIN] epoch=35, iter=22660/40000, loss=0.1950, lr=0.004713, batch_cost=0.7856, reader_cost=0.0001 | ETA 03:47:02 2021-01-18 14:57:35 [INFO] [TRAIN] epoch=35, iter=22670/40000, loss=0.1562, lr=0.004711, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:49:13 2021-01-18 14:57:43 [INFO] [TRAIN] epoch=35, iter=22680/40000, loss=0.1966, lr=0.004708, batch_cost=0.7870, reader_cost=0.0001 | ETA 03:47:10 2021-01-18 14:57:50 [INFO] [TRAIN] epoch=35, iter=22690/40000, loss=0.2229, lr=0.004706, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:48:20 2021-01-18 14:57:58 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.2414, lr=0.004703, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:47:55 2021-01-18 14:58:06 [INFO] [TRAIN] epoch=35, iter=22710/40000, loss=0.2344, lr=0.004701, batch_cost=0.7846, reader_cost=0.0001 | ETA 03:46:05 2021-01-18 14:58:14 [INFO] [TRAIN] epoch=35, iter=22720/40000, loss=0.2087, lr=0.004698, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:48:26 2021-01-18 14:58:22 [INFO] [TRAIN] epoch=35, iter=22730/40000, loss=0.2164, lr=0.004696, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:48:01 2021-01-18 14:58:30 [INFO] [TRAIN] epoch=35, iter=22740/40000, loss=0.1651, lr=0.004694, batch_cost=0.7891, reader_cost=0.0001 | ETA 03:46:59 2021-01-18 14:58:38 [INFO] [TRAIN] epoch=35, iter=22750/40000, loss=0.1865, lr=0.004691, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:47:05 2021-01-18 14:58:46 [INFO] [TRAIN] epoch=35, iter=22760/40000, loss=0.1334, lr=0.004689, batch_cost=0.7901, reader_cost=0.0001 | ETA 03:47:00 2021-01-18 14:58:54 [INFO] [TRAIN] epoch=35, iter=22770/40000, loss=0.1716, lr=0.004686, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:47:24 2021-01-18 14:59:02 [INFO] [TRAIN] epoch=35, iter=22780/40000, loss=0.1923, lr=0.004684, batch_cost=0.7894, reader_cost=0.0001 | ETA 03:46:33 2021-01-18 14:59:09 [INFO] [TRAIN] epoch=35, iter=22790/40000, loss=0.1399, lr=0.004681, batch_cost=0.7884, reader_cost=0.0001 | ETA 03:46:08 2021-01-18 14:59:17 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.1856, lr=0.004679, batch_cost=0.7853, reader_cost=0.0001 | ETA 03:45:07 2021-01-18 14:59:25 [INFO] [TRAIN] epoch=35, iter=22810/40000, loss=0.1687, lr=0.004676, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:46:26 2021-01-18 14:59:33 [INFO] [TRAIN] epoch=35, iter=22820/40000, loss=0.2492, lr=0.004674, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:46:35 2021-01-18 14:59:41 [INFO] [TRAIN] epoch=35, iter=22830/40000, loss=0.2659, lr=0.004672, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:46:07 2021-01-18 14:59:49 [INFO] [TRAIN] epoch=35, iter=22840/40000, loss=0.2005, lr=0.004669, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:46:01 2021-01-18 14:59:57 [INFO] [TRAIN] epoch=35, iter=22850/40000, loss=0.1838, lr=0.004667, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:45:49 2021-01-18 15:00:05 [INFO] [TRAIN] epoch=35, iter=22860/40000, loss=0.2208, lr=0.004664, batch_cost=0.7847, reader_cost=0.0001 | ETA 03:44:10 2021-01-18 15:00:13 [INFO] [TRAIN] epoch=35, iter=22870/40000, loss=0.2498, lr=0.004662, batch_cost=0.7929, reader_cost=0.0001 | ETA 03:46:22 2021-01-18 15:00:21 [INFO] [TRAIN] epoch=35, iter=22880/40000, loss=0.1671, lr=0.004659, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:45:44 2021-01-18 15:00:28 [INFO] [TRAIN] epoch=35, iter=22890/40000, loss=0.1988, lr=0.004657, batch_cost=0.7940, reader_cost=0.0001 | ETA 03:46:24 2021-01-18 15:00:36 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.2918, lr=0.004654, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:45:51 2021-01-18 15:00:44 [INFO] [TRAIN] epoch=35, iter=22910/40000, loss=0.1789, lr=0.004652, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:45:14 2021-01-18 15:00:52 [INFO] [TRAIN] epoch=35, iter=22920/40000, loss=0.1700, lr=0.004650, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:45:03 2021-01-18 15:01:00 [INFO] [TRAIN] epoch=35, iter=22930/40000, loss=0.2271, lr=0.004647, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:45:40 2021-01-18 15:01:08 [INFO] [TRAIN] epoch=35, iter=22940/40000, loss=0.4403, lr=0.004645, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:44:36 2021-01-18 15:01:16 [INFO] [TRAIN] epoch=35, iter=22950/40000, loss=0.2448, lr=0.004642, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:44:39 2021-01-18 15:01:24 [INFO] [TRAIN] epoch=35, iter=22960/40000, loss=0.2286, lr=0.004640, batch_cost=0.7880, reader_cost=0.0001 | ETA 03:43:46 2021-01-18 15:01:32 [INFO] [TRAIN] epoch=35, iter=22970/40000, loss=0.2518, lr=0.004637, batch_cost=0.7946, reader_cost=0.0001 | ETA 03:45:32 2021-01-18 15:01:40 [INFO] [TRAIN] epoch=35, iter=22980/40000, loss=0.2445, lr=0.004635, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:44:31 2021-01-18 15:01:48 [INFO] [TRAIN] epoch=35, iter=22990/40000, loss=0.2234, lr=0.004632, batch_cost=0.7841, reader_cost=0.0001 | ETA 03:42:18 2021-01-18 15:01:55 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.2175, lr=0.004630, batch_cost=0.7943, reader_cost=0.0001 | ETA 03:45:03 2021-01-18 15:01:55 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 15:02:23 [INFO] [EVAL] #Images=1449 mIoU=0.7749 Acc=0.9508 Kappa=0.8914 2021-01-18 15:02:23 [INFO] [EVAL] Class IoU: [0.9465 0.8987 0.4223 0.9031 0.6934 0.6583 0.9057 0.8083 0.9175 0.4432 0.8998 0.5834 0.8832 0.8937 0.842 0.8714 0.6269 0.8737 0.5598 0.9086 0.7332] 2021-01-18 15:02:23 [INFO] [EVAL] Class Acc: [0.969 0.9318 0.4423 0.9493 0.7698 0.9538 0.9523 0.8596 0.9734 0.6854 0.965 0.9185 0.9186 0.9493 0.9128 0.9575 0.8187 0.9121 0.7151 0.9452 0.858 ] 2021-01-18 15:02:30 [INFO] [EVAL] The model with the best validation mIoU (0.7833) was saved at iter 22000. 2021-01-18 15:02:37 [INFO] [TRAIN] epoch=35, iter=23010/40000, loss=0.1805, lr=0.004627, batch_cost=0.7781, reader_cost=0.0001 | ETA 03:40:19 2021-01-18 15:02:45 [INFO] [TRAIN] epoch=35, iter=23020/40000, loss=0.2515, lr=0.004625, batch_cost=0.7883, reader_cost=0.0001 | ETA 03:43:04 2021-01-18 15:02:53 [INFO] [TRAIN] epoch=35, iter=23030/40000, loss=0.1640, lr=0.004623, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:44:08 2021-01-18 15:03:01 [INFO] [TRAIN] epoch=35, iter=23040/40000, loss=0.1833, lr=0.004620, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:43:34 2021-01-18 15:03:09 [INFO] [TRAIN] epoch=35, iter=23050/40000, loss=0.2022, lr=0.004618, batch_cost=0.7819, reader_cost=0.0001 | ETA 03:40:52 2021-01-18 15:03:17 [INFO] [TRAIN] epoch=35, iter=23060/40000, loss=0.1596, lr=0.004615, batch_cost=0.7872, reader_cost=0.0001 | ETA 03:42:14 2021-01-18 15:03:25 [INFO] [TRAIN] epoch=35, iter=23070/40000, loss=0.1832, lr=0.004613, batch_cost=0.7891, reader_cost=0.0001 | ETA 03:42:40 2021-01-18 15:03:33 [INFO] [TRAIN] epoch=35, iter=23080/40000, loss=0.2814, lr=0.004610, batch_cost=0.7942, reader_cost=0.0001 | ETA 03:43:58 2021-01-18 15:03:41 [INFO] [TRAIN] epoch=35, iter=23090/40000, loss=0.1416, lr=0.004608, batch_cost=0.7923, reader_cost=0.0001 | ETA 03:43:18 2021-01-18 15:03:48 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.1619, lr=0.004605, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:43:14 2021-01-18 15:03:56 [INFO] [TRAIN] epoch=35, iter=23110/40000, loss=0.2298, lr=0.004603, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:42:30 2021-01-18 15:04:04 [INFO] [TRAIN] epoch=35, iter=23120/40000, loss=0.1814, lr=0.004600, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:41:44 2021-01-18 15:04:12 [INFO] [TRAIN] epoch=35, iter=23130/40000, loss=0.1417, lr=0.004598, batch_cost=0.7887, reader_cost=0.0001 | ETA 03:41:44 2021-01-18 15:04:20 [INFO] [TRAIN] epoch=36, iter=23140/40000, loss=0.3128, lr=0.004596, batch_cost=0.7777, reader_cost=0.0165 | ETA 03:38:32 2021-01-18 15:04:28 [INFO] [TRAIN] epoch=36, iter=23150/40000, loss=0.1757, lr=0.004593, batch_cost=0.7884, reader_cost=0.0001 | ETA 03:41:24 2021-01-18 15:04:36 [INFO] [TRAIN] epoch=36, iter=23160/40000, loss=0.1894, lr=0.004591, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:42:28 2021-01-18 15:04:44 [INFO] [TRAIN] epoch=36, iter=23170/40000, loss=0.2403, lr=0.004588, batch_cost=0.7878, reader_cost=0.0001 | ETA 03:40:59 2021-01-18 15:04:51 [INFO] [TRAIN] epoch=36, iter=23180/40000, loss=0.1976, lr=0.004586, batch_cost=0.7810, reader_cost=0.0001 | ETA 03:38:56 2021-01-18 15:04:59 [INFO] [TRAIN] epoch=36, iter=23190/40000, loss=0.1996, lr=0.004583, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:41:41 2021-01-18 15:05:07 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.2326, lr=0.004581, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:41:33 2021-01-18 15:05:15 [INFO] [TRAIN] epoch=36, iter=23210/40000, loss=0.1331, lr=0.004578, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:41:39 2021-01-18 15:05:23 [INFO] [TRAIN] epoch=36, iter=23220/40000, loss=0.1778, lr=0.004576, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:41:27 2021-01-18 15:05:31 [INFO] [TRAIN] epoch=36, iter=23230/40000, loss=0.1815, lr=0.004574, batch_cost=0.7894, reader_cost=0.0001 | ETA 03:40:38 2021-01-18 15:05:39 [INFO] [TRAIN] epoch=36, iter=23240/40000, loss=0.2160, lr=0.004571, batch_cost=0.7970, reader_cost=0.0001 | ETA 03:42:37 2021-01-18 15:05:47 [INFO] [TRAIN] epoch=36, iter=23250/40000, loss=0.2243, lr=0.004569, batch_cost=0.7931, reader_cost=0.0001 | ETA 03:41:25 2021-01-18 15:05:55 [INFO] [TRAIN] epoch=36, iter=23260/40000, loss=0.1672, lr=0.004566, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:39:41 2021-01-18 15:06:03 [INFO] [TRAIN] epoch=36, iter=23270/40000, loss=0.1686, lr=0.004564, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:40:58 2021-01-18 15:06:11 [INFO] [TRAIN] epoch=36, iter=23280/40000, loss=0.1970, lr=0.004561, batch_cost=0.7858, reader_cost=0.0001 | ETA 03:38:58 2021-01-18 15:06:18 [INFO] [TRAIN] epoch=36, iter=23290/40000, loss=0.2382, lr=0.004559, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:40:42 2021-01-18 15:06:26 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.1635, lr=0.004556, batch_cost=0.7937, reader_cost=0.0001 | ETA 03:40:54 2021-01-18 15:06:34 [INFO] [TRAIN] epoch=36, iter=23310/40000, loss=0.1523, lr=0.004554, batch_cost=0.7960, reader_cost=0.0001 | ETA 03:41:25 2021-01-18 15:06:42 [INFO] [TRAIN] epoch=36, iter=23320/40000, loss=0.2059, lr=0.004551, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:40:41 2021-01-18 15:06:50 [INFO] [TRAIN] epoch=36, iter=23330/40000, loss=0.1840, lr=0.004549, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:40:32 2021-01-18 15:06:58 [INFO] [TRAIN] epoch=36, iter=23340/40000, loss=0.1786, lr=0.004546, batch_cost=0.7883, reader_cost=0.0001 | ETA 03:38:53 2021-01-18 15:07:06 [INFO] [TRAIN] epoch=36, iter=23350/40000, loss=0.2309, lr=0.004544, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:39:14 2021-01-18 15:07:14 [INFO] [TRAIN] epoch=36, iter=23360/40000, loss=0.2421, lr=0.004542, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:39:11 2021-01-18 15:07:22 [INFO] [TRAIN] epoch=36, iter=23370/40000, loss=0.2481, lr=0.004539, batch_cost=0.7910, reader_cost=0.0001 | ETA 03:39:13 2021-01-18 15:07:30 [INFO] [TRAIN] epoch=36, iter=23380/40000, loss=0.1648, lr=0.004537, batch_cost=0.7917, reader_cost=0.0001 | ETA 03:39:18 2021-01-18 15:07:38 [INFO] [TRAIN] epoch=36, iter=23390/40000, loss=0.2828, lr=0.004534, batch_cost=0.7961, reader_cost=0.0001 | ETA 03:40:23 2021-01-18 15:07:46 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.1707, lr=0.004532, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:39:31 2021-01-18 15:07:54 [INFO] [TRAIN] epoch=36, iter=23410/40000, loss=0.2075, lr=0.004529, batch_cost=0.7910, reader_cost=0.0001 | ETA 03:38:42 2021-01-18 15:08:01 [INFO] [TRAIN] epoch=36, iter=23420/40000, loss=0.2215, lr=0.004527, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:38:20 2021-01-18 15:08:09 [INFO] [TRAIN] epoch=36, iter=23430/40000, loss=0.1841, lr=0.004524, batch_cost=0.7969, reader_cost=0.0001 | ETA 03:40:04 2021-01-18 15:08:17 [INFO] [TRAIN] epoch=36, iter=23440/40000, loss=0.2459, lr=0.004522, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:38:01 2021-01-18 15:08:25 [INFO] [TRAIN] epoch=36, iter=23450/40000, loss=0.1590, lr=0.004519, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:38:18 2021-01-18 15:08:33 [INFO] [TRAIN] epoch=36, iter=23460/40000, loss=0.1589, lr=0.004517, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:37:46 2021-01-18 15:08:41 [INFO] [TRAIN] epoch=36, iter=23470/40000, loss=0.1297, lr=0.004515, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:38:20 2021-01-18 15:08:49 [INFO] [TRAIN] epoch=36, iter=23480/40000, loss=0.1601, lr=0.004512, batch_cost=0.7897, reader_cost=0.0001 | ETA 03:37:26 2021-01-18 15:08:57 [INFO] [TRAIN] epoch=36, iter=23490/40000, loss=0.1669, lr=0.004510, batch_cost=0.7918, reader_cost=0.0001 | ETA 03:37:53 2021-01-18 15:09:05 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.1674, lr=0.004507, batch_cost=0.7914, reader_cost=0.0001 | ETA 03:37:38 2021-01-18 15:09:13 [INFO] [TRAIN] epoch=36, iter=23510/40000, loss=0.1962, lr=0.004505, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:37:27 2021-01-18 15:09:21 [INFO] [TRAIN] epoch=36, iter=23520/40000, loss=0.2209, lr=0.004502, batch_cost=0.7894, reader_cost=0.0001 | ETA 03:36:48 2021-01-18 15:09:29 [INFO] [TRAIN] epoch=36, iter=23530/40000, loss=0.1182, lr=0.004500, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:37:25 2021-01-18 15:09:36 [INFO] [TRAIN] epoch=36, iter=23540/40000, loss=0.1993, lr=0.004497, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:36:08 2021-01-18 15:09:44 [INFO] [TRAIN] epoch=36, iter=23550/40000, loss=0.1940, lr=0.004495, batch_cost=0.7945, reader_cost=0.0001 | ETA 03:37:49 2021-01-18 15:09:52 [INFO] [TRAIN] epoch=36, iter=23560/40000, loss=0.1710, lr=0.004492, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:36:35 2021-01-18 15:10:00 [INFO] [TRAIN] epoch=36, iter=23570/40000, loss=0.1746, lr=0.004490, batch_cost=0.7878, reader_cost=0.0001 | ETA 03:35:43 2021-01-18 15:10:08 [INFO] [TRAIN] epoch=36, iter=23580/40000, loss=0.1647, lr=0.004488, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:37:01 2021-01-18 15:10:16 [INFO] [TRAIN] epoch=36, iter=23590/40000, loss=0.2264, lr=0.004485, batch_cost=0.7876, reader_cost=0.0001 | ETA 03:35:24 2021-01-18 15:10:24 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.2030, lr=0.004483, batch_cost=0.7892, reader_cost=0.0001 | ETA 03:35:42 2021-01-18 15:10:32 [INFO] [TRAIN] epoch=36, iter=23610/40000, loss=0.2158, lr=0.004480, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:35:39 2021-01-18 15:10:40 [INFO] [TRAIN] epoch=36, iter=23620/40000, loss=0.2089, lr=0.004478, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:36:35 2021-01-18 15:10:48 [INFO] [TRAIN] epoch=36, iter=23630/40000, loss=0.1652, lr=0.004475, batch_cost=0.7880, reader_cost=0.0001 | ETA 03:34:59 2021-01-18 15:10:55 [INFO] [TRAIN] epoch=36, iter=23640/40000, loss=0.1431, lr=0.004473, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:35:39 2021-01-18 15:11:03 [INFO] [TRAIN] epoch=36, iter=23650/40000, loss=0.1547, lr=0.004470, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:35:08 2021-01-18 15:11:11 [INFO] [TRAIN] epoch=36, iter=23660/40000, loss=0.1672, lr=0.004468, batch_cost=0.7841, reader_cost=0.0001 | ETA 03:33:31 2021-01-18 15:11:19 [INFO] [TRAIN] epoch=36, iter=23670/40000, loss=0.1712, lr=0.004465, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:35:06 2021-01-18 15:11:27 [INFO] [TRAIN] epoch=36, iter=23680/40000, loss=0.2255, lr=0.004463, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:34:51 2021-01-18 15:11:35 [INFO] [TRAIN] epoch=36, iter=23690/40000, loss=0.2198, lr=0.004460, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:34:29 2021-01-18 15:11:43 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.2243, lr=0.004458, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:33:59 2021-01-18 15:11:51 [INFO] [TRAIN] epoch=36, iter=23710/40000, loss=0.1817, lr=0.004456, batch_cost=0.7866, reader_cost=0.0001 | ETA 03:33:33 2021-01-18 15:11:59 [INFO] [TRAIN] epoch=36, iter=23720/40000, loss=0.2324, lr=0.004453, batch_cost=0.7966, reader_cost=0.0001 | ETA 03:36:08 2021-01-18 15:12:07 [INFO] [TRAIN] epoch=36, iter=23730/40000, loss=0.2378, lr=0.004451, batch_cost=0.7941, reader_cost=0.0001 | ETA 03:35:19 2021-01-18 15:12:15 [INFO] [TRAIN] epoch=36, iter=23740/40000, loss=0.2210, lr=0.004448, batch_cost=0.7978, reader_cost=0.0001 | ETA 03:36:12 2021-01-18 15:12:22 [INFO] [TRAIN] epoch=36, iter=23750/40000, loss=0.2130, lr=0.004446, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:33:27 2021-01-18 15:12:30 [INFO] [TRAIN] epoch=36, iter=23760/40000, loss=0.2780, lr=0.004443, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:33:14 2021-01-18 15:12:38 [INFO] [TRAIN] epoch=36, iter=23770/40000, loss=0.2036, lr=0.004441, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:33:41 2021-01-18 15:12:46 [INFO] [TRAIN] epoch=36, iter=23780/40000, loss=0.1935, lr=0.004438, batch_cost=0.7891, reader_cost=0.0001 | ETA 03:33:19 2021-01-18 15:12:54 [INFO] [TRAIN] epoch=36, iter=23790/40000, loss=0.1999, lr=0.004436, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:34:23 2021-01-18 15:13:02 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.1550, lr=0.004433, batch_cost=0.7773, reader_cost=0.0133 | ETA 03:29:51 2021-01-18 15:13:10 [INFO] [TRAIN] epoch=37, iter=23810/40000, loss=0.2330, lr=0.004431, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:32:38 2021-01-18 15:13:18 [INFO] [TRAIN] epoch=37, iter=23820/40000, loss=0.1709, lr=0.004428, batch_cost=0.7946, reader_cost=0.0001 | ETA 03:34:16 2021-01-18 15:13:26 [INFO] [TRAIN] epoch=37, iter=23830/40000, loss=0.1481, lr=0.004426, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:32:53 2021-01-18 15:13:33 [INFO] [TRAIN] epoch=37, iter=23840/40000, loss=0.2313, lr=0.004424, batch_cost=0.7884, reader_cost=0.0001 | ETA 03:32:20 2021-01-18 15:13:41 [INFO] [TRAIN] epoch=37, iter=23850/40000, loss=0.1371, lr=0.004421, batch_cost=0.7901, reader_cost=0.0001 | ETA 03:32:39 2021-01-18 15:13:49 [INFO] [TRAIN] epoch=37, iter=23860/40000, loss=0.2483, lr=0.004419, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:32:22 2021-01-18 15:13:57 [INFO] [TRAIN] epoch=37, iter=23870/40000, loss=0.2185, lr=0.004416, batch_cost=0.7950, reader_cost=0.0001 | ETA 03:33:43 2021-01-18 15:14:05 [INFO] [TRAIN] epoch=37, iter=23880/40000, loss=0.1517, lr=0.004414, batch_cost=0.7924, reader_cost=0.0001 | ETA 03:32:53 2021-01-18 15:14:13 [INFO] [TRAIN] epoch=37, iter=23890/40000, loss=0.2049, lr=0.004411, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:33:08 2021-01-18 15:14:21 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.1679, lr=0.004409, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:32:40 2021-01-18 15:14:29 [INFO] [TRAIN] epoch=37, iter=23910/40000, loss=0.2017, lr=0.004406, batch_cost=0.7969, reader_cost=0.0001 | ETA 03:33:42 2021-01-18 15:14:37 [INFO] [TRAIN] epoch=37, iter=23920/40000, loss=0.1442, lr=0.004404, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:31:39 2021-01-18 15:14:45 [INFO] [TRAIN] epoch=37, iter=23930/40000, loss=0.1966, lr=0.004401, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:30:58 2021-01-18 15:14:53 [INFO] [TRAIN] epoch=37, iter=23940/40000, loss=0.1693, lr=0.004399, batch_cost=0.7914, reader_cost=0.0001 | ETA 03:31:49 2021-01-18 15:15:01 [INFO] [TRAIN] epoch=37, iter=23950/40000, loss=0.1678, lr=0.004396, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:31:26 2021-01-18 15:15:08 [INFO] [TRAIN] epoch=37, iter=23960/40000, loss=0.1856, lr=0.004394, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:30:49 2021-01-18 15:15:16 [INFO] [TRAIN] epoch=37, iter=23970/40000, loss=0.1391, lr=0.004391, batch_cost=0.7883, reader_cost=0.0001 | ETA 03:30:37 2021-01-18 15:15:24 [INFO] [TRAIN] epoch=37, iter=23980/40000, loss=0.1550, lr=0.004389, batch_cost=0.7871, reader_cost=0.0001 | ETA 03:30:09 2021-01-18 15:15:32 [INFO] [TRAIN] epoch=37, iter=23990/40000, loss=0.1871, lr=0.004387, batch_cost=0.7810, reader_cost=0.0001 | ETA 03:28:24 2021-01-18 15:15:40 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.1720, lr=0.004384, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:31:19 2021-01-18 15:15:40 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 15:16:08 [INFO] [EVAL] #Images=1449 mIoU=0.7919 Acc=0.9529 Kappa=0.8961 2021-01-18 15:16:08 [INFO] [EVAL] Class IoU: [0.9475 0.859 0.4146 0.9006 0.7319 0.8338 0.9362 0.8756 0.915 0.4298 0.9129 0.6136 0.8707 0.8927 0.8792 0.8686 0.6244 0.9012 0.5485 0.8969 0.7763] 2021-01-18 15:16:08 [INFO] [EVAL] Class Acc: [0.9701 0.8776 0.4219 0.9513 0.7958 0.9026 0.9852 0.9369 0.9485 0.633 0.962 0.9009 0.9368 0.9465 0.9206 0.9474 0.781 0.9484 0.7675 0.9676 0.9443] 2021-01-18 15:16:18 [INFO] [EVAL] The model with the best validation mIoU (0.7919) was saved at iter 24000. 2021-01-18 15:16:26 [INFO] [TRAIN] epoch=37, iter=24010/40000, loss=0.1761, lr=0.004382, batch_cost=0.7868, reader_cost=0.0001 | ETA 03:29:40 2021-01-18 15:16:34 [INFO] [TRAIN] epoch=37, iter=24020/40000, loss=0.1482, lr=0.004379, batch_cost=0.7883, reader_cost=0.0001 | ETA 03:29:56 2021-01-18 15:16:42 [INFO] [TRAIN] epoch=37, iter=24030/40000, loss=0.1988, lr=0.004377, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:30:26 2021-01-18 15:16:50 [INFO] [TRAIN] epoch=37, iter=24040/40000, loss=0.1930, lr=0.004374, batch_cost=0.7924, reader_cost=0.0001 | ETA 03:30:47 2021-01-18 15:16:58 [INFO] [TRAIN] epoch=37, iter=24050/40000, loss=0.1952, lr=0.004372, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:29:44 2021-01-18 15:17:06 [INFO] [TRAIN] epoch=37, iter=24060/40000, loss=0.1606, lr=0.004369, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:29:23 2021-01-18 15:17:14 [INFO] [TRAIN] epoch=37, iter=24070/40000, loss=0.1967, lr=0.004367, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:30:20 2021-01-18 15:17:22 [INFO] [TRAIN] epoch=37, iter=24080/40000, loss=0.1417, lr=0.004364, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:30:18 2021-01-18 15:17:29 [INFO] [TRAIN] epoch=37, iter=24090/40000, loss=0.2998, lr=0.004362, batch_cost=0.7869, reader_cost=0.0001 | ETA 03:28:40 2021-01-18 15:17:37 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.2204, lr=0.004359, batch_cost=0.7944, reader_cost=0.0001 | ETA 03:30:30 2021-01-18 15:17:45 [INFO] [TRAIN] epoch=37, iter=24110/40000, loss=0.1738, lr=0.004357, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:29:24 2021-01-18 15:17:53 [INFO] [TRAIN] epoch=37, iter=24120/40000, loss=0.1902, lr=0.004354, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:28:36 2021-01-18 15:18:01 [INFO] [TRAIN] epoch=37, iter=24130/40000, loss=0.2211, lr=0.004352, batch_cost=0.7884, reader_cost=0.0001 | ETA 03:28:31 2021-01-18 15:18:09 [INFO] [TRAIN] epoch=37, iter=24140/40000, loss=0.1891, lr=0.004350, batch_cost=0.7867, reader_cost=0.0001 | ETA 03:27:56 2021-01-18 15:18:17 [INFO] [TRAIN] epoch=37, iter=24150/40000, loss=0.1736, lr=0.004347, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:29:32 2021-01-18 15:18:25 [INFO] [TRAIN] epoch=37, iter=24160/40000, loss=0.1199, lr=0.004345, batch_cost=0.7831, reader_cost=0.0001 | ETA 03:26:44 2021-01-18 15:18:33 [INFO] [TRAIN] epoch=37, iter=24170/40000, loss=0.2070, lr=0.004342, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:29:19 2021-01-18 15:18:41 [INFO] [TRAIN] epoch=37, iter=24180/40000, loss=0.2551, lr=0.004340, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:28:58 2021-01-18 15:18:48 [INFO] [TRAIN] epoch=37, iter=24190/40000, loss=0.3022, lr=0.004337, batch_cost=0.7861, reader_cost=0.0001 | ETA 03:27:08 2021-01-18 15:18:56 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.2143, lr=0.004335, batch_cost=0.7947, reader_cost=0.0001 | ETA 03:29:16 2021-01-18 15:19:04 [INFO] [TRAIN] epoch=37, iter=24210/40000, loss=0.1251, lr=0.004332, batch_cost=0.7868, reader_cost=0.0001 | ETA 03:27:02 2021-01-18 15:19:12 [INFO] [TRAIN] epoch=37, iter=24220/40000, loss=0.2267, lr=0.004330, batch_cost=0.7926, reader_cost=0.0001 | ETA 03:28:27 2021-01-18 15:19:20 [INFO] [TRAIN] epoch=37, iter=24230/40000, loss=0.1835, lr=0.004327, batch_cost=0.7954, reader_cost=0.0001 | ETA 03:29:03 2021-01-18 15:19:28 [INFO] [TRAIN] epoch=37, iter=24240/40000, loss=0.2108, lr=0.004325, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:27:47 2021-01-18 15:19:36 [INFO] [TRAIN] epoch=37, iter=24250/40000, loss=0.1524, lr=0.004322, batch_cost=0.7857, reader_cost=0.0001 | ETA 03:26:14 2021-01-18 15:19:44 [INFO] [TRAIN] epoch=37, iter=24260/40000, loss=0.1482, lr=0.004320, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:26:53 2021-01-18 15:19:52 [INFO] [TRAIN] epoch=37, iter=24270/40000, loss=0.1575, lr=0.004317, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:27:24 2021-01-18 15:20:00 [INFO] [TRAIN] epoch=37, iter=24280/40000, loss=0.2029, lr=0.004315, batch_cost=0.7952, reader_cost=0.0001 | ETA 03:28:20 2021-01-18 15:20:08 [INFO] [TRAIN] epoch=37, iter=24290/40000, loss=0.1729, lr=0.004313, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:27:05 2021-01-18 15:20:15 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.1984, lr=0.004310, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:26:50 2021-01-18 15:20:23 [INFO] [TRAIN] epoch=37, iter=24310/40000, loss=0.2004, lr=0.004308, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:26:12 2021-01-18 15:20:31 [INFO] [TRAIN] epoch=37, iter=24320/40000, loss=0.1436, lr=0.004305, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:27:23 2021-01-18 15:20:39 [INFO] [TRAIN] epoch=37, iter=24330/40000, loss=0.2834, lr=0.004303, batch_cost=0.7865, reader_cost=0.0001 | ETA 03:25:23 2021-01-18 15:20:47 [INFO] [TRAIN] epoch=37, iter=24340/40000, loss=0.2617, lr=0.004300, batch_cost=0.7883, reader_cost=0.0001 | ETA 03:25:44 2021-01-18 15:20:55 [INFO] [TRAIN] epoch=37, iter=24350/40000, loss=0.1469, lr=0.004298, batch_cost=0.7865, reader_cost=0.0001 | ETA 03:25:08 2021-01-18 15:21:03 [INFO] [TRAIN] epoch=37, iter=24360/40000, loss=0.1310, lr=0.004295, batch_cost=0.7971, reader_cost=0.0001 | ETA 03:27:47 2021-01-18 15:21:11 [INFO] [TRAIN] epoch=37, iter=24370/40000, loss=0.1497, lr=0.004293, batch_cost=0.7848, reader_cost=0.0001 | ETA 03:24:26 2021-01-18 15:21:19 [INFO] [TRAIN] epoch=37, iter=24380/40000, loss=0.2610, lr=0.004290, batch_cost=0.7873, reader_cost=0.0001 | ETA 03:24:56 2021-01-18 15:21:26 [INFO] [TRAIN] epoch=37, iter=24390/40000, loss=0.2037, lr=0.004288, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:25:16 2021-01-18 15:21:34 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.1694, lr=0.004285, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:25:24 2021-01-18 15:21:42 [INFO] [TRAIN] epoch=37, iter=24410/40000, loss=0.1628, lr=0.004283, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:25:34 2021-01-18 15:21:50 [INFO] [TRAIN] epoch=37, iter=24420/40000, loss=0.1455, lr=0.004280, batch_cost=0.7935, reader_cost=0.0001 | ETA 03:26:02 2021-01-18 15:21:58 [INFO] [TRAIN] epoch=37, iter=24430/40000, loss=0.1622, lr=0.004278, batch_cost=0.7916, reader_cost=0.0001 | ETA 03:25:25 2021-01-18 15:22:06 [INFO] [TRAIN] epoch=37, iter=24440/40000, loss=0.1938, lr=0.004275, batch_cost=0.7876, reader_cost=0.0001 | ETA 03:24:15 2021-01-18 15:22:14 [INFO] [TRAIN] epoch=37, iter=24450/40000, loss=0.1897, lr=0.004273, batch_cost=0.7864, reader_cost=0.0001 | ETA 03:23:48 2021-01-18 15:22:22 [INFO] [TRAIN] epoch=38, iter=24460/40000, loss=0.2484, lr=0.004270, batch_cost=0.7754, reader_cost=0.0135 | ETA 03:20:49 2021-01-18 15:22:29 [INFO] [TRAIN] epoch=38, iter=24470/40000, loss=0.1094, lr=0.004268, batch_cost=0.7872, reader_cost=0.0001 | ETA 03:23:44 2021-01-18 15:22:37 [INFO] [TRAIN] epoch=38, iter=24480/40000, loss=0.1374, lr=0.004266, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:24:19 2021-01-18 15:22:45 [INFO] [TRAIN] epoch=38, iter=24490/40000, loss=0.2142, lr=0.004263, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:23:51 2021-01-18 15:22:53 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.1662, lr=0.004261, batch_cost=0.7871, reader_cost=0.0001 | ETA 03:23:19 2021-01-18 15:23:01 [INFO] [TRAIN] epoch=38, iter=24510/40000, loss=0.1663, lr=0.004258, batch_cost=0.7894, reader_cost=0.0001 | ETA 03:23:47 2021-01-18 15:23:09 [INFO] [TRAIN] epoch=38, iter=24520/40000, loss=0.1526, lr=0.004256, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:24:45 2021-01-18 15:23:17 [INFO] [TRAIN] epoch=38, iter=24530/40000, loss=0.1754, lr=0.004253, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:24:00 2021-01-18 15:23:25 [INFO] [TRAIN] epoch=38, iter=24540/40000, loss=0.2189, lr=0.004251, batch_cost=0.7875, reader_cost=0.0001 | ETA 03:22:54 2021-01-18 15:23:33 [INFO] [TRAIN] epoch=38, iter=24550/40000, loss=0.1925, lr=0.004248, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:23:25 2021-01-18 15:23:41 [INFO] [TRAIN] epoch=38, iter=24560/40000, loss=0.1773, lr=0.004246, batch_cost=0.7917, reader_cost=0.0001 | ETA 03:23:43 2021-01-18 15:23:48 [INFO] [TRAIN] epoch=38, iter=24570/40000, loss=0.2177, lr=0.004243, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:23:21 2021-01-18 15:23:56 [INFO] [TRAIN] epoch=38, iter=24580/40000, loss=0.1407, lr=0.004241, batch_cost=0.7916, reader_cost=0.0001 | ETA 03:23:26 2021-01-18 15:24:04 [INFO] [TRAIN] epoch=38, iter=24590/40000, loss=0.2131, lr=0.004238, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:23:06 2021-01-18 15:24:12 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.1567, lr=0.004236, batch_cost=0.7927, reader_cost=0.0001 | ETA 03:23:28 2021-01-18 15:24:20 [INFO] [TRAIN] epoch=38, iter=24610/40000, loss=0.1794, lr=0.004233, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:23:30 2021-01-18 15:24:28 [INFO] [TRAIN] epoch=38, iter=24620/40000, loss=0.2261, lr=0.004231, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:22:48 2021-01-18 15:24:36 [INFO] [TRAIN] epoch=38, iter=24630/40000, loss=0.1450, lr=0.004228, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:22:26 2021-01-18 15:24:44 [INFO] [TRAIN] epoch=38, iter=24640/40000, loss=0.1928, lr=0.004226, batch_cost=0.7866, reader_cost=0.0001 | ETA 03:21:21 2021-01-18 15:24:52 [INFO] [TRAIN] epoch=38, iter=24650/40000, loss=0.1813, lr=0.004223, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:22:15 2021-01-18 15:25:00 [INFO] [TRAIN] epoch=38, iter=24660/40000, loss=0.1830, lr=0.004221, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:21:42 2021-01-18 15:25:07 [INFO] [TRAIN] epoch=38, iter=24670/40000, loss=0.2387, lr=0.004219, batch_cost=0.7857, reader_cost=0.0001 | ETA 03:20:45 2021-01-18 15:25:15 [INFO] [TRAIN] epoch=38, iter=24680/40000, loss=0.2839, lr=0.004216, batch_cost=0.7853, reader_cost=0.0001 | ETA 03:20:30 2021-01-18 15:25:23 [INFO] [TRAIN] epoch=38, iter=24690/40000, loss=0.1460, lr=0.004214, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:21:57 2021-01-18 15:25:31 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.1537, lr=0.004211, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:21:40 2021-01-18 15:25:39 [INFO] [TRAIN] epoch=38, iter=24710/40000, loss=0.1877, lr=0.004209, batch_cost=0.7892, reader_cost=0.0001 | ETA 03:21:06 2021-01-18 15:25:47 [INFO] [TRAIN] epoch=38, iter=24720/40000, loss=0.3097, lr=0.004206, batch_cost=0.7946, reader_cost=0.0001 | ETA 03:22:21 2021-01-18 15:25:55 [INFO] [TRAIN] epoch=38, iter=24730/40000, loss=0.2467, lr=0.004204, batch_cost=0.7872, reader_cost=0.0001 | ETA 03:20:20 2021-01-18 15:26:03 [INFO] [TRAIN] epoch=38, iter=24740/40000, loss=0.1598, lr=0.004201, batch_cost=0.7943, reader_cost=0.0001 | ETA 03:22:00 2021-01-18 15:26:11 [INFO] [TRAIN] epoch=38, iter=24750/40000, loss=0.2114, lr=0.004199, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:21:21 2021-01-18 15:26:19 [INFO] [TRAIN] epoch=38, iter=24760/40000, loss=0.2235, lr=0.004196, batch_cost=0.7969, reader_cost=0.0001 | ETA 03:22:25 2021-01-18 15:26:27 [INFO] [TRAIN] epoch=38, iter=24770/40000, loss=0.1994, lr=0.004194, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:20:04 2021-01-18 15:26:35 [INFO] [TRAIN] epoch=38, iter=24780/40000, loss=0.1666, lr=0.004191, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:21:16 2021-01-18 15:26:42 [INFO] [TRAIN] epoch=38, iter=24790/40000, loss=0.1795, lr=0.004189, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:20:49 2021-01-18 15:26:50 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.1679, lr=0.004186, batch_cost=0.7931, reader_cost=0.0001 | ETA 03:20:55 2021-01-18 15:26:58 [INFO] [TRAIN] epoch=38, iter=24810/40000, loss=0.2464, lr=0.004184, batch_cost=0.7958, reader_cost=0.0001 | ETA 03:21:28 2021-01-18 15:27:06 [INFO] [TRAIN] epoch=38, iter=24820/40000, loss=0.1463, lr=0.004181, batch_cost=0.7885, reader_cost=0.0001 | ETA 03:19:29 2021-01-18 15:27:14 [INFO] [TRAIN] epoch=38, iter=24830/40000, loss=0.1695, lr=0.004179, batch_cost=0.7943, reader_cost=0.0001 | ETA 03:20:49 2021-01-18 15:27:22 [INFO] [TRAIN] epoch=38, iter=24840/40000, loss=0.2013, lr=0.004176, batch_cost=0.7887, reader_cost=0.0001 | ETA 03:19:16 2021-01-18 15:27:30 [INFO] [TRAIN] epoch=38, iter=24850/40000, loss=0.1629, lr=0.004174, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:19:26 2021-01-18 15:27:38 [INFO] [TRAIN] epoch=38, iter=24860/40000, loss=0.1421, lr=0.004171, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:19:48 2021-01-18 15:27:46 [INFO] [TRAIN] epoch=38, iter=24870/40000, loss=0.2000, lr=0.004169, batch_cost=0.7920, reader_cost=0.0001 | ETA 03:19:42 2021-01-18 15:27:54 [INFO] [TRAIN] epoch=38, iter=24880/40000, loss=0.1593, lr=0.004166, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:19:21 2021-01-18 15:28:02 [INFO] [TRAIN] epoch=38, iter=24890/40000, loss=0.2099, lr=0.004164, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:19:13 2021-01-18 15:28:10 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.1584, lr=0.004162, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:18:57 2021-01-18 15:28:17 [INFO] [TRAIN] epoch=38, iter=24910/40000, loss=0.1989, lr=0.004159, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:18:32 2021-01-18 15:28:25 [INFO] [TRAIN] epoch=38, iter=24920/40000, loss=0.2281, lr=0.004157, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:19:11 2021-01-18 15:28:33 [INFO] [TRAIN] epoch=38, iter=24930/40000, loss=0.1663, lr=0.004154, batch_cost=0.7944, reader_cost=0.0001 | ETA 03:19:31 2021-01-18 15:28:41 [INFO] [TRAIN] epoch=38, iter=24940/40000, loss=0.1868, lr=0.004152, batch_cost=0.7880, reader_cost=0.0001 | ETA 03:17:47 2021-01-18 15:28:49 [INFO] [TRAIN] epoch=38, iter=24950/40000, loss=0.2425, lr=0.004149, batch_cost=0.7867, reader_cost=0.0001 | ETA 03:17:19 2021-01-18 15:28:57 [INFO] [TRAIN] epoch=38, iter=24960/40000, loss=0.1842, lr=0.004147, batch_cost=0.7919, reader_cost=0.0001 | ETA 03:18:29 2021-01-18 15:29:05 [INFO] [TRAIN] epoch=38, iter=24970/40000, loss=0.1631, lr=0.004144, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:17:50 2021-01-18 15:29:13 [INFO] [TRAIN] epoch=38, iter=24980/40000, loss=0.1998, lr=0.004142, batch_cost=0.7892, reader_cost=0.0001 | ETA 03:17:33 2021-01-18 15:29:21 [INFO] [TRAIN] epoch=38, iter=24990/40000, loss=0.2131, lr=0.004139, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:18:11 2021-01-18 15:29:29 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.1388, lr=0.004137, batch_cost=0.7889, reader_cost=0.0001 | ETA 03:17:13 2021-01-18 15:29:29 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 15:29:56 [INFO] [EVAL] #Images=1449 mIoU=0.7926 Acc=0.9541 Kappa=0.8991 2021-01-18 15:29:56 [INFO] [EVAL] Class IoU: [0.9485 0.9143 0.4175 0.902 0.6571 0.8404 0.9426 0.8867 0.9199 0.4404 0.8832 0.6184 0.8805 0.8886 0.8671 0.8768 0.6358 0.8451 0.5832 0.9202 0.7769] 2021-01-18 15:29:56 [INFO] [EVAL] Class Acc: [0.9721 0.9396 0.4306 0.9458 0.7498 0.9089 0.9864 0.9465 0.9503 0.6567 0.9241 0.9133 0.9446 0.9425 0.951 0.9279 0.7848 0.9414 0.7842 0.9545 0.9167] 2021-01-18 15:30:07 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 15:30:15 [INFO] [TRAIN] epoch=38, iter=25010/40000, loss=0.1782, lr=0.004134, batch_cost=0.7805, reader_cost=0.0001 | ETA 03:15:00 2021-01-18 15:30:23 [INFO] [TRAIN] epoch=38, iter=25020/40000, loss=0.2126, lr=0.004132, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:16:42 2021-01-18 15:30:31 [INFO] [TRAIN] epoch=38, iter=25030/40000, loss=0.2251, lr=0.004129, batch_cost=0.7860, reader_cost=0.0001 | ETA 03:16:06 2021-01-18 15:30:39 [INFO] [TRAIN] epoch=38, iter=25040/40000, loss=0.1200, lr=0.004127, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:17:02 2021-01-18 15:30:47 [INFO] [TRAIN] epoch=38, iter=25050/40000, loss=0.1878, lr=0.004124, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:17:03 2021-01-18 15:30:54 [INFO] [TRAIN] epoch=38, iter=25060/40000, loss=0.1815, lr=0.004122, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:16:47 2021-01-18 15:31:02 [INFO] [TRAIN] epoch=38, iter=25070/40000, loss=0.1936, lr=0.004119, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:16:52 2021-01-18 15:31:10 [INFO] [TRAIN] epoch=38, iter=25080/40000, loss=0.2566, lr=0.004117, batch_cost=0.7944, reader_cost=0.0001 | ETA 03:17:31 2021-01-18 15:31:18 [INFO] [TRAIN] epoch=38, iter=25090/40000, loss=0.1521, lr=0.004114, batch_cost=0.7880, reader_cost=0.0001 | ETA 03:15:49 2021-01-18 15:31:26 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.1794, lr=0.004112, batch_cost=0.7910, reader_cost=0.0001 | ETA 03:16:25 2021-01-18 15:31:34 [INFO] [TRAIN] epoch=38, iter=25110/40000, loss=0.1709, lr=0.004109, batch_cost=0.7944, reader_cost=0.0001 | ETA 03:17:09 2021-01-18 15:31:42 [INFO] [TRAIN] epoch=39, iter=25120/40000, loss=0.1622, lr=0.004107, batch_cost=0.7792, reader_cost=0.0136 | ETA 03:13:14 2021-01-18 15:31:50 [INFO] [TRAIN] epoch=39, iter=25130/40000, loss=0.1834, lr=0.004104, batch_cost=0.7896, reader_cost=0.0001 | ETA 03:15:41 2021-01-18 15:31:58 [INFO] [TRAIN] epoch=39, iter=25140/40000, loss=0.2300, lr=0.004102, batch_cost=0.7878, reader_cost=0.0001 | ETA 03:15:06 2021-01-18 15:32:06 [INFO] [TRAIN] epoch=39, iter=25150/40000, loss=0.1509, lr=0.004099, batch_cost=0.7859, reader_cost=0.0001 | ETA 03:14:30 2021-01-18 15:32:13 [INFO] [TRAIN] epoch=39, iter=25160/40000, loss=0.1974, lr=0.004097, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:16:14 2021-01-18 15:32:21 [INFO] [TRAIN] epoch=39, iter=25170/40000, loss=0.1482, lr=0.004094, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:15:00 2021-01-18 15:32:29 [INFO] [TRAIN] epoch=39, iter=25180/40000, loss=0.1328, lr=0.004092, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:15:00 2021-01-18 15:32:37 [INFO] [TRAIN] epoch=39, iter=25190/40000, loss=0.1754, lr=0.004090, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:14:31 2021-01-18 15:32:45 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.1534, lr=0.004087, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:15:03 2021-01-18 15:32:53 [INFO] [TRAIN] epoch=39, iter=25210/40000, loss=0.2014, lr=0.004085, batch_cost=0.7918, reader_cost=0.0001 | ETA 03:15:11 2021-01-18 15:33:01 [INFO] [TRAIN] epoch=39, iter=25220/40000, loss=0.1372, lr=0.004082, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:14:15 2021-01-18 15:33:09 [INFO] [TRAIN] epoch=39, iter=25230/40000, loss=0.1523, lr=0.004080, batch_cost=0.7872, reader_cost=0.0001 | ETA 03:13:47 2021-01-18 15:33:17 [INFO] [TRAIN] epoch=39, iter=25240/40000, loss=0.1671, lr=0.004077, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:14:31 2021-01-18 15:33:24 [INFO] [TRAIN] epoch=39, iter=25250/40000, loss=0.1233, lr=0.004075, batch_cost=0.7836, reader_cost=0.0001 | ETA 03:12:37 2021-01-18 15:33:32 [INFO] [TRAIN] epoch=39, iter=25260/40000, loss=0.1538, lr=0.004072, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:14:09 2021-01-18 15:33:40 [INFO] [TRAIN] epoch=39, iter=25270/40000, loss=0.1265, lr=0.004070, batch_cost=0.7911, reader_cost=0.0001 | ETA 03:14:12 2021-01-18 15:33:48 [INFO] [TRAIN] epoch=39, iter=25280/40000, loss=0.1312, lr=0.004067, batch_cost=0.7951, reader_cost=0.0001 | ETA 03:15:04 2021-01-18 15:33:56 [INFO] [TRAIN] epoch=39, iter=25290/40000, loss=0.1542, lr=0.004065, batch_cost=0.7928, reader_cost=0.0001 | ETA 03:14:22 2021-01-18 15:34:04 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.2010, lr=0.004062, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:14:04 2021-01-18 15:34:12 [INFO] [TRAIN] epoch=39, iter=25310/40000, loss=0.1681, lr=0.004060, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:14:11 2021-01-18 15:34:20 [INFO] [TRAIN] epoch=39, iter=25320/40000, loss=0.1488, lr=0.004057, batch_cost=0.7957, reader_cost=0.0001 | ETA 03:14:41 2021-01-18 15:34:28 [INFO] [TRAIN] epoch=39, iter=25330/40000, loss=0.1625, lr=0.004055, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:13:16 2021-01-18 15:34:36 [INFO] [TRAIN] epoch=39, iter=25340/40000, loss=0.1298, lr=0.004052, batch_cost=0.7882, reader_cost=0.0001 | ETA 03:12:35 2021-01-18 15:34:44 [INFO] [TRAIN] epoch=39, iter=25350/40000, loss=0.1387, lr=0.004050, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:13:01 2021-01-18 15:34:52 [INFO] [TRAIN] epoch=39, iter=25360/40000, loss=0.1897, lr=0.004047, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:12:43 2021-01-18 15:34:59 [INFO] [TRAIN] epoch=39, iter=25370/40000, loss=0.1908, lr=0.004045, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:12:22 2021-01-18 15:35:07 [INFO] [TRAIN] epoch=39, iter=25380/40000, loss=0.2032, lr=0.004042, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:12:14 2021-01-18 15:35:15 [INFO] [TRAIN] epoch=39, iter=25390/40000, loss=0.1825, lr=0.004040, batch_cost=0.7983, reader_cost=0.0001 | ETA 03:14:23 2021-01-18 15:35:23 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.2334, lr=0.004037, batch_cost=0.7923, reader_cost=0.0001 | ETA 03:12:47 2021-01-18 15:35:31 [INFO] [TRAIN] epoch=39, iter=25410/40000, loss=0.1782, lr=0.004035, batch_cost=0.7917, reader_cost=0.0001 | ETA 03:12:31 2021-01-18 15:35:39 [INFO] [TRAIN] epoch=39, iter=25420/40000, loss=0.1984, lr=0.004032, batch_cost=0.7901, reader_cost=0.0001 | ETA 03:11:59 2021-01-18 15:35:47 [INFO] [TRAIN] epoch=39, iter=25430/40000, loss=0.1752, lr=0.004030, batch_cost=0.7898, reader_cost=0.0001 | ETA 03:11:47 2021-01-18 15:35:55 [INFO] [TRAIN] epoch=39, iter=25440/40000, loss=0.2064, lr=0.004027, batch_cost=0.7912, reader_cost=0.0001 | ETA 03:11:59 2021-01-18 15:36:03 [INFO] [TRAIN] epoch=39, iter=25450/40000, loss=0.2698, lr=0.004025, batch_cost=0.7933, reader_cost=0.0001 | ETA 03:12:23 2021-01-18 15:36:11 [INFO] [TRAIN] epoch=39, iter=25460/40000, loss=0.1957, lr=0.004022, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:11:18 2021-01-18 15:36:19 [INFO] [TRAIN] epoch=39, iter=25470/40000, loss=0.1797, lr=0.004020, batch_cost=0.7858, reader_cost=0.0001 | ETA 03:10:17 2021-01-18 15:36:26 [INFO] [TRAIN] epoch=39, iter=25480/40000, loss=0.1563, lr=0.004017, batch_cost=0.7886, reader_cost=0.0001 | ETA 03:10:50 2021-01-18 15:36:34 [INFO] [TRAIN] epoch=39, iter=25490/40000, loss=0.1769, lr=0.004015, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:11:46 2021-01-18 15:36:42 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.2598, lr=0.004012, batch_cost=0.7917, reader_cost=0.0001 | ETA 03:11:19 2021-01-18 15:36:50 [INFO] [TRAIN] epoch=39, iter=25510/40000, loss=0.2073, lr=0.004010, batch_cost=0.7941, reader_cost=0.0001 | ETA 03:11:46 2021-01-18 15:36:58 [INFO] [TRAIN] epoch=39, iter=25520/40000, loss=0.1857, lr=0.004007, batch_cost=0.7897, reader_cost=0.0001 | ETA 03:10:34 2021-01-18 15:37:06 [INFO] [TRAIN] epoch=39, iter=25530/40000, loss=0.1606, lr=0.004005, batch_cost=0.7847, reader_cost=0.0001 | ETA 03:09:14 2021-01-18 15:37:14 [INFO] [TRAIN] epoch=39, iter=25540/40000, loss=0.2637, lr=0.004002, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:10:53 2021-01-18 15:37:22 [INFO] [TRAIN] epoch=39, iter=25550/40000, loss=0.1680, lr=0.004000, batch_cost=0.7955, reader_cost=0.0001 | ETA 03:11:35 2021-01-18 15:37:30 [INFO] [TRAIN] epoch=39, iter=25560/40000, loss=0.1827, lr=0.003997, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:10:53 2021-01-18 15:37:38 [INFO] [TRAIN] epoch=39, iter=25570/40000, loss=0.1482, lr=0.003995, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:10:30 2021-01-18 15:37:46 [INFO] [TRAIN] epoch=39, iter=25580/40000, loss=0.1786, lr=0.003992, batch_cost=0.7918, reader_cost=0.0001 | ETA 03:10:18 2021-01-18 15:37:54 [INFO] [TRAIN] epoch=39, iter=25590/40000, loss=0.1487, lr=0.003990, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:10:02 2021-01-18 15:38:01 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.2067, lr=0.003987, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:09:08 2021-01-18 15:38:09 [INFO] [TRAIN] epoch=39, iter=25610/40000, loss=0.1615, lr=0.003985, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:09:59 2021-01-18 15:38:17 [INFO] [TRAIN] epoch=39, iter=25620/40000, loss=0.1960, lr=0.003983, batch_cost=0.7889, reader_cost=0.0001 | ETA 03:09:04 2021-01-18 15:38:25 [INFO] [TRAIN] epoch=39, iter=25630/40000, loss=0.1878, lr=0.003980, batch_cost=0.7921, reader_cost=0.0001 | ETA 03:09:42 2021-01-18 15:38:33 [INFO] [TRAIN] epoch=39, iter=25640/40000, loss=0.1588, lr=0.003978, batch_cost=0.7929, reader_cost=0.0001 | ETA 03:09:45 2021-01-18 15:38:41 [INFO] [TRAIN] epoch=39, iter=25650/40000, loss=0.1865, lr=0.003975, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:08:55 2021-01-18 15:38:49 [INFO] [TRAIN] epoch=39, iter=25660/40000, loss=0.1615, lr=0.003973, batch_cost=0.7909, reader_cost=0.0001 | ETA 03:09:01 2021-01-18 15:38:57 [INFO] [TRAIN] epoch=39, iter=25670/40000, loss=0.1359, lr=0.003970, batch_cost=0.7892, reader_cost=0.0001 | ETA 03:08:29 2021-01-18 15:39:05 [INFO] [TRAIN] epoch=39, iter=25680/40000, loss=0.1366, lr=0.003968, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:08:44 2021-01-18 15:39:13 [INFO] [TRAIN] epoch=39, iter=25690/40000, loss=0.1979, lr=0.003965, batch_cost=0.7928, reader_cost=0.0001 | ETA 03:09:04 2021-01-18 15:39:21 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.1663, lr=0.003963, batch_cost=0.7941, reader_cost=0.0001 | ETA 03:09:15 2021-01-18 15:39:29 [INFO] [TRAIN] epoch=39, iter=25710/40000, loss=0.1413, lr=0.003960, batch_cost=0.7951, reader_cost=0.0001 | ETA 03:09:22 2021-01-18 15:39:36 [INFO] [TRAIN] epoch=39, iter=25720/40000, loss=0.1647, lr=0.003958, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:08:37 2021-01-18 15:39:44 [INFO] [TRAIN] epoch=39, iter=25730/40000, loss=0.1550, lr=0.003955, batch_cost=0.7866, reader_cost=0.0001 | ETA 03:07:05 2021-01-18 15:39:52 [INFO] [TRAIN] epoch=39, iter=25740/40000, loss=0.1397, lr=0.003953, batch_cost=0.7895, reader_cost=0.0001 | ETA 03:07:38 2021-01-18 15:40:00 [INFO] [TRAIN] epoch=39, iter=25750/40000, loss=0.2272, lr=0.003950, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:07:00 2021-01-18 15:40:08 [INFO] [TRAIN] epoch=39, iter=25760/40000, loss=0.1580, lr=0.003948, batch_cost=0.7874, reader_cost=0.0001 | ETA 03:06:52 2021-01-18 15:40:16 [INFO] [TRAIN] epoch=39, iter=25770/40000, loss=0.2161, lr=0.003945, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:07:27 2021-01-18 15:40:24 [INFO] [TRAIN] epoch=40, iter=25780/40000, loss=0.1787, lr=0.003943, batch_cost=0.7765, reader_cost=0.0169 | ETA 03:04:01 2021-01-18 15:40:32 [INFO] [TRAIN] epoch=40, iter=25790/40000, loss=0.1926, lr=0.003940, batch_cost=0.7897, reader_cost=0.0001 | ETA 03:07:01 2021-01-18 15:40:39 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.1569, lr=0.003938, batch_cost=0.7965, reader_cost=0.0001 | ETA 03:08:30 2021-01-18 15:40:47 [INFO] [TRAIN] epoch=40, iter=25810/40000, loss=0.1314, lr=0.003935, batch_cost=0.7902, reader_cost=0.0001 | ETA 03:06:53 2021-01-18 15:40:55 [INFO] [TRAIN] epoch=40, iter=25820/40000, loss=0.1528, lr=0.003933, batch_cost=0.7932, reader_cost=0.0001 | ETA 03:07:28 2021-01-18 15:41:03 [INFO] [TRAIN] epoch=40, iter=25830/40000, loss=0.2174, lr=0.003930, batch_cost=0.7959, reader_cost=0.0001 | ETA 03:07:58 2021-01-18 15:41:11 [INFO] [TRAIN] epoch=40, iter=25840/40000, loss=0.2054, lr=0.003928, batch_cost=0.7915, reader_cost=0.0001 | ETA 03:06:47 2021-01-18 15:41:19 [INFO] [TRAIN] epoch=40, iter=25850/40000, loss=0.1844, lr=0.003925, batch_cost=0.7922, reader_cost=0.0001 | ETA 03:06:49 2021-01-18 15:41:27 [INFO] [TRAIN] epoch=40, iter=25860/40000, loss=0.1503, lr=0.003923, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:06:09 2021-01-18 15:41:35 [INFO] [TRAIN] epoch=40, iter=25870/40000, loss=0.1484, lr=0.003920, batch_cost=0.7903, reader_cost=0.0001 | ETA 03:06:06 2021-01-18 15:41:43 [INFO] [TRAIN] epoch=40, iter=25880/40000, loss=0.1660, lr=0.003918, batch_cost=0.7965, reader_cost=0.0001 | ETA 03:07:26 2021-01-18 15:41:51 [INFO] [TRAIN] epoch=40, iter=25890/40000, loss=0.1485, lr=0.003915, batch_cost=0.7894, reader_cost=0.0001 | ETA 03:05:38 2021-01-18 15:41:59 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.2123, lr=0.003913, batch_cost=0.7901, reader_cost=0.0001 | ETA 03:05:40 2021-01-18 15:42:07 [INFO] [TRAIN] epoch=40, iter=25910/40000, loss=0.1668, lr=0.003910, batch_cost=0.7896, reader_cost=0.0001 | ETA 03:05:25 2021-01-18 15:42:14 [INFO] [TRAIN] epoch=40, iter=25920/40000, loss=0.1604, lr=0.003908, batch_cost=0.7925, reader_cost=0.0001 | ETA 03:05:57 2021-01-18 15:42:22 [INFO] [TRAIN] epoch=40, iter=25930/40000, loss=0.1420, lr=0.003905, batch_cost=0.7831, reader_cost=0.0001 | ETA 03:03:38 2021-01-18 15:42:30 [INFO] [TRAIN] epoch=40, iter=25940/40000, loss=0.2060, lr=0.003903, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:05:55 2021-01-18 15:42:38 [INFO] [TRAIN] epoch=40, iter=25950/40000, loss=0.1989, lr=0.003900, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:04:29 2021-01-18 15:42:46 [INFO] [TRAIN] epoch=40, iter=25960/40000, loss=0.1335, lr=0.003898, batch_cost=0.7906, reader_cost=0.0001 | ETA 03:04:59 2021-01-18 15:42:54 [INFO] [TRAIN] epoch=40, iter=25970/40000, loss=0.1882, lr=0.003895, batch_cost=0.7940, reader_cost=0.0001 | ETA 03:05:39 2021-01-18 15:43:02 [INFO] [TRAIN] epoch=40, iter=25980/40000, loss=0.1111, lr=0.003893, batch_cost=0.7873, reader_cost=0.0001 | ETA 03:03:57 2021-01-18 15:43:10 [INFO] [TRAIN] epoch=40, iter=25990/40000, loss=0.1336, lr=0.003890, batch_cost=0.7951, reader_cost=0.0001 | ETA 03:05:38 2021-01-18 15:43:18 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.1815, lr=0.003888, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:03:48 2021-01-18 15:43:18 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 15:43:45 [INFO] [EVAL] #Images=1449 mIoU=0.7831 Acc=0.9526 Kappa=0.8939 2021-01-18 15:43:45 [INFO] [EVAL] Class IoU: [0.9458 0.9131 0.4221 0.9076 0.6701 0.8405 0.9243 0.8803 0.9201 0.453 0.89 0.4335 0.8785 0.894 0.8661 0.883 0.6248 0.8479 0.5591 0.9122 0.7783] 2021-01-18 15:43:45 [INFO] [EVAL] Class Acc: [0.963 0.9384 0.4403 0.9512 0.8274 0.8841 0.984 0.94 0.9525 0.7288 0.9368 0.9192 0.9402 0.9613 0.922 0.9575 0.8467 0.9561 0.8821 0.9629 0.9249] 2021-01-18 15:43:52 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 15:44:00 [INFO] [TRAIN] epoch=40, iter=26010/40000, loss=0.1454, lr=0.003885, batch_cost=0.7885, reader_cost=0.0001 | ETA 03:03:51 2021-01-18 15:44:08 [INFO] [TRAIN] epoch=40, iter=26020/40000, loss=0.1393, lr=0.003883, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:04:57 2021-01-18 15:44:16 [INFO] [TRAIN] epoch=40, iter=26030/40000, loss=0.2528, lr=0.003880, batch_cost=0.7879, reader_cost=0.0001 | ETA 03:03:27 2021-01-18 15:44:23 [INFO] [TRAIN] epoch=40, iter=26040/40000, loss=0.1360, lr=0.003878, batch_cost=0.7934, reader_cost=0.0001 | ETA 03:04:36 2021-01-18 15:44:31 [INFO] [TRAIN] epoch=40, iter=26050/40000, loss=0.2589, lr=0.003875, batch_cost=0.7913, reader_cost=0.0001 | ETA 03:03:59 2021-01-18 15:44:39 [INFO] [TRAIN] epoch=40, iter=26060/40000, loss=0.2606, lr=0.003873, batch_cost=0.7907, reader_cost=0.0001 | ETA 03:03:42 2021-01-18 15:44:47 [INFO] [TRAIN] epoch=40, iter=26070/40000, loss=0.2690, lr=0.003870, batch_cost=0.7901, reader_cost=0.0001 | ETA 03:03:26 2021-01-18 15:44:55 [INFO] [TRAIN] epoch=40, iter=26080/40000, loss=0.1337, lr=0.003868, batch_cost=0.7899, reader_cost=0.0001 | ETA 03:03:14 2021-01-18 15:45:03 [INFO] [TRAIN] epoch=40, iter=26090/40000, loss=0.2279, lr=0.003865, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:04:01 2021-01-18 15:45:11 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.1812, lr=0.003863, batch_cost=0.7966, reader_cost=0.0001 | ETA 03:04:32 2021-01-18 15:45:19 [INFO] [TRAIN] epoch=40, iter=26110/40000, loss=0.1669, lr=0.003860, batch_cost=0.7896, reader_cost=0.0001 | ETA 03:02:47 2021-01-18 15:45:27 [INFO] [TRAIN] epoch=40, iter=26120/40000, loss=0.1874, lr=0.003858, batch_cost=0.7877, reader_cost=0.0001 | ETA 03:02:13 2021-01-18 15:45:35 [INFO] [TRAIN] epoch=40, iter=26130/40000, loss=0.1618, lr=0.003855, batch_cost=0.7900, reader_cost=0.0001 | ETA 03:02:37 2021-01-18 15:45:43 [INFO] [TRAIN] epoch=40, iter=26140/40000, loss=0.2012, lr=0.003853, batch_cost=0.7943, reader_cost=0.0001 | ETA 03:03:28 2021-01-18 15:45:50 [INFO] [TRAIN] epoch=40, iter=26150/40000, loss=0.1836, lr=0.003850, batch_cost=0.7881, reader_cost=0.0001 | ETA 03:01:55 2021-01-18 15:45:58 [INFO] [TRAIN] epoch=40, iter=26160/40000, loss=0.2330, lr=0.003848, batch_cost=0.7914, reader_cost=0.0001 | ETA 03:02:33 2021-01-18 15:46:06 [INFO] [TRAIN] epoch=40, iter=26170/40000, loss=0.1683, lr=0.003845, batch_cost=0.7908, reader_cost=0.0001 | ETA 03:02:17 2021-01-18 15:46:14 [INFO] [TRAIN] epoch=40, iter=26180/40000, loss=0.2585, lr=0.003843, batch_cost=0.7888, reader_cost=0.0001 | ETA 03:01:41 2021-01-18 15:46:22 [INFO] [TRAIN] epoch=40, iter=26190/40000, loss=0.1883, lr=0.003840, batch_cost=0.7936, reader_cost=0.0001 | ETA 03:02:40 2021-01-18 15:46:30 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.1832, lr=0.003838, batch_cost=0.7904, reader_cost=0.0001 | ETA 03:01:46 2021-01-18 15:46:38 [INFO] [TRAIN] epoch=40, iter=26210/40000, loss=0.1613, lr=0.003835, batch_cost=0.7891, reader_cost=0.0001 | ETA 03:01:21 2021-01-18 15:46:46 [INFO] [TRAIN] epoch=40, iter=26220/40000, loss=0.2251, lr=0.003833, batch_cost=0.7918, reader_cost=0.0001 | ETA 03:01:51 2021-01-18 15:46:54 [INFO] [TRAIN] epoch=40, iter=26230/40000, loss=0.2012, lr=0.003830, batch_cost=0.7968, reader_cost=0.0001 | ETA 03:02:51 2021-01-18 15:47:02 [INFO] [TRAIN] epoch=40, iter=26240/40000, loss=0.1367, lr=0.003828, batch_cost=0.7860, reader_cost=0.0001 | ETA 03:00:15 2021-01-18 15:47:10 [INFO] [TRAIN] epoch=40, iter=26250/40000, loss=0.1272, lr=0.003825, batch_cost=0.7905, reader_cost=0.0001 | ETA 03:01:09 2021-01-18 15:47:17 [INFO] [TRAIN] epoch=40, iter=26260/40000, loss=0.1666, lr=0.003823, batch_cost=0.7890, reader_cost=0.0001 | ETA 03:00:40 2021-01-18 15:47:25 [INFO] [TRAIN] epoch=40, iter=26270/40000, loss=0.1284, lr=0.003820, batch_cost=0.7942, reader_cost=0.0001 | ETA 03:01:43 2021-01-18 15:47:33 [INFO] [TRAIN] epoch=40, iter=26280/40000, loss=0.2046, lr=0.003818, batch_cost=0.7868, reader_cost=0.0001 | ETA 02:59:55 2021-01-18 15:47:41 [INFO] [TRAIN] epoch=40, iter=26290/40000, loss=0.1368, lr=0.003815, batch_cost=0.7930, reader_cost=0.0001 | ETA 03:01:12 2021-01-18 15:47:49 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.1690, lr=0.003813, batch_cost=0.7917, reader_cost=0.0001 | ETA 03:00:46 2021-01-18 15:47:57 [INFO] [TRAIN] epoch=40, iter=26310/40000, loss=0.1808, lr=0.003810, batch_cost=0.7920, reader_cost=0.0001 | ETA 03:00:42 2021-01-18 15:48:05 [INFO] [TRAIN] epoch=40, iter=26320/40000, loss=0.1542, lr=0.003808, batch_cost=0.7927, reader_cost=0.0001 | ETA 03:00:44 2021-01-18 15:48:13 [INFO] [TRAIN] epoch=40, iter=26330/40000, loss=0.1990, lr=0.003805, batch_cost=0.7938, reader_cost=0.0001 | ETA 03:00:50 2021-01-18 15:48:21 [INFO] [TRAIN] epoch=40, iter=26340/40000, loss=0.1312, lr=0.003803, batch_cost=0.7910, reader_cost=0.0001 | ETA 03:00:04 2021-01-18 15:48:29 [INFO] [TRAIN] epoch=40, iter=26350/40000, loss=0.2180, lr=0.003800, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:59:49 2021-01-18 15:48:37 [INFO] [TRAIN] epoch=40, iter=26360/40000, loss=0.1544, lr=0.003798, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:59:41 2021-01-18 15:48:45 [INFO] [TRAIN] epoch=40, iter=26370/40000, loss=0.1822, lr=0.003795, batch_cost=0.7914, reader_cost=0.0001 | ETA 02:59:47 2021-01-18 15:48:52 [INFO] [TRAIN] epoch=40, iter=26380/40000, loss=0.1635, lr=0.003793, batch_cost=0.7931, reader_cost=0.0001 | ETA 03:00:02 2021-01-18 15:49:00 [INFO] [TRAIN] epoch=40, iter=26390/40000, loss=0.1594, lr=0.003790, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:59:23 2021-01-18 15:49:08 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.1942, lr=0.003788, batch_cost=0.7939, reader_cost=0.0001 | ETA 02:59:56 2021-01-18 15:49:16 [INFO] [TRAIN] epoch=40, iter=26410/40000, loss=0.1576, lr=0.003785, batch_cost=0.7871, reader_cost=0.0001 | ETA 02:58:16 2021-01-18 15:49:24 [INFO] [TRAIN] epoch=40, iter=26420/40000, loss=0.1883, lr=0.003783, batch_cost=0.7942, reader_cost=0.0001 | ETA 02:59:44 2021-01-18 15:49:32 [INFO] [TRAIN] epoch=40, iter=26430/40000, loss=0.2246, lr=0.003780, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:59:09 2021-01-18 15:49:40 [INFO] [TRAIN] epoch=40, iter=26440/40000, loss=0.1151, lr=0.003778, batch_cost=0.7560, reader_cost=0.0001 | ETA 02:50:51 2021-01-18 15:49:48 [INFO] [TRAIN] epoch=41, iter=26450/40000, loss=0.2228, lr=0.003775, batch_cost=0.8062, reader_cost=0.0137 | ETA 03:02:04 2021-01-18 15:49:56 [INFO] [TRAIN] epoch=41, iter=26460/40000, loss=0.1619, lr=0.003773, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:58:16 2021-01-18 15:50:04 [INFO] [TRAIN] epoch=41, iter=26470/40000, loss=0.2350, lr=0.003770, batch_cost=0.7964, reader_cost=0.0001 | ETA 02:59:35 2021-01-18 15:50:11 [INFO] [TRAIN] epoch=41, iter=26480/40000, loss=0.1665, lr=0.003768, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:58:08 2021-01-18 15:50:19 [INFO] [TRAIN] epoch=41, iter=26490/40000, loss=0.1789, lr=0.003765, batch_cost=0.7972, reader_cost=0.0001 | ETA 02:59:30 2021-01-18 15:50:27 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.1859, lr=0.003762, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:58:11 2021-01-18 15:50:35 [INFO] [TRAIN] epoch=41, iter=26510/40000, loss=0.1038, lr=0.003760, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:57:53 2021-01-18 15:50:43 [INFO] [TRAIN] epoch=41, iter=26520/40000, loss=0.1830, lr=0.003757, batch_cost=0.7955, reader_cost=0.0001 | ETA 02:58:43 2021-01-18 15:50:51 [INFO] [TRAIN] epoch=41, iter=26530/40000, loss=0.2001, lr=0.003755, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:57:56 2021-01-18 15:50:59 [INFO] [TRAIN] epoch=41, iter=26540/40000, loss=0.1691, lr=0.003752, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:56:53 2021-01-18 15:51:07 [INFO] [TRAIN] epoch=41, iter=26550/40000, loss=0.1637, lr=0.003750, batch_cost=0.7944, reader_cost=0.0001 | ETA 02:58:05 2021-01-18 15:51:15 [INFO] [TRAIN] epoch=41, iter=26560/40000, loss=0.1342, lr=0.003747, batch_cost=0.7961, reader_cost=0.0001 | ETA 02:58:19 2021-01-18 15:51:23 [INFO] [TRAIN] epoch=41, iter=26570/40000, loss=0.1462, lr=0.003745, batch_cost=0.7950, reader_cost=0.0001 | ETA 02:57:57 2021-01-18 15:51:31 [INFO] [TRAIN] epoch=41, iter=26580/40000, loss=0.1689, lr=0.003742, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:57:07 2021-01-18 15:51:39 [INFO] [TRAIN] epoch=41, iter=26590/40000, loss=0.1703, lr=0.003740, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:56:40 2021-01-18 15:51:47 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.1592, lr=0.003737, batch_cost=0.7961, reader_cost=0.0001 | ETA 02:57:48 2021-01-18 15:51:55 [INFO] [TRAIN] epoch=41, iter=26610/40000, loss=0.1751, lr=0.003735, batch_cost=0.7907, reader_cost=0.0001 | ETA 02:56:27 2021-01-18 15:52:02 [INFO] [TRAIN] epoch=41, iter=26620/40000, loss=0.2446, lr=0.003732, batch_cost=0.7906, reader_cost=0.0001 | ETA 02:56:17 2021-01-18 15:52:10 [INFO] [TRAIN] epoch=41, iter=26630/40000, loss=0.1521, lr=0.003730, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:55:51 2021-01-18 15:52:18 [INFO] [TRAIN] epoch=41, iter=26640/40000, loss=0.1668, lr=0.003727, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:56:06 2021-01-18 15:52:26 [INFO] [TRAIN] epoch=41, iter=26650/40000, loss=0.2095, lr=0.003725, batch_cost=0.7942, reader_cost=0.0001 | ETA 02:56:42 2021-01-18 15:52:34 [INFO] [TRAIN] epoch=41, iter=26660/40000, loss=0.2074, lr=0.003722, batch_cost=0.7927, reader_cost=0.0001 | ETA 02:56:14 2021-01-18 15:52:42 [INFO] [TRAIN] epoch=41, iter=26670/40000, loss=0.1392, lr=0.003720, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:55:48 2021-01-18 15:52:50 [INFO] [TRAIN] epoch=41, iter=26680/40000, loss=0.1837, lr=0.003717, batch_cost=0.7877, reader_cost=0.0001 | ETA 02:54:52 2021-01-18 15:52:58 [INFO] [TRAIN] epoch=41, iter=26690/40000, loss=0.1115, lr=0.003715, batch_cost=0.7916, reader_cost=0.0001 | ETA 02:55:36 2021-01-18 15:53:06 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.1937, lr=0.003712, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:54:58 2021-01-18 15:53:14 [INFO] [TRAIN] epoch=41, iter=26710/40000, loss=0.1818, lr=0.003710, batch_cost=0.7887, reader_cost=0.0001 | ETA 02:54:42 2021-01-18 15:53:21 [INFO] [TRAIN] epoch=41, iter=26720/40000, loss=0.1187, lr=0.003707, batch_cost=0.7849, reader_cost=0.0001 | ETA 02:53:43 2021-01-18 15:53:29 [INFO] [TRAIN] epoch=41, iter=26730/40000, loss=0.1920, lr=0.003705, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:54:56 2021-01-18 15:53:37 [INFO] [TRAIN] epoch=41, iter=26740/40000, loss=0.1612, lr=0.003702, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:55:00 2021-01-18 15:53:45 [INFO] [TRAIN] epoch=41, iter=26750/40000, loss=0.1639, lr=0.003700, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:54:17 2021-01-18 15:53:53 [INFO] [TRAIN] epoch=41, iter=26760/40000, loss=0.2029, lr=0.003697, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:54:46 2021-01-18 15:54:01 [INFO] [TRAIN] epoch=41, iter=26770/40000, loss=0.1455, lr=0.003695, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:54:05 2021-01-18 15:54:09 [INFO] [TRAIN] epoch=41, iter=26780/40000, loss=0.1614, lr=0.003692, batch_cost=0.7917, reader_cost=0.0001 | ETA 02:54:25 2021-01-18 15:54:17 [INFO] [TRAIN] epoch=41, iter=26790/40000, loss=0.2643, lr=0.003690, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:54:01 2021-01-18 15:54:25 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.1571, lr=0.003687, batch_cost=0.7902, reader_cost=0.0001 | ETA 02:53:51 2021-01-18 15:54:33 [INFO] [TRAIN] epoch=41, iter=26810/40000, loss=0.2072, lr=0.003685, batch_cost=0.7948, reader_cost=0.0001 | ETA 02:54:43 2021-01-18 15:54:41 [INFO] [TRAIN] epoch=41, iter=26820/40000, loss=0.2184, lr=0.003682, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:54:06 2021-01-18 15:54:49 [INFO] [TRAIN] epoch=41, iter=26830/40000, loss=0.1474, lr=0.003680, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:53:03 2021-01-18 15:54:56 [INFO] [TRAIN] epoch=41, iter=26840/40000, loss=0.1383, lr=0.003677, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:52:54 2021-01-18 15:55:04 [INFO] [TRAIN] epoch=41, iter=26850/40000, loss=0.2073, lr=0.003675, batch_cost=0.7871, reader_cost=0.0001 | ETA 02:52:30 2021-01-18 15:55:12 [INFO] [TRAIN] epoch=41, iter=26860/40000, loss=0.1300, lr=0.003672, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:52:26 2021-01-18 15:55:20 [INFO] [TRAIN] epoch=41, iter=26870/40000, loss=0.1319, lr=0.003670, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:52:46 2021-01-18 15:55:28 [INFO] [TRAIN] epoch=41, iter=26880/40000, loss=0.1680, lr=0.003667, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:52:36 2021-01-18 15:55:36 [INFO] [TRAIN] epoch=41, iter=26890/40000, loss=0.1992, lr=0.003665, batch_cost=0.7940, reader_cost=0.0001 | ETA 02:53:29 2021-01-18 15:55:44 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.1964, lr=0.003662, batch_cost=0.7939, reader_cost=0.0001 | ETA 02:53:19 2021-01-18 15:55:52 [INFO] [TRAIN] epoch=41, iter=26910/40000, loss=0.1259, lr=0.003659, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:52:12 2021-01-18 15:56:00 [INFO] [TRAIN] epoch=41, iter=26920/40000, loss=0.1335, lr=0.003657, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:52:07 2021-01-18 15:56:07 [INFO] [TRAIN] epoch=41, iter=26930/40000, loss=0.1985, lr=0.003654, batch_cost=0.7859, reader_cost=0.0001 | ETA 02:51:11 2021-01-18 15:56:15 [INFO] [TRAIN] epoch=41, iter=26940/40000, loss=0.1548, lr=0.003652, batch_cost=0.7845, reader_cost=0.0001 | ETA 02:50:45 2021-01-18 15:56:23 [INFO] [TRAIN] epoch=41, iter=26950/40000, loss=0.1751, lr=0.003649, batch_cost=0.7868, reader_cost=0.0001 | ETA 02:51:07 2021-01-18 15:56:31 [INFO] [TRAIN] epoch=41, iter=26960/40000, loss=0.1460, lr=0.003647, batch_cost=0.7933, reader_cost=0.0001 | ETA 02:52:24 2021-01-18 15:56:39 [INFO] [TRAIN] epoch=41, iter=26970/40000, loss=0.2043, lr=0.003644, batch_cost=0.7943, reader_cost=0.0001 | ETA 02:52:29 2021-01-18 15:56:47 [INFO] [TRAIN] epoch=41, iter=26980/40000, loss=0.1514, lr=0.003642, batch_cost=0.7894, reader_cost=0.0001 | ETA 02:51:18 2021-01-18 15:56:55 [INFO] [TRAIN] epoch=41, iter=26990/40000, loss=0.1167, lr=0.003639, batch_cost=0.7953, reader_cost=0.0001 | ETA 02:52:26 2021-01-18 15:57:03 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.2021, lr=0.003637, batch_cost=0.7964, reader_cost=0.0001 | ETA 02:52:32 2021-01-18 15:57:03 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 15:57:30 [INFO] [EVAL] #Images=1449 mIoU=0.7874 Acc=0.9532 Kappa=0.8960 2021-01-18 15:57:30 [INFO] [EVAL] Class IoU: [0.9474 0.9073 0.4181 0.896 0.6859 0.8543 0.9123 0.8871 0.9307 0.4355 0.865 0.6281 0.8773 0.9066 0.8825 0.8833 0.6397 0.8502 0.469 0.9013 0.7574] 2021-01-18 15:57:30 [INFO] [EVAL] Class Acc: [0.9665 0.9369 0.4397 0.9484 0.7777 0.913 0.9556 0.9414 0.9638 0.6486 0.9374 0.9022 0.9352 0.9694 0.9361 0.9536 0.8116 0.9512 0.851 0.9554 0.9326] 2021-01-18 15:57:37 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 15:57:45 [INFO] [TRAIN] epoch=41, iter=27010/40000, loss=0.1316, lr=0.003634, batch_cost=0.7907, reader_cost=0.0001 | ETA 02:51:11 2021-01-18 15:57:53 [INFO] [TRAIN] epoch=41, iter=27020/40000, loss=0.2344, lr=0.003632, batch_cost=0.7888, reader_cost=0.0001 | ETA 02:50:38 2021-01-18 15:58:01 [INFO] [TRAIN] epoch=41, iter=27030/40000, loss=0.1292, lr=0.003629, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:51:09 2021-01-18 15:58:09 [INFO] [TRAIN] epoch=41, iter=27040/40000, loss=0.1729, lr=0.003627, batch_cost=0.7881, reader_cost=0.0001 | ETA 02:50:13 2021-01-18 15:58:16 [INFO] [TRAIN] epoch=41, iter=27050/40000, loss=0.1958, lr=0.003624, batch_cost=0.7879, reader_cost=0.0001 | ETA 02:50:03 2021-01-18 15:58:24 [INFO] [TRAIN] epoch=41, iter=27060/40000, loss=0.1794, lr=0.003622, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:50:13 2021-01-18 15:58:32 [INFO] [TRAIN] epoch=41, iter=27070/40000, loss=0.1860, lr=0.003619, batch_cost=0.7907, reader_cost=0.0001 | ETA 02:50:23 2021-01-18 15:58:40 [INFO] [TRAIN] epoch=41, iter=27080/40000, loss=0.1590, lr=0.003617, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:50:19 2021-01-18 15:58:48 [INFO] [TRAIN] epoch=41, iter=27090/40000, loss=0.1782, lr=0.003614, batch_cost=0.7916, reader_cost=0.0001 | ETA 02:50:19 2021-01-18 15:58:56 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.1945, lr=0.003612, batch_cost=0.7650, reader_cost=0.0001 | ETA 02:44:28 2021-01-18 15:59:04 [INFO] [TRAIN] epoch=42, iter=27110/40000, loss=0.2306, lr=0.003609, batch_cost=0.8028, reader_cost=0.0163 | ETA 02:52:28 2021-01-18 15:59:12 [INFO] [TRAIN] epoch=42, iter=27120/40000, loss=0.1618, lr=0.003607, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:50:00 2021-01-18 15:59:20 [INFO] [TRAIN] epoch=42, iter=27130/40000, loss=0.1349, lr=0.003604, batch_cost=0.7970, reader_cost=0.0001 | ETA 02:50:56 2021-01-18 15:59:27 [INFO] [TRAIN] epoch=42, iter=27140/40000, loss=0.1792, lr=0.003602, batch_cost=0.7883, reader_cost=0.0001 | ETA 02:48:56 2021-01-18 15:59:35 [INFO] [TRAIN] epoch=42, iter=27150/40000, loss=0.1275, lr=0.003599, batch_cost=0.7953, reader_cost=0.0001 | ETA 02:50:19 2021-01-18 15:59:43 [INFO] [TRAIN] epoch=42, iter=27160/40000, loss=0.1479, lr=0.003597, batch_cost=0.7942, reader_cost=0.0001 | ETA 02:49:57 2021-01-18 15:59:51 [INFO] [TRAIN] epoch=42, iter=27170/40000, loss=0.1681, lr=0.003594, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:49:18 2021-01-18 15:59:59 [INFO] [TRAIN] epoch=42, iter=27180/40000, loss=0.1498, lr=0.003591, batch_cost=0.7879, reader_cost=0.0001 | ETA 02:48:20 2021-01-18 16:00:07 [INFO] [TRAIN] epoch=42, iter=27190/40000, loss=0.2007, lr=0.003589, batch_cost=0.7891, reader_cost=0.0001 | ETA 02:48:28 2021-01-18 16:00:15 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.1548, lr=0.003586, batch_cost=0.7869, reader_cost=0.0001 | ETA 02:47:52 2021-01-18 16:00:23 [INFO] [TRAIN] epoch=42, iter=27210/40000, loss=0.1921, lr=0.003584, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:48:51 2021-01-18 16:00:31 [INFO] [TRAIN] epoch=42, iter=27220/40000, loss=0.2135, lr=0.003581, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:48:54 2021-01-18 16:00:39 [INFO] [TRAIN] epoch=42, iter=27230/40000, loss=0.1313, lr=0.003579, batch_cost=0.7861, reader_cost=0.0001 | ETA 02:47:19 2021-01-18 16:00:47 [INFO] [TRAIN] epoch=42, iter=27240/40000, loss=0.1403, lr=0.003576, batch_cost=0.7927, reader_cost=0.0001 | ETA 02:48:34 2021-01-18 16:00:54 [INFO] [TRAIN] epoch=42, iter=27250/40000, loss=0.1282, lr=0.003574, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:47:41 2021-01-18 16:01:02 [INFO] [TRAIN] epoch=42, iter=27260/40000, loss=0.1364, lr=0.003571, batch_cost=0.7923, reader_cost=0.0001 | ETA 02:48:14 2021-01-18 16:01:10 [INFO] [TRAIN] epoch=42, iter=27270/40000, loss=0.1510, lr=0.003569, batch_cost=0.7888, reader_cost=0.0001 | ETA 02:47:20 2021-01-18 16:01:18 [INFO] [TRAIN] epoch=42, iter=27280/40000, loss=0.1565, lr=0.003566, batch_cost=0.7964, reader_cost=0.0001 | ETA 02:48:49 2021-01-18 16:01:26 [INFO] [TRAIN] epoch=42, iter=27290/40000, loss=0.1491, lr=0.003564, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:47:11 2021-01-18 16:01:34 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.1248, lr=0.003561, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:47:51 2021-01-18 16:01:42 [INFO] [TRAIN] epoch=42, iter=27310/40000, loss=0.1717, lr=0.003559, batch_cost=0.7933, reader_cost=0.0001 | ETA 02:47:46 2021-01-18 16:01:50 [INFO] [TRAIN] epoch=42, iter=27320/40000, loss=0.1474, lr=0.003556, batch_cost=0.7949, reader_cost=0.0001 | ETA 02:47:59 2021-01-18 16:01:58 [INFO] [TRAIN] epoch=42, iter=27330/40000, loss=0.1532, lr=0.003554, batch_cost=0.7899, reader_cost=0.0001 | ETA 02:46:47 2021-01-18 16:02:06 [INFO] [TRAIN] epoch=42, iter=27340/40000, loss=0.1392, lr=0.003551, batch_cost=0.7923, reader_cost=0.0001 | ETA 02:47:10 2021-01-18 16:02:14 [INFO] [TRAIN] epoch=42, iter=27350/40000, loss=0.1679, lr=0.003549, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:46:47 2021-01-18 16:02:22 [INFO] [TRAIN] epoch=42, iter=27360/40000, loss=0.1995, lr=0.003546, batch_cost=0.7878, reader_cost=0.0001 | ETA 02:45:57 2021-01-18 16:02:29 [INFO] [TRAIN] epoch=42, iter=27370/40000, loss=0.1490, lr=0.003544, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:46:32 2021-01-18 16:02:37 [INFO] [TRAIN] epoch=42, iter=27380/40000, loss=0.1722, lr=0.003541, batch_cost=0.7858, reader_cost=0.0001 | ETA 02:45:17 2021-01-18 16:02:45 [INFO] [TRAIN] epoch=42, iter=27390/40000, loss=0.2350, lr=0.003539, batch_cost=0.7957, reader_cost=0.0001 | ETA 02:47:14 2021-01-18 16:02:53 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.1340, lr=0.003536, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:45:51 2021-01-18 16:03:01 [INFO] [TRAIN] epoch=42, iter=27410/40000, loss=0.1650, lr=0.003533, batch_cost=0.7897, reader_cost=0.0001 | ETA 02:45:41 2021-01-18 16:03:09 [INFO] [TRAIN] epoch=42, iter=27420/40000, loss=0.2112, lr=0.003531, batch_cost=0.7861, reader_cost=0.0001 | ETA 02:44:48 2021-01-18 16:03:17 [INFO] [TRAIN] epoch=42, iter=27430/40000, loss=0.1765, lr=0.003528, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:45:25 2021-01-18 16:03:25 [INFO] [TRAIN] epoch=42, iter=27440/40000, loss=0.1242, lr=0.003526, batch_cost=0.7897, reader_cost=0.0001 | ETA 02:45:18 2021-01-18 16:03:33 [INFO] [TRAIN] epoch=42, iter=27450/40000, loss=0.1809, lr=0.003523, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:45:19 2021-01-18 16:03:41 [INFO] [TRAIN] epoch=42, iter=27460/40000, loss=0.1465, lr=0.003521, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:45:03 2021-01-18 16:03:48 [INFO] [TRAIN] epoch=42, iter=27470/40000, loss=0.1395, lr=0.003518, batch_cost=0.7928, reader_cost=0.0001 | ETA 02:45:33 2021-01-18 16:03:56 [INFO] [TRAIN] epoch=42, iter=27480/40000, loss=0.1336, lr=0.003516, batch_cost=0.7872, reader_cost=0.0001 | ETA 02:44:16 2021-01-18 16:04:04 [INFO] [TRAIN] epoch=42, iter=27490/40000, loss=0.1959, lr=0.003513, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:44:10 2021-01-18 16:04:12 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.1461, lr=0.003511, batch_cost=0.7859, reader_cost=0.0001 | ETA 02:43:44 2021-01-18 16:04:20 [INFO] [TRAIN] epoch=42, iter=27510/40000, loss=0.2228, lr=0.003508, batch_cost=0.7873, reader_cost=0.0001 | ETA 02:43:52 2021-01-18 16:04:28 [INFO] [TRAIN] epoch=42, iter=27520/40000, loss=0.1512, lr=0.003506, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:43:59 2021-01-18 16:04:36 [INFO] [TRAIN] epoch=42, iter=27530/40000, loss=0.1163, lr=0.003503, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:43:52 2021-01-18 16:04:44 [INFO] [TRAIN] epoch=42, iter=27540/40000, loss=0.1647, lr=0.003501, batch_cost=0.7870, reader_cost=0.0001 | ETA 02:43:26 2021-01-18 16:04:52 [INFO] [TRAIN] epoch=42, iter=27550/40000, loss=0.1216, lr=0.003498, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:44:21 2021-01-18 16:04:59 [INFO] [TRAIN] epoch=42, iter=27560/40000, loss=0.1303, lr=0.003496, batch_cost=0.7906, reader_cost=0.0001 | ETA 02:43:55 2021-01-18 16:05:07 [INFO] [TRAIN] epoch=42, iter=27570/40000, loss=0.1856, lr=0.003493, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:43:31 2021-01-18 16:05:15 [INFO] [TRAIN] epoch=42, iter=27580/40000, loss=0.1411, lr=0.003490, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:42:59 2021-01-18 16:05:23 [INFO] [TRAIN] epoch=42, iter=27590/40000, loss=0.1750, lr=0.003488, batch_cost=0.7876, reader_cost=0.0001 | ETA 02:42:54 2021-01-18 16:05:31 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.2166, lr=0.003485, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:43:42 2021-01-18 16:05:39 [INFO] [TRAIN] epoch=42, iter=27610/40000, loss=0.1915, lr=0.003483, batch_cost=0.7936, reader_cost=0.0001 | ETA 02:43:53 2021-01-18 16:05:47 [INFO] [TRAIN] epoch=42, iter=27620/40000, loss=0.1615, lr=0.003480, batch_cost=0.7966, reader_cost=0.0001 | ETA 02:44:22 2021-01-18 16:05:55 [INFO] [TRAIN] epoch=42, iter=27630/40000, loss=0.1860, lr=0.003478, batch_cost=0.7887, reader_cost=0.0001 | ETA 02:42:36 2021-01-18 16:06:03 [INFO] [TRAIN] epoch=42, iter=27640/40000, loss=0.1784, lr=0.003475, batch_cost=0.7834, reader_cost=0.0001 | ETA 02:41:22 2021-01-18 16:06:11 [INFO] [TRAIN] epoch=42, iter=27650/40000, loss=0.1025, lr=0.003473, batch_cost=0.7941, reader_cost=0.0001 | ETA 02:43:27 2021-01-18 16:06:18 [INFO] [TRAIN] epoch=42, iter=27660/40000, loss=0.1616, lr=0.003470, batch_cost=0.7883, reader_cost=0.0001 | ETA 02:42:08 2021-01-18 16:06:26 [INFO] [TRAIN] epoch=42, iter=27670/40000, loss=0.2033, lr=0.003468, batch_cost=0.7931, reader_cost=0.0001 | ETA 02:42:58 2021-01-18 16:06:34 [INFO] [TRAIN] epoch=42, iter=27680/40000, loss=0.1429, lr=0.003465, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:42:16 2021-01-18 16:06:42 [INFO] [TRAIN] epoch=42, iter=27690/40000, loss=0.1720, lr=0.003463, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:42:19 2021-01-18 16:06:50 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.1855, lr=0.003460, batch_cost=0.7924, reader_cost=0.0001 | ETA 02:42:26 2021-01-18 16:06:58 [INFO] [TRAIN] epoch=42, iter=27710/40000, loss=0.2194, lr=0.003458, batch_cost=0.7871, reader_cost=0.0001 | ETA 02:41:13 2021-01-18 16:07:06 [INFO] [TRAIN] epoch=42, iter=27720/40000, loss=0.2032, lr=0.003455, batch_cost=0.7906, reader_cost=0.0001 | ETA 02:41:48 2021-01-18 16:07:14 [INFO] [TRAIN] epoch=42, iter=27730/40000, loss=0.2118, lr=0.003453, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:41:34 2021-01-18 16:07:22 [INFO] [TRAIN] epoch=42, iter=27740/40000, loss=0.1496, lr=0.003450, batch_cost=0.7977, reader_cost=0.0001 | ETA 02:43:00 2021-01-18 16:07:30 [INFO] [TRAIN] epoch=42, iter=27750/40000, loss=0.1409, lr=0.003447, batch_cost=0.7937, reader_cost=0.0001 | ETA 02:42:02 2021-01-18 16:07:37 [INFO] [TRAIN] epoch=42, iter=27760/40000, loss=0.1526, lr=0.003445, batch_cost=0.7705, reader_cost=0.0001 | ETA 02:37:11 2021-01-18 16:07:45 [INFO] [TRAIN] epoch=43, iter=27770/40000, loss=0.1357, lr=0.003442, batch_cost=0.7998, reader_cost=0.0119 | ETA 02:43:01 2021-01-18 16:07:53 [INFO] [TRAIN] epoch=43, iter=27780/40000, loss=0.1702, lr=0.003440, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:40:48 2021-01-18 16:08:01 [INFO] [TRAIN] epoch=43, iter=27790/40000, loss=0.1348, lr=0.003437, batch_cost=0.7880, reader_cost=0.0001 | ETA 02:40:21 2021-01-18 16:08:09 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.1517, lr=0.003435, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:41:05 2021-01-18 16:08:17 [INFO] [TRAIN] epoch=43, iter=27810/40000, loss=0.1379, lr=0.003432, batch_cost=0.7894, reader_cost=0.0001 | ETA 02:40:22 2021-01-18 16:08:25 [INFO] [TRAIN] epoch=43, iter=27820/40000, loss=0.1702, lr=0.003430, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:40:54 2021-01-18 16:08:33 [INFO] [TRAIN] epoch=43, iter=27830/40000, loss=0.1831, lr=0.003427, batch_cost=0.7838, reader_cost=0.0001 | ETA 02:38:59 2021-01-18 16:08:41 [INFO] [TRAIN] epoch=43, iter=27840/40000, loss=0.1715, lr=0.003425, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:40:38 2021-01-18 16:08:49 [INFO] [TRAIN] epoch=43, iter=27850/40000, loss=0.1535, lr=0.003422, batch_cost=0.7947, reader_cost=0.0001 | ETA 02:40:56 2021-01-18 16:08:57 [INFO] [TRAIN] epoch=43, iter=27860/40000, loss=0.1549, lr=0.003420, batch_cost=0.7914, reader_cost=0.0001 | ETA 02:40:07 2021-01-18 16:09:04 [INFO] [TRAIN] epoch=43, iter=27870/40000, loss=0.1615, lr=0.003417, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:39:53 2021-01-18 16:09:12 [INFO] [TRAIN] epoch=43, iter=27880/40000, loss=0.1556, lr=0.003415, batch_cost=0.7855, reader_cost=0.0001 | ETA 02:38:40 2021-01-18 16:09:20 [INFO] [TRAIN] epoch=43, iter=27890/40000, loss=0.1263, lr=0.003412, batch_cost=0.7942, reader_cost=0.0001 | ETA 02:40:18 2021-01-18 16:09:28 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.1061, lr=0.003409, batch_cost=0.7936, reader_cost=0.0001 | ETA 02:40:02 2021-01-18 16:09:36 [INFO] [TRAIN] epoch=43, iter=27910/40000, loss=0.1699, lr=0.003407, batch_cost=0.7870, reader_cost=0.0001 | ETA 02:38:35 2021-01-18 16:09:44 [INFO] [TRAIN] epoch=43, iter=27920/40000, loss=0.2047, lr=0.003404, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:38:31 2021-01-18 16:09:52 [INFO] [TRAIN] epoch=43, iter=27930/40000, loss=0.1153, lr=0.003402, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:38:49 2021-01-18 16:10:00 [INFO] [TRAIN] epoch=43, iter=27940/40000, loss=0.1362, lr=0.003399, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:38:41 2021-01-18 16:10:08 [INFO] [TRAIN] epoch=43, iter=27950/40000, loss=0.1358, lr=0.003397, batch_cost=0.7863, reader_cost=0.0001 | ETA 02:37:54 2021-01-18 16:10:16 [INFO] [TRAIN] epoch=43, iter=27960/40000, loss=0.1436, lr=0.003394, batch_cost=0.7958, reader_cost=0.0001 | ETA 02:39:41 2021-01-18 16:10:23 [INFO] [TRAIN] epoch=43, iter=27970/40000, loss=0.1419, lr=0.003392, batch_cost=0.7883, reader_cost=0.0001 | ETA 02:38:03 2021-01-18 16:10:31 [INFO] [TRAIN] epoch=43, iter=27980/40000, loss=0.1492, lr=0.003389, batch_cost=0.7958, reader_cost=0.0001 | ETA 02:39:25 2021-01-18 16:10:39 [INFO] [TRAIN] epoch=43, iter=27990/40000, loss=0.2023, lr=0.003387, batch_cost=0.7879, reader_cost=0.0001 | ETA 02:37:42 2021-01-18 16:10:47 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.1457, lr=0.003384, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:37:51 2021-01-18 16:10:47 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 16:11:15 [INFO] [EVAL] #Images=1449 mIoU=0.7876 Acc=0.9534 Kappa=0.8962 2021-01-18 16:11:15 [INFO] [EVAL] Class IoU: [0.9483 0.9106 0.4311 0.9095 0.6998 0.8612 0.9044 0.8787 0.9237 0.4707 0.8734 0.5258 0.8919 0.8858 0.8799 0.8832 0.6388 0.8444 0.5122 0.884 0.7829] 2021-01-18 16:11:15 [INFO] [EVAL] Class Acc: [0.9664 0.9343 0.4462 0.9552 0.8167 0.918 0.9578 0.9424 0.9635 0.6488 0.9244 0.9272 0.9483 0.9365 0.9603 0.9547 0.786 0.9564 0.836 0.9608 0.9474] 2021-01-18 16:11:21 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 16:11:29 [INFO] [TRAIN] epoch=43, iter=28010/40000, loss=0.1961, lr=0.003382, batch_cost=0.7824, reader_cost=0.0001 | ETA 02:36:21 2021-01-18 16:11:37 [INFO] [TRAIN] epoch=43, iter=28020/40000, loss=0.1518, lr=0.003379, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:37:58 2021-01-18 16:11:45 [INFO] [TRAIN] epoch=43, iter=28030/40000, loss=0.1519, lr=0.003376, batch_cost=0.7917, reader_cost=0.0001 | ETA 02:37:56 2021-01-18 16:11:53 [INFO] [TRAIN] epoch=43, iter=28040/40000, loss=0.1817, lr=0.003374, batch_cost=0.7955, reader_cost=0.0001 | ETA 02:38:33 2021-01-18 16:12:01 [INFO] [TRAIN] epoch=43, iter=28050/40000, loss=0.1740, lr=0.003371, batch_cost=0.7858, reader_cost=0.0001 | ETA 02:36:30 2021-01-18 16:12:09 [INFO] [TRAIN] epoch=43, iter=28060/40000, loss=0.2291, lr=0.003369, batch_cost=0.7967, reader_cost=0.0001 | ETA 02:38:33 2021-01-18 16:12:17 [INFO] [TRAIN] epoch=43, iter=28070/40000, loss=0.1530, lr=0.003366, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:37:10 2021-01-18 16:12:25 [INFO] [TRAIN] epoch=43, iter=28080/40000, loss=0.1458, lr=0.003364, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:37:02 2021-01-18 16:12:32 [INFO] [TRAIN] epoch=43, iter=28090/40000, loss=0.1907, lr=0.003361, batch_cost=0.7886, reader_cost=0.0001 | ETA 02:36:31 2021-01-18 16:12:40 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.1421, lr=0.003359, batch_cost=0.7882, reader_cost=0.0001 | ETA 02:36:19 2021-01-18 16:12:48 [INFO] [TRAIN] epoch=43, iter=28110/40000, loss=0.1191, lr=0.003356, batch_cost=0.7939, reader_cost=0.0001 | ETA 02:37:19 2021-01-18 16:12:56 [INFO] [TRAIN] epoch=43, iter=28120/40000, loss=0.2106, lr=0.003354, batch_cost=0.7941, reader_cost=0.0001 | ETA 02:37:13 2021-01-18 16:13:04 [INFO] [TRAIN] epoch=43, iter=28130/40000, loss=0.1336, lr=0.003351, batch_cost=0.7868, reader_cost=0.0001 | ETA 02:35:39 2021-01-18 16:13:12 [INFO] [TRAIN] epoch=43, iter=28140/40000, loss=0.1442, lr=0.003349, batch_cost=0.7864, reader_cost=0.0001 | ETA 02:35:27 2021-01-18 16:13:20 [INFO] [TRAIN] epoch=43, iter=28150/40000, loss=0.2281, lr=0.003346, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:35:43 2021-01-18 16:13:28 [INFO] [TRAIN] epoch=43, iter=28160/40000, loss=0.1819, lr=0.003343, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:35:48 2021-01-18 16:13:36 [INFO] [TRAIN] epoch=43, iter=28170/40000, loss=0.2155, lr=0.003341, batch_cost=0.7851, reader_cost=0.0001 | ETA 02:34:48 2021-01-18 16:13:44 [INFO] [TRAIN] epoch=43, iter=28180/40000, loss=0.1351, lr=0.003338, batch_cost=0.7962, reader_cost=0.0001 | ETA 02:36:50 2021-01-18 16:13:51 [INFO] [TRAIN] epoch=43, iter=28190/40000, loss=0.1767, lr=0.003336, batch_cost=0.7863, reader_cost=0.0001 | ETA 02:34:46 2021-01-18 16:13:59 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.2639, lr=0.003333, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:35:52 2021-01-18 16:14:07 [INFO] [TRAIN] epoch=43, iter=28210/40000, loss=0.2044, lr=0.003331, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:35:07 2021-01-18 16:14:15 [INFO] [TRAIN] epoch=43, iter=28220/40000, loss=0.1591, lr=0.003328, batch_cost=0.7945, reader_cost=0.0001 | ETA 02:35:58 2021-01-18 16:14:23 [INFO] [TRAIN] epoch=43, iter=28230/40000, loss=0.1924, lr=0.003326, batch_cost=0.8018, reader_cost=0.0001 | ETA 02:37:16 2021-01-18 16:14:31 [INFO] [TRAIN] epoch=43, iter=28240/40000, loss=0.1701, lr=0.003323, batch_cost=0.7914, reader_cost=0.0001 | ETA 02:35:07 2021-01-18 16:14:39 [INFO] [TRAIN] epoch=43, iter=28250/40000, loss=0.2122, lr=0.003321, batch_cost=0.7949, reader_cost=0.0001 | ETA 02:35:39 2021-01-18 16:14:47 [INFO] [TRAIN] epoch=43, iter=28260/40000, loss=0.1741, lr=0.003318, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:34:39 2021-01-18 16:14:55 [INFO] [TRAIN] epoch=43, iter=28270/40000, loss=0.1732, lr=0.003315, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:34:36 2021-01-18 16:15:03 [INFO] [TRAIN] epoch=43, iter=28280/40000, loss=0.1520, lr=0.003313, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:34:32 2021-01-18 16:15:11 [INFO] [TRAIN] epoch=43, iter=28290/40000, loss=0.2470, lr=0.003310, batch_cost=0.7890, reader_cost=0.0001 | ETA 02:33:58 2021-01-18 16:15:19 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.1317, lr=0.003308, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:33:54 2021-01-18 16:15:26 [INFO] [TRAIN] epoch=43, iter=28310/40000, loss=0.1149, lr=0.003305, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:34:07 2021-01-18 16:15:34 [INFO] [TRAIN] epoch=43, iter=28320/40000, loss=0.1838, lr=0.003303, batch_cost=0.7882, reader_cost=0.0001 | ETA 02:33:25 2021-01-18 16:15:42 [INFO] [TRAIN] epoch=43, iter=28330/40000, loss=0.1569, lr=0.003300, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:33:42 2021-01-18 16:15:50 [INFO] [TRAIN] epoch=43, iter=28340/40000, loss=0.1673, lr=0.003298, batch_cost=0.7937, reader_cost=0.0001 | ETA 02:34:15 2021-01-18 16:15:58 [INFO] [TRAIN] epoch=43, iter=28350/40000, loss=0.1202, lr=0.003295, batch_cost=0.7849, reader_cost=0.0001 | ETA 02:32:24 2021-01-18 16:16:06 [INFO] [TRAIN] epoch=43, iter=28360/40000, loss=0.1592, lr=0.003293, batch_cost=0.7969, reader_cost=0.0001 | ETA 02:34:36 2021-01-18 16:16:14 [INFO] [TRAIN] epoch=43, iter=28370/40000, loss=0.1777, lr=0.003290, batch_cost=0.7880, reader_cost=0.0001 | ETA 02:32:44 2021-01-18 16:16:22 [INFO] [TRAIN] epoch=43, iter=28380/40000, loss=0.1278, lr=0.003287, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:33:02 2021-01-18 16:16:30 [INFO] [TRAIN] epoch=43, iter=28390/40000, loss=0.1986, lr=0.003285, batch_cost=0.7839, reader_cost=0.0001 | ETA 02:31:41 2021-01-18 16:16:38 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.1647, lr=0.003282, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:33:04 2021-01-18 16:16:45 [INFO] [TRAIN] epoch=43, iter=28410/40000, loss=0.1812, lr=0.003280, batch_cost=0.7928, reader_cost=0.0001 | ETA 02:33:08 2021-01-18 16:16:53 [INFO] [TRAIN] epoch=43, iter=28420/40000, loss=0.1812, lr=0.003277, batch_cost=0.7762, reader_cost=0.0001 | ETA 02:29:48 2021-01-18 16:17:01 [INFO] [TRAIN] epoch=44, iter=28430/40000, loss=0.1747, lr=0.003275, batch_cost=0.7939, reader_cost=0.0144 | ETA 02:33:05 2021-01-18 16:17:09 [INFO] [TRAIN] epoch=44, iter=28440/40000, loss=0.1577, lr=0.003272, batch_cost=0.7859, reader_cost=0.0001 | ETA 02:31:25 2021-01-18 16:17:17 [INFO] [TRAIN] epoch=44, iter=28450/40000, loss=0.1788, lr=0.003270, batch_cost=0.7928, reader_cost=0.0001 | ETA 02:32:37 2021-01-18 16:17:25 [INFO] [TRAIN] epoch=44, iter=28460/40000, loss=0.1531, lr=0.003267, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:31:56 2021-01-18 16:17:33 [INFO] [TRAIN] epoch=44, iter=28470/40000, loss=0.1543, lr=0.003265, batch_cost=0.7891, reader_cost=0.0001 | ETA 02:31:37 2021-01-18 16:17:41 [INFO] [TRAIN] epoch=44, iter=28480/40000, loss=0.1888, lr=0.003262, batch_cost=0.7848, reader_cost=0.0001 | ETA 02:30:40 2021-01-18 16:17:49 [INFO] [TRAIN] epoch=44, iter=28490/40000, loss=0.1501, lr=0.003259, batch_cost=0.7926, reader_cost=0.0001 | ETA 02:32:02 2021-01-18 16:17:56 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.1586, lr=0.003257, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:31:47 2021-01-18 16:18:04 [INFO] [TRAIN] epoch=44, iter=28510/40000, loss=0.1235, lr=0.003254, batch_cost=0.7934, reader_cost=0.0001 | ETA 02:31:56 2021-01-18 16:18:12 [INFO] [TRAIN] epoch=44, iter=28520/40000, loss=0.1490, lr=0.003252, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:31:12 2021-01-18 16:18:20 [INFO] [TRAIN] epoch=44, iter=28530/40000, loss=0.1300, lr=0.003249, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:31:10 2021-01-18 16:18:28 [INFO] [TRAIN] epoch=44, iter=28540/40000, loss=0.1520, lr=0.003247, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:31:04 2021-01-18 16:18:36 [INFO] [TRAIN] epoch=44, iter=28550/40000, loss=0.1389, lr=0.003244, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:31:09 2021-01-18 16:18:44 [INFO] [TRAIN] epoch=44, iter=28560/40000, loss=0.1520, lr=0.003242, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:31:01 2021-01-18 16:18:52 [INFO] [TRAIN] epoch=44, iter=28570/40000, loss=0.1385, lr=0.003239, batch_cost=0.7882, reader_cost=0.0001 | ETA 02:30:08 2021-01-18 16:19:00 [INFO] [TRAIN] epoch=44, iter=28580/40000, loss=0.1455, lr=0.003237, batch_cost=0.7932, reader_cost=0.0001 | ETA 02:30:57 2021-01-18 16:19:08 [INFO] [TRAIN] epoch=44, iter=28590/40000, loss=0.1361, lr=0.003234, batch_cost=0.7924, reader_cost=0.0001 | ETA 02:30:41 2021-01-18 16:19:16 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.1793, lr=0.003231, batch_cost=0.7975, reader_cost=0.0001 | ETA 02:31:31 2021-01-18 16:19:24 [INFO] [TRAIN] epoch=44, iter=28610/40000, loss=0.1816, lr=0.003229, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:30:18 2021-01-18 16:19:32 [INFO] [TRAIN] epoch=44, iter=28620/40000, loss=0.1515, lr=0.003226, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:30:14 2021-01-18 16:19:39 [INFO] [TRAIN] epoch=44, iter=28630/40000, loss=0.1702, lr=0.003224, batch_cost=0.7837, reader_cost=0.0001 | ETA 02:28:30 2021-01-18 16:19:47 [INFO] [TRAIN] epoch=44, iter=28640/40000, loss=0.1589, lr=0.003221, batch_cost=0.7947, reader_cost=0.0001 | ETA 02:30:28 2021-01-18 16:19:55 [INFO] [TRAIN] epoch=44, iter=28650/40000, loss=0.1342, lr=0.003219, batch_cost=0.7873, reader_cost=0.0001 | ETA 02:28:56 2021-01-18 16:20:03 [INFO] [TRAIN] epoch=44, iter=28660/40000, loss=0.1309, lr=0.003216, batch_cost=0.7873, reader_cost=0.0001 | ETA 02:28:47 2021-01-18 16:20:11 [INFO] [TRAIN] epoch=44, iter=28670/40000, loss=0.1390, lr=0.003214, batch_cost=0.7934, reader_cost=0.0001 | ETA 02:29:48 2021-01-18 16:20:19 [INFO] [TRAIN] epoch=44, iter=28680/40000, loss=0.1314, lr=0.003211, batch_cost=0.7882, reader_cost=0.0001 | ETA 02:28:41 2021-01-18 16:20:27 [INFO] [TRAIN] epoch=44, iter=28690/40000, loss=0.1319, lr=0.003208, batch_cost=0.7948, reader_cost=0.0001 | ETA 02:29:49 2021-01-18 16:20:35 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.1290, lr=0.003206, batch_cost=0.7858, reader_cost=0.0001 | ETA 02:27:59 2021-01-18 16:20:43 [INFO] [TRAIN] epoch=44, iter=28710/40000, loss=0.1422, lr=0.003203, batch_cost=0.7866, reader_cost=0.0001 | ETA 02:28:00 2021-01-18 16:20:50 [INFO] [TRAIN] epoch=44, iter=28720/40000, loss=0.1658, lr=0.003201, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:28:13 2021-01-18 16:20:58 [INFO] [TRAIN] epoch=44, iter=28730/40000, loss=0.1274, lr=0.003198, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:28:45 2021-01-18 16:21:06 [INFO] [TRAIN] epoch=44, iter=28740/40000, loss=0.2141, lr=0.003196, batch_cost=0.7947, reader_cost=0.0001 | ETA 02:29:08 2021-01-18 16:21:14 [INFO] [TRAIN] epoch=44, iter=28750/40000, loss=0.0946, lr=0.003193, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:28:13 2021-01-18 16:21:22 [INFO] [TRAIN] epoch=44, iter=28760/40000, loss=0.1701, lr=0.003191, batch_cost=0.7936, reader_cost=0.0001 | ETA 02:28:40 2021-01-18 16:21:30 [INFO] [TRAIN] epoch=44, iter=28770/40000, loss=0.1619, lr=0.003188, batch_cost=0.7854, reader_cost=0.0001 | ETA 02:26:59 2021-01-18 16:21:38 [INFO] [TRAIN] epoch=44, iter=28780/40000, loss=0.1168, lr=0.003185, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:27:52 2021-01-18 16:21:46 [INFO] [TRAIN] epoch=44, iter=28790/40000, loss=0.3005, lr=0.003183, batch_cost=0.7924, reader_cost=0.0001 | ETA 02:28:02 2021-01-18 16:21:54 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.1778, lr=0.003180, batch_cost=0.7949, reader_cost=0.0001 | ETA 02:28:23 2021-01-18 16:22:02 [INFO] [TRAIN] epoch=44, iter=28810/40000, loss=0.1455, lr=0.003178, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:27:29 2021-01-18 16:22:10 [INFO] [TRAIN] epoch=44, iter=28820/40000, loss=0.1959, lr=0.003175, batch_cost=0.7897, reader_cost=0.0001 | ETA 02:27:08 2021-01-18 16:22:18 [INFO] [TRAIN] epoch=44, iter=28830/40000, loss=0.1549, lr=0.003173, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:27:17 2021-01-18 16:22:25 [INFO] [TRAIN] epoch=44, iter=28840/40000, loss=0.1407, lr=0.003170, batch_cost=0.7939, reader_cost=0.0001 | ETA 02:27:40 2021-01-18 16:22:33 [INFO] [TRAIN] epoch=44, iter=28850/40000, loss=0.1116, lr=0.003168, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:26:52 2021-01-18 16:22:41 [INFO] [TRAIN] epoch=44, iter=28860/40000, loss=0.1511, lr=0.003165, batch_cost=0.7927, reader_cost=0.0001 | ETA 02:27:10 2021-01-18 16:22:49 [INFO] [TRAIN] epoch=44, iter=28870/40000, loss=0.1434, lr=0.003162, batch_cost=0.7962, reader_cost=0.0001 | ETA 02:27:42 2021-01-18 16:22:57 [INFO] [TRAIN] epoch=44, iter=28880/40000, loss=0.1331, lr=0.003160, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:26:46 2021-01-18 16:23:05 [INFO] [TRAIN] epoch=44, iter=28890/40000, loss=0.1134, lr=0.003157, batch_cost=0.7863, reader_cost=0.0001 | ETA 02:25:36 2021-01-18 16:23:13 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.1749, lr=0.003155, batch_cost=0.7914, reader_cost=0.0001 | ETA 02:26:24 2021-01-18 16:23:21 [INFO] [TRAIN] epoch=44, iter=28910/40000, loss=0.1439, lr=0.003152, batch_cost=0.7886, reader_cost=0.0001 | ETA 02:25:45 2021-01-18 16:23:29 [INFO] [TRAIN] epoch=44, iter=28920/40000, loss=0.1555, lr=0.003150, batch_cost=0.7836, reader_cost=0.0001 | ETA 02:24:42 2021-01-18 16:23:37 [INFO] [TRAIN] epoch=44, iter=28930/40000, loss=0.1357, lr=0.003147, batch_cost=0.7944, reader_cost=0.0001 | ETA 02:26:34 2021-01-18 16:23:45 [INFO] [TRAIN] epoch=44, iter=28940/40000, loss=0.1435, lr=0.003145, batch_cost=0.7890, reader_cost=0.0001 | ETA 02:25:26 2021-01-18 16:23:52 [INFO] [TRAIN] epoch=44, iter=28950/40000, loss=0.1754, lr=0.003142, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:25:49 2021-01-18 16:24:00 [INFO] [TRAIN] epoch=44, iter=28960/40000, loss=0.1754, lr=0.003139, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:25:26 2021-01-18 16:24:08 [INFO] [TRAIN] epoch=44, iter=28970/40000, loss=0.1555, lr=0.003137, batch_cost=0.7945, reader_cost=0.0001 | ETA 02:26:03 2021-01-18 16:24:16 [INFO] [TRAIN] epoch=44, iter=28980/40000, loss=0.1467, lr=0.003134, batch_cost=0.7907, reader_cost=0.0001 | ETA 02:25:13 2021-01-18 16:24:24 [INFO] [TRAIN] epoch=44, iter=28990/40000, loss=0.1432, lr=0.003132, batch_cost=0.7929, reader_cost=0.0001 | ETA 02:25:29 2021-01-18 16:24:32 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.1690, lr=0.003129, batch_cost=0.7854, reader_cost=0.0001 | ETA 02:23:59 2021-01-18 16:24:32 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 16:25:00 [INFO] [EVAL] #Images=1449 mIoU=0.7796 Acc=0.9526 Kappa=0.8949 2021-01-18 16:25:00 [INFO] [EVAL] Class IoU: [0.9474 0.8998 0.4132 0.9038 0.6355 0.8351 0.8978 0.8852 0.935 0.4352 0.8531 0.5215 0.8953 0.8713 0.887 0.8818 0.5936 0.8409 0.5771 0.893 0.769 ] 2021-01-18 16:25:00 [INFO] [EVAL] Class Acc: [0.9677 0.9239 0.4258 0.9465 0.7341 0.9027 0.9868 0.9414 0.9585 0.66 0.9333 0.9412 0.9474 0.9218 0.9193 0.9491 0.8357 0.937 0.8484 0.9242 0.9192] 2021-01-18 16:25:06 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 16:25:14 [INFO] [TRAIN] epoch=44, iter=29010/40000, loss=0.1315, lr=0.003127, batch_cost=0.7865, reader_cost=0.0001 | ETA 02:24:03 2021-01-18 16:25:22 [INFO] [TRAIN] epoch=44, iter=29020/40000, loss=0.1457, lr=0.003124, batch_cost=0.7865, reader_cost=0.0001 | ETA 02:23:55 2021-01-18 16:25:30 [INFO] [TRAIN] epoch=44, iter=29030/40000, loss=0.1171, lr=0.003122, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:24:31 2021-01-18 16:25:38 [INFO] [TRAIN] epoch=44, iter=29040/40000, loss=0.1441, lr=0.003119, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:24:22 2021-01-18 16:25:46 [INFO] [TRAIN] epoch=44, iter=29050/40000, loss=0.1810, lr=0.003116, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:24:31 2021-01-18 16:25:54 [INFO] [TRAIN] epoch=44, iter=29060/40000, loss=0.1082, lr=0.003114, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:24:35 2021-01-18 16:26:02 [INFO] [TRAIN] epoch=44, iter=29070/40000, loss=0.1637, lr=0.003111, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:24:06 2021-01-18 16:26:09 [INFO] [TRAIN] epoch=44, iter=29080/40000, loss=0.1298, lr=0.003109, batch_cost=0.7854, reader_cost=0.0001 | ETA 02:22:56 2021-01-18 16:26:17 [INFO] [TRAIN] epoch=45, iter=29090/40000, loss=0.1720, lr=0.003106, batch_cost=0.7816, reader_cost=0.0151 | ETA 02:22:06 2021-01-18 16:26:25 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.1348, lr=0.003104, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:23:50 2021-01-18 16:26:33 [INFO] [TRAIN] epoch=45, iter=29110/40000, loss=0.1463, lr=0.003101, batch_cost=0.7937, reader_cost=0.0001 | ETA 02:24:03 2021-01-18 16:26:41 [INFO] [TRAIN] epoch=45, iter=29120/40000, loss=0.1701, lr=0.003098, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:23:37 2021-01-18 16:26:49 [INFO] [TRAIN] epoch=45, iter=29130/40000, loss=0.1650, lr=0.003096, batch_cost=0.7889, reader_cost=0.0001 | ETA 02:22:55 2021-01-18 16:26:57 [INFO] [TRAIN] epoch=45, iter=29140/40000, loss=0.1619, lr=0.003093, batch_cost=0.7869, reader_cost=0.0001 | ETA 02:22:25 2021-01-18 16:27:05 [INFO] [TRAIN] epoch=45, iter=29150/40000, loss=0.1707, lr=0.003091, batch_cost=0.7859, reader_cost=0.0001 | ETA 02:22:06 2021-01-18 16:27:12 [INFO] [TRAIN] epoch=45, iter=29160/40000, loss=0.1294, lr=0.003088, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:22:54 2021-01-18 16:27:20 [INFO] [TRAIN] epoch=45, iter=29170/40000, loss=0.1262, lr=0.003086, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:22:59 2021-01-18 16:27:28 [INFO] [TRAIN] epoch=45, iter=29180/40000, loss=0.1361, lr=0.003083, batch_cost=0.7897, reader_cost=0.0001 | ETA 02:22:24 2021-01-18 16:27:36 [INFO] [TRAIN] epoch=45, iter=29190/40000, loss=0.1223, lr=0.003081, batch_cost=0.7875, reader_cost=0.0001 | ETA 02:21:52 2021-01-18 16:27:44 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.2012, lr=0.003078, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:22:09 2021-01-18 16:27:52 [INFO] [TRAIN] epoch=45, iter=29210/40000, loss=0.1342, lr=0.003075, batch_cost=0.7946, reader_cost=0.0001 | ETA 02:22:53 2021-01-18 16:28:00 [INFO] [TRAIN] epoch=45, iter=29220/40000, loss=0.1227, lr=0.003073, batch_cost=0.7882, reader_cost=0.0001 | ETA 02:21:37 2021-01-18 16:28:08 [INFO] [TRAIN] epoch=45, iter=29230/40000, loss=0.1822, lr=0.003070, batch_cost=0.7856, reader_cost=0.0001 | ETA 02:21:01 2021-01-18 16:28:16 [INFO] [TRAIN] epoch=45, iter=29240/40000, loss=0.0886, lr=0.003068, batch_cost=0.7924, reader_cost=0.0001 | ETA 02:22:06 2021-01-18 16:28:24 [INFO] [TRAIN] epoch=45, iter=29250/40000, loss=0.1594, lr=0.003065, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:21:28 2021-01-18 16:28:32 [INFO] [TRAIN] epoch=45, iter=29260/40000, loss=0.1478, lr=0.003063, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:21:38 2021-01-18 16:28:39 [INFO] [TRAIN] epoch=45, iter=29270/40000, loss=0.1711, lr=0.003060, batch_cost=0.7875, reader_cost=0.0001 | ETA 02:20:49 2021-01-18 16:28:47 [INFO] [TRAIN] epoch=45, iter=29280/40000, loss=0.1537, lr=0.003057, batch_cost=0.7908, reader_cost=0.0001 | ETA 02:21:17 2021-01-18 16:28:55 [INFO] [TRAIN] epoch=45, iter=29290/40000, loss=0.1255, lr=0.003055, batch_cost=0.7929, reader_cost=0.0001 | ETA 02:21:32 2021-01-18 16:29:03 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.1455, lr=0.003052, batch_cost=0.7889, reader_cost=0.0001 | ETA 02:20:41 2021-01-18 16:29:11 [INFO] [TRAIN] epoch=45, iter=29310/40000, loss=0.1812, lr=0.003050, batch_cost=0.7857, reader_cost=0.0001 | ETA 02:19:59 2021-01-18 16:29:19 [INFO] [TRAIN] epoch=45, iter=29320/40000, loss=0.1099, lr=0.003047, batch_cost=0.7928, reader_cost=0.0001 | ETA 02:21:07 2021-01-18 16:29:27 [INFO] [TRAIN] epoch=45, iter=29330/40000, loss=0.1429, lr=0.003045, batch_cost=0.7949, reader_cost=0.0001 | ETA 02:21:21 2021-01-18 16:29:35 [INFO] [TRAIN] epoch=45, iter=29340/40000, loss=0.1272, lr=0.003042, batch_cost=0.7886, reader_cost=0.0001 | ETA 02:20:06 2021-01-18 16:29:43 [INFO] [TRAIN] epoch=45, iter=29350/40000, loss=0.1935, lr=0.003039, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:19:57 2021-01-18 16:29:51 [INFO] [TRAIN] epoch=45, iter=29360/40000, loss=0.1977, lr=0.003037, batch_cost=0.7929, reader_cost=0.0001 | ETA 02:20:36 2021-01-18 16:29:58 [INFO] [TRAIN] epoch=45, iter=29370/40000, loss=0.1798, lr=0.003034, batch_cost=0.7887, reader_cost=0.0001 | ETA 02:19:44 2021-01-18 16:30:06 [INFO] [TRAIN] epoch=45, iter=29380/40000, loss=0.1398, lr=0.003032, batch_cost=0.7860, reader_cost=0.0001 | ETA 02:19:07 2021-01-18 16:30:14 [INFO] [TRAIN] epoch=45, iter=29390/40000, loss=0.1414, lr=0.003029, batch_cost=0.7929, reader_cost=0.0001 | ETA 02:20:12 2021-01-18 16:30:22 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.1762, lr=0.003027, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:19:26 2021-01-18 16:30:30 [INFO] [TRAIN] epoch=45, iter=29410/40000, loss=0.1121, lr=0.003024, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:19:40 2021-01-18 16:30:38 [INFO] [TRAIN] epoch=45, iter=29420/40000, loss=0.1510, lr=0.003021, batch_cost=0.7895, reader_cost=0.0001 | ETA 02:19:13 2021-01-18 16:30:46 [INFO] [TRAIN] epoch=45, iter=29430/40000, loss=0.1205, lr=0.003019, batch_cost=0.7938, reader_cost=0.0001 | ETA 02:19:50 2021-01-18 16:30:54 [INFO] [TRAIN] epoch=45, iter=29440/40000, loss=0.1473, lr=0.003016, batch_cost=0.7951, reader_cost=0.0001 | ETA 02:19:56 2021-01-18 16:31:02 [INFO] [TRAIN] epoch=45, iter=29450/40000, loss=0.2129, lr=0.003014, batch_cost=0.7869, reader_cost=0.0001 | ETA 02:18:22 2021-01-18 16:31:10 [INFO] [TRAIN] epoch=45, iter=29460/40000, loss=0.1559, lr=0.003011, batch_cost=0.7932, reader_cost=0.0001 | ETA 02:19:19 2021-01-18 16:31:18 [INFO] [TRAIN] epoch=45, iter=29470/40000, loss=0.1688, lr=0.003009, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:18:39 2021-01-18 16:31:25 [INFO] [TRAIN] epoch=45, iter=29480/40000, loss=0.1298, lr=0.003006, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:18:50 2021-01-18 16:31:33 [INFO] [TRAIN] epoch=45, iter=29490/40000, loss=0.1764, lr=0.003003, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:18:35 2021-01-18 16:31:41 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0962, lr=0.003001, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:18:28 2021-01-18 16:31:49 [INFO] [TRAIN] epoch=45, iter=29510/40000, loss=0.1552, lr=0.002998, batch_cost=0.7925, reader_cost=0.0001 | ETA 02:18:33 2021-01-18 16:31:57 [INFO] [TRAIN] epoch=45, iter=29520/40000, loss=0.1984, lr=0.002996, batch_cost=0.7928, reader_cost=0.0001 | ETA 02:18:28 2021-01-18 16:32:05 [INFO] [TRAIN] epoch=45, iter=29530/40000, loss=0.1746, lr=0.002993, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:17:50 2021-01-18 16:32:13 [INFO] [TRAIN] epoch=45, iter=29540/40000, loss=0.1730, lr=0.002991, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:17:56 2021-01-18 16:32:21 [INFO] [TRAIN] epoch=45, iter=29550/40000, loss=0.1776, lr=0.002988, batch_cost=0.7897, reader_cost=0.0001 | ETA 02:17:32 2021-01-18 16:32:29 [INFO] [TRAIN] epoch=45, iter=29560/40000, loss=0.1322, lr=0.002985, batch_cost=0.7951, reader_cost=0.0001 | ETA 02:18:21 2021-01-18 16:32:37 [INFO] [TRAIN] epoch=45, iter=29570/40000, loss=0.1700, lr=0.002983, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:17:24 2021-01-18 16:32:45 [INFO] [TRAIN] epoch=45, iter=29580/40000, loss=0.1514, lr=0.002980, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:17:33 2021-01-18 16:32:52 [INFO] [TRAIN] epoch=45, iter=29590/40000, loss=0.1098, lr=0.002978, batch_cost=0.7863, reader_cost=0.0001 | ETA 02:16:24 2021-01-18 16:33:00 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.1299, lr=0.002975, batch_cost=0.8004, reader_cost=0.0001 | ETA 02:18:44 2021-01-18 16:33:08 [INFO] [TRAIN] epoch=45, iter=29610/40000, loss=0.1461, lr=0.002973, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:17:19 2021-01-18 16:33:16 [INFO] [TRAIN] epoch=45, iter=29620/40000, loss=0.1341, lr=0.002970, batch_cost=0.7854, reader_cost=0.0001 | ETA 02:15:52 2021-01-18 16:33:24 [INFO] [TRAIN] epoch=45, iter=29630/40000, loss=0.1377, lr=0.002967, batch_cost=0.7944, reader_cost=0.0001 | ETA 02:17:18 2021-01-18 16:33:32 [INFO] [TRAIN] epoch=45, iter=29640/40000, loss=0.1561, lr=0.002965, batch_cost=0.7889, reader_cost=0.0001 | ETA 02:16:13 2021-01-18 16:33:40 [INFO] [TRAIN] epoch=45, iter=29650/40000, loss=0.1408, lr=0.002962, batch_cost=0.7886, reader_cost=0.0001 | ETA 02:16:01 2021-01-18 16:33:48 [INFO] [TRAIN] epoch=45, iter=29660/40000, loss=0.1470, lr=0.002960, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:16:28 2021-01-18 16:33:56 [INFO] [TRAIN] epoch=45, iter=29670/40000, loss=0.1385, lr=0.002957, batch_cost=0.7872, reader_cost=0.0001 | ETA 02:15:31 2021-01-18 16:34:04 [INFO] [TRAIN] epoch=45, iter=29680/40000, loss=0.1128, lr=0.002955, batch_cost=0.7855, reader_cost=0.0001 | ETA 02:15:06 2021-01-18 16:34:12 [INFO] [TRAIN] epoch=45, iter=29690/40000, loss=0.1518, lr=0.002952, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:15:55 2021-01-18 16:34:19 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.1339, lr=0.002949, batch_cost=0.7891, reader_cost=0.0001 | ETA 02:15:27 2021-01-18 16:34:27 [INFO] [TRAIN] epoch=45, iter=29710/40000, loss=0.1670, lr=0.002947, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:15:27 2021-01-18 16:34:35 [INFO] [TRAIN] epoch=45, iter=29720/40000, loss=0.1269, lr=0.002944, batch_cost=0.7910, reader_cost=0.0001 | ETA 02:15:31 2021-01-18 16:34:43 [INFO] [TRAIN] epoch=45, iter=29730/40000, loss=0.1778, lr=0.002942, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:15:22 2021-01-18 16:34:51 [INFO] [TRAIN] epoch=45, iter=29740/40000, loss=0.1225, lr=0.002939, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:15:01 2021-01-18 16:34:59 [INFO] [TRAIN] epoch=46, iter=29750/40000, loss=0.1304, lr=0.002937, batch_cost=0.7706, reader_cost=0.0154 | ETA 02:11:38 2021-01-18 16:35:07 [INFO] [TRAIN] epoch=46, iter=29760/40000, loss=0.1746, lr=0.002934, batch_cost=0.7883, reader_cost=0.0001 | ETA 02:14:32 2021-01-18 16:35:14 [INFO] [TRAIN] epoch=46, iter=29770/40000, loss=0.1652, lr=0.002931, batch_cost=0.7828, reader_cost=0.0001 | ETA 02:13:27 2021-01-18 16:35:22 [INFO] [TRAIN] epoch=46, iter=29780/40000, loss=0.1246, lr=0.002929, batch_cost=0.7949, reader_cost=0.0001 | ETA 02:15:23 2021-01-18 16:35:30 [INFO] [TRAIN] epoch=46, iter=29790/40000, loss=0.0983, lr=0.002926, batch_cost=0.7870, reader_cost=0.0001 | ETA 02:13:54 2021-01-18 16:35:38 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.1382, lr=0.002924, batch_cost=0.7905, reader_cost=0.0001 | ETA 02:14:23 2021-01-18 16:35:46 [INFO] [TRAIN] epoch=46, iter=29810/40000, loss=0.2035, lr=0.002921, batch_cost=0.7841, reader_cost=0.0001 | ETA 02:13:09 2021-01-18 16:35:54 [INFO] [TRAIN] epoch=46, iter=29820/40000, loss=0.1942, lr=0.002918, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:13:45 2021-01-18 16:36:02 [INFO] [TRAIN] epoch=46, iter=29830/40000, loss=0.1948, lr=0.002916, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:13:54 2021-01-18 16:36:10 [INFO] [TRAIN] epoch=46, iter=29840/40000, loss=0.1337, lr=0.002913, batch_cost=0.7894, reader_cost=0.0001 | ETA 02:13:40 2021-01-18 16:36:18 [INFO] [TRAIN] epoch=46, iter=29850/40000, loss=0.1701, lr=0.002911, batch_cost=0.7947, reader_cost=0.0001 | ETA 02:14:26 2021-01-18 16:36:26 [INFO] [TRAIN] epoch=46, iter=29860/40000, loss=0.1367, lr=0.002908, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:13:43 2021-01-18 16:36:33 [INFO] [TRAIN] epoch=46, iter=29870/40000, loss=0.1823, lr=0.002906, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:13:20 2021-01-18 16:36:41 [INFO] [TRAIN] epoch=46, iter=29880/40000, loss=0.1514, lr=0.002903, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:13:37 2021-01-18 16:36:49 [INFO] [TRAIN] epoch=46, iter=29890/40000, loss=0.1110, lr=0.002900, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:12:50 2021-01-18 16:36:57 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.1291, lr=0.002898, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:13:20 2021-01-18 16:37:05 [INFO] [TRAIN] epoch=46, iter=29910/40000, loss=0.1133, lr=0.002895, batch_cost=0.7938, reader_cost=0.0001 | ETA 02:13:29 2021-01-18 16:37:13 [INFO] [TRAIN] epoch=46, iter=29920/40000, loss=0.1575, lr=0.002893, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:12:44 2021-01-18 16:37:21 [INFO] [TRAIN] epoch=46, iter=29930/40000, loss=0.1300, lr=0.002890, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:12:20 2021-01-18 16:37:29 [INFO] [TRAIN] epoch=46, iter=29940/40000, loss=0.1488, lr=0.002888, batch_cost=0.7891, reader_cost=0.0001 | ETA 02:12:18 2021-01-18 16:37:37 [INFO] [TRAIN] epoch=46, iter=29950/40000, loss=0.1097, lr=0.002885, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:12:17 2021-01-18 16:37:45 [INFO] [TRAIN] epoch=46, iter=29960/40000, loss=0.1656, lr=0.002882, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:12:12 2021-01-18 16:37:53 [INFO] [TRAIN] epoch=46, iter=29970/40000, loss=0.1547, lr=0.002880, batch_cost=0.7931, reader_cost=0.0001 | ETA 02:12:34 2021-01-18 16:38:00 [INFO] [TRAIN] epoch=46, iter=29980/40000, loss=0.1713, lr=0.002877, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:11:40 2021-01-18 16:38:08 [INFO] [TRAIN] epoch=46, iter=29990/40000, loss=0.1856, lr=0.002875, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:11:43 2021-01-18 16:38:16 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.1703, lr=0.002872, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:11:59 2021-01-18 16:38:16 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 16:38:44 [INFO] [EVAL] #Images=1449 mIoU=0.7843 Acc=0.9524 Kappa=0.8938 2021-01-18 16:38:44 [INFO] [EVAL] Class IoU: [0.946 0.8959 0.4312 0.9007 0.6884 0.8188 0.905 0.8755 0.926 0.4245 0.8778 0.5475 0.8903 0.889 0.8981 0.8847 0.6162 0.8733 0.4999 0.8951 0.7873] 2021-01-18 16:38:44 [INFO] [EVAL] Class Acc: [0.9642 0.9236 0.4476 0.9419 0.7621 0.9508 0.9824 0.9472 0.9602 0.7034 0.9618 0.899 0.9487 0.9533 0.9346 0.9591 0.8424 0.9426 0.7918 0.9326 0.9378] 2021-01-18 16:38:51 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 25000. 2021-01-18 16:38:58 [INFO] [TRAIN] epoch=46, iter=30010/40000, loss=0.1104, lr=0.002869, batch_cost=0.7872, reader_cost=0.0001 | ETA 02:11:04 2021-01-18 16:39:06 [INFO] [TRAIN] epoch=46, iter=30020/40000, loss=0.1361, lr=0.002867, batch_cost=0.7844, reader_cost=0.0001 | ETA 02:10:27 2021-01-18 16:39:14 [INFO] [TRAIN] epoch=46, iter=30030/40000, loss=0.1568, lr=0.002864, batch_cost=0.7893, reader_cost=0.0001 | ETA 02:11:09 2021-01-18 16:39:22 [INFO] [TRAIN] epoch=46, iter=30040/40000, loss=0.1878, lr=0.002862, batch_cost=0.7873, reader_cost=0.0001 | ETA 02:10:41 2021-01-18 16:39:30 [INFO] [TRAIN] epoch=46, iter=30050/40000, loss=0.1799, lr=0.002859, batch_cost=0.7849, reader_cost=0.0001 | ETA 02:10:09 2021-01-18 16:39:38 [INFO] [TRAIN] epoch=46, iter=30060/40000, loss=0.1680, lr=0.002856, batch_cost=0.7840, reader_cost=0.0001 | ETA 02:09:52 2021-01-18 16:39:46 [INFO] [TRAIN] epoch=46, iter=30070/40000, loss=0.1605, lr=0.002854, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:10:45 2021-01-18 16:39:54 [INFO] [TRAIN] epoch=46, iter=30080/40000, loss=0.1337, lr=0.002851, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:10:38 2021-01-18 16:40:01 [INFO] [TRAIN] epoch=46, iter=30090/40000, loss=0.0891, lr=0.002849, batch_cost=0.7925, reader_cost=0.0001 | ETA 02:10:53 2021-01-18 16:40:09 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.1504, lr=0.002846, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:10:06 2021-01-18 16:40:17 [INFO] [TRAIN] epoch=46, iter=30110/40000, loss=0.1212, lr=0.002844, batch_cost=0.7938, reader_cost=0.0001 | ETA 02:10:50 2021-01-18 16:40:25 [INFO] [TRAIN] epoch=46, iter=30120/40000, loss=0.1832, lr=0.002841, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:10:04 2021-01-18 16:40:33 [INFO] [TRAIN] epoch=46, iter=30130/40000, loss=0.1711, lr=0.002838, batch_cost=0.7936, reader_cost=0.0001 | ETA 02:10:32 2021-01-18 16:40:41 [INFO] [TRAIN] epoch=46, iter=30140/40000, loss=0.1369, lr=0.002836, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:09:52 2021-01-18 16:40:49 [INFO] [TRAIN] epoch=46, iter=30150/40000, loss=0.1315, lr=0.002833, batch_cost=0.7932, reader_cost=0.0001 | ETA 02:10:12 2021-01-18 16:40:57 [INFO] [TRAIN] epoch=46, iter=30160/40000, loss=0.1391, lr=0.002831, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:09:26 2021-01-18 16:41:05 [INFO] [TRAIN] epoch=46, iter=30170/40000, loss=0.1428, lr=0.002828, batch_cost=0.7904, reader_cost=0.0001 | ETA 02:09:29 2021-01-18 16:41:13 [INFO] [TRAIN] epoch=46, iter=30180/40000, loss=0.1631, lr=0.002825, batch_cost=0.7883, reader_cost=0.0001 | ETA 02:09:00 2021-01-18 16:41:21 [INFO] [TRAIN] epoch=46, iter=30190/40000, loss=0.1083, lr=0.002823, batch_cost=0.7906, reader_cost=0.0001 | ETA 02:09:16 2021-01-18 16:41:28 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.1346, lr=0.002820, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:08:58 2021-01-18 16:41:36 [INFO] [TRAIN] epoch=46, iter=30210/40000, loss=0.1717, lr=0.002818, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:09:12 2021-01-18 16:41:44 [INFO] [TRAIN] epoch=46, iter=30220/40000, loss=0.1640, lr=0.002815, batch_cost=0.7844, reader_cost=0.0001 | ETA 02:07:51 2021-01-18 16:41:52 [INFO] [TRAIN] epoch=46, iter=30230/40000, loss=0.1280, lr=0.002812, batch_cost=0.7919, reader_cost=0.0001 | ETA 02:08:57 2021-01-18 16:42:00 [INFO] [TRAIN] epoch=46, iter=30240/40000, loss=0.1229, lr=0.002810, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:09:00 2021-01-18 16:42:08 [INFO] [TRAIN] epoch=46, iter=30250/40000, loss=0.2018, lr=0.002807, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:08:06 2021-01-18 16:42:16 [INFO] [TRAIN] epoch=46, iter=30260/40000, loss=0.1991, lr=0.002805, batch_cost=0.7966, reader_cost=0.0001 | ETA 02:09:19 2021-01-18 16:42:24 [INFO] [TRAIN] epoch=46, iter=30270/40000, loss=0.1509, lr=0.002802, batch_cost=0.7884, reader_cost=0.0001 | ETA 02:07:50 2021-01-18 16:42:32 [INFO] [TRAIN] epoch=46, iter=30280/40000, loss=0.1358, lr=0.002800, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:07:58 2021-01-18 16:42:40 [INFO] [TRAIN] epoch=46, iter=30290/40000, loss=0.1795, lr=0.002797, batch_cost=0.7947, reader_cost=0.0001 | ETA 02:08:36 2021-01-18 16:42:48 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.1112, lr=0.002794, batch_cost=0.7901, reader_cost=0.0001 | ETA 02:07:43 2021-01-18 16:42:55 [INFO] [TRAIN] epoch=46, iter=30310/40000, loss=0.1232, lr=0.002792, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:07:54 2021-01-18 16:43:03 [INFO] [TRAIN] epoch=46, iter=30320/40000, loss=0.1425, lr=0.002789, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:07:30 2021-01-18 16:43:11 [INFO] [TRAIN] epoch=46, iter=30330/40000, loss=0.1251, lr=0.002787, batch_cost=0.7899, reader_cost=0.0001 | ETA 02:07:18 2021-01-18 16:43:19 [INFO] [TRAIN] epoch=46, iter=30340/40000, loss=0.1806, lr=0.002784, batch_cost=0.7921, reader_cost=0.0001 | ETA 02:07:31 2021-01-18 16:43:27 [INFO] [TRAIN] epoch=46, iter=30350/40000, loss=0.1159, lr=0.002781, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:06:48 2021-01-18 16:43:35 [INFO] [TRAIN] epoch=46, iter=30360/40000, loss=0.1275, lr=0.002779, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:07:04 2021-01-18 16:43:43 [INFO] [TRAIN] epoch=46, iter=30370/40000, loss=0.1622, lr=0.002776, batch_cost=0.7931, reader_cost=0.0001 | ETA 02:07:17 2021-01-18 16:43:51 [INFO] [TRAIN] epoch=46, iter=30380/40000, loss=0.1494, lr=0.002774, batch_cost=0.7881, reader_cost=0.0001 | ETA 02:06:21 2021-01-18 16:43:59 [INFO] [TRAIN] epoch=46, iter=30390/40000, loss=0.1572, lr=0.002771, batch_cost=0.7876, reader_cost=0.0001 | ETA 02:06:09 2021-01-18 16:44:07 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.1153, lr=0.002768, batch_cost=0.8006, reader_cost=0.0001 | ETA 02:08:05 2021-01-18 16:44:14 [INFO] [TRAIN] epoch=47, iter=30410/40000, loss=0.1187, lr=0.002766, batch_cost=0.7807, reader_cost=0.0131 | ETA 02:04:47 2021-01-18 16:44:22 [INFO] [TRAIN] epoch=47, iter=30420/40000, loss=0.2059, lr=0.002763, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:06:10 2021-01-18 16:44:30 [INFO] [TRAIN] epoch=47, iter=30430/40000, loss=0.1410, lr=0.002761, batch_cost=0.7922, reader_cost=0.0001 | ETA 02:06:21 2021-01-18 16:44:38 [INFO] [TRAIN] epoch=47, iter=30440/40000, loss=0.1272, lr=0.002758, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:06:09 2021-01-18 16:44:46 [INFO] [TRAIN] epoch=47, iter=30450/40000, loss=0.1310, lr=0.002755, batch_cost=0.7889, reader_cost=0.0001 | ETA 02:05:34 2021-01-18 16:44:54 [INFO] [TRAIN] epoch=47, iter=30460/40000, loss=0.1387, lr=0.002753, batch_cost=0.7885, reader_cost=0.0001 | ETA 02:05:22 2021-01-18 16:45:02 [INFO] [TRAIN] epoch=47, iter=30470/40000, loss=0.1729, lr=0.002750, batch_cost=0.7918, reader_cost=0.0001 | ETA 02:05:45 2021-01-18 16:45:10 [INFO] [TRAIN] epoch=47, iter=30480/40000, loss=0.1751, lr=0.002748, batch_cost=0.7898, reader_cost=0.0001 | ETA 02:05:18 2021-01-18 16:45:18 [INFO] [TRAIN] epoch=47, iter=30490/40000, loss=0.1436, lr=0.002745, batch_cost=0.7900, reader_cost=0.0001 | ETA 02:05:13 2021-01-18 16:45:26 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.1290, lr=0.002742, batch_cost=0.7911, reader_cost=0.0001 | ETA 02:05:15 2021-01-18 16:45:34 [INFO] [TRAIN] epoch=47, iter=30510/40000, loss=0.1303, lr=0.002740, batch_cost=0.7930, reader_cost=0.0001 | ETA 02:05:25 2021-01-18 16:45:41 [INFO] [TRAIN] epoch=47, iter=30520/40000, loss=0.1526, lr=0.002737, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:05:08 2021-01-18 16:45:49 [INFO] [TRAIN] epoch=47, iter=30530/40000, loss=0.1013, lr=0.002735, batch_cost=0.7931, reader_cost=0.0001 | ETA 02:05:10 2021-01-18 16:45:57 [INFO] [TRAIN] epoch=47, iter=30540/40000, loss=0.0946, lr=0.002732, batch_cost=0.7913, reader_cost=0.0001 | ETA 02:04:45 2021-01-18 16:46:05 [INFO] [TRAIN] epoch=47, iter=30550/40000, loss=0.1693, lr=0.002729, batch_cost=0.7964, reader_cost=0.0001 | ETA 02:05:26 2021-01-18 16:46:13 [INFO] [TRAIN] epoch=47, iter=30560/40000, loss=0.1230, lr=0.002727, batch_cost=0.7896, reader_cost=0.0001 | ETA 02:04:13 2021-01-18 16:46:21 [INFO] [TRAIN] epoch=47, iter=30570/40000, loss=0.1157, lr=0.002724, batch_cost=0.7878, reader_cost=0.0001 | ETA 02:03:49 2021-01-18 16:46:29 [INFO] [TRAIN] epoch=47, iter=30580/40000, loss=0.1512, lr=0.002722, batch_cost=0.7944, reader_cost=0.0001 | ETA 02:04:43 2021-01-18 16:46:37 [INFO] [TRAIN] epoch=47, iter=30590/40000, loss=0.0908, lr=0.002719, batch_cost=0.7946, reader_cost=0.0001 | ETA 02:04:36 2021-01-18 16:46:45 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.1264, lr=0.002716, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:03:49 2021-01-18 16:46:53 [INFO] [TRAIN] epoch=47, iter=30610/40000, loss=0.1022, lr=0.002714, batch_cost=0.7915, reader_cost=0.0001 | ETA 02:03:52 2021-01-18 16:47:01 [INFO] [TRAIN] epoch=47, iter=30620/40000, loss=0.1135, lr=0.002711, batch_cost=0.7915, reader_cost=0.0001 | ETA 02:03:44 2021-01-18 16:47:09 [INFO] [TRAIN] epoch=47, iter=30630/40000, loss=0.1419, lr=0.002709, batch_cost=0.7942, reader_cost=0.0001 | ETA 02:04:01 2021-01-18 16:47:17 [INFO] [TRAIN] epoch=47, iter=30640/40000, loss=0.2024, lr=0.002706, batch_cost=0.7917, reader_cost=0.0001 | ETA 02:03:30 2021-01-18 16:47:25 [INFO] [TRAIN] epoch=47, iter=30650/40000, loss=0.1621, lr=0.002703, batch_cost=0.7939, reader_cost=0.0001 | ETA 02:03:42 2021-01-18 16:47:32 [INFO] [TRAIN] epoch=47, iter=30660/40000, loss=0.1496, lr=0.002701, batch_cost=0.7890, reader_cost=0.0001 | ETA 02:02:48 2021-01-18 16:47:40 [INFO] [TRAIN] epoch=47, iter=30670/40000, loss=0.2020, lr=0.002698, batch_cost=0.7915, reader_cost=0.0001 | ETA 02:03:05 2021-01-18 16:47:48 [INFO] [TRAIN] epoch=47, iter=30680/40000, loss=0.0882, lr=0.002696, batch_cost=0.7923, reader_cost=0.0001 | ETA 02:03:04 2021-01-18 16:47:56 [INFO] [TRAIN] epoch=47, iter=30690/40000, loss=0.1363, lr=0.002693, batch_cost=0.7948, reader_cost=0.0001 | ETA 02:03:19 2021-01-18 16:48:04 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.1558, lr=0.002690, batch_cost=0.7965, reader_cost=0.0001 | ETA 02:03:27 2021-01-18 16:48:12 [INFO] [TRAIN] epoch=47, iter=30710/40000, loss=0.1373, lr=0.002688, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:02:11 2021-01-18 16:48:20 [INFO] [TRAIN] epoch=47, iter=30720/40000, loss=0.1204, lr=0.002685, batch_cost=0.7917, reader_cost=0.0001 | ETA 02:02:26 2021-01-18 16:48:28 [INFO] [TRAIN] epoch=47, iter=30730/40000, loss=0.1520, lr=0.002683, batch_cost=0.7899, reader_cost=0.0001 | ETA 02:02:02 2021-01-18 16:48:36 [INFO] [TRAIN] epoch=47, iter=30740/40000, loss=0.1346, lr=0.002680, batch_cost=0.7876, reader_cost=0.0001 | ETA 02:01:33 2021-01-18 16:48:44 [INFO] [TRAIN] epoch=47, iter=30750/40000, loss=0.1874, lr=0.002677, batch_cost=0.7946, reader_cost=0.0001 | ETA 02:02:29 2021-01-18 16:48:52 [INFO] [TRAIN] epoch=47, iter=30760/40000, loss=0.1463, lr=0.002675, batch_cost=0.7920, reader_cost=0.0001 | ETA 02:01:57 2021-01-18 16:48:59 [INFO] [TRAIN] epoch=47, iter=30770/40000, loss=0.1081, lr=0.002672, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:01:07 2021-01-18 16:49:07 [INFO] [TRAIN] epoch=47, iter=30780/40000, loss=0.1596, lr=0.002670, batch_cost=0.7927, reader_cost=0.0001 | ETA 02:01:48 2021-01-18 16:49:15 [INFO] [TRAIN] epoch=47, iter=30790/40000, loss=0.1138, lr=0.002667, batch_cost=0.7892, reader_cost=0.0001 | ETA 02:01:08 2021-01-18 16:49:23 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.1321, lr=0.002664, batch_cost=0.7903, reader_cost=0.0001 | ETA 02:01:10 2021-01-18 16:49:31 [INFO] [TRAIN] epoch=47, iter=30810/40000, loss=0.1447, lr=0.002662, batch_cost=0.7941, reader_cost=0.0001 | ETA 02:01:37 2021-01-18 16:49:39 [INFO] [TRAIN] epoch=47, iter=30820/40000, loss=0.1172, lr=0.002659, batch_cost=0.7889, reader_cost=0.0001 | ETA 02:00:41 2021-01-18 16:49:47 [INFO] [TRAIN] epoch=47, iter=30830/40000, loss=0.1229, lr=0.002657, batch_cost=0.7909, reader_cost=0.0001 | ETA 02:00:52 2021-01-18 16:49:55 [INFO] [TRAIN] epoch=47, iter=30840/40000, loss=0.1623, lr=0.002654, batch_cost=0.7912, reader_cost=0.0001 | ETA 02:00:47 2021-01-18 16:50:03 [INFO] [TRAIN] epoch=47, iter=30850/40000, loss=0.1582, lr=0.002651, batch_cost=0.7874, reader_cost=0.0001 | ETA 02:00:04 2021-01-18 16:50:11 [INFO] [TRAIN] epoch=47, iter=30860/40000, loss=0.1314, lr=0.002649, batch_cost=0.7940, reader_cost=0.0001 | ETA 02:00:56 2021-01-18 16:50:19 [INFO] [TRAIN] epoch=47, iter=30870/40000, loss=0.1402, lr=0.002646, batch_cost=0.7924, reader_cost=0.0001 | ETA 02:00:34 2021-01-18 16:50:26 [INFO] [TRAIN] epoch=47, iter=30880/40000, loss=0.1140, lr=0.002644, batch_cost=0.7849, reader_cost=0.0001 | ETA 01:59:18 2021-01-18 16:50:34 [INFO] [TRAIN] epoch=47, iter=30890/40000, loss=0.1229, lr=0.002641, batch_cost=0.7952, reader_cost=0.0001 | ETA 02:00:44 2021-01-18 16:50:42 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.1155, lr=0.002638, batch_cost=0.7936, reader_cost=0.0001 | ETA 02:00:21 2021-01-18 16:50:50 [INFO] [TRAIN] epoch=47, iter=30910/40000, loss=0.1559, lr=0.002636, batch_cost=0.7888, reader_cost=0.0001 | ETA 01:59:30 2021-01-18 16:50:58 [INFO] [TRAIN] epoch=47, iter=30920/40000, loss=0.1445, lr=0.002633, batch_cost=0.7904, reader_cost=0.0001 | ETA 01:59:37 2021-01-18 16:51:06 [INFO] [TRAIN] epoch=47, iter=30930/40000, loss=0.1294, lr=0.002630, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:58:55 2021-01-18 16:51:14 [INFO] [TRAIN] epoch=47, iter=30940/40000, loss=0.1076, lr=0.002628, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:59:09 2021-01-18 16:51:22 [INFO] [TRAIN] epoch=47, iter=30950/40000, loss=0.0998, lr=0.002625, batch_cost=0.7865, reader_cost=0.0001 | ETA 01:58:37 2021-01-18 16:51:30 [INFO] [TRAIN] epoch=47, iter=30960/40000, loss=0.1461, lr=0.002623, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:59:22 2021-01-18 16:51:38 [INFO] [TRAIN] epoch=47, iter=30970/40000, loss=0.0995, lr=0.002620, batch_cost=0.7893, reader_cost=0.0001 | ETA 01:58:46 2021-01-18 16:51:45 [INFO] [TRAIN] epoch=47, iter=30980/40000, loss=0.1242, lr=0.002617, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:59:06 2021-01-18 16:51:53 [INFO] [TRAIN] epoch=47, iter=30990/40000, loss=0.1412, lr=0.002615, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:58:59 2021-01-18 16:52:01 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.1497, lr=0.002612, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:58:59 2021-01-18 16:52:01 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 16:52:30 [INFO] [EVAL] #Images=1449 mIoU=0.8049 Acc=0.9569 Kappa=0.9048 2021-01-18 16:52:30 [INFO] [EVAL] Class IoU: [0.9513 0.9079 0.4282 0.9134 0.7014 0.8527 0.9379 0.8852 0.9318 0.423 0.8953 0.6317 0.8875 0.8965 0.884 0.8886 0.7099 0.8856 0.5817 0.9233 0.786 ] 2021-01-18 16:52:30 [INFO] [EVAL] Class Acc: [0.9715 0.9388 0.4431 0.9545 0.7813 0.9113 0.9868 0.9509 0.9576 0.6343 0.9483 0.877 0.9413 0.9517 0.9304 0.9524 0.8231 0.9542 0.8414 0.9617 0.9618] 2021-01-18 16:52:40 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 16:52:48 [INFO] [TRAIN] epoch=47, iter=31010/40000, loss=0.0861, lr=0.002610, batch_cost=0.7854, reader_cost=0.0001 | ETA 01:57:40 2021-01-18 16:52:56 [INFO] [TRAIN] epoch=47, iter=31020/40000, loss=0.1924, lr=0.002607, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:58:59 2021-01-18 16:53:04 [INFO] [TRAIN] epoch=47, iter=31030/40000, loss=0.1633, lr=0.002604, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:58:52 2021-01-18 16:53:12 [INFO] [TRAIN] epoch=47, iter=31040/40000, loss=0.1589, lr=0.002602, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:58:44 2021-01-18 16:53:20 [INFO] [TRAIN] epoch=47, iter=31050/40000, loss=0.1370, lr=0.002599, batch_cost=0.7882, reader_cost=0.0001 | ETA 01:57:34 2021-01-18 16:53:28 [INFO] [TRAIN] epoch=47, iter=31060/40000, loss=0.1188, lr=0.002597, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:57:43 2021-01-18 16:53:36 [INFO] [TRAIN] epoch=48, iter=31070/40000, loss=0.1390, lr=0.002594, batch_cost=0.7794, reader_cost=0.0166 | ETA 01:56:00 2021-01-18 16:53:44 [INFO] [TRAIN] epoch=48, iter=31080/40000, loss=0.1358, lr=0.002591, batch_cost=0.7865, reader_cost=0.0001 | ETA 01:56:55 2021-01-18 16:53:52 [INFO] [TRAIN] epoch=48, iter=31090/40000, loss=0.1708, lr=0.002589, batch_cost=0.7955, reader_cost=0.0001 | ETA 01:58:08 2021-01-18 16:53:59 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.2069, lr=0.002586, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:57:40 2021-01-18 16:54:07 [INFO] [TRAIN] epoch=48, iter=31110/40000, loss=0.1329, lr=0.002583, batch_cost=0.7960, reader_cost=0.0001 | ETA 01:57:56 2021-01-18 16:54:15 [INFO] [TRAIN] epoch=48, iter=31120/40000, loss=0.1020, lr=0.002581, batch_cost=0.7930, reader_cost=0.0001 | ETA 01:57:21 2021-01-18 16:54:23 [INFO] [TRAIN] epoch=48, iter=31130/40000, loss=0.1554, lr=0.002578, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:56:30 2021-01-18 16:54:31 [INFO] [TRAIN] epoch=48, iter=31140/40000, loss=0.1672, lr=0.002576, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:56:15 2021-01-18 16:54:39 [INFO] [TRAIN] epoch=48, iter=31150/40000, loss=0.1310, lr=0.002573, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:56:07 2021-01-18 16:54:47 [INFO] [TRAIN] epoch=48, iter=31160/40000, loss=0.0957, lr=0.002570, batch_cost=0.7877, reader_cost=0.0001 | ETA 01:56:03 2021-01-18 16:54:55 [INFO] [TRAIN] epoch=48, iter=31170/40000, loss=0.1495, lr=0.002568, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:55:54 2021-01-18 16:55:03 [INFO] [TRAIN] epoch=48, iter=31180/40000, loss=0.1156, lr=0.002565, batch_cost=0.7905, reader_cost=0.0001 | ETA 01:56:12 2021-01-18 16:55:10 [INFO] [TRAIN] epoch=48, iter=31190/40000, loss=0.1201, lr=0.002563, batch_cost=0.7816, reader_cost=0.0001 | ETA 01:54:45 2021-01-18 16:55:18 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.1263, lr=0.002560, batch_cost=0.7966, reader_cost=0.0001 | ETA 01:56:49 2021-01-18 16:55:26 [INFO] [TRAIN] epoch=48, iter=31210/40000, loss=0.1643, lr=0.002557, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:55:20 2021-01-18 16:55:34 [INFO] [TRAIN] epoch=48, iter=31220/40000, loss=0.2209, lr=0.002555, batch_cost=0.7877, reader_cost=0.0001 | ETA 01:55:15 2021-01-18 16:55:42 [INFO] [TRAIN] epoch=48, iter=31230/40000, loss=0.1269, lr=0.002552, batch_cost=0.7944, reader_cost=0.0001 | ETA 01:56:06 2021-01-18 16:55:50 [INFO] [TRAIN] epoch=48, iter=31240/40000, loss=0.0745, lr=0.002549, batch_cost=0.7854, reader_cost=0.0001 | ETA 01:54:40 2021-01-18 16:55:58 [INFO] [TRAIN] epoch=48, iter=31250/40000, loss=0.1476, lr=0.002547, batch_cost=0.7843, reader_cost=0.0001 | ETA 01:54:22 2021-01-18 16:56:06 [INFO] [TRAIN] epoch=48, iter=31260/40000, loss=0.1444, lr=0.002544, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:54:40 2021-01-18 16:56:14 [INFO] [TRAIN] epoch=48, iter=31270/40000, loss=0.1478, lr=0.002542, batch_cost=0.7977, reader_cost=0.0001 | ETA 01:56:04 2021-01-18 16:56:22 [INFO] [TRAIN] epoch=48, iter=31280/40000, loss=0.1346, lr=0.002539, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:54:40 2021-01-18 16:56:29 [INFO] [TRAIN] epoch=48, iter=31290/40000, loss=0.1257, lr=0.002536, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:54:23 2021-01-18 16:56:37 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.1570, lr=0.002534, batch_cost=0.7934, reader_cost=0.0001 | ETA 01:55:02 2021-01-18 16:56:45 [INFO] [TRAIN] epoch=48, iter=31310/40000, loss=0.1428, lr=0.002531, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:54:34 2021-01-18 16:56:53 [INFO] [TRAIN] epoch=48, iter=31320/40000, loss=0.1118, lr=0.002528, batch_cost=0.7946, reader_cost=0.0001 | ETA 01:54:56 2021-01-18 16:57:01 [INFO] [TRAIN] epoch=48, iter=31330/40000, loss=0.1370, lr=0.002526, batch_cost=0.7956, reader_cost=0.0001 | ETA 01:54:57 2021-01-18 16:57:09 [INFO] [TRAIN] epoch=48, iter=31340/40000, loss=0.1668, lr=0.002523, batch_cost=0.7860, reader_cost=0.0001 | ETA 01:53:26 2021-01-18 16:57:17 [INFO] [TRAIN] epoch=48, iter=31350/40000, loss=0.1501, lr=0.002521, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:54:04 2021-01-18 16:57:25 [INFO] [TRAIN] epoch=48, iter=31360/40000, loss=0.1551, lr=0.002518, batch_cost=0.7877, reader_cost=0.0001 | ETA 01:53:25 2021-01-18 16:57:33 [INFO] [TRAIN] epoch=48, iter=31370/40000, loss=0.1543, lr=0.002515, batch_cost=0.7879, reader_cost=0.0001 | ETA 01:53:19 2021-01-18 16:57:41 [INFO] [TRAIN] epoch=48, iter=31380/40000, loss=0.1422, lr=0.002513, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:53:58 2021-01-18 16:57:49 [INFO] [TRAIN] epoch=48, iter=31390/40000, loss=0.1289, lr=0.002510, batch_cost=0.7929, reader_cost=0.0001 | ETA 01:53:47 2021-01-18 16:57:56 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.1187, lr=0.002507, batch_cost=0.7860, reader_cost=0.0001 | ETA 01:52:39 2021-01-18 16:58:04 [INFO] [TRAIN] epoch=48, iter=31410/40000, loss=0.1606, lr=0.002505, batch_cost=0.7912, reader_cost=0.0001 | ETA 01:53:16 2021-01-18 16:58:12 [INFO] [TRAIN] epoch=48, iter=31420/40000, loss=0.1066, lr=0.002502, batch_cost=0.7854, reader_cost=0.0001 | ETA 01:52:18 2021-01-18 16:58:20 [INFO] [TRAIN] epoch=48, iter=31430/40000, loss=0.1428, lr=0.002500, batch_cost=0.7943, reader_cost=0.0001 | ETA 01:53:26 2021-01-18 16:58:28 [INFO] [TRAIN] epoch=48, iter=31440/40000, loss=0.1626, lr=0.002497, batch_cost=0.7857, reader_cost=0.0001 | ETA 01:52:05 2021-01-18 16:58:36 [INFO] [TRAIN] epoch=48, iter=31450/40000, loss=0.1211, lr=0.002494, batch_cost=0.7883, reader_cost=0.0001 | ETA 01:52:20 2021-01-18 16:58:44 [INFO] [TRAIN] epoch=48, iter=31460/40000, loss=0.1173, lr=0.002492, batch_cost=0.7890, reader_cost=0.0001 | ETA 01:52:18 2021-01-18 16:58:52 [INFO] [TRAIN] epoch=48, iter=31470/40000, loss=0.1504, lr=0.002489, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:52:41 2021-01-18 16:59:00 [INFO] [TRAIN] epoch=48, iter=31480/40000, loss=0.1157, lr=0.002486, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:52:17 2021-01-18 16:59:08 [INFO] [TRAIN] epoch=48, iter=31490/40000, loss=0.1680, lr=0.002484, batch_cost=0.7916, reader_cost=0.0001 | ETA 01:52:16 2021-01-18 16:59:15 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.1249, lr=0.002481, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:51:57 2021-01-18 16:59:23 [INFO] [TRAIN] epoch=48, iter=31510/40000, loss=0.1256, lr=0.002479, batch_cost=0.7855, reader_cost=0.0001 | ETA 01:51:09 2021-01-18 16:59:31 [INFO] [TRAIN] epoch=48, iter=31520/40000, loss=0.1214, lr=0.002476, batch_cost=0.7890, reader_cost=0.0001 | ETA 01:51:30 2021-01-18 16:59:39 [INFO] [TRAIN] epoch=48, iter=31530/40000, loss=0.1130, lr=0.002473, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:51:42 2021-01-18 16:59:47 [INFO] [TRAIN] epoch=48, iter=31540/40000, loss=0.1145, lr=0.002471, batch_cost=0.7905, reader_cost=0.0001 | ETA 01:51:27 2021-01-18 16:59:55 [INFO] [TRAIN] epoch=48, iter=31550/40000, loss=0.1362, lr=0.002468, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:50:58 2021-01-18 17:00:03 [INFO] [TRAIN] epoch=48, iter=31560/40000, loss=0.1095, lr=0.002465, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:50:51 2021-01-18 17:00:11 [INFO] [TRAIN] epoch=48, iter=31570/40000, loss=0.1312, lr=0.002463, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:50:39 2021-01-18 17:00:18 [INFO] [TRAIN] epoch=48, iter=31580/40000, loss=0.1297, lr=0.002460, batch_cost=0.7850, reader_cost=0.0001 | ETA 01:50:09 2021-01-18 17:00:26 [INFO] [TRAIN] epoch=48, iter=31590/40000, loss=0.1178, lr=0.002458, batch_cost=0.7939, reader_cost=0.0001 | ETA 01:51:16 2021-01-18 17:00:34 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.1463, lr=0.002455, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:50:50 2021-01-18 17:00:42 [INFO] [TRAIN] epoch=48, iter=31610/40000, loss=0.1199, lr=0.002452, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:50:26 2021-01-18 17:00:50 [INFO] [TRAIN] epoch=48, iter=31620/40000, loss=0.1737, lr=0.002450, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:50:35 2021-01-18 17:00:58 [INFO] [TRAIN] epoch=48, iter=31630/40000, loss=0.1341, lr=0.002447, batch_cost=0.7995, reader_cost=0.0001 | ETA 01:51:31 2021-01-18 17:01:06 [INFO] [TRAIN] epoch=48, iter=31640/40000, loss=0.1310, lr=0.002444, batch_cost=0.7905, reader_cost=0.0001 | ETA 01:50:08 2021-01-18 17:01:14 [INFO] [TRAIN] epoch=48, iter=31650/40000, loss=0.0984, lr=0.002442, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:50:10 2021-01-18 17:01:22 [INFO] [TRAIN] epoch=48, iter=31660/40000, loss=0.1433, lr=0.002439, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:50:09 2021-01-18 17:01:30 [INFO] [TRAIN] epoch=48, iter=31670/40000, loss=0.1115, lr=0.002437, batch_cost=0.7916, reader_cost=0.0001 | ETA 01:49:54 2021-01-18 17:01:38 [INFO] [TRAIN] epoch=48, iter=31680/40000, loss=0.1600, lr=0.002434, batch_cost=0.7934, reader_cost=0.0001 | ETA 01:50:01 2021-01-18 17:01:46 [INFO] [TRAIN] epoch=48, iter=31690/40000, loss=0.2025, lr=0.002431, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:49:18 2021-01-18 17:01:54 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.1162, lr=0.002429, batch_cost=0.7904, reader_cost=0.0001 | ETA 01:49:20 2021-01-18 17:02:01 [INFO] [TRAIN] epoch=48, iter=31710/40000, loss=0.1314, lr=0.002426, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:49:16 2021-01-18 17:02:09 [INFO] [TRAIN] epoch=48, iter=31720/40000, loss=0.1907, lr=0.002423, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:49:17 2021-01-18 17:02:17 [INFO] [TRAIN] epoch=49, iter=31730/40000, loss=0.1428, lr=0.002421, batch_cost=0.7772, reader_cost=0.0167 | ETA 01:47:07 2021-01-18 17:02:25 [INFO] [TRAIN] epoch=49, iter=31740/40000, loss=0.1382, lr=0.002418, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:48:52 2021-01-18 17:02:33 [INFO] [TRAIN] epoch=49, iter=31750/40000, loss=0.1396, lr=0.002415, batch_cost=0.7872, reader_cost=0.0001 | ETA 01:48:14 2021-01-18 17:02:41 [INFO] [TRAIN] epoch=49, iter=31760/40000, loss=0.1093, lr=0.002413, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:48:13 2021-01-18 17:02:49 [INFO] [TRAIN] epoch=49, iter=31770/40000, loss=0.1384, lr=0.002410, batch_cost=0.7871, reader_cost=0.0001 | ETA 01:47:57 2021-01-18 17:02:57 [INFO] [TRAIN] epoch=49, iter=31780/40000, loss=0.1291, lr=0.002408, batch_cost=0.7949, reader_cost=0.0001 | ETA 01:48:54 2021-01-18 17:03:05 [INFO] [TRAIN] epoch=49, iter=31790/40000, loss=0.1267, lr=0.002405, batch_cost=0.7884, reader_cost=0.0001 | ETA 01:47:52 2021-01-18 17:03:12 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.1066, lr=0.002402, batch_cost=0.7828, reader_cost=0.0001 | ETA 01:46:59 2021-01-18 17:03:20 [INFO] [TRAIN] epoch=49, iter=31810/40000, loss=0.1271, lr=0.002400, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:48:00 2021-01-18 17:03:28 [INFO] [TRAIN] epoch=49, iter=31820/40000, loss=0.1148, lr=0.002397, batch_cost=0.7915, reader_cost=0.0001 | ETA 01:47:54 2021-01-18 17:03:36 [INFO] [TRAIN] epoch=49, iter=31830/40000, loss=0.1235, lr=0.002394, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:47:14 2021-01-18 17:03:44 [INFO] [TRAIN] epoch=49, iter=31840/40000, loss=0.0911, lr=0.002392, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:47:31 2021-01-18 17:03:52 [INFO] [TRAIN] epoch=49, iter=31850/40000, loss=0.1022, lr=0.002389, batch_cost=0.7900, reader_cost=0.0001 | ETA 01:47:18 2021-01-18 17:04:00 [INFO] [TRAIN] epoch=49, iter=31860/40000, loss=0.1108, lr=0.002386, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:46:44 2021-01-18 17:04:08 [INFO] [TRAIN] epoch=49, iter=31870/40000, loss=0.1937, lr=0.002384, batch_cost=0.7860, reader_cost=0.0001 | ETA 01:46:30 2021-01-18 17:04:15 [INFO] [TRAIN] epoch=49, iter=31880/40000, loss=0.1690, lr=0.002381, batch_cost=0.7893, reader_cost=0.0001 | ETA 01:46:48 2021-01-18 17:04:23 [INFO] [TRAIN] epoch=49, iter=31890/40000, loss=0.1040, lr=0.002379, batch_cost=0.7945, reader_cost=0.0001 | ETA 01:47:23 2021-01-18 17:04:31 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0988, lr=0.002376, batch_cost=0.7958, reader_cost=0.0001 | ETA 01:47:26 2021-01-18 17:04:39 [INFO] [TRAIN] epoch=49, iter=31910/40000, loss=0.1188, lr=0.002373, batch_cost=0.7902, reader_cost=0.0001 | ETA 01:46:32 2021-01-18 17:04:47 [INFO] [TRAIN] epoch=49, iter=31920/40000, loss=0.1062, lr=0.002371, batch_cost=0.7872, reader_cost=0.0001 | ETA 01:46:00 2021-01-18 17:04:55 [INFO] [TRAIN] epoch=49, iter=31930/40000, loss=0.1674, lr=0.002368, batch_cost=0.7963, reader_cost=0.0001 | ETA 01:47:06 2021-01-18 17:05:03 [INFO] [TRAIN] epoch=49, iter=31940/40000, loss=0.1383, lr=0.002365, batch_cost=0.7940, reader_cost=0.0001 | ETA 01:46:39 2021-01-18 17:05:11 [INFO] [TRAIN] epoch=49, iter=31950/40000, loss=0.0900, lr=0.002363, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:46:01 2021-01-18 17:05:19 [INFO] [TRAIN] epoch=49, iter=31960/40000, loss=0.1173, lr=0.002360, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:46:18 2021-01-18 17:05:27 [INFO] [TRAIN] epoch=49, iter=31970/40000, loss=0.1593, lr=0.002357, batch_cost=0.7916, reader_cost=0.0001 | ETA 01:45:56 2021-01-18 17:05:35 [INFO] [TRAIN] epoch=49, iter=31980/40000, loss=0.1203, lr=0.002355, batch_cost=0.7920, reader_cost=0.0001 | ETA 01:45:52 2021-01-18 17:05:43 [INFO] [TRAIN] epoch=49, iter=31990/40000, loss=0.1337, lr=0.002352, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:45:33 2021-01-18 17:05:51 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.1057, lr=0.002350, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:45:04 2021-01-18 17:05:51 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 17:06:18 [INFO] [EVAL] #Images=1449 mIoU=0.7990 Acc=0.9562 Kappa=0.9031 2021-01-18 17:06:18 [INFO] [EVAL] Class IoU: [0.9508 0.9098 0.4279 0.9216 0.6943 0.8602 0.9202 0.891 0.9279 0.4642 0.8815 0.6147 0.8788 0.8986 0.8829 0.8911 0.684 0.8512 0.5679 0.8951 0.7648] 2021-01-18 17:06:18 [INFO] [EVAL] Class Acc: [0.9705 0.9363 0.4453 0.9667 0.7843 0.9139 0.9551 0.9406 0.9557 0.7193 0.929 0.9031 0.942 0.9511 0.9223 0.9589 0.7989 0.9569 0.8529 0.9565 0.8843] 2021-01-18 17:06:25 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 17:06:33 [INFO] [TRAIN] epoch=49, iter=32010/40000, loss=0.1170, lr=0.002347, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:44:46 2021-01-18 17:06:41 [INFO] [TRAIN] epoch=49, iter=32020/40000, loss=0.0978, lr=0.002344, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:45:14 2021-01-18 17:06:49 [INFO] [TRAIN] epoch=49, iter=32030/40000, loss=0.1354, lr=0.002342, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:44:49 2021-01-18 17:06:56 [INFO] [TRAIN] epoch=49, iter=32040/40000, loss=0.1683, lr=0.002339, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:45:08 2021-01-18 17:07:04 [INFO] [TRAIN] epoch=49, iter=32050/40000, loss=0.1441, lr=0.002336, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:44:39 2021-01-18 17:07:12 [INFO] [TRAIN] epoch=49, iter=32060/40000, loss=0.1598, lr=0.002334, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:44:40 2021-01-18 17:07:20 [INFO] [TRAIN] epoch=49, iter=32070/40000, loss=0.1451, lr=0.002331, batch_cost=0.7878, reader_cost=0.0001 | ETA 01:44:07 2021-01-18 17:07:28 [INFO] [TRAIN] epoch=49, iter=32080/40000, loss=0.1982, lr=0.002328, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:44:31 2021-01-18 17:07:36 [INFO] [TRAIN] epoch=49, iter=32090/40000, loss=0.1214, lr=0.002326, batch_cost=0.7910, reader_cost=0.0001 | ETA 01:44:17 2021-01-18 17:07:44 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.1215, lr=0.002323, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:44:14 2021-01-18 17:07:52 [INFO] [TRAIN] epoch=49, iter=32110/40000, loss=0.1386, lr=0.002320, batch_cost=0.7885, reader_cost=0.0001 | ETA 01:43:41 2021-01-18 17:08:00 [INFO] [TRAIN] epoch=49, iter=32120/40000, loss=0.1128, lr=0.002318, batch_cost=0.7866, reader_cost=0.0001 | ETA 01:43:18 2021-01-18 17:08:08 [INFO] [TRAIN] epoch=49, iter=32130/40000, loss=0.1579, lr=0.002315, batch_cost=0.7895, reader_cost=0.0001 | ETA 01:43:33 2021-01-18 17:08:15 [INFO] [TRAIN] epoch=49, iter=32140/40000, loss=0.1153, lr=0.002312, batch_cost=0.7946, reader_cost=0.0001 | ETA 01:44:05 2021-01-18 17:08:23 [INFO] [TRAIN] epoch=49, iter=32150/40000, loss=0.1669, lr=0.002310, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:44:01 2021-01-18 17:08:31 [INFO] [TRAIN] epoch=49, iter=32160/40000, loss=0.0961, lr=0.002307, batch_cost=0.7897, reader_cost=0.0001 | ETA 01:43:11 2021-01-18 17:08:39 [INFO] [TRAIN] epoch=49, iter=32170/40000, loss=0.1178, lr=0.002305, batch_cost=0.7916, reader_cost=0.0001 | ETA 01:43:18 2021-01-18 17:08:47 [INFO] [TRAIN] epoch=49, iter=32180/40000, loss=0.1535, lr=0.002302, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:43:07 2021-01-18 17:08:55 [INFO] [TRAIN] epoch=49, iter=32190/40000, loss=0.1078, lr=0.002299, batch_cost=0.7941, reader_cost=0.0001 | ETA 01:43:21 2021-01-18 17:09:03 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.1073, lr=0.002297, batch_cost=0.7924, reader_cost=0.0001 | ETA 01:43:00 2021-01-18 17:09:11 [INFO] [TRAIN] epoch=49, iter=32210/40000, loss=0.1469, lr=0.002294, batch_cost=0.7922, reader_cost=0.0001 | ETA 01:42:51 2021-01-18 17:09:19 [INFO] [TRAIN] epoch=49, iter=32220/40000, loss=0.1629, lr=0.002291, batch_cost=0.7899, reader_cost=0.0001 | ETA 01:42:25 2021-01-18 17:09:27 [INFO] [TRAIN] epoch=49, iter=32230/40000, loss=0.1526, lr=0.002289, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:42:26 2021-01-18 17:09:35 [INFO] [TRAIN] epoch=49, iter=32240/40000, loss=0.1511, lr=0.002286, batch_cost=0.7885, reader_cost=0.0001 | ETA 01:41:58 2021-01-18 17:09:43 [INFO] [TRAIN] epoch=49, iter=32250/40000, loss=0.1377, lr=0.002283, batch_cost=0.7916, reader_cost=0.0001 | ETA 01:42:15 2021-01-18 17:09:50 [INFO] [TRAIN] epoch=49, iter=32260/40000, loss=0.1370, lr=0.002281, batch_cost=0.7888, reader_cost=0.0001 | ETA 01:41:45 2021-01-18 17:09:58 [INFO] [TRAIN] epoch=49, iter=32270/40000, loss=0.2063, lr=0.002278, batch_cost=0.7922, reader_cost=0.0001 | ETA 01:42:03 2021-01-18 17:10:06 [INFO] [TRAIN] epoch=49, iter=32280/40000, loss=0.0955, lr=0.002275, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:41:39 2021-01-18 17:10:14 [INFO] [TRAIN] epoch=49, iter=32290/40000, loss=0.1319, lr=0.002273, batch_cost=0.7952, reader_cost=0.0001 | ETA 01:42:11 2021-01-18 17:10:22 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.1012, lr=0.002270, batch_cost=0.7864, reader_cost=0.0001 | ETA 01:40:55 2021-01-18 17:10:30 [INFO] [TRAIN] epoch=49, iter=32310/40000, loss=0.1173, lr=0.002267, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:40:59 2021-01-18 17:10:38 [INFO] [TRAIN] epoch=49, iter=32320/40000, loss=0.1842, lr=0.002265, batch_cost=0.7945, reader_cost=0.0001 | ETA 01:41:41 2021-01-18 17:10:46 [INFO] [TRAIN] epoch=49, iter=32330/40000, loss=0.1528, lr=0.002262, batch_cost=0.7930, reader_cost=0.0001 | ETA 01:41:22 2021-01-18 17:10:54 [INFO] [TRAIN] epoch=49, iter=32340/40000, loss=0.1498, lr=0.002259, batch_cost=0.7980, reader_cost=0.0001 | ETA 01:41:52 2021-01-18 17:11:02 [INFO] [TRAIN] epoch=49, iter=32350/40000, loss=0.1580, lr=0.002257, batch_cost=0.7945, reader_cost=0.0001 | ETA 01:41:17 2021-01-18 17:11:10 [INFO] [TRAIN] epoch=49, iter=32360/40000, loss=0.1270, lr=0.002254, batch_cost=0.7870, reader_cost=0.0001 | ETA 01:40:12 2021-01-18 17:11:18 [INFO] [TRAIN] epoch=49, iter=32370/40000, loss=0.1729, lr=0.002251, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:40:53 2021-01-18 17:11:25 [INFO] [TRAIN] epoch=49, iter=32380/40000, loss=0.1358, lr=0.002249, batch_cost=0.7882, reader_cost=0.0001 | ETA 01:40:05 2021-01-18 17:11:33 [INFO] [TRAIN] epoch=50, iter=32390/40000, loss=0.1626, lr=0.002246, batch_cost=0.7784, reader_cost=0.0182 | ETA 01:38:43 2021-01-18 17:11:41 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0970, lr=0.002244, batch_cost=0.7894, reader_cost=0.0001 | ETA 01:39:59 2021-01-18 17:11:49 [INFO] [TRAIN] epoch=50, iter=32410/40000, loss=0.1339, lr=0.002241, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:40:19 2021-01-18 17:11:57 [INFO] [TRAIN] epoch=50, iter=32420/40000, loss=0.1114, lr=0.002238, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:40:05 2021-01-18 17:12:05 [INFO] [TRAIN] epoch=50, iter=32430/40000, loss=0.1070, lr=0.002236, batch_cost=0.7943, reader_cost=0.0001 | ETA 01:40:12 2021-01-18 17:12:13 [INFO] [TRAIN] epoch=50, iter=32440/40000, loss=0.1368, lr=0.002233, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:39:52 2021-01-18 17:12:21 [INFO] [TRAIN] epoch=50, iter=32450/40000, loss=0.1258, lr=0.002230, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:39:38 2021-01-18 17:12:29 [INFO] [TRAIN] epoch=50, iter=32460/40000, loss=0.1196, lr=0.002228, batch_cost=0.7894, reader_cost=0.0001 | ETA 01:39:11 2021-01-18 17:12:37 [INFO] [TRAIN] epoch=50, iter=32470/40000, loss=0.1494, lr=0.002225, batch_cost=0.7861, reader_cost=0.0001 | ETA 01:38:39 2021-01-18 17:12:44 [INFO] [TRAIN] epoch=50, iter=32480/40000, loss=0.1384, lr=0.002222, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:39:14 2021-01-18 17:12:52 [INFO] [TRAIN] epoch=50, iter=32490/40000, loss=0.1153, lr=0.002220, batch_cost=0.7912, reader_cost=0.0001 | ETA 01:39:02 2021-01-18 17:13:00 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.1198, lr=0.002217, batch_cost=0.7874, reader_cost=0.0001 | ETA 01:38:25 2021-01-18 17:13:08 [INFO] [TRAIN] epoch=50, iter=32510/40000, loss=0.1274, lr=0.002214, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:38:57 2021-01-18 17:13:16 [INFO] [TRAIN] epoch=50, iter=32520/40000, loss=0.1562, lr=0.002212, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:38:27 2021-01-18 17:13:24 [INFO] [TRAIN] epoch=50, iter=32530/40000, loss=0.0944, lr=0.002209, batch_cost=0.7879, reader_cost=0.0001 | ETA 01:38:05 2021-01-18 17:13:32 [INFO] [TRAIN] epoch=50, iter=32540/40000, loss=0.2204, lr=0.002206, batch_cost=0.7863, reader_cost=0.0001 | ETA 01:37:45 2021-01-18 17:13:40 [INFO] [TRAIN] epoch=50, iter=32550/40000, loss=0.1571, lr=0.002204, batch_cost=0.7896, reader_cost=0.0001 | ETA 01:38:02 2021-01-18 17:13:48 [INFO] [TRAIN] epoch=50, iter=32560/40000, loss=0.1498, lr=0.002201, batch_cost=0.7920, reader_cost=0.0001 | ETA 01:38:12 2021-01-18 17:13:56 [INFO] [TRAIN] epoch=50, iter=32570/40000, loss=0.1526, lr=0.002198, batch_cost=0.7926, reader_cost=0.0001 | ETA 01:38:09 2021-01-18 17:14:03 [INFO] [TRAIN] epoch=50, iter=32580/40000, loss=0.1960, lr=0.002196, batch_cost=0.7851, reader_cost=0.0001 | ETA 01:37:05 2021-01-18 17:14:11 [INFO] [TRAIN] epoch=50, iter=32590/40000, loss=0.1879, lr=0.002193, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:37:46 2021-01-18 17:14:19 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.1045, lr=0.002190, batch_cost=0.7870, reader_cost=0.0001 | ETA 01:37:03 2021-01-18 17:14:27 [INFO] [TRAIN] epoch=50, iter=32610/40000, loss=0.1210, lr=0.002188, batch_cost=0.7842, reader_cost=0.0001 | ETA 01:36:35 2021-01-18 17:14:35 [INFO] [TRAIN] epoch=50, iter=32620/40000, loss=0.1462, lr=0.002185, batch_cost=0.7965, reader_cost=0.0001 | ETA 01:37:57 2021-01-18 17:14:43 [INFO] [TRAIN] epoch=50, iter=32630/40000, loss=0.1313, lr=0.002182, batch_cost=0.7852, reader_cost=0.0001 | ETA 01:36:26 2021-01-18 17:14:51 [INFO] [TRAIN] epoch=50, iter=32640/40000, loss=0.1217, lr=0.002180, batch_cost=0.7864, reader_cost=0.0001 | ETA 01:36:27 2021-01-18 17:14:59 [INFO] [TRAIN] epoch=50, iter=32650/40000, loss=0.1027, lr=0.002177, batch_cost=0.7971, reader_cost=0.0001 | ETA 01:37:38 2021-01-18 17:15:07 [INFO] [TRAIN] epoch=50, iter=32660/40000, loss=0.1725, lr=0.002174, batch_cost=0.7890, reader_cost=0.0001 | ETA 01:36:31 2021-01-18 17:15:15 [INFO] [TRAIN] epoch=50, iter=32670/40000, loss=0.1318, lr=0.002172, batch_cost=0.7943, reader_cost=0.0001 | ETA 01:37:02 2021-01-18 17:15:22 [INFO] [TRAIN] epoch=50, iter=32680/40000, loss=0.0843, lr=0.002169, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:36:30 2021-01-18 17:15:30 [INFO] [TRAIN] epoch=50, iter=32690/40000, loss=0.1341, lr=0.002166, batch_cost=0.7865, reader_cost=0.0001 | ETA 01:35:49 2021-01-18 17:15:38 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.1104, lr=0.002164, batch_cost=0.7912, reader_cost=0.0001 | ETA 01:36:15 2021-01-18 17:15:46 [INFO] [TRAIN] epoch=50, iter=32710/40000, loss=0.1081, lr=0.002161, batch_cost=0.7915, reader_cost=0.0001 | ETA 01:36:10 2021-01-18 17:15:54 [INFO] [TRAIN] epoch=50, iter=32720/40000, loss=0.1297, lr=0.002158, batch_cost=0.7940, reader_cost=0.0001 | ETA 01:36:20 2021-01-18 17:16:02 [INFO] [TRAIN] epoch=50, iter=32730/40000, loss=0.0910, lr=0.002156, batch_cost=0.7889, reader_cost=0.0001 | ETA 01:35:35 2021-01-18 17:16:10 [INFO] [TRAIN] epoch=50, iter=32740/40000, loss=0.1666, lr=0.002153, batch_cost=0.7858, reader_cost=0.0001 | ETA 01:35:04 2021-01-18 17:16:18 [INFO] [TRAIN] epoch=50, iter=32750/40000, loss=0.1639, lr=0.002150, batch_cost=0.7958, reader_cost=0.0001 | ETA 01:36:09 2021-01-18 17:16:26 [INFO] [TRAIN] epoch=50, iter=32760/40000, loss=0.1443, lr=0.002148, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:35:32 2021-01-18 17:16:34 [INFO] [TRAIN] epoch=50, iter=32770/40000, loss=0.1053, lr=0.002145, batch_cost=0.7884, reader_cost=0.0001 | ETA 01:34:59 2021-01-18 17:16:41 [INFO] [TRAIN] epoch=50, iter=32780/40000, loss=0.1423, lr=0.002142, batch_cost=0.7885, reader_cost=0.0001 | ETA 01:34:53 2021-01-18 17:16:49 [INFO] [TRAIN] epoch=50, iter=32790/40000, loss=0.1172, lr=0.002140, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:35:19 2021-01-18 17:16:57 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.1769, lr=0.002137, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:35:01 2021-01-18 17:17:05 [INFO] [TRAIN] epoch=50, iter=32810/40000, loss=0.1312, lr=0.002134, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:34:40 2021-01-18 17:17:13 [INFO] [TRAIN] epoch=50, iter=32820/40000, loss=0.1097, lr=0.002132, batch_cost=0.7884, reader_cost=0.0001 | ETA 01:34:20 2021-01-18 17:17:21 [INFO] [TRAIN] epoch=50, iter=32830/40000, loss=0.1694, lr=0.002129, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:34:30 2021-01-18 17:17:29 [INFO] [TRAIN] epoch=50, iter=32840/40000, loss=0.1043, lr=0.002126, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:34:24 2021-01-18 17:17:37 [INFO] [TRAIN] epoch=50, iter=32850/40000, loss=0.1593, lr=0.002124, batch_cost=0.7889, reader_cost=0.0001 | ETA 01:34:00 2021-01-18 17:17:45 [INFO] [TRAIN] epoch=50, iter=32860/40000, loss=0.1459, lr=0.002121, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:34:01 2021-01-18 17:17:53 [INFO] [TRAIN] epoch=50, iter=32870/40000, loss=0.1286, lr=0.002118, batch_cost=0.7912, reader_cost=0.0001 | ETA 01:34:01 2021-01-18 17:18:01 [INFO] [TRAIN] epoch=50, iter=32880/40000, loss=0.1648, lr=0.002116, batch_cost=0.7939, reader_cost=0.0001 | ETA 01:34:12 2021-01-18 17:18:08 [INFO] [TRAIN] epoch=50, iter=32890/40000, loss=0.1290, lr=0.002113, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:33:56 2021-01-18 17:18:16 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.1080, lr=0.002110, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:33:36 2021-01-18 17:18:24 [INFO] [TRAIN] epoch=50, iter=32910/40000, loss=0.1185, lr=0.002108, batch_cost=0.7945, reader_cost=0.0001 | ETA 01:33:53 2021-01-18 17:18:32 [INFO] [TRAIN] epoch=50, iter=32920/40000, loss=0.1352, lr=0.002105, batch_cost=0.7947, reader_cost=0.0001 | ETA 01:33:46 2021-01-18 17:18:40 [INFO] [TRAIN] epoch=50, iter=32930/40000, loss=0.1246, lr=0.002102, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:33:11 2021-01-18 17:18:48 [INFO] [TRAIN] epoch=50, iter=32940/40000, loss=0.0967, lr=0.002100, batch_cost=0.7950, reader_cost=0.0001 | ETA 01:33:32 2021-01-18 17:18:56 [INFO] [TRAIN] epoch=50, iter=32950/40000, loss=0.1539, lr=0.002097, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:32:51 2021-01-18 17:19:04 [INFO] [TRAIN] epoch=50, iter=32960/40000, loss=0.1233, lr=0.002094, batch_cost=0.7895, reader_cost=0.0001 | ETA 01:32:37 2021-01-18 17:19:12 [INFO] [TRAIN] epoch=50, iter=32970/40000, loss=0.1387, lr=0.002092, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:32:14 2021-01-18 17:19:20 [INFO] [TRAIN] epoch=50, iter=32980/40000, loss=0.1196, lr=0.002089, batch_cost=0.7914, reader_cost=0.0001 | ETA 01:32:35 2021-01-18 17:19:28 [INFO] [TRAIN] epoch=50, iter=32990/40000, loss=0.1502, lr=0.002086, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:31:59 2021-01-18 17:19:36 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.1231, lr=0.002083, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:32:31 2021-01-18 17:19:36 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 17:20:03 [INFO] [EVAL] #Images=1449 mIoU=0.7941 Acc=0.9550 Kappa=0.9001 2021-01-18 17:20:03 [INFO] [EVAL] Class IoU: [0.9488 0.9051 0.4273 0.9086 0.673 0.8584 0.9382 0.8836 0.9354 0.4501 0.9015 0.5247 0.8932 0.8999 0.8736 0.8899 0.6418 0.9017 0.528 0.9314 0.7626] 2021-01-18 17:20:03 [INFO] [EVAL] Class Acc: [0.9682 0.9306 0.4395 0.9456 0.7844 0.9126 0.9861 0.9307 0.9586 0.6481 0.949 0.9433 0.9395 0.955 0.9248 0.9648 0.8181 0.9495 0.8532 0.9517 0.9054] 2021-01-18 17:20:10 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 17:20:18 [INFO] [TRAIN] epoch=50, iter=33010/40000, loss=0.1467, lr=0.002081, batch_cost=0.7848, reader_cost=0.0001 | ETA 01:31:25 2021-01-18 17:20:26 [INFO] [TRAIN] epoch=50, iter=33020/40000, loss=0.1139, lr=0.002078, batch_cost=0.7846, reader_cost=0.0001 | ETA 01:31:16 2021-01-18 17:20:33 [INFO] [TRAIN] epoch=50, iter=33030/40000, loss=0.1144, lr=0.002075, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:31:52 2021-01-18 17:20:41 [INFO] [TRAIN] epoch=50, iter=33040/40000, loss=0.1016, lr=0.002073, batch_cost=0.7862, reader_cost=0.0001 | ETA 01:31:12 2021-01-18 17:20:49 [INFO] [TRAIN] epoch=50, iter=33050/40000, loss=0.1080, lr=0.002070, batch_cost=0.7531, reader_cost=0.0001 | ETA 01:27:14 2021-01-18 17:20:57 [INFO] [TRAIN] epoch=51, iter=33060/40000, loss=0.0975, lr=0.002067, batch_cost=0.8066, reader_cost=0.0146 | ETA 01:33:17 2021-01-18 17:21:05 [INFO] [TRAIN] epoch=51, iter=33070/40000, loss=0.1154, lr=0.002065, batch_cost=0.7935, reader_cost=0.0001 | ETA 01:31:38 2021-01-18 17:21:13 [INFO] [TRAIN] epoch=51, iter=33080/40000, loss=0.1163, lr=0.002062, batch_cost=0.7882, reader_cost=0.0001 | ETA 01:30:54 2021-01-18 17:21:21 [INFO] [TRAIN] epoch=51, iter=33090/40000, loss=0.1538, lr=0.002059, batch_cost=0.7937, reader_cost=0.0001 | ETA 01:31:24 2021-01-18 17:21:29 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.1519, lr=0.002057, batch_cost=0.7863, reader_cost=0.0001 | ETA 01:30:25 2021-01-18 17:21:36 [INFO] [TRAIN] epoch=51, iter=33110/40000, loss=0.1702, lr=0.002054, batch_cost=0.7853, reader_cost=0.0001 | ETA 01:30:10 2021-01-18 17:21:44 [INFO] [TRAIN] epoch=51, iter=33120/40000, loss=0.1166, lr=0.002051, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:30:48 2021-01-18 17:21:52 [INFO] [TRAIN] epoch=51, iter=33130/40000, loss=0.1636, lr=0.002049, batch_cost=0.7936, reader_cost=0.0001 | ETA 01:30:51 2021-01-18 17:22:00 [INFO] [TRAIN] epoch=51, iter=33140/40000, loss=0.0925, lr=0.002046, batch_cost=0.7900, reader_cost=0.0001 | ETA 01:30:19 2021-01-18 17:22:08 [INFO] [TRAIN] epoch=51, iter=33150/40000, loss=0.1546, lr=0.002043, batch_cost=0.7906, reader_cost=0.0001 | ETA 01:30:15 2021-01-18 17:22:16 [INFO] [TRAIN] epoch=51, iter=33160/40000, loss=0.1015, lr=0.002041, batch_cost=0.7869, reader_cost=0.0001 | ETA 01:29:42 2021-01-18 17:22:24 [INFO] [TRAIN] epoch=51, iter=33170/40000, loss=0.1002, lr=0.002038, batch_cost=0.7884, reader_cost=0.0001 | ETA 01:29:44 2021-01-18 17:22:32 [INFO] [TRAIN] epoch=51, iter=33180/40000, loss=0.0979, lr=0.002035, batch_cost=0.7885, reader_cost=0.0001 | ETA 01:29:37 2021-01-18 17:22:40 [INFO] [TRAIN] epoch=51, iter=33190/40000, loss=0.1231, lr=0.002033, batch_cost=0.7936, reader_cost=0.0001 | ETA 01:30:04 2021-01-18 17:22:48 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.1298, lr=0.002030, batch_cost=0.7900, reader_cost=0.0001 | ETA 01:29:31 2021-01-18 17:22:55 [INFO] [TRAIN] epoch=51, iter=33210/40000, loss=0.1202, lr=0.002027, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:29:45 2021-01-18 17:23:03 [INFO] [TRAIN] epoch=51, iter=33220/40000, loss=0.1103, lr=0.002024, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:29:11 2021-01-18 17:23:11 [INFO] [TRAIN] epoch=51, iter=33230/40000, loss=0.1087, lr=0.002022, batch_cost=0.7904, reader_cost=0.0001 | ETA 01:29:10 2021-01-18 17:23:19 [INFO] [TRAIN] epoch=51, iter=33240/40000, loss=0.1192, lr=0.002019, batch_cost=0.7966, reader_cost=0.0001 | ETA 01:29:44 2021-01-18 17:23:27 [INFO] [TRAIN] epoch=51, iter=33250/40000, loss=0.1169, lr=0.002016, batch_cost=0.7871, reader_cost=0.0001 | ETA 01:28:33 2021-01-18 17:23:35 [INFO] [TRAIN] epoch=51, iter=33260/40000, loss=0.1031, lr=0.002014, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:29:01 2021-01-18 17:23:43 [INFO] [TRAIN] epoch=51, iter=33270/40000, loss=0.1065, lr=0.002011, batch_cost=0.7902, reader_cost=0.0001 | ETA 01:28:37 2021-01-18 17:23:51 [INFO] [TRAIN] epoch=51, iter=33280/40000, loss=0.1164, lr=0.002008, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:28:15 2021-01-18 17:23:59 [INFO] [TRAIN] epoch=51, iter=33290/40000, loss=0.1365, lr=0.002006, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:28:41 2021-01-18 17:24:07 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.1447, lr=0.002003, batch_cost=0.7910, reader_cost=0.0001 | ETA 01:28:19 2021-01-18 17:24:15 [INFO] [TRAIN] epoch=51, iter=33310/40000, loss=0.1385, lr=0.002000, batch_cost=0.7935, reader_cost=0.0001 | ETA 01:28:28 2021-01-18 17:24:22 [INFO] [TRAIN] epoch=51, iter=33320/40000, loss=0.0987, lr=0.001998, batch_cost=0.7915, reader_cost=0.0001 | ETA 01:28:06 2021-01-18 17:24:30 [INFO] [TRAIN] epoch=51, iter=33330/40000, loss=0.1055, lr=0.001995, batch_cost=0.7846, reader_cost=0.0001 | ETA 01:27:13 2021-01-18 17:24:38 [INFO] [TRAIN] epoch=51, iter=33340/40000, loss=0.1018, lr=0.001992, batch_cost=0.7878, reader_cost=0.0001 | ETA 01:27:26 2021-01-18 17:24:46 [INFO] [TRAIN] epoch=51, iter=33350/40000, loss=0.1518, lr=0.001990, batch_cost=0.7871, reader_cost=0.0001 | ETA 01:27:13 2021-01-18 17:24:54 [INFO] [TRAIN] epoch=51, iter=33360/40000, loss=0.1295, lr=0.001987, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:27:09 2021-01-18 17:25:02 [INFO] [TRAIN] epoch=51, iter=33370/40000, loss=0.0873, lr=0.001984, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:26:56 2021-01-18 17:25:10 [INFO] [TRAIN] epoch=51, iter=33380/40000, loss=0.1317, lr=0.001981, batch_cost=0.7889, reader_cost=0.0001 | ETA 01:27:02 2021-01-18 17:25:18 [INFO] [TRAIN] epoch=51, iter=33390/40000, loss=0.1581, lr=0.001979, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:26:43 2021-01-18 17:25:25 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.1025, lr=0.001976, batch_cost=0.7888, reader_cost=0.0001 | ETA 01:26:45 2021-01-18 17:25:33 [INFO] [TRAIN] epoch=51, iter=33410/40000, loss=0.1205, lr=0.001973, batch_cost=0.7914, reader_cost=0.0001 | ETA 01:26:55 2021-01-18 17:25:41 [INFO] [TRAIN] epoch=51, iter=33420/40000, loss=0.1152, lr=0.001971, batch_cost=0.7946, reader_cost=0.0001 | ETA 01:27:08 2021-01-18 17:25:49 [INFO] [TRAIN] epoch=51, iter=33430/40000, loss=0.1596, lr=0.001968, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:27:03 2021-01-18 17:25:57 [INFO] [TRAIN] epoch=51, iter=33440/40000, loss=0.1221, lr=0.001965, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:26:09 2021-01-18 17:26:05 [INFO] [TRAIN] epoch=51, iter=33450/40000, loss=0.1283, lr=0.001963, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:26:26 2021-01-18 17:26:13 [INFO] [TRAIN] epoch=51, iter=33460/40000, loss=0.1080, lr=0.001960, batch_cost=0.7858, reader_cost=0.0001 | ETA 01:25:39 2021-01-18 17:26:21 [INFO] [TRAIN] epoch=51, iter=33470/40000, loss=0.0962, lr=0.001957, batch_cost=0.7942, reader_cost=0.0001 | ETA 01:26:26 2021-01-18 17:26:29 [INFO] [TRAIN] epoch=51, iter=33480/40000, loss=0.1110, lr=0.001954, batch_cost=0.7912, reader_cost=0.0001 | ETA 01:25:58 2021-01-18 17:26:37 [INFO] [TRAIN] epoch=51, iter=33490/40000, loss=0.1120, lr=0.001952, batch_cost=0.7896, reader_cost=0.0001 | ETA 01:25:40 2021-01-18 17:26:45 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.1628, lr=0.001949, batch_cost=0.7889, reader_cost=0.0001 | ETA 01:25:27 2021-01-18 17:26:53 [INFO] [TRAIN] epoch=51, iter=33510/40000, loss=0.1160, lr=0.001946, batch_cost=0.7987, reader_cost=0.0001 | ETA 01:26:23 2021-01-18 17:27:00 [INFO] [TRAIN] epoch=51, iter=33520/40000, loss=0.1262, lr=0.001944, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:25:13 2021-01-18 17:27:08 [INFO] [TRAIN] epoch=51, iter=33530/40000, loss=0.0985, lr=0.001941, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:25:32 2021-01-18 17:27:16 [INFO] [TRAIN] epoch=51, iter=33540/40000, loss=0.1753, lr=0.001938, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:25:14 2021-01-18 17:27:24 [INFO] [TRAIN] epoch=51, iter=33550/40000, loss=0.1430, lr=0.001936, batch_cost=0.7862, reader_cost=0.0001 | ETA 01:24:31 2021-01-18 17:27:32 [INFO] [TRAIN] epoch=51, iter=33560/40000, loss=0.1399, lr=0.001933, batch_cost=0.7976, reader_cost=0.0001 | ETA 01:25:36 2021-01-18 17:27:40 [INFO] [TRAIN] epoch=51, iter=33570/40000, loss=0.1293, lr=0.001930, batch_cost=0.7921, reader_cost=0.0001 | ETA 01:24:53 2021-01-18 17:27:48 [INFO] [TRAIN] epoch=51, iter=33580/40000, loss=0.1378, lr=0.001927, batch_cost=0.7947, reader_cost=0.0001 | ETA 01:25:01 2021-01-18 17:27:56 [INFO] [TRAIN] epoch=51, iter=33590/40000, loss=0.1167, lr=0.001925, batch_cost=0.7883, reader_cost=0.0001 | ETA 01:24:13 2021-01-18 17:28:04 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.1210, lr=0.001922, batch_cost=0.7882, reader_cost=0.0001 | ETA 01:24:04 2021-01-18 17:28:12 [INFO] [TRAIN] epoch=51, iter=33610/40000, loss=0.1200, lr=0.001919, batch_cost=0.7915, reader_cost=0.0001 | ETA 01:24:17 2021-01-18 17:28:20 [INFO] [TRAIN] epoch=51, iter=33620/40000, loss=0.1425, lr=0.001917, batch_cost=0.7943, reader_cost=0.0001 | ETA 01:24:27 2021-01-18 17:28:28 [INFO] [TRAIN] epoch=51, iter=33630/40000, loss=0.1531, lr=0.001914, batch_cost=0.7869, reader_cost=0.0001 | ETA 01:23:32 2021-01-18 17:28:35 [INFO] [TRAIN] epoch=51, iter=33640/40000, loss=0.1543, lr=0.001911, batch_cost=0.7904, reader_cost=0.0001 | ETA 01:23:47 2021-01-18 17:28:43 [INFO] [TRAIN] epoch=51, iter=33650/40000, loss=0.1131, lr=0.001909, batch_cost=0.7900, reader_cost=0.0001 | ETA 01:23:36 2021-01-18 17:28:51 [INFO] [TRAIN] epoch=51, iter=33660/40000, loss=0.1202, lr=0.001906, batch_cost=0.7872, reader_cost=0.0001 | ETA 01:23:10 2021-01-18 17:28:59 [INFO] [TRAIN] epoch=51, iter=33670/40000, loss=0.1788, lr=0.001903, batch_cost=0.7921, reader_cost=0.0001 | ETA 01:23:33 2021-01-18 17:29:07 [INFO] [TRAIN] epoch=51, iter=33680/40000, loss=0.1003, lr=0.001900, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:23:23 2021-01-18 17:29:15 [INFO] [TRAIN] epoch=51, iter=33690/40000, loss=0.1126, lr=0.001898, batch_cost=0.7935, reader_cost=0.0001 | ETA 01:23:27 2021-01-18 17:29:23 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.1264, lr=0.001895, batch_cost=0.7867, reader_cost=0.0001 | ETA 01:22:35 2021-01-18 17:29:30 [INFO] [TRAIN] epoch=51, iter=33710/40000, loss=0.1430, lr=0.001892, batch_cost=0.7595, reader_cost=0.0001 | ETA 01:19:37 2021-01-18 17:29:38 [INFO] [TRAIN] epoch=52, iter=33720/40000, loss=0.1075, lr=0.001890, batch_cost=0.8021, reader_cost=0.0145 | ETA 01:23:57 2021-01-18 17:29:46 [INFO] [TRAIN] epoch=52, iter=33730/40000, loss=0.1420, lr=0.001887, batch_cost=0.7883, reader_cost=0.0001 | ETA 01:22:22 2021-01-18 17:29:54 [INFO] [TRAIN] epoch=52, iter=33740/40000, loss=0.1095, lr=0.001884, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:22:31 2021-01-18 17:30:02 [INFO] [TRAIN] epoch=52, iter=33750/40000, loss=0.1404, lr=0.001881, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:22:18 2021-01-18 17:30:10 [INFO] [TRAIN] epoch=52, iter=33760/40000, loss=0.1307, lr=0.001879, batch_cost=0.7948, reader_cost=0.0001 | ETA 01:22:39 2021-01-18 17:30:18 [INFO] [TRAIN] epoch=52, iter=33770/40000, loss=0.1221, lr=0.001876, batch_cost=0.7884, reader_cost=0.0001 | ETA 01:21:51 2021-01-18 17:30:26 [INFO] [TRAIN] epoch=52, iter=33780/40000, loss=0.1238, lr=0.001873, batch_cost=0.7878, reader_cost=0.0001 | ETA 01:21:40 2021-01-18 17:30:34 [INFO] [TRAIN] epoch=52, iter=33790/40000, loss=0.1383, lr=0.001871, batch_cost=0.7926, reader_cost=0.0001 | ETA 01:22:01 2021-01-18 17:30:42 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.1279, lr=0.001868, batch_cost=0.7910, reader_cost=0.0001 | ETA 01:21:43 2021-01-18 17:30:50 [INFO] [TRAIN] epoch=52, iter=33810/40000, loss=0.1322, lr=0.001865, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:21:35 2021-01-18 17:30:57 [INFO] [TRAIN] epoch=52, iter=33820/40000, loss=0.1760, lr=0.001863, batch_cost=0.7854, reader_cost=0.0001 | ETA 01:20:53 2021-01-18 17:31:05 [INFO] [TRAIN] epoch=52, iter=33830/40000, loss=0.1346, lr=0.001860, batch_cost=0.7874, reader_cost=0.0001 | ETA 01:20:58 2021-01-18 17:31:13 [INFO] [TRAIN] epoch=52, iter=33840/40000, loss=0.1556, lr=0.001857, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:21:05 2021-01-18 17:31:21 [INFO] [TRAIN] epoch=52, iter=33850/40000, loss=0.1163, lr=0.001854, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:21:00 2021-01-18 17:31:29 [INFO] [TRAIN] epoch=52, iter=33860/40000, loss=0.1230, lr=0.001852, batch_cost=0.7869, reader_cost=0.0001 | ETA 01:20:31 2021-01-18 17:31:37 [INFO] [TRAIN] epoch=52, iter=33870/40000, loss=0.1259, lr=0.001849, batch_cost=0.7914, reader_cost=0.0001 | ETA 01:20:51 2021-01-18 17:31:45 [INFO] [TRAIN] epoch=52, iter=33880/40000, loss=0.1263, lr=0.001846, batch_cost=0.7965, reader_cost=0.0001 | ETA 01:21:14 2021-01-18 17:31:53 [INFO] [TRAIN] epoch=52, iter=33890/40000, loss=0.1666, lr=0.001844, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:20:28 2021-01-18 17:32:01 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.1703, lr=0.001841, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:20:23 2021-01-18 17:32:09 [INFO] [TRAIN] epoch=52, iter=33910/40000, loss=0.1147, lr=0.001838, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:20:09 2021-01-18 17:32:16 [INFO] [TRAIN] epoch=52, iter=33920/40000, loss=0.1005, lr=0.001835, batch_cost=0.7874, reader_cost=0.0001 | ETA 01:19:47 2021-01-18 17:32:24 [INFO] [TRAIN] epoch=52, iter=33930/40000, loss=0.1048, lr=0.001833, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:19:50 2021-01-18 17:32:32 [INFO] [TRAIN] epoch=52, iter=33940/40000, loss=0.1394, lr=0.001830, batch_cost=0.7859, reader_cost=0.0001 | ETA 01:19:22 2021-01-18 17:32:40 [INFO] [TRAIN] epoch=52, iter=33950/40000, loss=0.1196, lr=0.001827, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:19:50 2021-01-18 17:32:48 [INFO] [TRAIN] epoch=52, iter=33960/40000, loss=0.1087, lr=0.001825, batch_cost=0.7938, reader_cost=0.0001 | ETA 01:19:54 2021-01-18 17:32:56 [INFO] [TRAIN] epoch=52, iter=33970/40000, loss=0.1405, lr=0.001822, batch_cost=0.7915, reader_cost=0.0001 | ETA 01:19:33 2021-01-18 17:33:04 [INFO] [TRAIN] epoch=52, iter=33980/40000, loss=0.1334, lr=0.001819, batch_cost=0.7941, reader_cost=0.0001 | ETA 01:19:40 2021-01-18 17:33:12 [INFO] [TRAIN] epoch=52, iter=33990/40000, loss=0.1314, lr=0.001816, batch_cost=0.7973, reader_cost=0.0001 | ETA 01:19:51 2021-01-18 17:33:20 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.1231, lr=0.001814, batch_cost=0.7870, reader_cost=0.0001 | ETA 01:18:42 2021-01-18 17:33:20 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 17:33:48 [INFO] [EVAL] #Images=1449 mIoU=0.7981 Acc=0.9557 Kappa=0.9014 2021-01-18 17:33:48 [INFO] [EVAL] Class IoU: [0.9497 0.9116 0.4406 0.9088 0.7012 0.8534 0.9401 0.8868 0.9334 0.43 0.8947 0.567 0.8888 0.8954 0.8996 0.8882 0.6438 0.8818 0.5277 0.9208 0.796 ] 2021-01-18 17:33:48 [INFO] [EVAL] Class Acc: [0.9676 0.9416 0.4577 0.9551 0.8159 0.9225 0.9852 0.9393 0.9648 0.6631 0.9496 0.8952 0.9448 0.9447 0.9437 0.9583 0.8186 0.9607 0.8255 0.9563 0.9632] 2021-01-18 17:33:55 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 17:34:03 [INFO] [TRAIN] epoch=52, iter=34010/40000, loss=0.1292, lr=0.001811, batch_cost=0.7866, reader_cost=0.0001 | ETA 01:18:32 2021-01-18 17:34:10 [INFO] [TRAIN] epoch=52, iter=34020/40000, loss=0.1176, lr=0.001808, batch_cost=0.7842, reader_cost=0.0001 | ETA 01:18:09 2021-01-18 17:34:18 [INFO] [TRAIN] epoch=52, iter=34030/40000, loss=0.1068, lr=0.001805, batch_cost=0.7895, reader_cost=0.0001 | ETA 01:18:33 2021-01-18 17:34:26 [INFO] [TRAIN] epoch=52, iter=34040/40000, loss=0.1104, lr=0.001803, batch_cost=0.7934, reader_cost=0.0001 | ETA 01:18:48 2021-01-18 17:34:34 [INFO] [TRAIN] epoch=52, iter=34050/40000, loss=0.1127, lr=0.001800, batch_cost=0.7887, reader_cost=0.0001 | ETA 01:18:12 2021-01-18 17:34:42 [INFO] [TRAIN] epoch=52, iter=34060/40000, loss=0.1202, lr=0.001797, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:18:07 2021-01-18 17:34:50 [INFO] [TRAIN] epoch=52, iter=34070/40000, loss=0.1232, lr=0.001795, batch_cost=0.7879, reader_cost=0.0001 | ETA 01:17:52 2021-01-18 17:34:58 [INFO] [TRAIN] epoch=52, iter=34080/40000, loss=0.0814, lr=0.001792, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:17:37 2021-01-18 17:35:06 [INFO] [TRAIN] epoch=52, iter=34090/40000, loss=0.1287, lr=0.001789, batch_cost=0.7962, reader_cost=0.0001 | ETA 01:18:25 2021-01-18 17:35:14 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.1121, lr=0.001786, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:17:51 2021-01-18 17:35:22 [INFO] [TRAIN] epoch=52, iter=34110/40000, loss=0.0907, lr=0.001784, batch_cost=0.7910, reader_cost=0.0001 | ETA 01:17:39 2021-01-18 17:35:29 [INFO] [TRAIN] epoch=52, iter=34120/40000, loss=0.1778, lr=0.001781, batch_cost=0.7917, reader_cost=0.0001 | ETA 01:17:35 2021-01-18 17:35:37 [INFO] [TRAIN] epoch=52, iter=34130/40000, loss=0.1242, lr=0.001778, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:16:58 2021-01-18 17:35:45 [INFO] [TRAIN] epoch=52, iter=34140/40000, loss=0.1114, lr=0.001775, batch_cost=0.7894, reader_cost=0.0001 | ETA 01:17:06 2021-01-18 17:35:53 [INFO] [TRAIN] epoch=52, iter=34150/40000, loss=0.1225, lr=0.001773, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:17:09 2021-01-18 17:36:01 [INFO] [TRAIN] epoch=52, iter=34160/40000, loss=0.1357, lr=0.001770, batch_cost=0.7871, reader_cost=0.0001 | ETA 01:16:36 2021-01-18 17:36:09 [INFO] [TRAIN] epoch=52, iter=34170/40000, loss=0.0920, lr=0.001767, batch_cost=0.7932, reader_cost=0.0001 | ETA 01:17:04 2021-01-18 17:36:17 [INFO] [TRAIN] epoch=52, iter=34180/40000, loss=0.1130, lr=0.001765, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:16:42 2021-01-18 17:36:25 [INFO] [TRAIN] epoch=52, iter=34190/40000, loss=0.1230, lr=0.001762, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:16:18 2021-01-18 17:36:33 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.1060, lr=0.001759, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:16:41 2021-01-18 17:36:41 [INFO] [TRAIN] epoch=52, iter=34210/40000, loss=0.1376, lr=0.001756, batch_cost=0.7943, reader_cost=0.0001 | ETA 01:16:38 2021-01-18 17:36:48 [INFO] [TRAIN] epoch=52, iter=34220/40000, loss=0.1037, lr=0.001754, batch_cost=0.7862, reader_cost=0.0001 | ETA 01:15:44 2021-01-18 17:36:56 [INFO] [TRAIN] epoch=52, iter=34230/40000, loss=0.1424, lr=0.001751, batch_cost=0.7888, reader_cost=0.0001 | ETA 01:15:51 2021-01-18 17:37:04 [INFO] [TRAIN] epoch=52, iter=34240/40000, loss=0.1523, lr=0.001748, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:15:54 2021-01-18 17:37:12 [INFO] [TRAIN] epoch=52, iter=34250/40000, loss=0.1466, lr=0.001745, batch_cost=0.7958, reader_cost=0.0001 | ETA 01:16:15 2021-01-18 17:37:20 [INFO] [TRAIN] epoch=52, iter=34260/40000, loss=0.1211, lr=0.001743, batch_cost=0.7934, reader_cost=0.0001 | ETA 01:15:54 2021-01-18 17:37:28 [INFO] [TRAIN] epoch=52, iter=34270/40000, loss=0.1229, lr=0.001740, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:15:40 2021-01-18 17:37:36 [INFO] [TRAIN] epoch=52, iter=34280/40000, loss=0.1494, lr=0.001737, batch_cost=0.7882, reader_cost=0.0001 | ETA 01:15:08 2021-01-18 17:37:44 [INFO] [TRAIN] epoch=52, iter=34290/40000, loss=0.1198, lr=0.001735, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:15:18 2021-01-18 17:37:52 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.1127, lr=0.001732, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:14:57 2021-01-18 17:38:00 [INFO] [TRAIN] epoch=52, iter=34310/40000, loss=0.1076, lr=0.001729, batch_cost=0.7924, reader_cost=0.0001 | ETA 01:15:08 2021-01-18 17:38:08 [INFO] [TRAIN] epoch=52, iter=34320/40000, loss=0.1277, lr=0.001726, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:14:51 2021-01-18 17:38:16 [INFO] [TRAIN] epoch=52, iter=34330/40000, loss=0.1113, lr=0.001724, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:14:49 2021-01-18 17:38:23 [INFO] [TRAIN] epoch=52, iter=34340/40000, loss=0.1047, lr=0.001721, batch_cost=0.7887, reader_cost=0.0001 | ETA 01:14:23 2021-01-18 17:38:31 [INFO] [TRAIN] epoch=52, iter=34350/40000, loss=0.2031, lr=0.001718, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:14:33 2021-01-18 17:38:39 [INFO] [TRAIN] epoch=52, iter=34360/40000, loss=0.1838, lr=0.001715, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:14:21 2021-01-18 17:38:47 [INFO] [TRAIN] epoch=52, iter=34370/40000, loss=0.2155, lr=0.001713, batch_cost=0.7712, reader_cost=0.0001 | ETA 01:12:21 2021-01-18 17:38:55 [INFO] [TRAIN] epoch=53, iter=34380/40000, loss=0.1257, lr=0.001710, batch_cost=0.7967, reader_cost=0.0139 | ETA 01:14:37 2021-01-18 17:39:03 [INFO] [TRAIN] epoch=53, iter=34390/40000, loss=0.1283, lr=0.001707, batch_cost=0.7861, reader_cost=0.0001 | ETA 01:13:29 2021-01-18 17:39:11 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.1187, lr=0.001704, batch_cost=0.7948, reader_cost=0.0001 | ETA 01:14:10 2021-01-18 17:39:19 [INFO] [TRAIN] epoch=53, iter=34410/40000, loss=0.1341, lr=0.001702, batch_cost=0.7977, reader_cost=0.0001 | ETA 01:14:19 2021-01-18 17:39:27 [INFO] [TRAIN] epoch=53, iter=34420/40000, loss=0.1531, lr=0.001699, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:13:42 2021-01-18 17:39:35 [INFO] [TRAIN] epoch=53, iter=34430/40000, loss=0.1301, lr=0.001696, batch_cost=0.7894, reader_cost=0.0001 | ETA 01:13:17 2021-01-18 17:39:42 [INFO] [TRAIN] epoch=53, iter=34440/40000, loss=0.1196, lr=0.001693, batch_cost=0.7875, reader_cost=0.0001 | ETA 01:12:58 2021-01-18 17:39:50 [INFO] [TRAIN] epoch=53, iter=34450/40000, loss=0.1241, lr=0.001691, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:13:19 2021-01-18 17:39:58 [INFO] [TRAIN] epoch=53, iter=34460/40000, loss=0.1178, lr=0.001688, batch_cost=0.7937, reader_cost=0.0001 | ETA 01:13:17 2021-01-18 17:40:06 [INFO] [TRAIN] epoch=53, iter=34470/40000, loss=0.1346, lr=0.001685, batch_cost=0.7933, reader_cost=0.0001 | ETA 01:13:06 2021-01-18 17:40:14 [INFO] [TRAIN] epoch=53, iter=34480/40000, loss=0.0965, lr=0.001683, batch_cost=0.7996, reader_cost=0.0001 | ETA 01:13:33 2021-01-18 17:40:22 [INFO] [TRAIN] epoch=53, iter=34490/40000, loss=0.1068, lr=0.001680, batch_cost=0.7929, reader_cost=0.0001 | ETA 01:12:48 2021-01-18 17:40:30 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.1111, lr=0.001677, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:12:31 2021-01-18 17:40:38 [INFO] [TRAIN] epoch=53, iter=34510/40000, loss=0.1523, lr=0.001674, batch_cost=0.7901, reader_cost=0.0001 | ETA 01:12:17 2021-01-18 17:40:46 [INFO] [TRAIN] epoch=53, iter=34520/40000, loss=0.1214, lr=0.001672, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:12:04 2021-01-18 17:40:54 [INFO] [TRAIN] epoch=53, iter=34530/40000, loss=0.1485, lr=0.001669, batch_cost=0.7864, reader_cost=0.0001 | ETA 01:11:41 2021-01-18 17:41:02 [INFO] [TRAIN] epoch=53, iter=34540/40000, loss=0.1108, lr=0.001666, batch_cost=0.7833, reader_cost=0.0001 | ETA 01:11:16 2021-01-18 17:41:09 [INFO] [TRAIN] epoch=53, iter=34550/40000, loss=0.1516, lr=0.001663, batch_cost=0.7921, reader_cost=0.0001 | ETA 01:11:56 2021-01-18 17:41:17 [INFO] [TRAIN] epoch=53, iter=34560/40000, loss=0.2090, lr=0.001661, batch_cost=0.7859, reader_cost=0.0001 | ETA 01:11:15 2021-01-18 17:41:25 [INFO] [TRAIN] epoch=53, iter=34570/40000, loss=0.1566, lr=0.001658, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:11:33 2021-01-18 17:41:33 [INFO] [TRAIN] epoch=53, iter=34580/40000, loss=0.1560, lr=0.001655, batch_cost=0.7936, reader_cost=0.0001 | ETA 01:11:41 2021-01-18 17:41:41 [INFO] [TRAIN] epoch=53, iter=34590/40000, loss=0.0998, lr=0.001652, batch_cost=0.7865, reader_cost=0.0001 | ETA 01:10:55 2021-01-18 17:41:49 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.1140, lr=0.001650, batch_cost=0.7935, reader_cost=0.0001 | ETA 01:11:24 2021-01-18 17:41:57 [INFO] [TRAIN] epoch=53, iter=34610/40000, loss=0.1304, lr=0.001647, batch_cost=0.7950, reader_cost=0.0001 | ETA 01:11:24 2021-01-18 17:42:05 [INFO] [TRAIN] epoch=53, iter=34620/40000, loss=0.1130, lr=0.001644, batch_cost=0.7895, reader_cost=0.0001 | ETA 01:10:47 2021-01-18 17:42:13 [INFO] [TRAIN] epoch=53, iter=34630/40000, loss=0.1288, lr=0.001641, batch_cost=0.7862, reader_cost=0.0001 | ETA 01:10:21 2021-01-18 17:42:21 [INFO] [TRAIN] epoch=53, iter=34640/40000, loss=0.1661, lr=0.001639, batch_cost=0.7883, reader_cost=0.0001 | ETA 01:10:25 2021-01-18 17:42:28 [INFO] [TRAIN] epoch=53, iter=34650/40000, loss=0.0890, lr=0.001636, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:10:16 2021-01-18 17:42:36 [INFO] [TRAIN] epoch=53, iter=34660/40000, loss=0.1465, lr=0.001633, batch_cost=0.7942, reader_cost=0.0001 | ETA 01:10:41 2021-01-18 17:42:44 [INFO] [TRAIN] epoch=53, iter=34670/40000, loss=0.1599, lr=0.001630, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:09:58 2021-01-18 17:42:52 [INFO] [TRAIN] epoch=53, iter=34680/40000, loss=0.1630, lr=0.001628, batch_cost=0.7856, reader_cost=0.0001 | ETA 01:09:39 2021-01-18 17:43:00 [INFO] [TRAIN] epoch=53, iter=34690/40000, loss=0.1884, lr=0.001625, batch_cost=0.7905, reader_cost=0.0001 | ETA 01:09:57 2021-01-18 17:43:08 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.1212, lr=0.001622, batch_cost=0.7910, reader_cost=0.0001 | ETA 01:09:52 2021-01-18 17:43:16 [INFO] [TRAIN] epoch=53, iter=34710/40000, loss=0.0977, lr=0.001619, batch_cost=0.7925, reader_cost=0.0001 | ETA 01:09:52 2021-01-18 17:43:24 [INFO] [TRAIN] epoch=53, iter=34720/40000, loss=0.1156, lr=0.001617, batch_cost=0.7900, reader_cost=0.0001 | ETA 01:09:31 2021-01-18 17:43:32 [INFO] [TRAIN] epoch=53, iter=34730/40000, loss=0.0988, lr=0.001614, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:09:35 2021-01-18 17:43:40 [INFO] [TRAIN] epoch=53, iter=34740/40000, loss=0.0899, lr=0.001611, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:09:17 2021-01-18 17:43:47 [INFO] [TRAIN] epoch=53, iter=34750/40000, loss=0.1484, lr=0.001608, batch_cost=0.7891, reader_cost=0.0001 | ETA 01:09:02 2021-01-18 17:43:55 [INFO] [TRAIN] epoch=53, iter=34760/40000, loss=0.0836, lr=0.001606, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:08:55 2021-01-18 17:44:03 [INFO] [TRAIN] epoch=53, iter=34770/40000, loss=0.1411, lr=0.001603, batch_cost=0.7975, reader_cost=0.0001 | ETA 01:09:30 2021-01-18 17:44:11 [INFO] [TRAIN] epoch=53, iter=34780/40000, loss=0.1011, lr=0.001600, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:08:50 2021-01-18 17:44:19 [INFO] [TRAIN] epoch=53, iter=34790/40000, loss=0.1373, lr=0.001597, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:08:39 2021-01-18 17:44:27 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.1185, lr=0.001594, batch_cost=0.7926, reader_cost=0.0001 | ETA 01:08:41 2021-01-18 17:44:35 [INFO] [TRAIN] epoch=53, iter=34810/40000, loss=0.0924, lr=0.001592, batch_cost=0.7945, reader_cost=0.0001 | ETA 01:08:43 2021-01-18 17:44:43 [INFO] [TRAIN] epoch=53, iter=34820/40000, loss=0.1635, lr=0.001589, batch_cost=0.7898, reader_cost=0.0001 | ETA 01:08:11 2021-01-18 17:44:51 [INFO] [TRAIN] epoch=53, iter=34830/40000, loss=0.1143, lr=0.001586, batch_cost=0.7951, reader_cost=0.0001 | ETA 01:08:30 2021-01-18 17:44:59 [INFO] [TRAIN] epoch=53, iter=34840/40000, loss=0.2261, lr=0.001583, batch_cost=0.7899, reader_cost=0.0001 | ETA 01:07:56 2021-01-18 17:45:07 [INFO] [TRAIN] epoch=53, iter=34850/40000, loss=0.1056, lr=0.001581, batch_cost=0.7956, reader_cost=0.0001 | ETA 01:08:17 2021-01-18 17:45:15 [INFO] [TRAIN] epoch=53, iter=34860/40000, loss=0.1534, lr=0.001578, batch_cost=0.7927, reader_cost=0.0001 | ETA 01:07:54 2021-01-18 17:45:23 [INFO] [TRAIN] epoch=53, iter=34870/40000, loss=0.1619, lr=0.001575, batch_cost=0.7890, reader_cost=0.0001 | ETA 01:07:27 2021-01-18 17:45:30 [INFO] [TRAIN] epoch=53, iter=34880/40000, loss=0.1085, lr=0.001572, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:07:31 2021-01-18 17:45:38 [INFO] [TRAIN] epoch=53, iter=34890/40000, loss=0.1169, lr=0.001570, batch_cost=0.7858, reader_cost=0.0001 | ETA 01:06:55 2021-01-18 17:45:46 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.1110, lr=0.001567, batch_cost=0.7859, reader_cost=0.0001 | ETA 01:06:47 2021-01-18 17:45:54 [INFO] [TRAIN] epoch=53, iter=34910/40000, loss=0.0947, lr=0.001564, batch_cost=0.7906, reader_cost=0.0001 | ETA 01:07:04 2021-01-18 17:46:02 [INFO] [TRAIN] epoch=53, iter=34920/40000, loss=0.1011, lr=0.001561, batch_cost=0.7887, reader_cost=0.0001 | ETA 01:06:46 2021-01-18 17:46:10 [INFO] [TRAIN] epoch=53, iter=34930/40000, loss=0.1145, lr=0.001559, batch_cost=0.7874, reader_cost=0.0001 | ETA 01:06:32 2021-01-18 17:46:18 [INFO] [TRAIN] epoch=53, iter=34940/40000, loss=0.0954, lr=0.001556, batch_cost=0.7952, reader_cost=0.0001 | ETA 01:07:03 2021-01-18 17:46:26 [INFO] [TRAIN] epoch=53, iter=34950/40000, loss=0.1255, lr=0.001553, batch_cost=0.7899, reader_cost=0.0001 | ETA 01:06:28 2021-01-18 17:46:34 [INFO] [TRAIN] epoch=53, iter=34960/40000, loss=0.1207, lr=0.001550, batch_cost=0.7960, reader_cost=0.0001 | ETA 01:06:51 2021-01-18 17:46:42 [INFO] [TRAIN] epoch=53, iter=34970/40000, loss=0.1142, lr=0.001548, batch_cost=0.7965, reader_cost=0.0001 | ETA 01:06:46 2021-01-18 17:46:50 [INFO] [TRAIN] epoch=53, iter=34980/40000, loss=0.1155, lr=0.001545, batch_cost=0.7880, reader_cost=0.0001 | ETA 01:05:55 2021-01-18 17:46:57 [INFO] [TRAIN] epoch=53, iter=34990/40000, loss=0.0974, lr=0.001542, batch_cost=0.7868, reader_cost=0.0001 | ETA 01:05:41 2021-01-18 17:47:05 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.1443, lr=0.001539, batch_cost=0.7894, reader_cost=0.0001 | ETA 01:05:46 2021-01-18 17:47:05 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 17:47:35 [INFO] [EVAL] #Images=1449 mIoU=0.7995 Acc=0.9557 Kappa=0.9016 2021-01-18 17:47:35 [INFO] [EVAL] Class IoU: [0.949 0.9133 0.4362 0.9099 0.6598 0.8459 0.9418 0.8954 0.9328 0.4651 0.9131 0.6025 0.8921 0.9004 0.8942 0.8878 0.6807 0.8912 0.4849 0.9192 0.7742] 2021-01-18 17:47:35 [INFO] [EVAL] Class Acc: [0.9673 0.9429 0.4538 0.953 0.7861 0.9034 0.9865 0.9469 0.958 0.6939 0.9616 0.9089 0.9458 0.9599 0.9349 0.9555 0.8486 0.9433 0.8619 0.959 0.9289] 2021-01-18 17:47:42 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 17:47:50 [INFO] [TRAIN] epoch=53, iter=35010/40000, loss=0.1565, lr=0.001536, batch_cost=0.7805, reader_cost=0.0001 | ETA 01:04:54 2021-01-18 17:47:58 [INFO] [TRAIN] epoch=53, iter=35020/40000, loss=0.1397, lr=0.001534, batch_cost=0.7881, reader_cost=0.0001 | ETA 01:05:24 2021-01-18 17:48:05 [INFO] [TRAIN] epoch=53, iter=35030/40000, loss=0.1112, lr=0.001531, batch_cost=0.7770, reader_cost=0.0001 | ETA 01:04:21 2021-01-18 17:48:13 [INFO] [TRAIN] epoch=54, iter=35040/40000, loss=0.0979, lr=0.001528, batch_cost=0.7909, reader_cost=0.0159 | ETA 01:05:23 2021-01-18 17:48:21 [INFO] [TRAIN] epoch=54, iter=35050/40000, loss=0.1495, lr=0.001525, batch_cost=0.7878, reader_cost=0.0001 | ETA 01:04:59 2021-01-18 17:48:29 [INFO] [TRAIN] epoch=54, iter=35060/40000, loss=0.1387, lr=0.001523, batch_cost=0.7924, reader_cost=0.0001 | ETA 01:05:14 2021-01-18 17:48:37 [INFO] [TRAIN] epoch=54, iter=35070/40000, loss=0.1136, lr=0.001520, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:04:58 2021-01-18 17:48:45 [INFO] [TRAIN] epoch=54, iter=35080/40000, loss=0.1079, lr=0.001517, batch_cost=0.7852, reader_cost=0.0001 | ETA 01:04:23 2021-01-18 17:48:53 [INFO] [TRAIN] epoch=54, iter=35090/40000, loss=0.1071, lr=0.001514, batch_cost=0.7904, reader_cost=0.0001 | ETA 01:04:40 2021-01-18 17:49:01 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.1360, lr=0.001511, batch_cost=0.7854, reader_cost=0.0001 | ETA 01:04:08 2021-01-18 17:49:08 [INFO] [TRAIN] epoch=54, iter=35110/40000, loss=0.1180, lr=0.001509, batch_cost=0.7883, reader_cost=0.0001 | ETA 01:04:14 2021-01-18 17:49:16 [INFO] [TRAIN] epoch=54, iter=35120/40000, loss=0.1171, lr=0.001506, batch_cost=0.7886, reader_cost=0.0001 | ETA 01:04:08 2021-01-18 17:49:24 [INFO] [TRAIN] epoch=54, iter=35130/40000, loss=0.0949, lr=0.001503, batch_cost=0.7947, reader_cost=0.0001 | ETA 01:04:30 2021-01-18 17:49:32 [INFO] [TRAIN] epoch=54, iter=35140/40000, loss=0.1277, lr=0.001500, batch_cost=0.7905, reader_cost=0.0001 | ETA 01:04:01 2021-01-18 17:49:40 [INFO] [TRAIN] epoch=54, iter=35150/40000, loss=0.1111, lr=0.001498, batch_cost=0.7960, reader_cost=0.0001 | ETA 01:04:20 2021-01-18 17:49:48 [INFO] [TRAIN] epoch=54, iter=35160/40000, loss=0.1751, lr=0.001495, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:03:58 2021-01-18 17:49:56 [INFO] [TRAIN] epoch=54, iter=35170/40000, loss=0.0964, lr=0.001492, batch_cost=0.7831, reader_cost=0.0001 | ETA 01:03:02 2021-01-18 17:50:04 [INFO] [TRAIN] epoch=54, iter=35180/40000, loss=0.0857, lr=0.001489, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:03:37 2021-01-18 17:50:12 [INFO] [TRAIN] epoch=54, iter=35190/40000, loss=0.1136, lr=0.001486, batch_cost=0.7935, reader_cost=0.0001 | ETA 01:03:36 2021-01-18 17:50:20 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.1220, lr=0.001484, batch_cost=0.7903, reader_cost=0.0001 | ETA 01:03:13 2021-01-18 17:50:28 [INFO] [TRAIN] epoch=54, iter=35210/40000, loss=0.1341, lr=0.001481, batch_cost=0.7918, reader_cost=0.0001 | ETA 01:03:12 2021-01-18 17:50:35 [INFO] [TRAIN] epoch=54, iter=35220/40000, loss=0.1416, lr=0.001478, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:03:07 2021-01-18 17:50:43 [INFO] [TRAIN] epoch=54, iter=35230/40000, loss=0.1312, lr=0.001475, batch_cost=0.7914, reader_cost=0.0001 | ETA 01:02:55 2021-01-18 17:50:51 [INFO] [TRAIN] epoch=54, iter=35240/40000, loss=0.1548, lr=0.001473, batch_cost=0.7911, reader_cost=0.0001 | ETA 01:02:45 2021-01-18 17:50:59 [INFO] [TRAIN] epoch=54, iter=35250/40000, loss=0.1007, lr=0.001470, batch_cost=0.7908, reader_cost=0.0001 | ETA 01:02:36 2021-01-18 17:51:07 [INFO] [TRAIN] epoch=54, iter=35260/40000, loss=0.1643, lr=0.001467, batch_cost=0.7931, reader_cost=0.0001 | ETA 01:02:39 2021-01-18 17:51:15 [INFO] [TRAIN] epoch=54, iter=35270/40000, loss=0.1278, lr=0.001464, batch_cost=0.7934, reader_cost=0.0001 | ETA 01:02:32 2021-01-18 17:51:23 [INFO] [TRAIN] epoch=54, iter=35280/40000, loss=0.1059, lr=0.001461, batch_cost=0.7913, reader_cost=0.0001 | ETA 01:02:15 2021-01-18 17:51:31 [INFO] [TRAIN] epoch=54, iter=35290/40000, loss=0.1405, lr=0.001459, batch_cost=0.7909, reader_cost=0.0001 | ETA 01:02:05 2021-01-18 17:51:39 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.1116, lr=0.001456, batch_cost=0.7958, reader_cost=0.0001 | ETA 01:02:20 2021-01-18 17:51:47 [INFO] [TRAIN] epoch=54, iter=35310/40000, loss=0.1285, lr=0.001453, batch_cost=0.7886, reader_cost=0.0001 | ETA 01:01:38 2021-01-18 17:51:55 [INFO] [TRAIN] epoch=54, iter=35320/40000, loss=0.1246, lr=0.001450, batch_cost=0.7873, reader_cost=0.0001 | ETA 01:01:24 2021-01-18 17:52:03 [INFO] [TRAIN] epoch=54, iter=35330/40000, loss=0.0916, lr=0.001447, batch_cost=0.7948, reader_cost=0.0001 | ETA 01:01:51 2021-01-18 17:52:11 [INFO] [TRAIN] epoch=54, iter=35340/40000, loss=0.1380, lr=0.001445, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:01:30 2021-01-18 17:52:18 [INFO] [TRAIN] epoch=54, iter=35350/40000, loss=0.1206, lr=0.001442, batch_cost=0.7856, reader_cost=0.0001 | ETA 01:00:53 2021-01-18 17:52:26 [INFO] [TRAIN] epoch=54, iter=35360/40000, loss=0.1070, lr=0.001439, batch_cost=0.7923, reader_cost=0.0001 | ETA 01:01:16 2021-01-18 17:52:34 [INFO] [TRAIN] epoch=54, iter=35370/40000, loss=0.1107, lr=0.001436, batch_cost=0.7876, reader_cost=0.0001 | ETA 01:00:46 2021-01-18 17:52:42 [INFO] [TRAIN] epoch=54, iter=35380/40000, loss=0.1110, lr=0.001434, batch_cost=0.7899, reader_cost=0.0001 | ETA 01:00:49 2021-01-18 17:52:50 [INFO] [TRAIN] epoch=54, iter=35390/40000, loss=0.1338, lr=0.001431, batch_cost=0.7890, reader_cost=0.0001 | ETA 01:00:37 2021-01-18 17:52:58 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.1418, lr=0.001428, batch_cost=0.7919, reader_cost=0.0001 | ETA 01:00:42 2021-01-18 17:53:06 [INFO] [TRAIN] epoch=54, iter=35410/40000, loss=0.1079, lr=0.001425, batch_cost=0.7907, reader_cost=0.0001 | ETA 01:00:29 2021-01-18 17:53:14 [INFO] [TRAIN] epoch=54, iter=35420/40000, loss=0.1047, lr=0.001422, batch_cost=0.7892, reader_cost=0.0001 | ETA 01:00:14 2021-01-18 17:53:22 [INFO] [TRAIN] epoch=54, iter=35430/40000, loss=0.0842, lr=0.001420, batch_cost=0.7841, reader_cost=0.0001 | ETA 00:59:43 2021-01-18 17:53:29 [INFO] [TRAIN] epoch=54, iter=35440/40000, loss=0.1580, lr=0.001417, batch_cost=0.7941, reader_cost=0.0001 | ETA 01:00:21 2021-01-18 17:53:37 [INFO] [TRAIN] epoch=54, iter=35450/40000, loss=0.1132, lr=0.001414, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:59:54 2021-01-18 17:53:45 [INFO] [TRAIN] epoch=54, iter=35460/40000, loss=0.1539, lr=0.001411, batch_cost=0.7864, reader_cost=0.0001 | ETA 00:59:30 2021-01-18 17:53:53 [INFO] [TRAIN] epoch=54, iter=35470/40000, loss=0.1072, lr=0.001408, batch_cost=0.7879, reader_cost=0.0001 | ETA 00:59:29 2021-01-18 17:54:01 [INFO] [TRAIN] epoch=54, iter=35480/40000, loss=0.1399, lr=0.001406, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:59:25 2021-01-18 17:54:09 [INFO] [TRAIN] epoch=54, iter=35490/40000, loss=0.0975, lr=0.001403, batch_cost=0.7929, reader_cost=0.0001 | ETA 00:59:36 2021-01-18 17:54:17 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0900, lr=0.001400, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:59:15 2021-01-18 17:54:25 [INFO] [TRAIN] epoch=54, iter=35510/40000, loss=0.1287, lr=0.001397, batch_cost=0.7859, reader_cost=0.0001 | ETA 00:58:48 2021-01-18 17:54:33 [INFO] [TRAIN] epoch=54, iter=35520/40000, loss=0.0809, lr=0.001394, batch_cost=0.7961, reader_cost=0.0001 | ETA 00:59:26 2021-01-18 17:54:40 [INFO] [TRAIN] epoch=54, iter=35530/40000, loss=0.1414, lr=0.001392, batch_cost=0.7832, reader_cost=0.0001 | ETA 00:58:20 2021-01-18 17:54:48 [INFO] [TRAIN] epoch=54, iter=35540/40000, loss=0.1034, lr=0.001389, batch_cost=0.7896, reader_cost=0.0001 | ETA 00:58:41 2021-01-18 17:54:56 [INFO] [TRAIN] epoch=54, iter=35550/40000, loss=0.1459, lr=0.001386, batch_cost=0.7908, reader_cost=0.0001 | ETA 00:58:39 2021-01-18 17:55:04 [INFO] [TRAIN] epoch=54, iter=35560/40000, loss=0.1034, lr=0.001383, batch_cost=0.7902, reader_cost=0.0001 | ETA 00:58:28 2021-01-18 17:55:12 [INFO] [TRAIN] epoch=54, iter=35570/40000, loss=0.1528, lr=0.001380, batch_cost=0.7887, reader_cost=0.0001 | ETA 00:58:13 2021-01-18 17:55:20 [INFO] [TRAIN] epoch=54, iter=35580/40000, loss=0.1757, lr=0.001378, batch_cost=0.7864, reader_cost=0.0001 | ETA 00:57:55 2021-01-18 17:55:28 [INFO] [TRAIN] epoch=54, iter=35590/40000, loss=0.0997, lr=0.001375, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:58:11 2021-01-18 17:55:36 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.1453, lr=0.001372, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:57:53 2021-01-18 17:55:44 [INFO] [TRAIN] epoch=54, iter=35610/40000, loss=0.1181, lr=0.001369, batch_cost=0.7930, reader_cost=0.0001 | ETA 00:58:01 2021-01-18 17:55:52 [INFO] [TRAIN] epoch=54, iter=35620/40000, loss=0.0901, lr=0.001366, batch_cost=0.7877, reader_cost=0.0001 | ETA 00:57:30 2021-01-18 17:55:59 [INFO] [TRAIN] epoch=54, iter=35630/40000, loss=0.1550, lr=0.001364, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:57:47 2021-01-18 17:56:07 [INFO] [TRAIN] epoch=54, iter=35640/40000, loss=0.1267, lr=0.001361, batch_cost=0.7919, reader_cost=0.0001 | ETA 00:57:32 2021-01-18 17:56:15 [INFO] [TRAIN] epoch=54, iter=35650/40000, loss=0.1098, lr=0.001358, batch_cost=0.7852, reader_cost=0.0001 | ETA 00:56:55 2021-01-18 17:56:23 [INFO] [TRAIN] epoch=54, iter=35660/40000, loss=0.1323, lr=0.001355, batch_cost=0.7850, reader_cost=0.0001 | ETA 00:56:46 2021-01-18 17:56:31 [INFO] [TRAIN] epoch=54, iter=35670/40000, loss=0.1409, lr=0.001352, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:56:56 2021-01-18 17:56:39 [INFO] [TRAIN] epoch=54, iter=35680/40000, loss=0.0770, lr=0.001349, batch_cost=0.7879, reader_cost=0.0001 | ETA 00:56:43 2021-01-18 17:56:47 [INFO] [TRAIN] epoch=54, iter=35690/40000, loss=0.1083, lr=0.001347, batch_cost=0.7872, reader_cost=0.0001 | ETA 00:56:32 2021-01-18 17:56:55 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.1100, lr=0.001344, batch_cost=0.7819, reader_cost=0.0154 | ETA 00:56:02 2021-01-18 17:57:03 [INFO] [TRAIN] epoch=55, iter=35710/40000, loss=0.1118, lr=0.001341, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:56:45 2021-01-18 17:57:10 [INFO] [TRAIN] epoch=55, iter=35720/40000, loss=0.0895, lr=0.001338, batch_cost=0.7946, reader_cost=0.0001 | ETA 00:56:40 2021-01-18 17:57:18 [INFO] [TRAIN] epoch=55, iter=35730/40000, loss=0.0996, lr=0.001335, batch_cost=0.7874, reader_cost=0.0001 | ETA 00:56:02 2021-01-18 17:57:26 [INFO] [TRAIN] epoch=55, iter=35740/40000, loss=0.1191, lr=0.001333, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:56:12 2021-01-18 17:57:34 [INFO] [TRAIN] epoch=55, iter=35750/40000, loss=0.1318, lr=0.001330, batch_cost=0.7883, reader_cost=0.0001 | ETA 00:55:50 2021-01-18 17:57:42 [INFO] [TRAIN] epoch=55, iter=35760/40000, loss=0.1513, lr=0.001327, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:55:52 2021-01-18 17:57:50 [INFO] [TRAIN] epoch=55, iter=35770/40000, loss=0.0879, lr=0.001324, batch_cost=0.7954, reader_cost=0.0001 | ETA 00:56:04 2021-01-18 17:57:58 [INFO] [TRAIN] epoch=55, iter=35780/40000, loss=0.1019, lr=0.001321, batch_cost=0.7875, reader_cost=0.0001 | ETA 00:55:23 2021-01-18 17:58:06 [INFO] [TRAIN] epoch=55, iter=35790/40000, loss=0.1114, lr=0.001319, batch_cost=0.7912, reader_cost=0.0001 | ETA 00:55:30 2021-01-18 17:58:14 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0901, lr=0.001316, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:55:29 2021-01-18 17:58:22 [INFO] [TRAIN] epoch=55, iter=35810/40000, loss=0.1374, lr=0.001313, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:55:23 2021-01-18 17:58:30 [INFO] [TRAIN] epoch=55, iter=35820/40000, loss=0.1124, lr=0.001310, batch_cost=0.7876, reader_cost=0.0001 | ETA 00:54:52 2021-01-18 17:58:37 [INFO] [TRAIN] epoch=55, iter=35830/40000, loss=0.1315, lr=0.001307, batch_cost=0.7839, reader_cost=0.0001 | ETA 00:54:28 2021-01-18 17:58:45 [INFO] [TRAIN] epoch=55, iter=35840/40000, loss=0.0970, lr=0.001304, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:54:38 2021-01-18 17:58:53 [INFO] [TRAIN] epoch=55, iter=35850/40000, loss=0.1188, lr=0.001302, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:54:31 2021-01-18 17:59:01 [INFO] [TRAIN] epoch=55, iter=35860/40000, loss=0.1086, lr=0.001299, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:54:35 2021-01-18 17:59:09 [INFO] [TRAIN] epoch=55, iter=35870/40000, loss=0.1826, lr=0.001296, batch_cost=0.7903, reader_cost=0.0001 | ETA 00:54:23 2021-01-18 17:59:17 [INFO] [TRAIN] epoch=55, iter=35880/40000, loss=0.0919, lr=0.001293, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:54:24 2021-01-18 17:59:25 [INFO] [TRAIN] epoch=55, iter=35890/40000, loss=0.1236, lr=0.001290, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:53:58 2021-01-18 17:59:33 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.1479, lr=0.001287, batch_cost=0.7898, reader_cost=0.0001 | ETA 00:53:58 2021-01-18 17:59:41 [INFO] [TRAIN] epoch=55, iter=35910/40000, loss=0.1483, lr=0.001285, batch_cost=0.7870, reader_cost=0.0001 | ETA 00:53:38 2021-01-18 17:59:48 [INFO] [TRAIN] epoch=55, iter=35920/40000, loss=0.1000, lr=0.001282, batch_cost=0.7888, reader_cost=0.0001 | ETA 00:53:38 2021-01-18 17:59:56 [INFO] [TRAIN] epoch=55, iter=35930/40000, loss=0.1383, lr=0.001279, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:53:45 2021-01-18 18:00:04 [INFO] [TRAIN] epoch=55, iter=35940/40000, loss=0.1101, lr=0.001276, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:53:30 2021-01-18 18:00:12 [INFO] [TRAIN] epoch=55, iter=35950/40000, loss=0.1173, lr=0.001273, batch_cost=0.7907, reader_cost=0.0001 | ETA 00:53:22 2021-01-18 18:00:20 [INFO] [TRAIN] epoch=55, iter=35960/40000, loss=0.1104, lr=0.001271, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:53:16 2021-01-18 18:00:28 [INFO] [TRAIN] epoch=55, iter=35970/40000, loss=0.1241, lr=0.001268, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:52:50 2021-01-18 18:00:36 [INFO] [TRAIN] epoch=55, iter=35980/40000, loss=0.1232, lr=0.001265, batch_cost=0.7935, reader_cost=0.0001 | ETA 00:53:09 2021-01-18 18:00:44 [INFO] [TRAIN] epoch=55, iter=35990/40000, loss=0.1170, lr=0.001262, batch_cost=0.7885, reader_cost=0.0001 | ETA 00:52:42 2021-01-18 18:00:52 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.1966, lr=0.001259, batch_cost=0.7940, reader_cost=0.0001 | ETA 00:52:55 2021-01-18 18:00:52 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 18:01:19 [INFO] [EVAL] #Images=1449 mIoU=0.7965 Acc=0.9553 Kappa=0.9005 2021-01-18 18:01:19 [INFO] [EVAL] Class IoU: [0.9483 0.9112 0.4389 0.9075 0.6961 0.8151 0.9349 0.8935 0.9359 0.4653 0.8724 0.5846 0.8933 0.8878 0.8955 0.8896 0.6628 0.862 0.5372 0.9105 0.7838] 2021-01-18 18:01:19 [INFO] [EVAL] Class Acc: [0.9663 0.9374 0.4507 0.9485 0.7758 0.9098 0.9869 0.9465 0.9624 0.7731 0.9428 0.9215 0.9435 0.9387 0.931 0.961 0.8295 0.9582 0.8356 0.959 0.9641] 2021-01-18 18:01:26 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 18:01:34 [INFO] [TRAIN] epoch=55, iter=36010/40000, loss=0.0989, lr=0.001256, batch_cost=0.7827, reader_cost=0.0001 | ETA 00:52:03 2021-01-18 18:01:42 [INFO] [TRAIN] epoch=55, iter=36020/40000, loss=0.1169, lr=0.001254, batch_cost=0.7886, reader_cost=0.0001 | ETA 00:52:18 2021-01-18 18:01:50 [INFO] [TRAIN] epoch=55, iter=36030/40000, loss=0.0917, lr=0.001251, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:52:09 2021-01-18 18:01:57 [INFO] [TRAIN] epoch=55, iter=36040/40000, loss=0.1683, lr=0.001248, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:52:05 2021-01-18 18:02:05 [INFO] [TRAIN] epoch=55, iter=36050/40000, loss=0.1059, lr=0.001245, batch_cost=0.7867, reader_cost=0.0001 | ETA 00:51:47 2021-01-18 18:02:13 [INFO] [TRAIN] epoch=55, iter=36060/40000, loss=0.0898, lr=0.001242, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:51:52 2021-01-18 18:02:21 [INFO] [TRAIN] epoch=55, iter=36070/40000, loss=0.0678, lr=0.001239, batch_cost=0.7859, reader_cost=0.0001 | ETA 00:51:28 2021-01-18 18:02:29 [INFO] [TRAIN] epoch=55, iter=36080/40000, loss=0.1086, lr=0.001237, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:51:29 2021-01-18 18:02:37 [INFO] [TRAIN] epoch=55, iter=36090/40000, loss=0.1178, lr=0.001234, batch_cost=0.8000, reader_cost=0.0001 | ETA 00:52:07 2021-01-18 18:02:45 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.1218, lr=0.001231, batch_cost=0.7945, reader_cost=0.0001 | ETA 00:51:38 2021-01-18 18:02:53 [INFO] [TRAIN] epoch=55, iter=36110/40000, loss=0.1131, lr=0.001228, batch_cost=0.7885, reader_cost=0.0001 | ETA 00:51:07 2021-01-18 18:03:01 [INFO] [TRAIN] epoch=55, iter=36120/40000, loss=0.1327, lr=0.001225, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:50:52 2021-01-18 18:03:09 [INFO] [TRAIN] epoch=55, iter=36130/40000, loss=0.1070, lr=0.001222, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:50:59 2021-01-18 18:03:16 [INFO] [TRAIN] epoch=55, iter=36140/40000, loss=0.1248, lr=0.001219, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:50:51 2021-01-18 18:03:24 [INFO] [TRAIN] epoch=55, iter=36150/40000, loss=0.1187, lr=0.001217, batch_cost=0.7905, reader_cost=0.0001 | ETA 00:50:43 2021-01-18 18:03:32 [INFO] [TRAIN] epoch=55, iter=36160/40000, loss=0.1682, lr=0.001214, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:50:37 2021-01-18 18:03:40 [INFO] [TRAIN] epoch=55, iter=36170/40000, loss=0.1386, lr=0.001211, batch_cost=0.7934, reader_cost=0.0001 | ETA 00:50:38 2021-01-18 18:03:48 [INFO] [TRAIN] epoch=55, iter=36180/40000, loss=0.1048, lr=0.001208, batch_cost=0.7865, reader_cost=0.0001 | ETA 00:50:04 2021-01-18 18:03:56 [INFO] [TRAIN] epoch=55, iter=36190/40000, loss=0.1489, lr=0.001205, batch_cost=0.7917, reader_cost=0.0001 | ETA 00:50:16 2021-01-18 18:04:04 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0989, lr=0.001202, batch_cost=0.7886, reader_cost=0.0001 | ETA 00:49:56 2021-01-18 18:04:12 [INFO] [TRAIN] epoch=55, iter=36210/40000, loss=0.1117, lr=0.001200, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:49:51 2021-01-18 18:04:20 [INFO] [TRAIN] epoch=55, iter=36220/40000, loss=0.1166, lr=0.001197, batch_cost=0.7919, reader_cost=0.0001 | ETA 00:49:53 2021-01-18 18:04:28 [INFO] [TRAIN] epoch=55, iter=36230/40000, loss=0.1585, lr=0.001194, batch_cost=0.7852, reader_cost=0.0001 | ETA 00:49:20 2021-01-18 18:04:35 [INFO] [TRAIN] epoch=55, iter=36240/40000, loss=0.1202, lr=0.001191, batch_cost=0.7893, reader_cost=0.0001 | ETA 00:49:27 2021-01-18 18:04:43 [INFO] [TRAIN] epoch=55, iter=36250/40000, loss=0.1428, lr=0.001188, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:49:34 2021-01-18 18:04:51 [INFO] [TRAIN] epoch=55, iter=36260/40000, loss=0.1111, lr=0.001185, batch_cost=0.7879, reader_cost=0.0001 | ETA 00:49:06 2021-01-18 18:04:59 [INFO] [TRAIN] epoch=55, iter=36270/40000, loss=0.1272, lr=0.001182, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:49:10 2021-01-18 18:05:07 [INFO] [TRAIN] epoch=55, iter=36280/40000, loss=0.1441, lr=0.001180, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:48:59 2021-01-18 18:05:15 [INFO] [TRAIN] epoch=55, iter=36290/40000, loss=0.1251, lr=0.001177, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:48:56 2021-01-18 18:05:23 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.1002, lr=0.001174, batch_cost=0.7905, reader_cost=0.0001 | ETA 00:48:45 2021-01-18 18:05:31 [INFO] [TRAIN] epoch=55, iter=36310/40000, loss=0.1195, lr=0.001171, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:48:27 2021-01-18 18:05:39 [INFO] [TRAIN] epoch=55, iter=36320/40000, loss=0.0869, lr=0.001168, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:48:32 2021-01-18 18:05:47 [INFO] [TRAIN] epoch=55, iter=36330/40000, loss=0.0867, lr=0.001165, batch_cost=0.7893, reader_cost=0.0001 | ETA 00:48:16 2021-01-18 18:05:54 [INFO] [TRAIN] epoch=55, iter=36340/40000, loss=0.1465, lr=0.001162, batch_cost=0.7903, reader_cost=0.0001 | ETA 00:48:12 2021-01-18 18:06:02 [INFO] [TRAIN] epoch=55, iter=36350/40000, loss=0.1055, lr=0.001160, batch_cost=0.7841, reader_cost=0.0001 | ETA 00:47:42 2021-01-18 18:06:10 [INFO] [TRAIN] epoch=56, iter=36360/40000, loss=0.1057, lr=0.001157, batch_cost=0.7786, reader_cost=0.0157 | ETA 00:47:14 2021-01-18 18:06:18 [INFO] [TRAIN] epoch=56, iter=36370/40000, loss=0.1353, lr=0.001154, batch_cost=0.7948, reader_cost=0.0001 | ETA 00:48:04 2021-01-18 18:06:26 [INFO] [TRAIN] epoch=56, iter=36380/40000, loss=0.0880, lr=0.001151, batch_cost=0.7851, reader_cost=0.0001 | ETA 00:47:22 2021-01-18 18:06:34 [INFO] [TRAIN] epoch=56, iter=36390/40000, loss=0.1016, lr=0.001148, batch_cost=0.7929, reader_cost=0.0001 | ETA 00:47:42 2021-01-18 18:06:42 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.1319, lr=0.001145, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:47:17 2021-01-18 18:06:50 [INFO] [TRAIN] epoch=56, iter=36410/40000, loss=0.1284, lr=0.001142, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:47:29 2021-01-18 18:06:58 [INFO] [TRAIN] epoch=56, iter=36420/40000, loss=0.0827, lr=0.001140, batch_cost=0.7874, reader_cost=0.0001 | ETA 00:46:58 2021-01-18 18:07:05 [INFO] [TRAIN] epoch=56, iter=36430/40000, loss=0.1547, lr=0.001137, batch_cost=0.7888, reader_cost=0.0001 | ETA 00:46:56 2021-01-18 18:07:13 [INFO] [TRAIN] epoch=56, iter=36440/40000, loss=0.1235, lr=0.001134, batch_cost=0.7970, reader_cost=0.0001 | ETA 00:47:17 2021-01-18 18:07:21 [INFO] [TRAIN] epoch=56, iter=36450/40000, loss=0.1001, lr=0.001131, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:46:37 2021-01-18 18:07:29 [INFO] [TRAIN] epoch=56, iter=36460/40000, loss=0.1267, lr=0.001128, batch_cost=0.7907, reader_cost=0.0001 | ETA 00:46:39 2021-01-18 18:07:37 [INFO] [TRAIN] epoch=56, iter=36470/40000, loss=0.1190, lr=0.001125, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:46:32 2021-01-18 18:07:45 [INFO] [TRAIN] epoch=56, iter=36480/40000, loss=0.1438, lr=0.001122, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:46:13 2021-01-18 18:07:53 [INFO] [TRAIN] epoch=56, iter=36490/40000, loss=0.1053, lr=0.001120, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:46:15 2021-01-18 18:08:01 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.1133, lr=0.001117, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:46:14 2021-01-18 18:08:09 [INFO] [TRAIN] epoch=56, iter=36510/40000, loss=0.0844, lr=0.001114, batch_cost=0.7912, reader_cost=0.0001 | ETA 00:46:01 2021-01-18 18:08:17 [INFO] [TRAIN] epoch=56, iter=36520/40000, loss=0.1031, lr=0.001111, batch_cost=0.7917, reader_cost=0.0001 | ETA 00:45:55 2021-01-18 18:08:25 [INFO] [TRAIN] epoch=56, iter=36530/40000, loss=0.1053, lr=0.001108, batch_cost=0.7971, reader_cost=0.0001 | ETA 00:46:05 2021-01-18 18:08:32 [INFO] [TRAIN] epoch=56, iter=36540/40000, loss=0.1085, lr=0.001105, batch_cost=0.7874, reader_cost=0.0001 | ETA 00:45:24 2021-01-18 18:08:40 [INFO] [TRAIN] epoch=56, iter=36550/40000, loss=0.1562, lr=0.001102, batch_cost=0.7912, reader_cost=0.0001 | ETA 00:45:29 2021-01-18 18:08:48 [INFO] [TRAIN] epoch=56, iter=36560/40000, loss=0.1325, lr=0.001099, batch_cost=0.7945, reader_cost=0.0001 | ETA 00:45:33 2021-01-18 18:08:56 [INFO] [TRAIN] epoch=56, iter=36570/40000, loss=0.1262, lr=0.001097, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:45:05 2021-01-18 18:09:04 [INFO] [TRAIN] epoch=56, iter=36580/40000, loss=0.1202, lr=0.001094, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:45:01 2021-01-18 18:09:12 [INFO] [TRAIN] epoch=56, iter=36590/40000, loss=0.1244, lr=0.001091, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:45:02 2021-01-18 18:09:20 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.1075, lr=0.001088, batch_cost=0.7875, reader_cost=0.0001 | ETA 00:44:37 2021-01-18 18:09:28 [INFO] [TRAIN] epoch=56, iter=36610/40000, loss=0.1278, lr=0.001085, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:44:50 2021-01-18 18:09:36 [INFO] [TRAIN] epoch=56, iter=36620/40000, loss=0.0922, lr=0.001082, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:44:39 2021-01-18 18:09:44 [INFO] [TRAIN] epoch=56, iter=36630/40000, loss=0.1295, lr=0.001079, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:44:19 2021-01-18 18:09:52 [INFO] [TRAIN] epoch=56, iter=36640/40000, loss=0.1334, lr=0.001076, batch_cost=0.7873, reader_cost=0.0001 | ETA 00:44:05 2021-01-18 18:09:59 [INFO] [TRAIN] epoch=56, iter=36650/40000, loss=0.1076, lr=0.001074, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:44:00 2021-01-18 18:10:07 [INFO] [TRAIN] epoch=56, iter=36660/40000, loss=0.1038, lr=0.001071, batch_cost=0.7864, reader_cost=0.0001 | ETA 00:43:46 2021-01-18 18:10:15 [INFO] [TRAIN] epoch=56, iter=36670/40000, loss=0.1169, lr=0.001068, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:43:52 2021-01-18 18:10:23 [INFO] [TRAIN] epoch=56, iter=36680/40000, loss=0.1146, lr=0.001065, batch_cost=0.7941, reader_cost=0.0001 | ETA 00:43:56 2021-01-18 18:10:31 [INFO] [TRAIN] epoch=56, iter=36690/40000, loss=0.1089, lr=0.001062, batch_cost=0.7948, reader_cost=0.0001 | ETA 00:43:50 2021-01-18 18:10:39 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.1078, lr=0.001059, batch_cost=0.7851, reader_cost=0.0001 | ETA 00:43:10 2021-01-18 18:10:47 [INFO] [TRAIN] epoch=56, iter=36710/40000, loss=0.0887, lr=0.001056, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:43:19 2021-01-18 18:10:55 [INFO] [TRAIN] epoch=56, iter=36720/40000, loss=0.1502, lr=0.001053, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:43:18 2021-01-18 18:11:03 [INFO] [TRAIN] epoch=56, iter=36730/40000, loss=0.0900, lr=0.001050, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:43:09 2021-01-18 18:11:11 [INFO] [TRAIN] epoch=56, iter=36740/40000, loss=0.1150, lr=0.001048, batch_cost=0.7886, reader_cost=0.0001 | ETA 00:42:50 2021-01-18 18:11:19 [INFO] [TRAIN] epoch=56, iter=36750/40000, loss=0.1350, lr=0.001045, batch_cost=0.7974, reader_cost=0.0001 | ETA 00:43:11 2021-01-18 18:11:26 [INFO] [TRAIN] epoch=56, iter=36760/40000, loss=0.1479, lr=0.001042, batch_cost=0.7875, reader_cost=0.0001 | ETA 00:42:31 2021-01-18 18:11:34 [INFO] [TRAIN] epoch=56, iter=36770/40000, loss=0.1153, lr=0.001039, batch_cost=0.7945, reader_cost=0.0001 | ETA 00:42:46 2021-01-18 18:11:42 [INFO] [TRAIN] epoch=56, iter=36780/40000, loss=0.1227, lr=0.001036, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:42:31 2021-01-18 18:11:50 [INFO] [TRAIN] epoch=56, iter=36790/40000, loss=0.1198, lr=0.001033, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:42:22 2021-01-18 18:11:58 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.1188, lr=0.001030, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:42:02 2021-01-18 18:12:06 [INFO] [TRAIN] epoch=56, iter=36810/40000, loss=0.1008, lr=0.001027, batch_cost=0.7931, reader_cost=0.0001 | ETA 00:42:09 2021-01-18 18:12:14 [INFO] [TRAIN] epoch=56, iter=36820/40000, loss=0.1208, lr=0.001024, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:41:49 2021-01-18 18:12:22 [INFO] [TRAIN] epoch=56, iter=36830/40000, loss=0.1092, lr=0.001021, batch_cost=0.7895, reader_cost=0.0001 | ETA 00:41:42 2021-01-18 18:12:30 [INFO] [TRAIN] epoch=56, iter=36840/40000, loss=0.1307, lr=0.001019, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:41:33 2021-01-18 18:12:38 [INFO] [TRAIN] epoch=56, iter=36850/40000, loss=0.1286, lr=0.001016, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:41:22 2021-01-18 18:12:46 [INFO] [TRAIN] epoch=56, iter=36860/40000, loss=0.1086, lr=0.001013, batch_cost=0.7903, reader_cost=0.0001 | ETA 00:41:21 2021-01-18 18:12:53 [INFO] [TRAIN] epoch=56, iter=36870/40000, loss=0.1246, lr=0.001010, batch_cost=0.7878, reader_cost=0.0001 | ETA 00:41:05 2021-01-18 18:13:01 [INFO] [TRAIN] epoch=56, iter=36880/40000, loss=0.1024, lr=0.001007, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:41:04 2021-01-18 18:13:09 [INFO] [TRAIN] epoch=56, iter=36890/40000, loss=0.0961, lr=0.001004, batch_cost=0.7865, reader_cost=0.0001 | ETA 00:40:45 2021-01-18 18:13:17 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.1124, lr=0.001001, batch_cost=0.7952, reader_cost=0.0001 | ETA 00:41:05 2021-01-18 18:13:25 [INFO] [TRAIN] epoch=56, iter=36910/40000, loss=0.1253, lr=0.000998, batch_cost=0.7896, reader_cost=0.0001 | ETA 00:40:39 2021-01-18 18:13:33 [INFO] [TRAIN] epoch=56, iter=36920/40000, loss=0.1414, lr=0.000995, batch_cost=0.7869, reader_cost=0.0001 | ETA 00:40:23 2021-01-18 18:13:41 [INFO] [TRAIN] epoch=56, iter=36930/40000, loss=0.1060, lr=0.000992, batch_cost=0.7893, reader_cost=0.0001 | ETA 00:40:23 2021-01-18 18:13:49 [INFO] [TRAIN] epoch=56, iter=36940/40000, loss=0.1683, lr=0.000990, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:40:19 2021-01-18 18:13:57 [INFO] [TRAIN] epoch=56, iter=36950/40000, loss=0.1223, lr=0.000987, batch_cost=0.7896, reader_cost=0.0001 | ETA 00:40:08 2021-01-18 18:14:04 [INFO] [TRAIN] epoch=56, iter=36960/40000, loss=0.1471, lr=0.000984, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:39:51 2021-01-18 18:14:12 [INFO] [TRAIN] epoch=56, iter=36970/40000, loss=0.1352, lr=0.000981, batch_cost=0.7924, reader_cost=0.0001 | ETA 00:40:01 2021-01-18 18:14:20 [INFO] [TRAIN] epoch=56, iter=36980/40000, loss=0.1312, lr=0.000978, batch_cost=0.7903, reader_cost=0.0001 | ETA 00:39:46 2021-01-18 18:14:28 [INFO] [TRAIN] epoch=56, iter=36990/40000, loss=0.1252, lr=0.000975, batch_cost=0.7828, reader_cost=0.0001 | ETA 00:39:16 2021-01-18 18:14:36 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.1324, lr=0.000972, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:39:33 2021-01-18 18:14:36 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 18:15:04 [INFO] [EVAL] #Images=1449 mIoU=0.7891 Acc=0.9534 Kappa=0.8956 2021-01-18 18:15:04 [INFO] [EVAL] Class IoU: [0.9462 0.9101 0.4431 0.9084 0.663 0.7982 0.9387 0.8878 0.9326 0.4523 0.8967 0.5247 0.8932 0.8982 0.8988 0.89 0.6541 0.8882 0.452 0.9108 0.7844] 2021-01-18 18:15:04 [INFO] [EVAL] Class Acc: [0.9631 0.9432 0.4592 0.9527 0.7768 0.934 0.9882 0.9505 0.9618 0.6586 0.9612 0.9297 0.9455 0.9627 0.95 0.9576 0.8209 0.9494 0.8565 0.9623 0.9662] 2021-01-18 18:15:10 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 18:15:18 [INFO] [TRAIN] epoch=56, iter=37010/40000, loss=0.1227, lr=0.000969, batch_cost=0.7877, reader_cost=0.0001 | ETA 00:39:15 2021-01-18 18:15:26 [INFO] [TRAIN] epoch=57, iter=37020/40000, loss=0.1048, lr=0.000966, batch_cost=0.7752, reader_cost=0.0128 | ETA 00:38:30 2021-01-18 18:15:34 [INFO] [TRAIN] epoch=57, iter=37030/40000, loss=0.1252, lr=0.000963, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:39:06 2021-01-18 18:15:42 [INFO] [TRAIN] epoch=57, iter=37040/40000, loss=0.1344, lr=0.000960, batch_cost=0.7926, reader_cost=0.0001 | ETA 00:39:05 2021-01-18 18:15:50 [INFO] [TRAIN] epoch=57, iter=37050/40000, loss=0.1308, lr=0.000957, batch_cost=0.7945, reader_cost=0.0001 | ETA 00:39:03 2021-01-18 18:15:57 [INFO] [TRAIN] epoch=57, iter=37060/40000, loss=0.1024, lr=0.000955, batch_cost=0.7890, reader_cost=0.0001 | ETA 00:38:39 2021-01-18 18:16:05 [INFO] [TRAIN] epoch=57, iter=37070/40000, loss=0.1302, lr=0.000952, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:38:29 2021-01-18 18:16:13 [INFO] [TRAIN] epoch=57, iter=37080/40000, loss=0.0936, lr=0.000949, batch_cost=0.7902, reader_cost=0.0001 | ETA 00:38:27 2021-01-18 18:16:21 [INFO] [TRAIN] epoch=57, iter=37090/40000, loss=0.1358, lr=0.000946, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:38:21 2021-01-18 18:16:29 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0973, lr=0.000943, batch_cost=0.7954, reader_cost=0.0001 | ETA 00:38:26 2021-01-18 18:16:37 [INFO] [TRAIN] epoch=57, iter=37110/40000, loss=0.1007, lr=0.000940, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:38:00 2021-01-18 18:16:45 [INFO] [TRAIN] epoch=57, iter=37120/40000, loss=0.1428, lr=0.000937, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:38:06 2021-01-18 18:16:53 [INFO] [TRAIN] epoch=57, iter=37130/40000, loss=0.1178, lr=0.000934, batch_cost=0.7926, reader_cost=0.0001 | ETA 00:37:54 2021-01-18 18:17:01 [INFO] [TRAIN] epoch=57, iter=37140/40000, loss=0.1400, lr=0.000931, batch_cost=0.7876, reader_cost=0.0001 | ETA 00:37:32 2021-01-18 18:17:09 [INFO] [TRAIN] epoch=57, iter=37150/40000, loss=0.1390, lr=0.000928, batch_cost=0.7844, reader_cost=0.0001 | ETA 00:37:15 2021-01-18 18:17:17 [INFO] [TRAIN] epoch=57, iter=37160/40000, loss=0.0823, lr=0.000925, batch_cost=0.7961, reader_cost=0.0001 | ETA 00:37:40 2021-01-18 18:17:24 [INFO] [TRAIN] epoch=57, iter=37170/40000, loss=0.1167, lr=0.000922, batch_cost=0.7878, reader_cost=0.0001 | ETA 00:37:09 2021-01-18 18:17:32 [INFO] [TRAIN] epoch=57, iter=37180/40000, loss=0.1850, lr=0.000919, batch_cost=0.7916, reader_cost=0.0001 | ETA 00:37:12 2021-01-18 18:17:40 [INFO] [TRAIN] epoch=57, iter=37190/40000, loss=0.1091, lr=0.000916, batch_cost=0.7866, reader_cost=0.0001 | ETA 00:36:50 2021-01-18 18:17:48 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0926, lr=0.000914, batch_cost=0.7848, reader_cost=0.0001 | ETA 00:36:37 2021-01-18 18:17:56 [INFO] [TRAIN] epoch=57, iter=37210/40000, loss=0.1145, lr=0.000911, batch_cost=0.7871, reader_cost=0.0001 | ETA 00:36:35 2021-01-18 18:18:04 [INFO] [TRAIN] epoch=57, iter=37220/40000, loss=0.1526, lr=0.000908, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:36:36 2021-01-18 18:18:12 [INFO] [TRAIN] epoch=57, iter=37230/40000, loss=0.1078, lr=0.000905, batch_cost=0.7951, reader_cost=0.0001 | ETA 00:36:42 2021-01-18 18:18:20 [INFO] [TRAIN] epoch=57, iter=37240/40000, loss=0.1218, lr=0.000902, batch_cost=0.7908, reader_cost=0.0001 | ETA 00:36:22 2021-01-18 18:18:28 [INFO] [TRAIN] epoch=57, iter=37250/40000, loss=0.1319, lr=0.000899, batch_cost=0.7941, reader_cost=0.0001 | ETA 00:36:23 2021-01-18 18:18:36 [INFO] [TRAIN] epoch=57, iter=37260/40000, loss=0.0773, lr=0.000896, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:36:14 2021-01-18 18:18:43 [INFO] [TRAIN] epoch=57, iter=37270/40000, loss=0.0870, lr=0.000893, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:35:59 2021-01-18 18:18:51 [INFO] [TRAIN] epoch=57, iter=37280/40000, loss=0.1363, lr=0.000890, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:35:50 2021-01-18 18:18:59 [INFO] [TRAIN] epoch=57, iter=37290/40000, loss=0.0899, lr=0.000887, batch_cost=0.7888, reader_cost=0.0001 | ETA 00:35:37 2021-01-18 18:19:07 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0918, lr=0.000884, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:35:37 2021-01-18 18:19:15 [INFO] [TRAIN] epoch=57, iter=37310/40000, loss=0.0968, lr=0.000881, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:35:35 2021-01-18 18:19:23 [INFO] [TRAIN] epoch=57, iter=37320/40000, loss=0.1138, lr=0.000878, batch_cost=0.7875, reader_cost=0.0001 | ETA 00:35:10 2021-01-18 18:19:31 [INFO] [TRAIN] epoch=57, iter=37330/40000, loss=0.0953, lr=0.000875, batch_cost=0.7834, reader_cost=0.0001 | ETA 00:34:51 2021-01-18 18:19:39 [INFO] [TRAIN] epoch=57, iter=37340/40000, loss=0.0869, lr=0.000872, batch_cost=0.7885, reader_cost=0.0001 | ETA 00:34:57 2021-01-18 18:19:47 [INFO] [TRAIN] epoch=57, iter=37350/40000, loss=0.1080, lr=0.000869, batch_cost=0.7953, reader_cost=0.0001 | ETA 00:35:07 2021-01-18 18:19:55 [INFO] [TRAIN] epoch=57, iter=37360/40000, loss=0.1098, lr=0.000866, batch_cost=0.7870, reader_cost=0.0001 | ETA 00:34:37 2021-01-18 18:20:02 [INFO] [TRAIN] epoch=57, iter=37370/40000, loss=0.1085, lr=0.000863, batch_cost=0.7844, reader_cost=0.0001 | ETA 00:34:22 2021-01-18 18:20:10 [INFO] [TRAIN] epoch=57, iter=37380/40000, loss=0.0997, lr=0.000861, batch_cost=0.7907, reader_cost=0.0001 | ETA 00:34:31 2021-01-18 18:20:18 [INFO] [TRAIN] epoch=57, iter=37390/40000, loss=0.1040, lr=0.000858, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:34:25 2021-01-18 18:20:26 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.1212, lr=0.000855, batch_cost=0.7908, reader_cost=0.0001 | ETA 00:34:16 2021-01-18 18:20:34 [INFO] [TRAIN] epoch=57, iter=37410/40000, loss=0.1125, lr=0.000852, batch_cost=0.7860, reader_cost=0.0001 | ETA 00:33:55 2021-01-18 18:20:42 [INFO] [TRAIN] epoch=57, iter=37420/40000, loss=0.1038, lr=0.000849, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:33:58 2021-01-18 18:20:50 [INFO] [TRAIN] epoch=57, iter=37430/40000, loss=0.1216, lr=0.000846, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:33:59 2021-01-18 18:20:58 [INFO] [TRAIN] epoch=57, iter=37440/40000, loss=0.1152, lr=0.000843, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:33:40 2021-01-18 18:21:06 [INFO] [TRAIN] epoch=57, iter=37450/40000, loss=0.1357, lr=0.000840, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:33:34 2021-01-18 18:21:14 [INFO] [TRAIN] epoch=57, iter=37460/40000, loss=0.0877, lr=0.000837, batch_cost=0.7953, reader_cost=0.0001 | ETA 00:33:40 2021-01-18 18:21:21 [INFO] [TRAIN] epoch=57, iter=37470/40000, loss=0.1342, lr=0.000834, batch_cost=0.7895, reader_cost=0.0001 | ETA 00:33:17 2021-01-18 18:21:29 [INFO] [TRAIN] epoch=57, iter=37480/40000, loss=0.1417, lr=0.000831, batch_cost=0.7916, reader_cost=0.0001 | ETA 00:33:14 2021-01-18 18:21:37 [INFO] [TRAIN] epoch=57, iter=37490/40000, loss=0.1185, lr=0.000828, batch_cost=0.7930, reader_cost=0.0001 | ETA 00:33:10 2021-01-18 18:21:45 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.1279, lr=0.000825, batch_cost=0.7959, reader_cost=0.0001 | ETA 00:33:09 2021-01-18 18:21:53 [INFO] [TRAIN] epoch=57, iter=37510/40000, loss=0.0985, lr=0.000822, batch_cost=0.7856, reader_cost=0.0001 | ETA 00:32:36 2021-01-18 18:22:01 [INFO] [TRAIN] epoch=57, iter=37520/40000, loss=0.1180, lr=0.000819, batch_cost=0.7841, reader_cost=0.0001 | ETA 00:32:24 2021-01-18 18:22:09 [INFO] [TRAIN] epoch=57, iter=37530/40000, loss=0.1302, lr=0.000816, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:32:36 2021-01-18 18:22:17 [INFO] [TRAIN] epoch=57, iter=37540/40000, loss=0.1085, lr=0.000813, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:32:15 2021-01-18 18:22:25 [INFO] [TRAIN] epoch=57, iter=37550/40000, loss=0.0905, lr=0.000810, batch_cost=0.7896, reader_cost=0.0001 | ETA 00:32:14 2021-01-18 18:22:33 [INFO] [TRAIN] epoch=57, iter=37560/40000, loss=0.1260, lr=0.000807, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:32:09 2021-01-18 18:22:40 [INFO] [TRAIN] epoch=57, iter=37570/40000, loss=0.1206, lr=0.000804, batch_cost=0.7853, reader_cost=0.0001 | ETA 00:31:48 2021-01-18 18:22:48 [INFO] [TRAIN] epoch=57, iter=37580/40000, loss=0.1366, lr=0.000801, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:31:49 2021-01-18 18:22:56 [INFO] [TRAIN] epoch=57, iter=37590/40000, loss=0.1287, lr=0.000798, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:31:39 2021-01-18 18:23:04 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0954, lr=0.000795, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:31:31 2021-01-18 18:23:12 [INFO] [TRAIN] epoch=57, iter=37610/40000, loss=0.1202, lr=0.000792, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:31:32 2021-01-18 18:23:20 [INFO] [TRAIN] epoch=57, iter=37620/40000, loss=0.1271, lr=0.000789, batch_cost=0.7859, reader_cost=0.0001 | ETA 00:31:10 2021-01-18 18:23:28 [INFO] [TRAIN] epoch=57, iter=37630/40000, loss=0.1377, lr=0.000786, batch_cost=0.7932, reader_cost=0.0001 | ETA 00:31:19 2021-01-18 18:23:36 [INFO] [TRAIN] epoch=57, iter=37640/40000, loss=0.0979, lr=0.000783, batch_cost=0.7874, reader_cost=0.0001 | ETA 00:30:58 2021-01-18 18:23:44 [INFO] [TRAIN] epoch=57, iter=37650/40000, loss=0.1098, lr=0.000780, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:30:59 2021-01-18 18:23:51 [INFO] [TRAIN] epoch=57, iter=37660/40000, loss=0.0957, lr=0.000777, batch_cost=0.7854, reader_cost=0.0001 | ETA 00:30:37 2021-01-18 18:23:59 [INFO] [TRAIN] epoch=57, iter=37670/40000, loss=0.1030, lr=0.000774, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:30:45 2021-01-18 18:24:07 [INFO] [TRAIN] epoch=58, iter=37680/40000, loss=0.1083, lr=0.000771, batch_cost=0.7794, reader_cost=0.0145 | ETA 00:30:08 2021-01-18 18:24:15 [INFO] [TRAIN] epoch=58, iter=37690/40000, loss=0.1334, lr=0.000768, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:30:29 2021-01-18 18:24:23 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.1290, lr=0.000765, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:30:12 2021-01-18 18:24:31 [INFO] [TRAIN] epoch=58, iter=37710/40000, loss=0.1227, lr=0.000762, batch_cost=0.7895, reader_cost=0.0001 | ETA 00:30:07 2021-01-18 18:24:39 [INFO] [TRAIN] epoch=58, iter=37720/40000, loss=0.0882, lr=0.000759, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:30:03 2021-01-18 18:24:47 [INFO] [TRAIN] epoch=58, iter=37730/40000, loss=0.1006, lr=0.000756, batch_cost=0.7864, reader_cost=0.0001 | ETA 00:29:45 2021-01-18 18:24:55 [INFO] [TRAIN] epoch=58, iter=37740/40000, loss=0.1149, lr=0.000753, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:29:50 2021-01-18 18:25:02 [INFO] [TRAIN] epoch=58, iter=37750/40000, loss=0.0932, lr=0.000750, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:29:40 2021-01-18 18:25:10 [INFO] [TRAIN] epoch=58, iter=37760/40000, loss=0.1092, lr=0.000747, batch_cost=0.7924, reader_cost=0.0001 | ETA 00:29:35 2021-01-18 18:25:18 [INFO] [TRAIN] epoch=58, iter=37770/40000, loss=0.0844, lr=0.000744, batch_cost=0.7872, reader_cost=0.0001 | ETA 00:29:15 2021-01-18 18:25:26 [INFO] [TRAIN] epoch=58, iter=37780/40000, loss=0.0983, lr=0.000741, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:29:21 2021-01-18 18:25:34 [INFO] [TRAIN] epoch=58, iter=37790/40000, loss=0.1037, lr=0.000738, batch_cost=0.7864, reader_cost=0.0001 | ETA 00:28:58 2021-01-18 18:25:42 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.1098, lr=0.000735, batch_cost=0.7955, reader_cost=0.0001 | ETA 00:29:10 2021-01-18 18:25:50 [INFO] [TRAIN] epoch=58, iter=37810/40000, loss=0.1521, lr=0.000732, batch_cost=0.7908, reader_cost=0.0001 | ETA 00:28:51 2021-01-18 18:25:58 [INFO] [TRAIN] epoch=58, iter=37820/40000, loss=0.1198, lr=0.000729, batch_cost=0.7960, reader_cost=0.0001 | ETA 00:28:55 2021-01-18 18:26:06 [INFO] [TRAIN] epoch=58, iter=37830/40000, loss=0.1245, lr=0.000726, batch_cost=0.7917, reader_cost=0.0001 | ETA 00:28:37 2021-01-18 18:26:14 [INFO] [TRAIN] epoch=58, iter=37840/40000, loss=0.1003, lr=0.000723, batch_cost=0.7932, reader_cost=0.0001 | ETA 00:28:33 2021-01-18 18:26:22 [INFO] [TRAIN] epoch=58, iter=37850/40000, loss=0.1262, lr=0.000720, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:28:18 2021-01-18 18:26:30 [INFO] [TRAIN] epoch=58, iter=37860/40000, loss=0.0948, lr=0.000717, batch_cost=0.7942, reader_cost=0.0001 | ETA 00:28:19 2021-01-18 18:26:38 [INFO] [TRAIN] epoch=58, iter=37870/40000, loss=0.1050, lr=0.000714, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:28:07 2021-01-18 18:26:45 [INFO] [TRAIN] epoch=58, iter=37880/40000, loss=0.1636, lr=0.000711, batch_cost=0.7934, reader_cost=0.0001 | ETA 00:28:02 2021-01-18 18:26:53 [INFO] [TRAIN] epoch=58, iter=37890/40000, loss=0.1223, lr=0.000708, batch_cost=0.7877, reader_cost=0.0001 | ETA 00:27:42 2021-01-18 18:27:01 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.1314, lr=0.000705, batch_cost=0.7951, reader_cost=0.0001 | ETA 00:27:49 2021-01-18 18:27:09 [INFO] [TRAIN] epoch=58, iter=37910/40000, loss=0.1301, lr=0.000702, batch_cost=0.7873, reader_cost=0.0001 | ETA 00:27:25 2021-01-18 18:27:17 [INFO] [TRAIN] epoch=58, iter=37920/40000, loss=0.1266, lr=0.000699, batch_cost=0.7963, reader_cost=0.0001 | ETA 00:27:36 2021-01-18 18:27:25 [INFO] [TRAIN] epoch=58, iter=37930/40000, loss=0.1213, lr=0.000696, batch_cost=0.7859, reader_cost=0.0001 | ETA 00:27:06 2021-01-18 18:27:33 [INFO] [TRAIN] epoch=58, iter=37940/40000, loss=0.1125, lr=0.000693, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:27:05 2021-01-18 18:27:41 [INFO] [TRAIN] epoch=58, iter=37950/40000, loss=0.1239, lr=0.000690, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:27:02 2021-01-18 18:27:49 [INFO] [TRAIN] epoch=58, iter=37960/40000, loss=0.1180, lr=0.000687, batch_cost=0.7887, reader_cost=0.0001 | ETA 00:26:48 2021-01-18 18:27:57 [INFO] [TRAIN] epoch=58, iter=37970/40000, loss=0.0999, lr=0.000684, batch_cost=0.7930, reader_cost=0.0001 | ETA 00:26:49 2021-01-18 18:28:04 [INFO] [TRAIN] epoch=58, iter=37980/40000, loss=0.1028, lr=0.000681, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:26:37 2021-01-18 18:28:12 [INFO] [TRAIN] epoch=58, iter=37990/40000, loss=0.0984, lr=0.000678, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:26:26 2021-01-18 18:28:20 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.1368, lr=0.000675, batch_cost=0.7845, reader_cost=0.0001 | ETA 00:26:08 2021-01-18 18:28:20 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 18:28:48 [INFO] [EVAL] #Images=1449 mIoU=0.7981 Acc=0.9556 Kappa=0.9011 2021-01-18 18:28:48 [INFO] [EVAL] Class IoU: [0.9486 0.9127 0.4436 0.9104 0.6816 0.8481 0.9404 0.8925 0.935 0.4712 0.9029 0.5837 0.8921 0.8977 0.8953 0.891 0.6502 0.8836 0.4872 0.9213 0.7711] 2021-01-18 18:28:48 [INFO] [EVAL] Class Acc: [0.9665 0.9383 0.4614 0.952 0.7738 0.917 0.9871 0.9485 0.9632 0.7087 0.9493 0.925 0.9462 0.957 0.94 0.9574 0.8318 0.957 0.8537 0.9601 0.9243] 2021-01-18 18:28:55 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 18:29:02 [INFO] [TRAIN] epoch=58, iter=38010/40000, loss=0.1058, lr=0.000672, batch_cost=0.7784, reader_cost=0.0001 | ETA 00:25:49 2021-01-18 18:29:10 [INFO] [TRAIN] epoch=58, iter=38020/40000, loss=0.0931, lr=0.000669, batch_cost=0.7887, reader_cost=0.0001 | ETA 00:26:01 2021-01-18 18:29:18 [INFO] [TRAIN] epoch=58, iter=38030/40000, loss=0.1296, lr=0.000666, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:26:00 2021-01-18 18:29:26 [INFO] [TRAIN] epoch=58, iter=38040/40000, loss=0.1158, lr=0.000663, batch_cost=0.7888, reader_cost=0.0001 | ETA 00:25:46 2021-01-18 18:29:34 [INFO] [TRAIN] epoch=58, iter=38050/40000, loss=0.0993, lr=0.000660, batch_cost=0.7908, reader_cost=0.0001 | ETA 00:25:42 2021-01-18 18:29:42 [INFO] [TRAIN] epoch=58, iter=38060/40000, loss=0.0704, lr=0.000657, batch_cost=0.7897, reader_cost=0.0001 | ETA 00:25:31 2021-01-18 18:29:50 [INFO] [TRAIN] epoch=58, iter=38070/40000, loss=0.1233, lr=0.000654, batch_cost=0.7858, reader_cost=0.0001 | ETA 00:25:16 2021-01-18 18:29:58 [INFO] [TRAIN] epoch=58, iter=38080/40000, loss=0.1525, lr=0.000651, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:25:24 2021-01-18 18:30:06 [INFO] [TRAIN] epoch=58, iter=38090/40000, loss=0.1250, lr=0.000648, batch_cost=0.7974, reader_cost=0.0001 | ETA 00:25:22 2021-01-18 18:30:14 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.1276, lr=0.000645, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:25:02 2021-01-18 18:30:22 [INFO] [TRAIN] epoch=58, iter=38110/40000, loss=0.1192, lr=0.000641, batch_cost=0.7948, reader_cost=0.0001 | ETA 00:25:02 2021-01-18 18:30:29 [INFO] [TRAIN] epoch=58, iter=38120/40000, loss=0.1056, lr=0.000638, batch_cost=0.7907, reader_cost=0.0001 | ETA 00:24:46 2021-01-18 18:30:37 [INFO] [TRAIN] epoch=58, iter=38130/40000, loss=0.1209, lr=0.000635, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:24:43 2021-01-18 18:30:45 [INFO] [TRAIN] epoch=58, iter=38140/40000, loss=0.1224, lr=0.000632, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:24:23 2021-01-18 18:30:53 [INFO] [TRAIN] epoch=58, iter=38150/40000, loss=0.0972, lr=0.000629, batch_cost=0.7916, reader_cost=0.0001 | ETA 00:24:24 2021-01-18 18:31:01 [INFO] [TRAIN] epoch=58, iter=38160/40000, loss=0.0805, lr=0.000626, batch_cost=0.7876, reader_cost=0.0001 | ETA 00:24:09 2021-01-18 18:31:09 [INFO] [TRAIN] epoch=58, iter=38170/40000, loss=0.1068, lr=0.000623, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:24:09 2021-01-18 18:31:17 [INFO] [TRAIN] epoch=58, iter=38180/40000, loss=0.1217, lr=0.000620, batch_cost=0.7905, reader_cost=0.0001 | ETA 00:23:58 2021-01-18 18:31:25 [INFO] [TRAIN] epoch=58, iter=38190/40000, loss=0.1310, lr=0.000617, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:23:48 2021-01-18 18:31:33 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.1343, lr=0.000614, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:23:44 2021-01-18 18:31:41 [INFO] [TRAIN] epoch=58, iter=38210/40000, loss=0.1390, lr=0.000611, batch_cost=0.7970, reader_cost=0.0001 | ETA 00:23:46 2021-01-18 18:31:49 [INFO] [TRAIN] epoch=58, iter=38220/40000, loss=0.1029, lr=0.000608, batch_cost=0.7901, reader_cost=0.0001 | ETA 00:23:26 2021-01-18 18:31:56 [INFO] [TRAIN] epoch=58, iter=38230/40000, loss=0.1370, lr=0.000605, batch_cost=0.7890, reader_cost=0.0001 | ETA 00:23:16 2021-01-18 18:32:04 [INFO] [TRAIN] epoch=58, iter=38240/40000, loss=0.1263, lr=0.000602, batch_cost=0.7835, reader_cost=0.0001 | ETA 00:22:58 2021-01-18 18:32:12 [INFO] [TRAIN] epoch=58, iter=38250/40000, loss=0.1173, lr=0.000599, batch_cost=0.7961, reader_cost=0.0001 | ETA 00:23:13 2021-01-18 18:32:20 [INFO] [TRAIN] epoch=58, iter=38260/40000, loss=0.0992, lr=0.000595, batch_cost=0.7922, reader_cost=0.0001 | ETA 00:22:58 2021-01-18 18:32:28 [INFO] [TRAIN] epoch=58, iter=38270/40000, loss=0.1004, lr=0.000592, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:22:45 2021-01-18 18:32:36 [INFO] [TRAIN] epoch=58, iter=38280/40000, loss=0.0990, lr=0.000589, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:22:35 2021-01-18 18:32:44 [INFO] [TRAIN] epoch=58, iter=38290/40000, loss=0.0922, lr=0.000586, batch_cost=0.7895, reader_cost=0.0001 | ETA 00:22:29 2021-01-18 18:32:52 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.1249, lr=0.000583, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:22:29 2021-01-18 18:33:00 [INFO] [TRAIN] epoch=58, iter=38310/40000, loss=0.1182, lr=0.000580, batch_cost=0.7866, reader_cost=0.0001 | ETA 00:22:09 2021-01-18 18:33:08 [INFO] [TRAIN] epoch=58, iter=38320/40000, loss=0.1175, lr=0.000577, batch_cost=0.7943, reader_cost=0.0001 | ETA 00:22:14 2021-01-18 18:33:16 [INFO] [TRAIN] epoch=58, iter=38330/40000, loss=0.1558, lr=0.000574, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:22:05 2021-01-18 18:33:23 [INFO] [TRAIN] epoch=59, iter=38340/40000, loss=0.1224, lr=0.000571, batch_cost=0.7752, reader_cost=0.0151 | ETA 00:21:26 2021-01-18 18:33:31 [INFO] [TRAIN] epoch=59, iter=38350/40000, loss=0.1022, lr=0.000568, batch_cost=0.7931, reader_cost=0.0001 | ETA 00:21:48 2021-01-18 18:33:39 [INFO] [TRAIN] epoch=59, iter=38360/40000, loss=0.1204, lr=0.000565, batch_cost=0.7917, reader_cost=0.0001 | ETA 00:21:38 2021-01-18 18:33:47 [INFO] [TRAIN] epoch=59, iter=38370/40000, loss=0.1052, lr=0.000562, batch_cost=0.7878, reader_cost=0.0001 | ETA 00:21:24 2021-01-18 18:33:55 [INFO] [TRAIN] epoch=59, iter=38380/40000, loss=0.1372, lr=0.000558, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:21:25 2021-01-18 18:34:03 [INFO] [TRAIN] epoch=59, iter=38390/40000, loss=0.1171, lr=0.000555, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:21:10 2021-01-18 18:34:11 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.1007, lr=0.000552, batch_cost=0.7958, reader_cost=0.0001 | ETA 00:21:13 2021-01-18 18:34:19 [INFO] [TRAIN] epoch=59, iter=38410/40000, loss=0.1031, lr=0.000549, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:20:58 2021-01-18 18:34:27 [INFO] [TRAIN] epoch=59, iter=38420/40000, loss=0.1178, lr=0.000546, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:20:49 2021-01-18 18:34:35 [INFO] [TRAIN] epoch=59, iter=38430/40000, loss=0.1059, lr=0.000543, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:20:43 2021-01-18 18:34:43 [INFO] [TRAIN] epoch=59, iter=38440/40000, loss=0.1178, lr=0.000540, batch_cost=0.7981, reader_cost=0.0001 | ETA 00:20:45 2021-01-18 18:34:50 [INFO] [TRAIN] epoch=59, iter=38450/40000, loss=0.0937, lr=0.000537, batch_cost=0.7930, reader_cost=0.0001 | ETA 00:20:29 2021-01-18 18:34:58 [INFO] [TRAIN] epoch=59, iter=38460/40000, loss=0.1298, lr=0.000534, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:20:18 2021-01-18 18:35:06 [INFO] [TRAIN] epoch=59, iter=38470/40000, loss=0.1365, lr=0.000530, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:20:11 2021-01-18 18:35:14 [INFO] [TRAIN] epoch=59, iter=38480/40000, loss=0.1272, lr=0.000527, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:19:57 2021-01-18 18:35:22 [INFO] [TRAIN] epoch=59, iter=38490/40000, loss=0.1269, lr=0.000524, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:19:55 2021-01-18 18:35:30 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.1029, lr=0.000521, batch_cost=0.7946, reader_cost=0.0001 | ETA 00:19:51 2021-01-18 18:35:38 [INFO] [TRAIN] epoch=59, iter=38510/40000, loss=0.1199, lr=0.000518, batch_cost=0.7978, reader_cost=0.0001 | ETA 00:19:48 2021-01-18 18:35:46 [INFO] [TRAIN] epoch=59, iter=38520/40000, loss=0.1105, lr=0.000515, batch_cost=0.7927, reader_cost=0.0001 | ETA 00:19:33 2021-01-18 18:35:54 [INFO] [TRAIN] epoch=59, iter=38530/40000, loss=0.1003, lr=0.000512, batch_cost=0.7870, reader_cost=0.0001 | ETA 00:19:16 2021-01-18 18:36:02 [INFO] [TRAIN] epoch=59, iter=38540/40000, loss=0.1148, lr=0.000509, batch_cost=0.7907, reader_cost=0.0001 | ETA 00:19:14 2021-01-18 18:36:10 [INFO] [TRAIN] epoch=59, iter=38550/40000, loss=0.0992, lr=0.000505, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:19:05 2021-01-18 18:36:18 [INFO] [TRAIN] epoch=59, iter=38560/40000, loss=0.1049, lr=0.000502, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:18:59 2021-01-18 18:36:25 [INFO] [TRAIN] epoch=59, iter=38570/40000, loss=0.0818, lr=0.000499, batch_cost=0.7912, reader_cost=0.0001 | ETA 00:18:51 2021-01-18 18:36:33 [INFO] [TRAIN] epoch=59, iter=38580/40000, loss=0.1327, lr=0.000496, batch_cost=0.7936, reader_cost=0.0001 | ETA 00:18:46 2021-01-18 18:36:41 [INFO] [TRAIN] epoch=59, iter=38590/40000, loss=0.1087, lr=0.000493, batch_cost=0.7873, reader_cost=0.0001 | ETA 00:18:30 2021-01-18 18:36:49 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0792, lr=0.000490, batch_cost=0.7935, reader_cost=0.0001 | ETA 00:18:30 2021-01-18 18:36:57 [INFO] [TRAIN] epoch=59, iter=38610/40000, loss=0.0920, lr=0.000487, batch_cost=0.7866, reader_cost=0.0001 | ETA 00:18:13 2021-01-18 18:37:05 [INFO] [TRAIN] epoch=59, iter=38620/40000, loss=0.1083, lr=0.000483, batch_cost=0.7882, reader_cost=0.0001 | ETA 00:18:07 2021-01-18 18:37:13 [INFO] [TRAIN] epoch=59, iter=38630/40000, loss=0.1136, lr=0.000480, batch_cost=0.7874, reader_cost=0.0001 | ETA 00:17:58 2021-01-18 18:37:21 [INFO] [TRAIN] epoch=59, iter=38640/40000, loss=0.1297, lr=0.000477, batch_cost=0.7859, reader_cost=0.0001 | ETA 00:17:48 2021-01-18 18:37:29 [INFO] [TRAIN] epoch=59, iter=38650/40000, loss=0.1131, lr=0.000474, batch_cost=0.7890, reader_cost=0.0001 | ETA 00:17:45 2021-01-18 18:37:36 [INFO] [TRAIN] epoch=59, iter=38660/40000, loss=0.1002, lr=0.000471, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:17:38 2021-01-18 18:37:44 [INFO] [TRAIN] epoch=59, iter=38670/40000, loss=0.1306, lr=0.000468, batch_cost=0.7866, reader_cost=0.0001 | ETA 00:17:26 2021-01-18 18:37:52 [INFO] [TRAIN] epoch=59, iter=38680/40000, loss=0.0928, lr=0.000464, batch_cost=0.7880, reader_cost=0.0001 | ETA 00:17:20 2021-01-18 18:38:00 [INFO] [TRAIN] epoch=59, iter=38690/40000, loss=0.1258, lr=0.000461, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:17:18 2021-01-18 18:38:08 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.1291, lr=0.000458, batch_cost=0.7860, reader_cost=0.0001 | ETA 00:17:01 2021-01-18 18:38:16 [INFO] [TRAIN] epoch=59, iter=38710/40000, loss=0.1056, lr=0.000455, batch_cost=0.7931, reader_cost=0.0001 | ETA 00:17:03 2021-01-18 18:38:24 [INFO] [TRAIN] epoch=59, iter=38720/40000, loss=0.1283, lr=0.000452, batch_cost=0.7897, reader_cost=0.0001 | ETA 00:16:50 2021-01-18 18:38:32 [INFO] [TRAIN] epoch=59, iter=38730/40000, loss=0.1217, lr=0.000449, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:16:45 2021-01-18 18:38:40 [INFO] [TRAIN] epoch=59, iter=38740/40000, loss=0.1195, lr=0.000445, batch_cost=0.7932, reader_cost=0.0001 | ETA 00:16:39 2021-01-18 18:38:48 [INFO] [TRAIN] epoch=59, iter=38750/40000, loss=0.0763, lr=0.000442, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:16:26 2021-01-18 18:38:55 [INFO] [TRAIN] epoch=59, iter=38760/40000, loss=0.1380, lr=0.000439, batch_cost=0.7873, reader_cost=0.0001 | ETA 00:16:16 2021-01-18 18:39:03 [INFO] [TRAIN] epoch=59, iter=38770/40000, loss=0.0786, lr=0.000436, batch_cost=0.7972, reader_cost=0.0001 | ETA 00:16:20 2021-01-18 18:39:11 [INFO] [TRAIN] epoch=59, iter=38780/40000, loss=0.1429, lr=0.000433, batch_cost=0.7865, reader_cost=0.0001 | ETA 00:15:59 2021-01-18 18:39:19 [INFO] [TRAIN] epoch=59, iter=38790/40000, loss=0.1388, lr=0.000430, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:15:58 2021-01-18 18:39:27 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0884, lr=0.000426, batch_cost=0.7827, reader_cost=0.0001 | ETA 00:15:39 2021-01-18 18:39:35 [INFO] [TRAIN] epoch=59, iter=38810/40000, loss=0.0972, lr=0.000423, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:15:43 2021-01-18 18:39:43 [INFO] [TRAIN] epoch=59, iter=38820/40000, loss=0.1047, lr=0.000420, batch_cost=0.7867, reader_cost=0.0001 | ETA 00:15:28 2021-01-18 18:39:51 [INFO] [TRAIN] epoch=59, iter=38830/40000, loss=0.1109, lr=0.000417, batch_cost=0.7962, reader_cost=0.0001 | ETA 00:15:31 2021-01-18 18:39:59 [INFO] [TRAIN] epoch=59, iter=38840/40000, loss=0.1366, lr=0.000414, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:15:18 2021-01-18 18:40:07 [INFO] [TRAIN] epoch=59, iter=38850/40000, loss=0.1214, lr=0.000410, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:15:11 2021-01-18 18:40:15 [INFO] [TRAIN] epoch=59, iter=38860/40000, loss=0.0945, lr=0.000407, batch_cost=0.7916, reader_cost=0.0001 | ETA 00:15:02 2021-01-18 18:40:22 [INFO] [TRAIN] epoch=59, iter=38870/40000, loss=0.1178, lr=0.000404, batch_cost=0.7883, reader_cost=0.0001 | ETA 00:14:50 2021-01-18 18:40:30 [INFO] [TRAIN] epoch=59, iter=38880/40000, loss=0.1565, lr=0.000401, batch_cost=0.7927, reader_cost=0.0001 | ETA 00:14:47 2021-01-18 18:40:38 [INFO] [TRAIN] epoch=59, iter=38890/40000, loss=0.1133, lr=0.000397, batch_cost=0.7906, reader_cost=0.0001 | ETA 00:14:37 2021-01-18 18:40:46 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.1290, lr=0.000394, batch_cost=0.7887, reader_cost=0.0001 | ETA 00:14:27 2021-01-18 18:40:54 [INFO] [TRAIN] epoch=59, iter=38910/40000, loss=0.1215, lr=0.000391, batch_cost=0.7871, reader_cost=0.0001 | ETA 00:14:17 2021-01-18 18:41:02 [INFO] [TRAIN] epoch=59, iter=38920/40000, loss=0.1270, lr=0.000388, batch_cost=0.7942, reader_cost=0.0001 | ETA 00:14:17 2021-01-18 18:41:10 [INFO] [TRAIN] epoch=59, iter=38930/40000, loss=0.0985, lr=0.000385, batch_cost=0.7848, reader_cost=0.0001 | ETA 00:13:59 2021-01-18 18:41:18 [INFO] [TRAIN] epoch=59, iter=38940/40000, loss=0.1216, lr=0.000381, batch_cost=0.7855, reader_cost=0.0001 | ETA 00:13:52 2021-01-18 18:41:26 [INFO] [TRAIN] epoch=59, iter=38950/40000, loss=0.1070, lr=0.000378, batch_cost=0.7904, reader_cost=0.0001 | ETA 00:13:49 2021-01-18 18:41:33 [INFO] [TRAIN] epoch=59, iter=38960/40000, loss=0.1160, lr=0.000375, batch_cost=0.7872, reader_cost=0.0001 | ETA 00:13:38 2021-01-18 18:41:41 [INFO] [TRAIN] epoch=59, iter=38970/40000, loss=0.1141, lr=0.000372, batch_cost=0.7872, reader_cost=0.0001 | ETA 00:13:30 2021-01-18 18:41:49 [INFO] [TRAIN] epoch=59, iter=38980/40000, loss=0.0942, lr=0.000368, batch_cost=0.7860, reader_cost=0.0001 | ETA 00:13:21 2021-01-18 18:41:57 [INFO] [TRAIN] epoch=59, iter=38990/40000, loss=0.1005, lr=0.000365, batch_cost=0.7886, reader_cost=0.0001 | ETA 00:13:16 2021-01-18 18:42:05 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.1226, lr=0.000362, batch_cost=0.7743, reader_cost=0.0162 | ETA 00:12:54 2021-01-18 18:42:05 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 18:42:33 [INFO] [EVAL] #Images=1449 mIoU=0.7946 Acc=0.9548 Kappa=0.8990 2021-01-18 18:42:33 [INFO] [EVAL] Class IoU: [0.9473 0.9109 0.4448 0.9102 0.6592 0.8113 0.9386 0.8918 0.9362 0.4794 0.9019 0.5546 0.894 0.8965 0.894 0.8901 0.6507 0.8856 0.4915 0.9169 0.7807] 2021-01-18 18:42:33 [INFO] [EVAL] Class Acc: [0.9646 0.9371 0.4621 0.9512 0.7691 0.9196 0.9882 0.9468 0.9635 0.7453 0.9535 0.9089 0.9453 0.9589 0.9441 0.9598 0.8456 0.9533 0.8454 0.9596 0.9581] 2021-01-18 18:42:39 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000. 2021-01-18 18:42:47 [INFO] [TRAIN] epoch=60, iter=39010/40000, loss=0.1267, lr=0.000359, batch_cost=0.7871, reader_cost=0.0001 | ETA 00:12:59 2021-01-18 18:42:55 [INFO] [TRAIN] epoch=60, iter=39020/40000, loss=0.1299, lr=0.000355, batch_cost=0.7934, reader_cost=0.0001 | ETA 00:12:57 2021-01-18 18:43:03 [INFO] [TRAIN] epoch=60, iter=39030/40000, loss=0.1271, lr=0.000352, batch_cost=0.7902, reader_cost=0.0001 | ETA 00:12:46 2021-01-18 18:43:11 [INFO] [TRAIN] epoch=60, iter=39040/40000, loss=0.1362, lr=0.000349, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:12:40 2021-01-18 18:43:19 [INFO] [TRAIN] epoch=60, iter=39050/40000, loss=0.1076, lr=0.000346, batch_cost=0.7895, reader_cost=0.0001 | ETA 00:12:30 2021-01-18 18:43:27 [INFO] [TRAIN] epoch=60, iter=39060/40000, loss=0.1298, lr=0.000342, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:12:25 2021-01-18 18:43:35 [INFO] [TRAIN] epoch=60, iter=39070/40000, loss=0.1150, lr=0.000339, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:12:16 2021-01-18 18:43:43 [INFO] [TRAIN] epoch=60, iter=39080/40000, loss=0.1042, lr=0.000336, batch_cost=0.7942, reader_cost=0.0001 | ETA 00:12:10 2021-01-18 18:43:51 [INFO] [TRAIN] epoch=60, iter=39090/40000, loss=0.0895, lr=0.000332, batch_cost=0.7976, reader_cost=0.0001 | ETA 00:12:05 2021-01-18 18:43:59 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.1144, lr=0.000329, batch_cost=0.7959, reader_cost=0.0001 | ETA 00:11:56 2021-01-18 18:44:07 [INFO] [TRAIN] epoch=60, iter=39110/40000, loss=0.1093, lr=0.000326, batch_cost=0.7884, reader_cost=0.0001 | ETA 00:11:41 2021-01-18 18:44:14 [INFO] [TRAIN] epoch=60, iter=39120/40000, loss=0.1318, lr=0.000323, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:11:34 2021-01-18 18:44:22 [INFO] [TRAIN] epoch=60, iter=39130/40000, loss=0.1120, lr=0.000319, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:11:29 2021-01-18 18:44:30 [INFO] [TRAIN] epoch=60, iter=39140/40000, loss=0.1212, lr=0.000316, batch_cost=0.7887, reader_cost=0.0001 | ETA 00:11:18 2021-01-18 18:44:38 [INFO] [TRAIN] epoch=60, iter=39150/40000, loss=0.1011, lr=0.000313, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:11:12 2021-01-18 18:44:46 [INFO] [TRAIN] epoch=60, iter=39160/40000, loss=0.0972, lr=0.000309, batch_cost=0.7862, reader_cost=0.0001 | ETA 00:11:00 2021-01-18 18:44:54 [INFO] [TRAIN] epoch=60, iter=39170/40000, loss=0.1287, lr=0.000306, batch_cost=0.7993, reader_cost=0.0001 | ETA 00:11:03 2021-01-18 18:45:02 [INFO] [TRAIN] epoch=60, iter=39180/40000, loss=0.1323, lr=0.000303, batch_cost=0.7892, reader_cost=0.0001 | ETA 00:10:47 2021-01-18 18:45:10 [INFO] [TRAIN] epoch=60, iter=39190/40000, loss=0.1353, lr=0.000299, batch_cost=0.7939, reader_cost=0.0001 | ETA 00:10:43 2021-01-18 18:45:18 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.1246, lr=0.000296, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:10:34 2021-01-18 18:45:26 [INFO] [TRAIN] epoch=60, iter=39210/40000, loss=0.1018, lr=0.000293, batch_cost=0.7855, reader_cost=0.0001 | ETA 00:10:20 2021-01-18 18:45:34 [INFO] [TRAIN] epoch=60, iter=39220/40000, loss=0.0918, lr=0.000289, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:10:15 2021-01-18 18:45:41 [INFO] [TRAIN] epoch=60, iter=39230/40000, loss=0.1217, lr=0.000286, batch_cost=0.7924, reader_cost=0.0001 | ETA 00:10:10 2021-01-18 18:45:49 [INFO] [TRAIN] epoch=60, iter=39240/40000, loss=0.1678, lr=0.000283, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:10:01 2021-01-18 18:45:57 [INFO] [TRAIN] epoch=60, iter=39250/40000, loss=0.0969, lr=0.000279, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:09:52 2021-01-18 18:46:05 [INFO] [TRAIN] epoch=60, iter=39260/40000, loss=0.1167, lr=0.000276, batch_cost=0.7863, reader_cost=0.0001 | ETA 00:09:41 2021-01-18 18:46:13 [INFO] [TRAIN] epoch=60, iter=39270/40000, loss=0.1124, lr=0.000273, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:09:38 2021-01-18 18:46:21 [INFO] [TRAIN] epoch=60, iter=39280/40000, loss=0.1206, lr=0.000269, batch_cost=0.7930, reader_cost=0.0001 | ETA 00:09:30 2021-01-18 18:46:29 [INFO] [TRAIN] epoch=60, iter=39290/40000, loss=0.1311, lr=0.000266, batch_cost=0.7877, reader_cost=0.0001 | ETA 00:09:19 2021-01-18 18:46:37 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.1486, lr=0.000263, batch_cost=0.7876, reader_cost=0.0001 | ETA 00:09:11 2021-01-18 18:46:45 [INFO] [TRAIN] epoch=60, iter=39310/40000, loss=0.1097, lr=0.000259, batch_cost=0.7927, reader_cost=0.0001 | ETA 00:09:06 2021-01-18 18:46:53 [INFO] [TRAIN] epoch=60, iter=39320/40000, loss=0.1351, lr=0.000256, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:08:57 2021-01-18 18:47:00 [INFO] [TRAIN] epoch=60, iter=39330/40000, loss=0.1138, lr=0.000252, batch_cost=0.7928, reader_cost=0.0001 | ETA 00:08:51 2021-01-18 18:47:08 [INFO] [TRAIN] epoch=60, iter=39340/40000, loss=0.1029, lr=0.000249, batch_cost=0.7888, reader_cost=0.0001 | ETA 00:08:40 2021-01-18 18:47:16 [INFO] [TRAIN] epoch=60, iter=39350/40000, loss=0.1172, lr=0.000246, batch_cost=0.7941, reader_cost=0.0001 | ETA 00:08:36 2021-01-18 18:47:24 [INFO] [TRAIN] epoch=60, iter=39360/40000, loss=0.0950, lr=0.000242, batch_cost=0.7929, reader_cost=0.0001 | ETA 00:08:27 2021-01-18 18:47:32 [INFO] [TRAIN] epoch=60, iter=39370/40000, loss=0.0869, lr=0.000239, batch_cost=0.7940, reader_cost=0.0001 | ETA 00:08:20 2021-01-18 18:47:40 [INFO] [TRAIN] epoch=60, iter=39380/40000, loss=0.1331, lr=0.000235, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:08:09 2021-01-18 18:47:48 [INFO] [TRAIN] epoch=60, iter=39390/40000, loss=0.1041, lr=0.000232, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:08:03 2021-01-18 18:47:56 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.1202, lr=0.000229, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:07:53 2021-01-18 18:48:04 [INFO] [TRAIN] epoch=60, iter=39410/40000, loss=0.1148, lr=0.000225, batch_cost=0.7918, reader_cost=0.0001 | ETA 00:07:47 2021-01-18 18:48:12 [INFO] [TRAIN] epoch=60, iter=39420/40000, loss=0.0745, lr=0.000222, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:07:38 2021-01-18 18:48:20 [INFO] [TRAIN] epoch=60, iter=39430/40000, loss=0.1041, lr=0.000218, batch_cost=0.7869, reader_cost=0.0001 | ETA 00:07:28 2021-01-18 18:48:27 [INFO] [TRAIN] epoch=60, iter=39440/40000, loss=0.1002, lr=0.000215, batch_cost=0.7913, reader_cost=0.0001 | ETA 00:07:23 2021-01-18 18:48:35 [INFO] [TRAIN] epoch=60, iter=39450/40000, loss=0.1136, lr=0.000211, batch_cost=0.7922, reader_cost=0.0001 | ETA 00:07:15 2021-01-18 18:48:43 [INFO] [TRAIN] epoch=60, iter=39460/40000, loss=0.1127, lr=0.000208, batch_cost=0.7884, reader_cost=0.0001 | ETA 00:07:05 2021-01-18 18:48:51 [INFO] [TRAIN] epoch=60, iter=39470/40000, loss=0.1127, lr=0.000205, batch_cost=0.7831, reader_cost=0.0001 | ETA 00:06:55 2021-01-18 18:48:59 [INFO] [TRAIN] epoch=60, iter=39480/40000, loss=0.0924, lr=0.000201, batch_cost=0.7868, reader_cost=0.0001 | ETA 00:06:49 2021-01-18 18:49:07 [INFO] [TRAIN] epoch=60, iter=39490/40000, loss=0.1364, lr=0.000198, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:06:43 2021-01-18 18:49:15 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.1688, lr=0.000194, batch_cost=0.7869, reader_cost=0.0001 | ETA 00:06:33 2021-01-18 18:49:23 [INFO] [TRAIN] epoch=60, iter=39510/40000, loss=0.1207, lr=0.000191, batch_cost=0.7900, reader_cost=0.0001 | ETA 00:06:27 2021-01-18 18:49:31 [INFO] [TRAIN] epoch=60, iter=39520/40000, loss=0.0949, lr=0.000187, batch_cost=0.7863, reader_cost=0.0001 | ETA 00:06:17 2021-01-18 18:49:38 [INFO] [TRAIN] epoch=60, iter=39530/40000, loss=0.0959, lr=0.000184, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:06:11 2021-01-18 18:49:46 [INFO] [TRAIN] epoch=60, iter=39540/40000, loss=0.0853, lr=0.000180, batch_cost=0.7962, reader_cost=0.0001 | ETA 00:06:06 2021-01-18 18:49:54 [INFO] [TRAIN] epoch=60, iter=39550/40000, loss=0.1126, lr=0.000177, batch_cost=0.7891, reader_cost=0.0001 | ETA 00:05:55 2021-01-18 18:50:02 [INFO] [TRAIN] epoch=60, iter=39560/40000, loss=0.1054, lr=0.000173, batch_cost=0.7920, reader_cost=0.0001 | ETA 00:05:48 2021-01-18 18:50:10 [INFO] [TRAIN] epoch=60, iter=39570/40000, loss=0.1078, lr=0.000170, batch_cost=0.7886, reader_cost=0.0001 | ETA 00:05:39 2021-01-18 18:50:18 [INFO] [TRAIN] epoch=60, iter=39580/40000, loss=0.1826, lr=0.000166, batch_cost=0.7923, reader_cost=0.0001 | ETA 00:05:32 2021-01-18 18:50:26 [INFO] [TRAIN] epoch=60, iter=39590/40000, loss=0.0855, lr=0.000162, batch_cost=0.7931, reader_cost=0.0001 | ETA 00:05:25 2021-01-18 18:50:34 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.1011, lr=0.000159, batch_cost=0.7924, reader_cost=0.0001 | ETA 00:05:16 2021-01-18 18:50:42 [INFO] [TRAIN] epoch=60, iter=39610/40000, loss=0.1053, lr=0.000155, batch_cost=0.7948, reader_cost=0.0001 | ETA 00:05:09 2021-01-18 18:50:50 [INFO] [TRAIN] epoch=60, iter=39620/40000, loss=0.0913, lr=0.000152, batch_cost=0.7911, reader_cost=0.0001 | ETA 00:05:00 2021-01-18 18:50:58 [INFO] [TRAIN] epoch=60, iter=39630/40000, loss=0.1064, lr=0.000148, batch_cost=0.7897, reader_cost=0.0001 | ETA 00:04:52 2021-01-18 18:51:06 [INFO] [TRAIN] epoch=60, iter=39640/40000, loss=0.0968, lr=0.000145, batch_cost=0.7878, reader_cost=0.0001 | ETA 00:04:43 2021-01-18 18:51:13 [INFO] [TRAIN] epoch=60, iter=39650/40000, loss=0.1273, lr=0.000141, batch_cost=0.7950, reader_cost=0.0001 | ETA 00:04:38 2021-01-18 18:51:21 [INFO] [TRAIN] epoch=60, iter=39660/40000, loss=0.1182, lr=0.000137, batch_cost=0.7585, reader_cost=0.0001 | ETA 00:04:17 2021-01-18 18:51:29 [INFO] [TRAIN] epoch=61, iter=39670/40000, loss=0.0924, lr=0.000134, batch_cost=0.8087, reader_cost=0.0167 | ETA 00:04:26 2021-01-18 18:51:37 [INFO] [TRAIN] epoch=61, iter=39680/40000, loss=0.1010, lr=0.000130, batch_cost=0.7922, reader_cost=0.0001 | ETA 00:04:13 2021-01-18 18:51:45 [INFO] [TRAIN] epoch=61, iter=39690/40000, loss=0.1074, lr=0.000126, batch_cost=0.7933, reader_cost=0.0001 | ETA 00:04:05 2021-01-18 18:51:53 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.1058, lr=0.000123, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:03:56 2021-01-18 18:52:01 [INFO] [TRAIN] epoch=61, iter=39710/40000, loss=0.1061, lr=0.000119, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:03:49 2021-01-18 18:52:09 [INFO] [TRAIN] epoch=61, iter=39720/40000, loss=0.0761, lr=0.000115, batch_cost=0.7934, reader_cost=0.0001 | ETA 00:03:42 2021-01-18 18:52:17 [INFO] [TRAIN] epoch=61, iter=39730/40000, loss=0.0882, lr=0.000112, batch_cost=0.7929, reader_cost=0.0001 | ETA 00:03:34 2021-01-18 18:52:25 [INFO] [TRAIN] epoch=61, iter=39740/40000, loss=0.1155, lr=0.000108, batch_cost=0.7926, reader_cost=0.0001 | ETA 00:03:26 2021-01-18 18:52:33 [INFO] [TRAIN] epoch=61, iter=39750/40000, loss=0.1061, lr=0.000104, batch_cost=0.7971, reader_cost=0.0001 | ETA 00:03:19 2021-01-18 18:52:40 [INFO] [TRAIN] epoch=61, iter=39760/40000, loss=0.1198, lr=0.000100, batch_cost=0.7876, reader_cost=0.0001 | ETA 00:03:09 2021-01-18 18:52:48 [INFO] [TRAIN] epoch=61, iter=39770/40000, loss=0.0908, lr=0.000097, batch_cost=0.7921, reader_cost=0.0001 | ETA 00:03:02 2021-01-18 18:52:56 [INFO] [TRAIN] epoch=61, iter=39780/40000, loss=0.1198, lr=0.000093, batch_cost=0.7938, reader_cost=0.0001 | ETA 00:02:54 2021-01-18 18:53:04 [INFO] [TRAIN] epoch=61, iter=39790/40000, loss=0.0858, lr=0.000089, batch_cost=0.7837, reader_cost=0.0001 | ETA 00:02:44 2021-01-18 18:53:12 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.1210, lr=0.000085, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:02:37 2021-01-18 18:53:20 [INFO] [TRAIN] epoch=61, iter=39810/40000, loss=0.1044, lr=0.000081, batch_cost=0.7837, reader_cost=0.0001 | ETA 00:02:28 2021-01-18 18:53:28 [INFO] [TRAIN] epoch=61, iter=39820/40000, loss=0.0848, lr=0.000078, batch_cost=0.7890, reader_cost=0.0001 | ETA 00:02:22 2021-01-18 18:53:36 [INFO] [TRAIN] epoch=61, iter=39830/40000, loss=0.0974, lr=0.000074, batch_cost=0.7945, reader_cost=0.0001 | ETA 00:02:15 2021-01-18 18:53:44 [INFO] [TRAIN] epoch=61, iter=39840/40000, loss=0.1104, lr=0.000070, batch_cost=0.7964, reader_cost=0.0001 | ETA 00:02:07 2021-01-18 18:53:52 [INFO] [TRAIN] epoch=61, iter=39850/40000, loss=0.1120, lr=0.000066, batch_cost=0.7903, reader_cost=0.0001 | ETA 00:01:58 2021-01-18 18:54:00 [INFO] [TRAIN] epoch=61, iter=39860/40000, loss=0.1347, lr=0.000062, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:01:50 2021-01-18 18:54:07 [INFO] [TRAIN] epoch=61, iter=39870/40000, loss=0.1115, lr=0.000058, batch_cost=0.7927, reader_cost=0.0001 | ETA 00:01:43 2021-01-18 18:54:15 [INFO] [TRAIN] epoch=61, iter=39880/40000, loss=0.1090, lr=0.000054, batch_cost=0.7909, reader_cost=0.0001 | ETA 00:01:34 2021-01-18 18:54:23 [INFO] [TRAIN] epoch=61, iter=39890/40000, loss=0.1143, lr=0.000050, batch_cost=0.7894, reader_cost=0.0001 | ETA 00:01:26 2021-01-18 18:54:31 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0858, lr=0.000046, batch_cost=0.7893, reader_cost=0.0001 | ETA 00:01:18 2021-01-18 18:54:39 [INFO] [TRAIN] epoch=61, iter=39910/40000, loss=0.0956, lr=0.000042, batch_cost=0.7968, reader_cost=0.0001 | ETA 00:01:11 2021-01-18 18:54:47 [INFO] [TRAIN] epoch=61, iter=39920/40000, loss=0.1255, lr=0.000038, batch_cost=0.7914, reader_cost=0.0001 | ETA 00:01:03 2021-01-18 18:54:55 [INFO] [TRAIN] epoch=61, iter=39930/40000, loss=0.1034, lr=0.000033, batch_cost=0.7889, reader_cost=0.0001 | ETA 00:00:55 2021-01-18 18:55:03 [INFO] [TRAIN] epoch=61, iter=39940/40000, loss=0.1104, lr=0.000029, batch_cost=0.7851, reader_cost=0.0001 | ETA 00:00:47 2021-01-18 18:55:11 [INFO] [TRAIN] epoch=61, iter=39950/40000, loss=0.1541, lr=0.000025, batch_cost=0.7905, reader_cost=0.0001 | ETA 00:00:39 2021-01-18 18:55:19 [INFO] [TRAIN] epoch=61, iter=39960/40000, loss=0.1402, lr=0.000020, batch_cost=0.7927, reader_cost=0.0001 | ETA 00:00:31 2021-01-18 18:55:26 [INFO] [TRAIN] epoch=61, iter=39970/40000, loss=0.1226, lr=0.000016, batch_cost=0.7881, reader_cost=0.0001 | ETA 00:00:23 2021-01-18 18:55:34 [INFO] [TRAIN] epoch=61, iter=39980/40000, loss=0.1187, lr=0.000011, batch_cost=0.7910, reader_cost=0.0001 | ETA 00:00:15 2021-01-18 18:55:42 [INFO] [TRAIN] epoch=61, iter=39990/40000, loss=0.1202, lr=0.000006, batch_cost=0.7915, reader_cost=0.0001 | ETA 00:00:07 2021-01-18 18:55:50 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.1169, lr=0.000001, batch_cost=0.7925, reader_cost=0.0001 | ETA 00:00:00 2021-01-18 18:55:50 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2021-01-18 18:56:19 [INFO] [EVAL] #Images=1449 mIoU=0.7972 Acc=0.9554 Kappa=0.9007 2021-01-18 18:56:19 [INFO] [EVAL] Class IoU: [0.9484 0.9107 0.438 0.9097 0.6595 0.8355 0.9401 0.893 0.9358 0.4749 0.9126 0.5518 0.8907 0.898 0.8957 0.8907 0.6525 0.8962 0.5086 0.9194 0.7785] 2021-01-18 18:56:19 [INFO] [EVAL] Class Acc: [0.9666 0.9353 0.4556 0.9507 0.7701 0.9102 0.9874 0.9466 0.9624 0.7084 0.9628 0.9064 0.938 0.9536 0.934 0.9597 0.842 0.9529 0.8542 0.9577 0.9487] 2021-01-18 18:56:24 [INFO] [EVAL] The model with the best validation mIoU (0.8049) was saved at iter 31000.