-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathout_DanQ_JASPAR.txt
149 lines (147 loc) · 11.5 KB
/
out_DanQ_JASPAR.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
2019-08-25 01:34:18.198574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-08-25 01:34:18.205890: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.206679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:03:00.0
2019-08-25 01:34:18.206841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-08-25 01:34:18.207720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-08-25 01:34:18.208485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-08-25 01:34:18.208691: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-08-25 01:34:18.209660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-08-25 01:34:18.210449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-08-25 01:34:18.212925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-08-25 01:34:18.213051: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.214147: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.215227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-08-25 01:34:18.215487: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-25 01:34:18.239051: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-08-25 01:34:18.239452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5627e6fddd80 executing computations on platform Host. Devices:
2019-08-25 01:34:18.239475: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-08-25 01:34:18.303142: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.303831: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5627e6fa26a0 executing computations on platform CUDA. Devices:
2019-08-25 01:34:18.303848: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2019-08-25 01:34:18.303985: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.304635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:03:00.0
2019-08-25 01:34:18.304671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-08-25 01:34:18.304684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-08-25 01:34:18.304696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-08-25 01:34:18.304715: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-08-25 01:34:18.304727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-08-25 01:34:18.304739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-08-25 01:34:18.304751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-08-25 01:34:18.304798: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.305452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.306082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-08-25 01:34:18.306110: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-08-25 01:34:18.307351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-25 01:34:18.307364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-08-25 01:34:18.307373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-08-25 01:34:18.307678: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.308402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-25 01:34:18.309054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10349 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
WARNING: Logging before flag parsing goes to stderr.
W0825 01:34:18.880836 140692385109824 deprecation.py:323] From /home/lijiawei/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/data/util/random_seed.py:58: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2019-08-25 01:34:19.430814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-08-25 01:34:20.444897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Initializing from scratch.
Begin to train the model.
Epoch: 1 | Train Loss: 0.06679
Epoch: 1 | Valid Loss: 0.05587
Epoch: 1 | Cost time: 5179.15836: second
Saved checkpoint for epoch 1: ./result/DanQ_JASPAR/checkpoint/ckpt-1
Epoch: 2 | Train Loss: 0.05985
Epoch: 2 | Valid Loss: 0.05384
Epoch: 2 | Cost time: 5062.81970: second
Saved checkpoint for epoch 2: ./result/DanQ_JASPAR/checkpoint/ckpt-2
Epoch: 3 | Train Loss: 0.05827
Epoch: 3 | Valid Loss: 0.05274
Epoch: 3 | Cost time: 5047.61440: second
Saved checkpoint for epoch 3: ./result/DanQ_JASPAR/checkpoint/ckpt-3
Epoch: 4 | Train Loss: 0.05745
Epoch: 4 | Valid Loss: 0.05275
Epoch: 4 | Cost time: 5155.70257: second
Epoch: 5 | Train Loss: 0.05693
Epoch: 5 | Valid Loss: 0.05211
Epoch: 5 | Cost time: 5806.53217: second
Saved checkpoint for epoch 5: ./result/DanQ_JASPAR/checkpoint/ckpt-5
Epoch: 6 | Train Loss: 0.05660
Epoch: 6 | Valid Loss: 0.05195
Epoch: 6 | Cost time: 5819.22163: second
Saved checkpoint for epoch 6: ./result/DanQ_JASPAR/checkpoint/ckpt-6
Epoch: 7 | Train Loss: 0.05634
Epoch: 7 | Valid Loss: 0.05183
Epoch: 7 | Cost time: 5802.55555: second
Saved checkpoint for epoch 7: ./result/DanQ_JASPAR/checkpoint/ckpt-7
Epoch: 8 | Train Loss: 0.05613
Epoch: 8 | Valid Loss: 0.05216
Epoch: 8 | Cost time: 5757.12484: second
Epoch: 9 | Train Loss: 0.05597
Epoch: 9 | Valid Loss: 0.05217
Epoch: 9 | Cost time: 5667.07176: second
Epoch: 10 | Train Loss: 0.05583
Epoch: 10 | Valid Loss: 0.05155
Epoch: 10 | Cost time: 5762.42855: second
Saved checkpoint for epoch 10: ./result/DanQ_JASPAR/checkpoint/ckpt-10
Epoch: 11 | Train Loss: 0.05568
Epoch: 11 | Valid Loss: 0.05172
Epoch: 11 | Cost time: 5699.60646: second
Epoch: 12 | Train Loss: 0.05557
Epoch: 12 | Valid Loss: 0.05170
Epoch: 12 | Cost time: 5789.25373: second
Epoch: 13 | Train Loss: 0.05546
Epoch: 13 | Valid Loss: 0.05209
Epoch: 13 | Cost time: 5753.80873: second
Epoch: 14 | Train Loss: 0.05539
Epoch: 14 | Valid Loss: 0.05207
Epoch: 14 | Cost time: 5795.42473: second
Epoch: 15 | Train Loss: 0.05529
Epoch: 15 | Valid Loss: 0.05155
Epoch: 15 | Cost time: 5771.54337: second
Saved checkpoint for epoch 15: ./result/DanQ_JASPAR/checkpoint/ckpt-15
Epoch: 16 | Train Loss: 0.05522
Epoch: 16 | Valid Loss: 0.05159
Epoch: 16 | Cost time: 5691.87030: second
Epoch: 17 | Train Loss: 0.05516
Epoch: 17 | Valid Loss: 0.05140
Epoch: 17 | Cost time: 5736.28442: second
Saved checkpoint for epoch 17: ./result/DanQ_JASPAR/checkpoint/ckpt-17
Epoch: 18 | Train Loss: 0.05508
Epoch: 18 | Valid Loss: 0.05161
Epoch: 18 | Cost time: 5784.03654: second
Epoch: 19 | Train Loss: 0.05501
Epoch: 19 | Valid Loss: 0.05175
Epoch: 19 | Cost time: 5771.50011: second
Epoch: 20 | Train Loss: 0.05497
Epoch: 20 | Valid Loss: 0.05175
Epoch: 20 | Cost time: 5781.82343: second
Epoch: 21 | Train Loss: 0.05491
Epoch: 21 | Valid Loss: 0.05180
Epoch: 21 | Cost time: 5803.33136: second
Epoch: 22 | Train Loss: 0.05487
Epoch: 22 | Valid Loss: 0.05140
Epoch: 22 | Cost time: 5785.77482: second
Saved checkpoint for epoch 22: ./result/DanQ_JASPAR/checkpoint/ckpt-22
Epoch: 23 | Train Loss: 0.05482
Epoch: 23 | Valid Loss: 0.05164
Epoch: 23 | Cost time: 5794.84337: second
Epoch: 24 | Train Loss: 0.05478
Epoch: 24 | Valid Loss: 0.05184
Epoch: 24 | Cost time: 5768.76995: second
Epoch: 25 | Train Loss: 0.05477
Epoch: 25 | Valid Loss: 0.05161
Epoch: 25 | Cost time: 5683.46408: second
Epoch: 26 | Train Loss: 0.05472
Epoch: 26 | Valid Loss: 0.05188
Epoch: 26 | Cost time: 5759.64747: second
Epoch: 27 | Train Loss: 0.05468
Epoch: 27 | Valid Loss: 0.05152
Epoch: 27 | Cost time: 5764.24519: second
Epoch: 28 | Train Loss: 0.05465
Epoch: 28 | Valid Loss: 0.05170
Epoch: 28 | Cost time: 5743.12116: second
Validation dice has not improved in 5 epochs. Stopped training.
history_dict: {'epoch': array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.]), 'train_loss': array([0.06678782, 0.05985317, 0.05826592, 0.05744836, 0.05692893, 0.05659513, 0.05633767, 0.05613344, 0.05596809, 0.05582597, 0.05568393, 0.05557226, 0.05546169, 0.05538688, 0.05529337, 0.05522459, 0.05515762, 0.05507536, 0.05501401, 0.05496603, 0.05491283, 0.05486777, 0.05482276, 0.05478211, 0.05477071, 0.05471827, 0.0546835 , 0.05465319]), 'valid_loss': array([0.05587105, 0.05383934, 0.05273974, 0.05274832, 0.05211249, 0.05195408, 0.05182694, 0.05216224, 0.05216955, 0.0515506 , 0.0517225 , 0.05170218, 0.05209092, 0.0520708 , 0.05154687, 0.05158537, 0.05139596, 0.05161155, 0.05175175, 0.05174986, 0.05180432, 0.05139541, 0.05164341, 0.05183591, 0.05161306, 0.0518752 , 0.05151938, 0.05169691])}