Skip to content

Commit 757fd6d

Browse files
committed
ALBERT vs BERT diff
1 parent 82e2b3b commit 757fd6d

15 files changed

+30794
-131
lines changed

LICENSE

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@
186186
same "printed page" as the copyright notice for easier
187187
identification within third-party archives.
188188

189-
Copyright [yyyy] [name of copyright owner]
189+
Copyright 2019 Tae Hwan Jung
190190

191191
Licensed under the Apache License, Version 2.0 (the "License");
192192
you may not use this file except in compliance with the License.
@@ -198,4 +198,4 @@
198198
distributed under the License is distributed on an "AS IS" BASIS,
199199
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200200
See the License for the specific language governing permissions and
201-
limitations under the License.
201+
limitations under the License.

classify.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ def forward(self, input_ids, segment_ids, input_mask):
171171

172172
def main(task='mrpc',
173173
train_cfg='config/train_mrpc.json',
174-
model_cfg='config/bert_base.json',
174+
model_cfg='config/albert_base.json',
175175
data_file='../glue/MRPC/train.tsv',
176176
model_file=None,
177177
pretrain_file='../uncased_L-12_H-768_A-12/bert_model.ckpt',

config/albert_base.json

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"embedding" : 128,
3+
4+
"hidden": 768,
5+
"hidden_ff": 3072,
6+
7+
"n_layers": 12,
8+
"n_heads": 12,
9+
"max_len": 512,
10+
"n_segments": 2,
11+
"vocab_size": 30522
12+
}
13+

config/albert_unittest.json

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"embedding" : 24,
3+
4+
"hidden": 64,
5+
"hidden_ff": 256,
6+
7+
"n_layers": 6,
8+
"n_heads": 1,
9+
"max_len": 512,
10+
"n_segments": 2,
11+
"vocab_size": 30522
12+
}

config/bert_base.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,3 @@
99
"n_segments": 2,
1010
"vocab_size": 30522
1111
}
12-

config/pretrain.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
{
22
"seed": 3431,
3-
"batch_size": 96,
3+
"batch_size": 8,
44
"lr": 1e-4,
55
"n_epochs": 25,
66
"warmup": 0.1,
77
"save_steps": 10000,
88
"total_steps": 1000000
9-
}
9+
}

data/.keep

Whitespace-only changes.

0 commit comments

Comments
 (0)