config.json 预训练模型调参
（来自BERT论文）
config文件定义了模型的超参数。
但是，由于此模型在64GB内存上训练，所以如果个人使用的话，需要调整超参数。
内存大小影响因子包括：
- max_seq_length，发布模型长度不超过512，你可以使用更短的。
- train_batch_size(成正比)
- Model type，Base和Large模型
- Optimizer(BERT默认Adam，后来的Roberta，Albert都有更改)(谷歌未测试其他优化器)，指的是SDG,SAG等迭代下降方法
用默认训练脚本 (run_classifier.py 和 run_squad.py), 获得基准后的maximum batch size 在一个单独的 Titan X GPU (12GB RAM) 和 TensorFlow 1.11.0:
BERT-large 的 max batch size相当小，以至于确实损害模型精度。我们正在努力增大batch size值。我们通过以下方法增加batch size值。
(后面与普通人无关)
-
Gradient accumulation: The samples in a minibatch are typically independent with respect to gradient computation (excluding batch normalization, which is not used here). This means that the gradients of multiple smaller minibatches can be accumulated before performing the weight update, and this will be exactly equivalent to a single larger update.
-
Gradient checkpointing: The major use of GPU/TPU memory during DNN training is caching the intermediate activations in the forward pass that are necessary for efficient computation in the backward pass. "Gradient checkpointing" trades memory for compute time by re-computing the activations in an intelligent way.