You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for developing these valuable scripts. We ran run_pretrain.py and encountered the following issue. We appreciate your guidance for the troubleshooting. Thanks.
07/10/2023 00:11:34 - INFO - main - Training new model from scratch
07/10/2023 00:11:36 - INFO - main - Training/evaluation parameters Namespace(adam_epsilon=1e-06, beta1=0.9, beta2=0.98, block_size=512, cache_dir=None, config_name='../src/transfo
rmers/dnabert-config/bert-config-6/config.json', device=device(type='cpu'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='../data/3k_6mer/1st_finished_asm_3k_
6mer.all', evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=25, learning_rate=0.0004, line_by_line=True, local_rank=-1, logging_steps=500, max grad_norm=1.0, max_steps=200000, mlm=True, mlm_probability=0.025, model_name_or_path=None, model_type='dna', n_gpu=0, n_process=16, no_cuda=False, num_train_epochs=1.0, output_dir='k6'
, overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=8, save_steps=500, save_total_limit=20, seed=42, server_ip='', server_port='', sh
ould_continue=False, tokenizer_name='dna6', train_data_file='../data/3k_6mer/1st_finished_asm_3k_6mer.all', warmup_steps=10000, weight_decay=0.01)
07/10/2023 00:11:36 - INFO - main - Creating features from dataset file at ../data/3k_6mer/1st_finished_asm_3k_6mer.all
07/10/2023 00:14:39 - INFO - main - Saving features into cached file ../data/3k_6mer/dna_cached_lm_512_1st_finished_asm_3k_6mer.all
07/10/2023 00:14:47 - INFO - main - ***** Running training *****
07/10/2023 00:14:47 - INFO - main - Num examples = 357566
07/10/2023 00:14:47 - INFO - main - Num Epochs = 112
07/10/2023 00:14:47 - INFO - main - Instantaneous batch size per GPU = 8
07/10/2023 00:14:47 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 200
07/10/2023 00:14:47 - INFO - main - Gradient Accumulation steps = 25
07/10/2023 00:14:47 - INFO - main - Total optimization steps = 200000
Iteration: 0%| | 0/44696 [00:00<?, ?it/s]
Epoch: 0%| | 0/112 [00:00<?, ?it/s]?it/s]
Traceback (most recent call last):
File "../scripts/run_pretrain.py", line 890, in
main()
File "../scripts/run_pretrain.py", line 840, in main
global_step, tr_loss = train(args, train_dataset, model, tokenizer)
File "../scripts/run_pretrain.py", line 426, in train
inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
File "../scripts/run_pretrain.py", line 272, in mask_tokens
probability_matrix.masked_fill(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
The text was updated successfully, but these errors were encountered:
Thanks for developing these valuable scripts. We ran
run_pretrain.py
and encountered the following issue. We appreciate your guidance for the troubleshooting. Thanks.07/10/2023 00:11:34 - INFO - main - Training new model from scratch
07/10/2023 00:11:36 - INFO - main - Training/evaluation parameters Namespace(adam_epsilon=1e-06, beta1=0.9, beta2=0.98, block_size=512, cache_dir=None, config_name='../src/transfo
rmers/dnabert-config/bert-config-6/config.json', device=device(type='cpu'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='../data/3k_6mer/1st_finished_asm_3k_
6mer.all', evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=25, learning_rate=0.0004, line_by_line=True, local_rank=-1, logging_steps=500, max
grad_norm=1.0, max_steps=200000, mlm=True, mlm_probability=0.025, model_name_or_path=None, model_type='dna', n_gpu=0, n_process=16, no_cuda=False, num_train_epochs=1.0, output_dir='k6'
, overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=8, save_steps=500, save_total_limit=20, seed=42, server_ip='', server_port='', sh
ould_continue=False, tokenizer_name='dna6', train_data_file='../data/3k_6mer/1st_finished_asm_3k_6mer.all', warmup_steps=10000, weight_decay=0.01)
07/10/2023 00:11:36 - INFO - main - Creating features from dataset file at ../data/3k_6mer/1st_finished_asm_3k_6mer.all
07/10/2023 00:14:39 - INFO - main - Saving features into cached file ../data/3k_6mer/dna_cached_lm_512_1st_finished_asm_3k_6mer.all
07/10/2023 00:14:47 - INFO - main - ***** Running training *****
07/10/2023 00:14:47 - INFO - main - Num examples = 357566
07/10/2023 00:14:47 - INFO - main - Num Epochs = 112
07/10/2023 00:14:47 - INFO - main - Instantaneous batch size per GPU = 8
07/10/2023 00:14:47 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 200
07/10/2023 00:14:47 - INFO - main - Gradient Accumulation steps = 25
07/10/2023 00:14:47 - INFO - main - Total optimization steps = 200000
Iteration: 0%| | 0/44696 [00:00<?, ?it/s]
Epoch: 0%| | 0/112 [00:00<?, ?it/s]?it/s]
Traceback (most recent call last):
File "../scripts/run_pretrain.py", line 890, in
main()
File "../scripts/run_pretrain.py", line 840, in main
global_step, tr_loss = train(args, train_dataset, model, tokenizer)
File "../scripts/run_pretrain.py", line 426, in train
inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
File "../scripts/run_pretrain.py", line 272, in mask_tokens
probability_matrix.masked_fill(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
The text was updated successfully, but these errors were encountered: